Dataset Consumption¶

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, config field names, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

Dataset consumption starts after records have been written. Training code needs to choose records from a store, batch them for a task, and hand tensors to a training framework such as PyTorch without rerunning generation.

The six producer layers (Emitters, Channels, Scenes, Labels, Annotations, Storage) describe how Record objects are produced and stored. Once stored, downstream training and evaluation pipelines consume them through three additional abstractions. These are not a seventh producer layer. They are a consumer-side toolkit that wraps the storage layer’s read API.

Place In The System¶

Producers run during dataset generation; consumers run during model training and evaluation. The two sides only communicate through stored Record objects: a producer writes a record, the consumer reads it. There is no shared in-process state.

Phase 1 is the synthetic radio-frequency (RF) generation pass described by the six producer layers. Phase 2 is the optional annotation pass that appends text metadata, such as captions or scene summaries, after in-phase/quadrature (IQ) samples and labels already exist.

Phase	Side	What runs
Phase 1	Producer	Generate records with the six producer layers; write via StoreHandle.
Phase 2	Producer	Annotator appends text fields to existing records.
Training	Consumer	BaseSampler selects records, BaseCollator shapes batches, RfgenTorchDataset adapts to PyTorch.

Data Flow¶

     BaseSampler ──► sample_id iteration order
            │
            ▼
StoreHandle.read(sample_id) ──► Record
            │
            ▼
     BaseCollator ──► batched tensors shaped for the task
            │
            ▼
     RfgenTorchDataset ──► torch.utils.data.DataLoader compatible

sample_id is the stable record identifier stored in the dataset index. The index is the store’s list of available records, usually represented by the storage backend’s metadata or manifest. The ID may be content-derived or supplied by the storage backend, but consumer code treats it as the key passed to StoreHandle when reading a record.

Abstract Base Classes¶

These abstract base classes (ABCs) are proposed extension contracts for the consumer side.

Concept	ABC	Module
Read order	BaseSampler	`rfgen.dataset.samplers`
Batch shape	BaseCollator	`rfgen.dataset.collators`
Framework adapter	RfgenTorchDataset	`rfgen.dataset` (optional, behind `rfgen[torch]`)

BaseSampler controls which sample_id values are yielded and in what order. Samplers read metadata from stored records and yield IDs in an order chosen for the training objective. Implementations may shuffle, stratify by class, balance by signal-to-noise ratio (SNR) bucket, or use curriculum ordering, where easier examples appear before harder examples.

BaseCollator turns a list of Record objects into batched tensors shaped for a specific task. For example, detection returns IQ tensors plus time-frequency bounding boxes around emitters, while classification returns IQ tensors plus class labels. Different downstream tasks need different batch shapes, so the collator is task-specific while the records are not.

RfgenTorchDataset is the optional PyTorch adapter behind the rfgen[torch] extra. It bridges BaseSampler and BaseCollator to torch.utils.data.DataLoader. When a collator is configured, RfgenTorchDataset.__iter__ already yields collated batches; use DataLoader(batch_size=None) so PyTorch does not add an unwanted outer batch dimension. JAX, TensorFlow, or framework-specific adapters can be added without touching the producer layers.

Boundaries¶

StoreHandle does not inherit from any consumer-side ABC. The store is producer-side; only the adapter layer touches consumer types.
Producers do not know about samplers or collators. A scene composer writes records; whether those records are later consumed in shuffled order or stratified order is a consumer choice.
Collators are task-specific; records are not. A single Record may feed both a detection task (bboxes) and a classification task (per-emitter labels) via different collators.

Minimal Example¶

from rfgen.dataset.collators import DetectionCollator
from rfgen.dataset.samplers import RandomSampler
from rfgen.dataset import RfgenTorchDataset
from rfgen.storage import open_store
from torch.utils.data import DataLoader

# This assumes the quickstart or local-dataset how-to has written
# detection records to ./out/local-smoke.
with open_store("./out/local-smoke") as store:
    dataset = RfgenTorchDataset(
        store=store,
        sampler=RandomSampler(seed=1337),
        collator=DetectionCollator(),
    )
    # batch_size=None keeps the DetectionCollator's batch shape unchanged.
    loader = DataLoader(dataset, batch_size=None, num_workers=4)
    for batch in loader:
        iq = batch["iq"]
        boxes = batch["bboxes"]
        ...

RandomSampler gives a deterministic random order over stored sample_id values. DetectionCollator returns detection batches in rfgen terms: IQ tensors, time-frequency bounding boxes, class IDs, and emitter counts. Exact tensor shapes and units belong in the dataset API reference.