Persist generated records with Signal Dataset

RFGen writes new generated datasets through Signal Dataset. The format supports records with several named numerical fields, arbitrary tensor ranks, nested JSON metadata, ordinal random access, and metadata-only reads. The same dataset lifecycle works on a local directory and on Google Cloud Storage (GCS).

Signal Dataset is an external file contract, not an RF coordinate system or a training framework. RFGen owns scene generation, labels, record composition, content-derived record IDs, and job scheduling. Signal Dataset owns immutable shards, the published root, and indexed reads. See the upstream Signal Dataset 0.2 documentation and PyPI release.

Generate and inspect a local dataset

The maintained templates already select the native backend:

rfgen init chirp-radar ./radar-data
cd ./radar-data
rfgen validate --config-dir .
rfgen generate --config-dir .
rfgen inspect ./rfgen-output

Generation writes each shard independently. After every expected shard has succeeded, the coordinator publishes the dataset root. Readers therefore see either no published dataset or a complete snapshot; a failed run does not publish root.json.

rfgen inspect opens that published snapshot without changing it. Its native report includes the record count, field names, shapes, dtypes, semantic axes, and a bounded list of record IDs. See Inspect and consume a dataset.

Understand one native record

A native record is a signal_dataset.Record. Each field combines a NumPy array with named axes.

A unified SceneObservationRecord always becomes exactly one native record. Its projection-owned primary fields use projections/<projection_id>/<local_field_name> names, and coordinate arrays may be emitted as auxiliary fields. The strict envelope under metadata["rfgen"] carries the canonical ScenePlan, projection and runtime identities, invocation seeds, interaction coverage, field ownership, and the primary/auxiliary classification. Each field retains its native rank and semantic axes. See Unified Scene Observations.

One ScenePlan produces one record; generation never fans a plan out by receiver. Record IDs cover field bytes, axes, scene ID, and metadata, so changing any of those changes the ID. Signal Dataset materializes a selected record’s numerical fields as a whole record; it does not page one tensor from inside the record. Configure observation.max_tensor_bytes to bound aggregate payloads, and use metadata-only reads when field arrays are unnecessary.

Read records for training

Use SignalDatasetStore when an RFGen application needs the configured storage boundary:

from rfgen.storage import SignalDatasetStore

dataset = SignalDatasetStore().open("./rfgen-output")
print(len(dataset), dataset.dataset_id, dataset.snapshot_id)

record = dataset[0]
metadata = dataset.metadata(0)  # does not fetch numerical field payloads

# Field names are namespaced by the projection that produced them, and each
# receiver is its own subtree. `rfgen inspect` prints the full inventory.
name = "projections/receiver/receivers/rx0/iq"
print(record[name].data.shape, metadata[name].axes)

DatasetAccess implements ordinal indexing and slicing. Its metadata(...) and iter_metadata() paths read the aligned metadata index without loading signal tensors. RFGen does not impose a batch or split policy on the persisted dataset.

Understand record identity

RFGen replaces a record’s caller-supplied ID with a SHA-256 identity derived from the public signal_dataset.record.encode(...) canonical encoding. Before encoding, it substitutes a fixed placeholder ID so identity is not recursive and does not depend on an arbitrary input label. Field bytes, field axes, scene ID, and metadata remain in the encoding, so changes to any persisted content change the resulting ID.

This RFGen rule makes retries and cross-shard collision checks independent of the producer’s temporary names while relying on Signal Dataset’s public canonical serializer instead of duplicating its wire format. Focused tests verify that input-ID-only changes preserve identity, content and semantic-axis changes alter it, and SignalDatasetStore.write_shard(...) rejects a record whose ID no longer matches its encoded content.

The one store

signal_dataset is the only registered store. It writes immutable shards on a local path or a gs:// root, gives ordinal random access, and holds arbitrary-rank fields, so a radar cube and a communications capture live in one record without either being reshaped.

storage.backend stays an open selector rather than a closed enum: a third-party store registers under rfgen.dataset_stores and is reachable without a schema change.

SigMF remains a calibration capture input, not a generation store.

Extend composition or storage

For unified observations, a custom projection registers under rfgen.observation_projections, declares a strict parameter model and interaction/capability contracts, and returns projection-owned fields. The central SDS adapter remains responsible for persistence. See Observation API.

A custom scene renderer registers under rfgen.scene_renderers, declares an exact Pydantic ParamsModel, and lists its required dataset-store capabilities. A custom native store registers separately under rfgen.dataset_stores and implements BaseDatasetStore. RFGen resolves both selectors and checks capabilities before generating a shard. See Storage API and Configuration schema.

Annotation boundary

A published snapshot is what annotation reads. Generated records stay immutable: a run publishes a dense annotation set under the snapshot root and rewrites no signal shard. See Annotations and Annotate an existing dataset.