Storage

Warning

Pre-implementation. Class signatures and defaults are proposals. Once code lands, the API page will be regenerated from docstrings and this concept page will be re-checked against it.

Storage writes generated in-phase/quadrature (IQ) arrays, labels, and metadata into a dataset. Later annotation jobs may attach text metadata, such as captions or reports, without changing the original generated records.

The storage layer persists every Record produced by Phase 1, the synthetic RF generation pass. In Phase 2, the optional text-annotation pass accepts append-only runs against already-written records. The rest of the pipeline speaks Record, not file paths or storage-specific handles.

The canonical Phase 1 dataset writes Zarr and WebDataset shards in one pass: Zarr is the random-access working store and source of truth; WebDataset is the tar-sharded, streaming-friendly training mirror. The exact paths, arrays, and metadata fields live in Reference / Storage Layout.

Source: Zarr’s storage guide documents arrays stored through local filesystems or cloud object stores such as S3, Google Cloud Storage, and Azure Blob Storage; WebDataset’s format guide defines tar shards where files sharing a basename form one training sample [1, 2].

Store and Handle Interfaces

The storage layer uses two abstract base classes (ABCs) because configuration and open input/output (I/O) sessions have different lifetimes:

This split keeps backend selection separate from per-open state such as shard indexes, buffers, byte counters, and annotation-run registries. It also lets Phase 2 reopen an existing dataset in append mode without reconstructing the dataset definition.

Exact methods and signatures belong in Reference / rfgen.storage API.

Minimal worked example

This contract sketch opens a local Zarr store, writes one generated Record, and reads it back by sample_id. The record value comes from the generation pipeline, for example a scene composer or quickstart run.

from rfgen.storage import ZarrStore
from rfgen.core_types import Record
from rfgen.enums import StoreMode

store = ZarrStore()
record: Record = ...  # supplied by generation code

with store.open("file://./out/demo/samples.zarr", mode=StoreMode.WRITE) as h:
    sid = h.write(record)          # returns the stable sample_id
    assert h.read(sid).iq.shape == record.iq.shape
    assert len(h) == 1

The same backend should work locally and in object storage by changing the URL, not the dataset logic. A file:// URL writes to the local filesystem; a gs:// URL writes to Google Cloud Storage when credentials are configured. sample_id is the stable record key. By default it is the content hash of the canonicalized record, so a retry that regenerates bit-identical content produces the same sample_id and addresses the same record; storage backends may also accept a caller-supplied value. The (global_seed, shard_id, sample_idx) tuple is the generation input that deterministically produces that content (see Reference / Determinism); the sample_id is the hash of the resulting bytes. See Storage layout naming and shard completion for the naming and stability contract.

Source: Zarr’s ObjectStore abstraction stores the hierarchy through object storage implementations, including Google Cloud Storage, S3, and Azure Blob Storage [1].

Backend matrix

The matrix separates working storage, training export, real-radio capture import, and metadata sidecars. Hardware-in-the-loop (HIL) means round-tripping through a real radio for validation. Signal Metadata Format (SigMF) is a standard file format for recorded RF sample data and JSON metadata.

Use case

Default

Alternates

Phase 1 canonical write (random access, audit, append target)

Zarr

(none)

Training-time sequential read at GPU scale

WebDataset

Zarr (slower, OK for small jobs)

TorchSig model interop

HDF5

(none)

HIL capture input (round-trip from real radio)

SigMF

HDF5 (legacy)

Manifest, statistics

JSON

(none)

The default policy: every Phase 1 dataset writes Zarr canonical and WebDataset shards in one pass. Other formats are adapters for training interop, capture ingest, or legacy handoff.

Source: WebDataset is the training-shard mirror because its documented format is tar-based and streamable; SigMF is the capture-ingest adapter because the SigMF specification defines RF sample recordings as binary sample data plus JSON metadata [2, 5].

Design principles

Phase 1 and Phase 2 have different access patterns and the design reflects that.

Phase 1 is write-once. Once a shard, a unit of generated records, lands, the IQ arrays and structured labels are immutable. Backfills, meaning generated records added or corrected after the fact, create a new dataset version.

Phase 2 is append-only. Each annotation run appends a row through StoreHandle.append_annotation_row. Old rows are never overwritten. Re-running Phase 2 is a no-op for records that already have the target (annotation_type, template_id, run_id) row; to re-annotate, bump run_id or template_id.

The remaining four principles hold across both phases:

  • Cloud-native by default. Production writes to Google Cloud Storage (GCS); the same backend writes locally given a file:// path. One backend, two URLs.

  • Zarr is the single source of truth. Every other format is derivable from Zarr without loss.

  • Deterministic, content-addressed naming. Re-runs with the same materialized config and shard index target the same shard IDs, so retries can skip already-written outputs.

  • Schema-versioned at every level. Top-level group, per-sample group, every WebDataset tar manifest.

Source: the first two principles follow from Zarr’s chunked, key/value storage model and object-store support; the WebDataset manifest convention is rfgen’s schema layer around WebDataset’s tar-shard format [1, 2].

Custom backend

Most users use the built-in Zarr and WebDataset stores. Custom backends are for teams integrating a new filesystem, archive format, database, or delivery format into the same store/handle split. The lifecycle excerpt below shows the Layer 9 integration point; it is not a complete subclass. See Add a custom store for the full shipped ABC skeleton.

Think of shard storage as record data plus one separate completion flag. The flag means every requested record finished; it is not written after a partial attempt.

Some record data may remain, but a retry regenerates the same content-derived IDs and treats identical existing records as no-ops. In this contract, committing a shard means publishing the flag, not rolling back every earlier record write like a database transaction.

from collections.abc import Iterable

from rfgen.core_types import Record
from rfgen.storage import StoreHandle

class MyStoreHandle(StoreHandle):
    def write_shard(
        self,
        shard_id: str,
        records: Iterable[Record],
    ) -> tuple[str, ...]:
        written = []
        for record in records:      # the Layer 9 iterator may raise here
            written.append(self.write(record))
        self._publish_completion_marker(shard_id, tuple(written))
        return tuple(written)

The completion-marker call occurs after the loop. An iterator or record-write exception therefore leaves the shard incomplete, while the safe no-op behavior for identical existing records lets a retry reuse records already stored. Register the complete backend through the rfgen.stores entry-point group and select it in configuration.

Why Zarr and not HDF5 as canonical

Most reference radio-frequency (RF) machine-learning datasets, including RadioML and TorchSig’s WidebandSig53 benchmark dataset, ship as HDF5. Three reasons Zarr fits this pipeline better.

Object-store native. Zarr stores each chunk as its own object; HDF5 is one monolithic file. On GCS or S3, a Spark job with hundreds of workers can write Zarr chunks in parallel directly to the bucket. HDF5 would require a local staging copy and a final merge step per worker.

Concurrent and resumable writes. Phase 1 (many Spark workers) and Phase 2 (inference-grounded annotation, separately resumable) both append to the same dataset. Zarr append is “create a new chunk object” with no locking. HDF5 Single Writer Multiple Reader (SWMR), HDF5’s concurrent-read mode, is fragile across cloud filesystems and forbids structural changes mid-write, which is incompatible with appending text columns after Phase 1 is done.

Random-access reads for re-annotation. Phase 2 reads per-emitter metadata for arbitrary samples without scanning the full file. Zarr’s chunk index makes this O(1); HDF5 random access on remote blob storage serializes badly under concurrent annotators.

HDF5 stays as a delivery format: TorchSig-trained models expect it, so the pipeline ships a byte-exact HDF5 exporter. The split is Zarr for working storage during generation, HDF5 for handoff to TorchSig consumers.

Source: Zarr stores array chunks as independently addressable objects in a key/value hierarchy; HDF5 SWMR requires a POSIX-write-compatible filesystem and cannot create new groups or datasets after entering SWMR mode; TorchSig interoperability is documented in rfgen’s TorchSig conversion contract [1, 3, 4].

Why content-hashed sample IDs?

Idempotency means a retried Spark worker can safely run the same shard again without creating duplicate records. A worker that re-runs a shard with the same seed produces bit-identical records and the same sample_id, so it writes to the same group. Partial-shard recovery becomes “skip if path exists.” No coordinator bookkeeping required.

References

  1. Zarr Developers. Storage guide. https://zarr.readthedocs.io/en/latest/user-guide/storage/

  2. WebDataset contributors. The WebDataset format. https://github.com/webdataset/webdataset#the-webdataset-format

  3. The HDF Group. Introduction to SWMR. https://support.hdfgroup.org/documentation/hdf5/latest/_s_w_m_r_t_n.html

  4. h5py contributors. Single Writer Multiple Reader (SWMR). https://docs.h5py.org/en/latest/swmr.html

  5. SigMF Contributors. Signal Metadata Format Specification. https://sigmf.org/

See Also