Design Decisions

The choices that shape the framework, with rationale. Quick reference; deeper rationale for each section follows inline.

Decision

Choice

Rationale

Substrate

Wrap TorchSig v2.1.x; do not fork

TorchSig provides BaseSignalGenerator, HDF5 I/O, 60+ comms generators, MIT-licensed. We add what it lacks (heterogeneous emitters, advanced channels, scene events, text). Forking would couple us to upstream changes we don’t need.

Scene composer

Custom (replaces TorchSig’s __generate_new_signal__)

TorchSig’s rectangle-overlap loop is single-RX, time-uniform, and rejects on overlap. We need event timing (PRI, hops, beacons), heavy-tailed power, optional cochannel, and multi-RX.

Channel backend

Pluggable: TorchSig / Sionna RT / Sionna PHY (TDL/CDL) / custom

One config switch swaps fast stochastic models for ray-traced site-specific channels. Sionna’s Apply*Channel accepts arbitrary IQ, so we keep emitter generation independent.

Hardware impairments

Custom torch module, applied per virtual device

Sionna ships a thin impairment library. We model CFO, SFO, IQ imbalance, phase noise (Leeson), PA nonlinearity (Rapp/Saleh), DAC quantization explicitly. These are the basis of RF fingerprinting.

Bandwidth mode

Single code path; num_signals_max and bandwidth are config knobs

Same engine produces narrowband (TorchSig Sig53-style) and wideband (TorchSigWideband-style) recordings. No separate dataset classes.

Label format

Bbox + per-bin segmentation + per-emitter metadata, all stored

Different downstream heads want different formats. Storing all three avoids regeneration.

Text annotations

Template, then LLM rewrite (RF-GPT / WavCaps pattern); 5 annotation types

Cheap ($10–60 per 100K samples with Gemini Flash), grounded in verified metadata, controllable hallucination. Bulk model + verification subset (ShareGPT4V pattern).

Taxonomy

Hierarchical tags across all source label spaces

Sig53, RadioML, DroneRF, WiSig, NIST CBRS, custom synth all use different label vocabularies. Hierarchical tags (Qwen-Audio pattern) unify them without lossy mapping.

Configuration

Hydra config groups + Pydantic v2 validation

Hydra wins on swappable backends; Pydantic catches invalid configs before expensive runs. Used by NeMo, AudioCraft, lerobot.

Storage

Zarr canonical + WebDataset training shards + SigMF for captures + HDF5 for TorchSig interop

Object-store native, concurrent writers, random-access re-annotation. See Why Zarr below.

Orchestration

GCP Managed Spark serverless (Phase 1) + Vertex AI Batch Prediction (Phase 2)

GCP-native, no cluster ops, autoscale, billed per-job. Spark fits embarrassingly-parallel shard generation; Vertex Batch Prediction handles Gemini rate limits and retries natively.

Validation

HIL (USRP/HackRF cabled + OTA) on a 1K-sample subset

Pure-sim datasets have no sim2real grounding. HIL validates that synthesized fingerprints/channels match real-radio statistics.

Closed enums

StrEnum from rfgen.enums; open-set names stay str

Single source of truth for fixed value sets; no magic strings in plugin code; YAML stays readable. See Closed enums use StrEnum below.

Why Zarr (and not HDF5) as the canonical store

Most reference RF datasets (RadioML, TorchSig’s WidebandSig53, etc.) ship as HDF5. We picked Zarr instead. Three reasons:

Object-store native. Zarr stores each chunk as its own object; HDF5 is one monolithic file. On GCS or S3, a Spark job with hundreds of workers can write Zarr chunks in parallel directly to the bucket. With HDF5, every worker would need a local staging copy and a final merge step. Cloud-native generation is a primary use case for us, so this asymmetry is decisive.

Concurrent and resumable writes. Phase 1 (Spark, many workers) and Phase 2 (inference-grounded annotation, separately resumable) both append to the same dataset. Zarr append is “create a new chunk object”, no locking required. HDF5 has SWMR, but it is fragile across cloud filesystems and forbids structural changes mid-write, which is incompatible with appending text columns to an already-stored dataset.

Random-access reads for re-annotation. Phase 2 re-reads per-emitter metadata for arbitrary samples without scanning the full file. Zarr’s chunk index makes this O(1); HDF5 random access on remote blob storage pulls byte ranges through one file handle, which serializes badly under concurrent annotators.

Why HDF5 is fine for RadioML and TorchSig. Those datasets are static, single-machine, write-once / read-many on local disk. Different workload, different optimum. We have nothing against the format for that case.

HDF5 stays as a delivery format. TorchSig-trained models expect HDF5, so we ship a byte-exact HDF5Store exporter. See Reference / TorchSig Interop for the round-trip contract and Storage Layout / HDF5 for the on-disk schema. The split is: Zarr for working storage during generation, HDF5 for handoff to TorchSig consumers.

What we reuse vs. what we build

Reuse

  • TorchSig: BaseSignalGenerator, Impairments, HDF5 file handler, comms emitter library, rectangle-based bbox utilities.

  • Sionna: sionna.rt (ray tracing), sionna.phy.channel (TDL/CDL), Apply*Channel (arbitrary-IQ channel application), 5G NR PHY.

  • 3GPP / ITU: TDL-A/B/C tap definitions, ITU multipath profiles.

  • Protocol stacks: pyModeS (ADS-B bits), gr-lora_sdr / LoRaPHY (LoRa TX), gr-ieee802-11 (Wi-Fi a/g/p), gr-ieee802-15-4 (Zigbee), srsRAN_4G via ZMQ (LTE TX waveforms).

  • Infra: Hydra, PySpark, Managed Spark serverless, Vertex AI Batch Prediction, Zarr, WebDataset, SigMF.

  • Patterns: RF-GPT prompt patterns, WavCaps 3-stage pipeline, RF-Analyzer’s PAES metric.

Build (the framework’s distinctive contribution)

  • Heterogeneous emitter zoo wrapping all of the above under one BaseEmitter-compatible interface.

  • Per-device fingerprint module (CFO + SFO + IQ imbalance + phase noise + PA + DAC).

  • Custom multi-emitter scene composer with event timing, density realism, optional multi-RX.

  • Joint label writer (bbox + segmentation + metadata).

  • Inference-grounded annotation pipeline backed by verified metadata; hallucination control + verifier.

  • Taxonomy unifier across all source datasets.

  • Hydra config groups for the full surface.

  • HIL validation harness.

  • Sim2real benchmark + dataset-statistics auditor.

Tensor library: PyTorch canonical, NumPy at boundaries

torch.Tensor is the canonical tensor dtype across the framework. Every layer of the pipeline produces and consumes torch tensors for IQ, spectrograms, segmentation masks, and any per-sample numeric payload. The canonical IQ alias is IQ = NewType("IQ", torch.Tensor).

Why PyTorch:

  • TorchSig is PyTorch-native, and Sionna 1.x has a PyTorch port (Sionna RT and Sionna PHY). Our two largest backend dependencies are PyTorch-first; matching them removes a conversion step at every layer boundary.

  • Foundation-model training pipelines that consume our datasets are overwhelmingly PyTorch. Same dtype, same device semantics, no bridge code at the read side.

  • Heterogeneous scene shapes (variable emitter counts, variable burst durations, mixed sample rates per emitter) suit PyTorch’s eager mode. JAX-style static-shape tracing would force padding or lax.scan for marginal benefit on data-generation workloads.

  • Tooling adoption: FlashAttention, vLLM, NVIDIA Modulus, Triton kernels, and most CUDA-side ecosystem work is PyTorch-first.

Why not JAX as the canonical dtype:

  • No JAX-native RF library equivalent to TorchSig; we would reimplement signal generators we get for free elsewhere.

  • Random-key threading, immutable arrays, and traced debugging add onboarding cost without a use-case-specific payoff for batched data generation.

  • Adoption gravity outside Google is concentrated in research labs, not the ML training stacks our datasets feed.

Why not NumPy as the canonical dtype:

  • No native GPU. Sionna RT and large FFTs need GPU to be tractable at production dataset scale.

  • Conversion tax at every torch boundary (training, TorchSig, Sionna PHY).

Where NumPy is correct, by policy:

  • TorchSig interop. TorchSig is numpy at the wire; conversion happens once at the interop boundary.

  • Zarr structured-array dtypes. np.dtype([(...)]) for label arrays and per-emitter records. No torch equivalent.

  • Audit and statistics. Aggregate scalar arrays consumed by SciPy. Numpy is the conventional dtype for these consumers.

  • HDF5 export. h5py is numpy-native.

  • Build dependency. TorchSig 2.x requires NumPy 2.x; numpy is a transitive dep.

What this means in practice:

  • Every dataclass field, function signature, and code example in this repo that holds IQ or per-sample tensor data MUST type it as torch.Tensor (or the IQ / Spectrogram aliases).

  • Conversion to numpy happens explicitly via .cpu().numpy() at the boundaries listed above, not implicitly mid-pipeline.

  • If a future consumer needs JAX, the bridge lives at the read side (e.g., a thin torch_to_jax adapter), not upstream in the generator.

ABC Pluggability Principle

The framework owns ABCs as the contract between layers. Concrete backends inherit from ABCs. The hierarchy rule is strict:

  • Allowed: abstract → abstract → concrete. A per-slot abstract subclass under a layer ABC is permitted, and concrete implementations then inherit from the per-slot ABC.

  • Forbidden: concrete → concrete. Subclassing one concrete class with another concrete class breaks the pluggability model. A concrete umbrella subclassed by other concrete classes prevents independent swapping of each backend. The per-slot ABC pattern exists specifically to eliminate this: each backend at a given slot inherits from the slot ABC, not from a sibling backend.

Per-layer ABCs are the layer’s primary contract:

Layer

Primary ABC

Emitters

BaseEmitter

Channels

BaseChannel

Scenes

BaseSceneComposer

Labels

BaseLabeler

Annotations

BaseAnnotator

Storage

BaseStore

Per-transformation ABCs are allowed where multiple implementations exist or are planned at the same pipeline transformation. They narrow the contract to one transformation slot and pin its Transformation ClassVar. The Channels layer uses this pattern extensively: 14 per-transformation ABCs partition the pipeline into the four groups (TX, CHANNEL, RX_CAPTURE, RX_HARDWARE) defined by Group.

Per-transformation ABC

Operation

Group

Parent

BaseDACQuantization

DAC quantization

Group.TX

BaseChannel

BasePANonlinearity

PA nonlinearity

Group.TX

BaseChannel

BaseTXPhaseNoise

TX phase noise

Group.TX

BaseChannel

BaseTXIQImbalance

TX IQ imbalance

Group.TX

BaseChannel

BaseCFO

CFO

Group.TX

BaseChannel

(single propagation slot)

Channel propagation

Group.CHANNEL

BaseChannel directly

BaseRXMixer

RX mixer

Group.RX_CAPTURE

BaseChannel

BaseIFFilter

IF filter

Group.RX_CAPTURE

BaseChannel

BaseResampler

Resampler

Group.RX_CAPTURE

BaseChannel

BaseLNANoise

LNA noise

Group.RX_CAPTURE

BaseChannel

BaseADCQuantization

ADC quantization

Group.RX_HARDWARE

BaseChannel

BaseRXPhaseNoise

RX phase noise

Group.RX_HARDWARE

BaseChannel

BaseRXIQImbalance

RX IQ imbalance

Group.RX_HARDWARE

BaseChannel

BaseAGC

AGC

Group.RX_HARDWARE

BaseChannel

Channel propagation reuses BaseChannel directly because Sionna’s channel objects already bundle path loss, multipath, shadowing, and Doppler; splitting that slot would invent an interface no real backend implements (see Channel pipeline: 4 groups, 14 transformations, no scene rate below).

The Scenes layer uses per-slot ABCs for placement strategies: BaseTimePlacement and BaseFrequencyPlacement under BaseSceneComposer.

Supporting ABCs are part of a layer’s pluggability contract when the layer requires more than one ABC for different lifetimes or roles:

This principle is enforced by documentation convention and by the registry, which rejects registrations that do not inherit from the expected per-slot or per-layer ABC.

Closed enums use StrEnum, open-set names stay str

Every config field whose set of valid values is fixed and owned by the framework is typed as a StrEnum member from rfgen.enums, not as Literal[str, ...]. Every config field whose value is a registered plugin name stays str and is resolved through the registry at runtime.

The split:

Field

Type

Why

density.mode, frequency_placement.strategy, time_placement.strategy, power.mode, overlap.policy, array, pa_model, family, snr_distribution, provider, types, backend, compression, record_axis

StrEnum (DensityMode, FrequencyPlacementStrategy, TimePlacementStrategy, ScenePowerMode, SceneOverlapPolicy, ArrayGeometry, …)

Closed; framework version controls the value set

ExecutorConfig.name, LabelConfig.name

str

Open; any pip install-supplied plugin can register a new name

Why this over Literal[str, ...]:

  • Single source of truth. Each enumeration is defined once in rfgen.enums. Renaming a value is a one-file change, not a global grep over plugin code.

  • No magic strings inside the framework. Plugin code references RecordAxis.PER_RX, not "per_rx". IDE autocomplete works; typos surface at edit time.

  • Pydantic validates at the boundary, identical to Literal. Unknown values are rejected at config load with a clear error.

  • YAML stays readable. record_axis: per_rx deserializes to RecordAxis.PER_RX because StrEnum members are strings at runtime.

Why registered names stay str: the channel and executor registries are open. A new plugin shipped via pip install rfgen-acme-channel registers name="acme" at import time. Closing that to a StrEnum would force a framework release every time a plugin shipped, defeating the registry.

This convention is rolled out incrementally. Phase 1 (this entry) covers the user-facing config schema in api/config and config-schema. Phase 2 will sweep the remaining Literal[...] sites in api/labels.md (segmentation modes, overlap strategies), api/executors.md (shard partitioning), api/registry.md (plugin kind), api/credentials.md (cloud discriminators), reference/schemas/annotation-templates.md (QA types), and reference/schemas/label-schema.md (segmentation modes).

Channel pipeline: 4 groups, 14 transformations, no scene rate (2026-06-17)

The channel pipeline was originally a five-stage chain (TX_HARDWARE → FREQUENCY_SHIFT → PROPAGATION → RX_FRONTEND → AWGN) running at one shared scene sample rate. That model carried two anchored assumptions: every emitter shares a baseband with every other emitter in the same scene, and every receiver sees the same band. Both assumptions break down for real customer scenarios (a multi-band device with a 5G tuner at 3.5 GHz and a Wi-Fi tuner at 2.4 GHz on the same chassis), and both are unnecessary even for the homogeneous case.

The redesign drops the shared scene baseband and explicitly names every transformation that happens between an emitter and a stored record.

What changed

  • Five stages → four groups. Group ∈ {TX, CHANNEL, RX_CAPTURE, RX_HARDWARE}. The old Stage enum is gone. FREQUENCY_SHIFT disappears: in the new model, carrier frequency is metadata throughout pre-sum stages and is applied as a per-RX mixer at the head of RX_CAPTURE. AWGN disappears as a standalone stage: thermal noise enters at the LNA inside RX_CAPTURE, where it physically belongs.

  • Coarse stages → 14 explicit transformations. Each transformation is its own pluggable ABC (BaseDACQuantization, BasePANonlinearity, …, BaseAGC). The boundary between transformation and group follows what mature libraries actually expose. TorchSig and similar provide one class per impairment; we split those into individual slots in TX impairments, RX capture, and RX hardware. Sionna provides one channel object per scenario that bundles path loss, multipath, shadowing, Doppler, and antenna patterns together; we collapse channel propagation to a single BaseChannel slot to match Sionna’s API rather than invent a sub-stage decomposition no real backend implements.

  • No stored scene RF anchor. Each emitter sets SignalMetadata.realized_carrier_hz (absolute Hz) directly. Stored SceneMetadata does not mirror center_hz, bandwidth_hz, or sample_rate_hz. SceneConfig still carries those fields as scene-construction defaults, and ReceiverConfig.center_freq_hz, bandwidth_hz, and sample_rate_hz override them per receiver when set. Multi-band devices are first-class because stored records preserve per-emitter carriers and per-receiver tuning, not one scene-wide RF anchor.

  • Scene-level post-sum chain. Groups 1 and 2 share one pre-sum chain per scene. Groups 3 and 4 are applied per receiver after the combine point at the head of Group.RX_CAPTURE, but they come from the single scene-level ChannelPipeline, not from an rx_chain field on ReceiverConfig. Receiver-specific inputs are threaded through ChannelRxParams.

  • Three sample-rate roles. R_emitter is per emitter, native, and set by the plugin. The channel rate defaults to R_emitter. R_rx is the effective per-receiver output rate, resolved from ReceiverConfig.sample_rate_hz or, when omitted, SceneConfig.sample_rate_hz. The BaseResampler step in Group.RX_CAPTURE is the one place where the rate transitions from R_emitter to R_rx.

Why explicit transformations instead of coarse stages

The previous five-stage model bundled multiple physically distinct operations under one Stage. RX_FRONTEND carried mix, IF filter, ADC sample-rate conversion, LNA noise figure, ADC quantization, RX phase noise, RX IQ imbalance, and AGC, all behind one ABC. Plugin authors wanting to override one piece had to reimplement the whole thing or accept the bundled default.

Explicit transformations expose each operation as its own slot. Plugin authors customize what they need; defaults cover the rest. The cost is more ABCs (14 vs 5); the benefit is precise pluggability and a documentation surface that names every operation a reader can expect to see in the pipeline.

Why one slot for BaseChannel

Sionna RT, Sionna PHY (CDL, TDL, UMa, UMi, RMa), and TorchSig fading all expose a single channel object whose internals cover path loss, multipath, shadowing, and Doppler. Antenna patterns are configured per TX/RX entity, not per channel call. Splitting channel propagation into five sub-transformations would invent an interface no real backend implements; collapsing to one slot matches the upstream APIs and the current shipped execution model, which applies propagation once per (emitter, receiver) pair. A later implementation may batch several pairs into one Sionna call for throughput, but only as an internal optimization that preserves the same per-pair observable behavior.

A custom backend that wants per-effect control writes a BaseChannel subclass and is free to use any internal decomposition.

Why the framework gains from this even without multi-band scenarios

Even single-band, single-RX datasets benefit:

  • A narrowband emitter (BLE at 1 MHz BW) no longer pays the upsampling cost of being squeezed into a wideband scene rate.

  • The framework is honest about what each operation does. FREQUENCY_SHIFT was a legacy of the shared-baseband assumption; in any reasonable physical model it does not exist as a sample-domain operation.

  • The mathematical link between the framework and the literature is direct. Each named transformation maps to an operation an RF engineer would draw on a whiteboard.

Migration

This is pre-implementation. No shipped code uses the old Stage enum or the old shared-scene-baseband model. The redesign is a doc-and-schema change today; implementation will start fresh against the new contracts.

References: