IQ Layout Policy¶

The canonical shapes, dtypes, and conversion boundaries for IQ data throughout the rfgen pipeline. This page is the single source of truth; all other pages link here.

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, config field names, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

Summary table¶

Context	Shape	Dtype	Notes
In-memory Python API (single-RX)	`(2, N)`	`float32`	Channel-first real/imag
In-memory Python API (multi-RX)	`(num_rx, 2, N)`	`float32`	RX-major, channel-first per RX
Zarr canonical store (single-RX)	`[N]`	`complex64`	Interleaved float32 pairs internally
Zarr canonical store (multi-RX)	`[num_rx, N]`	`complex64`	Interleaved float32 pairs internally
WebDataset shards (`.iq.npy`)	`[N]` or `[num_rx, N]`	`complex64`	Same as Zarr
HDF5 TorchSig interop	`[N]` or `[num_rx, N]`	`complex64`	Byte-exact round-trip required
SigMF captures (HIL input)	`[N]`	`complex64` or `complex32`	Per SigMF spec; converted at ingest

In-memory representation (Python API)¶

The type alias IQ is defined as:

from typing import NewType
import torch

IQ = NewType("IQ", torch.Tensor)
# Single-RX:  shape (2, N), dtype float32
# Multi-RX:   shape (num_rx, 2, N), dtype float32

Rules:

Channel-first layout. Dimension 0 (for single-RX) or dimension 1 (for multi-RX) is the real/imag channel. Index 0 is the real component, index 1 is the imaginary component.
float32, not complex64. PyTorch autograd and many neural network layers require real-valued tensors. The framework uses float32 in-memory to avoid implicit conversions that break gradient flow.
No complex torch.Tensor in the public API. Any stage that receives or produces complex data converts to (2, N) float32 before returning.
IQ is a NewType, not a subclass. Type checkers see IQ as a more specific form of torch.Tensor; at runtime it is a plain tensor.

Multi-RX shape¶

For scenes with more than one receiver, IQ shape is (num_rx, 2, N):

Axis 0: receiver index (0-indexed, matching rx_index in SignalMetadata).
Axis 1: real/imag channel (0 = real, 1 = imag).
Axis 2: sample index.

When StorageConfig.record_axis is RecordAxis.PER_RX, the framework splits the (num_rx, 2, N) tensor into num_rx records each with shape (2, N) before writing.

When record_axis is RecordAxis.JOINT, the full (num_rx, 2, N) tensor is stored as a single record.

Storage representation (Zarr canonical)¶

Zarr stores IQ as complex64:

Single-RX: shape [N], dtype complex64.
Multi-RX (joint): shape [num_rx, N], dtype complex64.
complex64 is two interleaved float32 values per sample (real then imaginary). This is the NumPy / Zarr default.

Zarr chunk parameters:

Property	Value
dtype	`complex64`
shape	`[N]` (single-RX) or `[num_rx, N]` (multi-RX joint)
chunks	`(min(1<<20, N),)` for 1-D; `(num_rx, min(1<<20, N))` for 2-D
codec	`blosc:zstd` level 5, byte-shuffle

See Storage layout IQ array for the full Zarr array spec.

WebDataset shards¶

WebDataset shards mirror the Zarr dtype. Each record is serialized as a .iq.npy file inside a .tar shard:

dtype: complex64
shape: [N] (single-RX) or [num_rx, N] (multi-RX joint)
Format: NumPy .npy with full header (version 1.0)

The .npy header encodes shape and dtype. Loaders reconstruct shape from the header; no separate metadata is needed to distinguish single-RX from multi-RX records.

HDF5 TorchSig interop¶

The HDF5 TorchSig-compatible shard uses complex64 with the same shape as Zarr. The byte-exact round-trip test confirms that:

complex64 Zarr → read → float32 (2, N) → write → complex64 HDF5

produces byte-identical values. See Reference / TorchSig Interop for the converter contract.

SigMF captures (HIL input)¶

SigMF files from hardware-in-the-loop captures are complex64 (or complex32 for some SDR platforms) per the SigMF spec. The capture ingest path converts to (2, N) float32 at load time, before any emitter or channel transformation sees the data. The original SigMF metadata is preserved in extras["sigmf_meta"].

Conversion boundary¶

The store layer owns the only authorized conversion between in-memory (2, N) float32 and storage [N] complex64.

BaseStore.write(record): converts torch.Tensor shape (2, N) float32 to complex64 before writing.
BaseStore.read(sample_id): converts complex64 back to torch.Tensor shape (2, N) float32 after reading.

The per-record contract test confirms round-trip is bit-identical:

# Proposed contract test shape
iq_original: IQ  # shape (2, N), float32
store.write(record_with(iq=iq_original))
record_back = store.read(sample_id)
assert torch.equal(iq_original, record_back.iq)

No emitter, channel transformation, labeler, or annotator converts between these two representations. Any transformation that needs complex arithmetic receives (2, N) float32 and must produce (2, N) float32.

Prohibited patterns¶

These patterns are explicitly disallowed and will fail the contract test:

Returning complex64 from BaseEmitter.generate.
Returning complex64 from BaseChannel.apply.
Storing (2, N) float32 directly in Zarr (wrong dtype).
Storing complex64 directly as a torch.Tensor in-memory (wrong dtype for autograd).

Dataset Sizing Formula¶

Use this formula to estimate raw IQ bytes before compression:

raw_iq_bytes = num_samples × duration_s × sample_rate_hz × 8_bytes_per_complex64 × num_rx

Blosc zstd level 5 achieves roughly 10-20x compression on typical RF IQ (the ratio depends heavily on signal content; noise-floor-dominated recordings compress less). Use 10x as a conservative estimate.

compressed_iq_bytes ≈ raw_iq_bytes / 10

Worked examples¶

Scenario	Samples	Duration	Sample rate	RX	Raw IQ	Compressed IQ (est. ×10)
Quickstart narrowband	100	100 ms	10 Msps	1	~800 MB	~80 MB
First scene (multi-emitter)	1 000	100 ms	200 Msps	1	~160 GB	~16 GB
Dense urban `_md`	100 000	100 ms	200 Msps	1	~16 TB	~1.6 TB
Dense urban `_lg`	1 000 000	100 ms	200 Msps	1	~160 TB	~16 TB
Dense urban `_lg` (4-RX)	1 000 000	100 ms	200 Msps	4	~640 TB	~64 TB

Labels (bbox + segmentation + per-emitter metadata) are typically 1-5 % of the IQ footprint at this sample rate and density. Text annotations add less than 1 KB per sample (a few MB total for 1M samples).