Storage layout¶

Signal Dataset is the only layout RFGen writes. There is one store, one record shape, and one on-disk format; the HDF5 and WebDataset layouts that once sat beside it are gone, along with the configuration fields that described them.

Signal Dataset root¶

RFGen delegates the physical shard and index format to Signal Dataset 0.2. The upstream persisted-format contract is authoritative; applications must not construct internal filenames or list a dataset directory or GCS prefix.

Workers call signal_dataset.write_shard(...) with a work_id and attempt. Each successful worker returns a ShardArtifact. The coordinator supplies all receipts and expected work IDs to signal_dataset.publish(...). Publication writes the root last, so readers see no snapshot until all expected shards are present.

RFGen adds rfgen_expected_work_ids to root metadata. Record IDs are content-derived from the canonical Signal Dataset encoding after substituting RFGen’s fixed identity placeholder. IDs cover fields, field axes, scene ID, and record metadata. RFGen calls the public signal_dataset.record.encode(...); the placeholder removes the caller-supplied ID from the identity calculation and avoids a recursive self-hash. Tests cover input-ID insensitivity, field/axis/metadata sensitivity, and write-time identity verification.

Open the root with SignalDatasetStore rather than addressing shards:

from rfgen.storage import SignalDatasetStore

dataset = SignalDatasetStore().open("./rfgen-output")
record = dataset[0]
metadata = dataset.metadata(0)

Local and GCS roots have the same logical contract. GCS access requires the rfgen[gcs] extra and generation-pinned reads supplied by Signal Dataset.

Root build provenance¶

Publication stamps exactly three keys on the dataset root, all plain strings and all always present. Read them with rfgen inspect <dataset>, which prints them under provenance.

Key	Answers	Values
`rfgen.version`	Which generator code	The installed distribution version. Never absent — the running code knows its own version.
`rfgen.git_commit`	Which generator code, precisely	`<40-hex sha>`, `<40-hex sha>-dirty`, or `absent` when the commit cannot be established honestly.
`rfgen.backends`	Which backends the corpus could depend on	See below.

rfgen.backends lists the backend capabilities declared by the node classes this graph can reach — through any branch of any select, conditional or repeat body, not just the branches a particular sample took. A node that declares no capability but is marked environment-sensitive is included too, as the bare marker undeclared: it says something here depends on an environment and will not name it. Entries are joined with |, and each entry has the form capability=distribution@version, with @absent for a distribution the environment does not have installed:

sionna=sionna@2.0.1|torchsig=torchsig@2.1.1

Three values are not backend lists, and they mean three different things. Do not read any of them as a stronger claim than it makes:

Value	Means	Does not mean
`none`	No node in this graph declares a backend capability, and none is marked environment-sensitive.	Not “this corpus is reproducible anywhere”. A node can depend on an unversioned backend without declaring one, and without being marked.
`unrecorded`	The publisher was never told which configuration produced this corpus, so it could not look.	Not “there are no backends”.
`unavailable`	The publisher had the configuration but could not bind its graph in that process, so the question went unanswered.	Not “there are no backends”, and not a corpus defect — the shards are unaffected.

What this does and does not let you conclude. Two corpora with the same rfgen.backends were built against the same declared backend versions; two with different values were not, and that difference is a candidate explanation when their records differ. The list is deliberately a superset of what record identity folds — it records what nodes declare, not what their reproducibility class claims — so it can make two identical corpora look distinguishable, but it will not make two different ones look identical. It says nothing about dependencies no node names, and it is not a dependency lock.

Native record fields¶

Unified scene-observation envelope¶

One SceneObservationRecord is encoded as exactly one signal_dataset.Record. Primary field names have the form projections/<projection_id>/<local_field_name>. There is no fixed rank: communications receiver IQ, radar pulse/fast-time tensors, labels, and future projection fields retain their semantic axes. Coordinate arrays generated from nonuniform coordinate contracts are stored as separate auxiliary fields.

The strict metadata["rfgen"] value uses envelope schema rfgen.scene_observation.v1 and contains:

record_type, record_schema_version, observation_id, and the complete canonical ScenePlan reference and payload;
projection descriptors with selector, contract version, validated-parameter identity, and implementation identity, which binds the runtime identity;
field_owners, primary_fields, and auxiliary_fields mappings;
projection interaction coverage and metadata; and
invocation provenance with the sample ordinal, sample seed, and each projection seed.

The SDS adapter validates exact envelope keys on decode. Consumers should use its metadata_view or store metadata reads when they do not need tensors. Signal Dataset loads the numerical fields of a selected record together, so observation.max_tensor_bytes is both an encoding and consumer-memory boundary.

What one record actually holds¶

A single-receiver record from a run declaring both shipped projections, read back with the SignalDatasetStore().open(...) route above:

projections/communications_rx/receivers/rx0/iq                  complex64  (20000,)      (time)
projections/communications_rx/labels/boxes/extent               float64    (1, 4)        (box, extent)
projections/communications_rx/labels/boxes/identity             int32      (1, 2)        (box, attribute)
projections/communications_rx/labels/segmentation               int16      (1024, 79)    (frequency, time)
projections/communications_rx/labels/segmentation/coordinates/time_offset_s
                                                                float64    (79,)         (time)
projections/radar_rx/components/baseband                        complex64  (1, 4, 400)   (frame_tx_rx_channel, pulse, fast_time_sample)
projections/radar_rx/components/noise                           complex64  (1, 4, 400)   (frame_tx_rx_channel, pulse, fast_time_sample)
projections/radar_rx/components/baseband/coordinates/pulse_start_offset_s
                                                                float64    (4,)          (pulse)
projections/radar_rx/components/noise/coordinates/pulse_start_offset_s
                                                                float64    (4,)          (pulse)

Nine fields, one record, one scene_plan_reference, and field_owners mapping every name back to the projection that produced it. Neither domain’s rank, rate, or axis names resemble the other’s, and neither was adjusted to accommodate it.

Two properties of that listing are contracts rather than incidental:

Radar publishes baseband and noise as separate fields, for the reason given in Communications sums, radar separates.
Receivers are named subtrees, never separate records. A second receiver appears as receivers/rx1/iq beside rx0, and the segmentation raster gains a leading receiver axis; one sample stays one record either way. See Multi-RX record layout.

Time coordinates, and how two domains align¶

A record holds fields on unrelated grids — communications IQ at one sample rate, radar pulses at another — and never resamples them onto a common one. What makes them comparable is that every time-bearing field carries a time_coordinates block stating how to turn its own indices into scene time. It sits at the top level of the field’s metadata mapping — not inside the nested metadata key that the box fields’ frames use:

from rfgen.storage import SignalDatasetStore

record = SignalDatasetStore().open("./rfgen-output")[0]
coordinates = record["projections/radar_rx/components/baseband"].metadata[
    "time_coordinates"
]

The typed route is equivalent and does the arithmetic for you: SDSObservationAdapter().decode(record) returns fields whose time_coordinates expose scene_time((pulse, sample)) and its single-float approximation scene_time_s(...).

Two fields describe one timeline when their time_reference strings are equal. That string is the minted plan’s content-bound reference (scene-plan:<scene_id>:<plan hash prefix>), so equality means the same scene, not merely the same run. It is the only join a consumer needs; there is no shared axis object and no requirement that the grids match.

Unequal time_reference strings mean the fields belong to different minted scenes, and no arithmetic on this page relates them. There is no fallback through scene_id or a run id: two scenes have independent epochs, so a difference here is not a scaling problem to repair but a statement that the two observations were never of the same world.

Every block carries kind, time_reference, and a time_composition naming the arithmetic. The composition is stated rather than implied so a reader never has to guess whether an offset is absolute or relative to the epoch. Axis and count keys are named per kind rather than universally, because a pulsed grid has two temporal axes and the others have one:

`kind`	`time_composition`	Scene time of index	Axis and count keys	Used by
`uniform`	`reference_origin_s_plus_offset_s`	`origin_s + n / sample_rate_hz`	`axis`, `count`	communications receiver IQ
`pulsed`	`event_origin_s_plus_pulse_start_offset_s_plus_fast_time_offset_s`	`event_origin_s + event_start_offset_s + pulse_start_offset_s[p] + k / sample_rate_hz`	`pulse_axis`, `pulse_count`, `fast_time_axis`, `fast_time_count` — no `axis` or `count`	radar baseband and noise
`explicit`	`origin_s_plus_explicit_offset_s`	`origin_s + offsets_s[i]`	`axis`, `count`	segmentation rasters
`segmented`	`origin_s_plus_segment_offset_s_plus_sample_offset_s`	`origin_s + segment.origin_offset_s + (i - segment.start_index) / segment.sample_rate_hz`	`axis`, `count`, plus per-segment `start_index`, `count`, `origin_offset_s`, `sample_rate_hz`	nonuniform schedules

Indexing a pulsed block for axis or count raises KeyError. It is the one kind whose grid is two-dimensional in time, and it is the kind the cross-domain alignment below depends on.

The epoch and the offsets are kept apart on purpose. A scene epoch can be large while activity offsets are microseconds; folding them into one float would spend the precision the offsets need on magnitude they do not have.

pulsed and explicit publish their per-index offsets as a separate auxiliary field beside the capture — <field>/coordinates/pulse_start_offset_s and <field>/coordinates/time_offset_s — because those offsets are data, not a formula. pulsed additionally declares schedule_extent_offset_s (what the pulse schedule reserves, the final PRI included) and support_span_offset_s (where samples actually exist). They differ whenever the last pulse’s dwell is shorter than its PRI, and conflating them overstates the occupied window.

A worked alignment¶

To place one field’s index on another’s grid, evaluate each field’s own composition to a scene time, then invert the target’s. For a uniform communications capture that inversion is n = (t_scene - origin_s) * sample_rate_hz.

From a single-receiver record carrying both projections — communications at 1 MHz for 20 000 samples, radar at 10 MHz over four 60 µs PRIs with a 400-sample (40 µs) dwell, both on a scene epoch of 0.25 s:

communications  origin_s 0.25, 1 MHz, 20000 samples
                first sample 0.250000 s, last sample starts 0.269999 s
                span_offset_s [0.0, 0.020000]      (end is one sample period past the last start)

radar           event_origin_s 0.25, four 60 us PRIs, 400-sample dwell
                schedule_extent_offset_s [0.0, 0.000240]   4 x 60 us the schedule reserves
                support_span_offset_s    [0.0, 0.000220]   180 us + 40 us of actual samples

pulse 0 starts at 0.250000 s = communications sample     0
pulse 1 starts at 0.250060 s = communications sample    60
pulse 2 starts at 0.250120 s = communications sample   120
pulse 3 starts at 0.250180 s = communications sample   180

This is the schedule_extent_offset_s versus support_span_offset_s case above, with numbers: the schedule reserves 240 µs, the samples occupy 220 µs, and the 20 µs difference is the last PRI minus its dwell. Reporting 240 µs as the occupied window would overstate it.

The radar burst therefore occupies the first 220 µs of a 20 ms communications capture. A consumer recovers that by evaluating each field’s own composition — no resampling, and no assumption that either grid is the reference.

Alignment is not permission to add. Two fields landing on the same instant may still be unsummable — radar baseband sits at a dechirped beat-frequency plane and communications IQ does not, so those two align in time and are never summed. The summation-eligibility rule is owned by Measurement planes are load-bearing.

The active-radar projection is available; its scientific qualification is bounded. Consult Radar Response before making a dataset claim from the fields above.

Label fields¶

A projection publishes its labels as typed fields alongside its captures. The names below are the local names; as stored, every field is prefixed by the projection that produced it, so the box extent of the receiver projection is read as projections/receiver/labels/boxes/extent.

Field	dtype	axes	Columns
`labels/boxes/extent`	`float64`	`box, extent`	`start_sample`, `duration_samples`, `low_freq_hz`, `high_freq_hz`
`labels/boxes/identity`	`int32`	`box, attribute`	`class_id`, `emitter_index`
`labels/segmentation`	`int16`	`[receiver,] frequency, time`	single-label: the cell’s `class_id`, or `-1` for background
`labels/segmentation`	`uint8`	`[receiver,] class, frequency, time`	multi-label: one plane per class

The leading receiver axis on the raster is present when the scene has more than one receiver and absent when it has one, so a consumer must size the array from the field’s declared axes rather than from a fixed rank. The box fields never gain that axis: boxes are declared once per record.

Two box fields rather than one, because a box carries two kinds of number: storing a class identifier as a float invites arithmetic on it, and a frequency edge is not an integer.

Read the frame off the field, never assume it. labels/boxes/extent carries frequency_frame, receiver_center_hz, time_frame, and sample_rate_hz in its field metadata. The edges are offsets from the receiver’s tuned centre, not absolute RF, and the sample indices are in the receiver’s rate. An emitter at 2.4005 GHz with 100 kHz of bandwidth, seen by a receiver centred at 2.4 GHz, publishes high_freq_hz = 550000.0 (2.4005e9 - 2.4e9 + 100e3 / 2).

One frame per record is unambiguous rather than approximate: a scene whose receivers do not share a centre frequency is refused at label time, so a record that exists has exactly one receiver frame for its boxes to be in.

duration_samples is a duration, not a stop index: the interval is half-open [start_sample, start_sample + duration_samples).

A scene with no declared emitters writes neither box field; an observation axis of length zero is refused. The record’s bboxes provenance metadata is present either way, and carries the same boxes — annotation reads metadata only, and a metadata shard holds field descriptors but no field values.

Post-generation annotation artifacts¶

A run publishes a dense annotation set as an overlay under the snapshot root. It rewrites no signal shard.

Signal Dataset annotation sets¶

<dataset_uri>/
    rfgen/annotations/<annotation_type>/<template_id>/<run_id>/
        work-<digest>/config.json               staged batch configuration
        work-<digest>/bootstrap.py              the PySpark wrapper the batch runs
    annotations/
        catalog.json                            pointer to the current catalog revision
        catalogs/<sha256>.json                  the revision listing every published set
        sets/
            <run_id>/
                <publication_id>/
                    shards/<xx>/<sha256>.json     create-only staged-shard pointers
                    objects/<xx>/<id>.arrayrecord the annotation rows
                    root.json                     the published set

The set is named for the run’s run_id. Staged shards live under a publication_id derived from everything that decides their contents (run identity, model, source snapshot, shard size, and the requested subset), so a re-run whose configuration changed stages fresh work instead of adopting the previous attempt’s answers. A run publishes nothing outside sets/<run_id>/<publication_id>/: publication writes that prefix’s root.json and only then updates annotations/catalog.json to point at it. No signal shard is ever modified.

The work-<digest> prefix is written only by an execution_mode: dataproc_serverless submission, and it is written at submit time rather than at publication: the batch has to read its configuration from somewhere the cluster can reach. It sits outside annotations/, which the Signal Dataset format reserves, and under a work-<digest> prefix naming everything that decides what the batch would do, so different work stages beside the old work rather than colliding with it. Those objects are create-only and survive a failed batch on purpose: a later submit re-uses them instead of re-staging. They are inert to every reader, because the format resolves everything by deterministic path derivation and never lists a directory.

A published set is dense: one row per source ordinal. RFGen’s own AnnotationRowV1.status (succeeded, failed, skipped) is translated at the publisher boundary onto the format’s vocabulary, so a row read back carries success, failed, or skipped. Read a set through the signal_dataset package against the same root, for example signal_dataset.open(dataset_uri).annotations[run_id]. rfgen inspect does not read annotation sets; its --annotations flag named a store this build cannot open and was removed with it.

The set root also carries an rfgen.row_counts mapping in its metadata: one count per row status, written at publish from the shards being published. It is there so a reader can answer did this run work? without one object read per record. A set published by an RFGen build older than the tally has no such key at all, which is how a reader tells “no failures” from “not tallied”.

Non-finite declared metadata¶

Declared metadata is normally finite. Two quantities are exceptions, and both because the generator can legitimately produce them. Everything else non-finite is refused at write time by rfgen.storage.ensure_declared_metadata_is_finite.

Value	Where it is legitimate	Meaning
`±inf`	any path whose leaf name contains `snr_db` or `sinr_db`	`10*log10` of a component with no noise, or with no power
`NaN`	`scene.realized_snr_db_stats.`, only* when `scene.extras["realized_snr_db_stats_status"] == "unavailable"`	the statistic was not computed

The exemptions are classes in rfgen.storage.nonfinite_exemptions, so a new declared quantity registers one and states its own reasoning; the fragment match on the ratio names is deliberate, so a new variant is not refused on the day it is added.

A data-quality filter must not simply drop non-finite values. Every propagation backend except AWGN declares no SNR, so +inf is the ordinary case rather than the edge one, and dropna over these fields discards most of a corpus. NaN is the signal worth acting on, and it cannot reach a published record.

Non-finite values are stored as tagged JSON objects and restored with rfgen.core.metadata.decode_rfgen_metadata.

Format boundaries¶

StorageConfig carries backend, path, and assets_path and nothing else. compression, chunk_samples, dataset_filename, and record_axis described the HDF5 and WebDataset layouts; neither store ships, so the fields are gone from the model and a config that sets one is refused by name.
SigMF capture files may be read by calibration utilities, but SigMF is not a generation backend.
Annotation reads a published Signal Dataset snapshot and writes a dense annotation set beside it.
The declared-evidence builder does not yet interpret the unified scene-observation envelope. Projection-aware aggregate annotation is deferred.