Storage Layout

Shipped on-disk layouts for the Layer 7 storage backends in src/rfgen/storage.py. Zarr is the canonical backend. WebDataset, HDF5, and SigMF persist the same Record contract through backend-specific container shapes.

Shared rules

  • Default sample_id is sha256(canonical_cbor(record)).

  • A caller may supply an explicit sample_id.

  • Re-writing the same sample_id with identical canonical bytes is idempotent.

  • Canonical record bytes persist: scene_cbor, emitters_cbor, bboxes_cbor, optional text_cbor, IQ, and optional segmentation.

  • Phase 2 annotation appends do not rewrite canonical record bytes. The transactional annotation overlay below stores new annotation rows separately.

  • IQ is stored as complex64 on disk, [N] for per-RX records and [num_rx, N] for joint records.

  • Segmentation uses canonical storage dtype: int16 for single-label masks and uint8 for multi-label masks.

  • Content-addressed geometry assets live under assets/<kind>/<content_hash>/ with a required manifest.cbor.

Run manifest schema

The run manifest is run-level provenance, not a record, shard-completion marker, or asset bundle description. rfgen.manifest defines version 1 as an RFC 8785 canonical JSON envelope:

Envelope member

Contract

schema

SchemaIdentity with name rfgen.run-manifest, major at least 1, and non-negative minor version.

payload

Strict major-one RunManifestPayloadV1 data. Unknown payload fields are rejected.

extensions

Optional JSON semantic trees for forward-compatible data. Unknown top-level envelope members are preserved on parsing for the same purpose.

The major-one payload contains UUID run_id and dataset_id; UTC RFC 3339 created_at and updated_at; STAGING, COMMITTED, or ABORTED status; canonical resolved-config bytes and their SHA-256; a non-negative root seed and seed-schedule ID; sorted package and plugin provenance; source revision; asset provenance; ordered output objects; optional annotation, audit, and parent revisions; and manifest_sha256. Asset provenance requires a lowercase SHA-256 digest, an SPDX 2.3 license expression, and an absolute license URI.

Each declared output object has a non-empty key, non-negative size_bytes, lowercase SHA-256, media type, one of EXPECTED, STAGED, COMMITTED, MISSING, or CORRUPT, and optional provider version, generation, and ETag. Object order is UTF-8 key-byte order. Package and plugin rows sort by (normalized_name, version, entry_point).

The manifest self-hash is SHA-256 of RFC 8785 canonical payload bytes after omitting only manifest_sha256. The resolved-config field must already be UTF-8 RFC 8785 JSON bytes, and its digest must match those exact bytes. Hashes do not depend on received whitespace or object-member order.

parse_manifest validates the known payload and preserves unknown envelope and extension values as JSON semantic trees. It rejects an unsupported schema major, unknown major-one payload fields, invalid ordering or hashes, and values outside RFC 8785’s JSON number domain. A newer minor version or any forward-compatible envelope data is rejected when preserve_unknown=False. Packaged Draft 2020-12 JSON Schemas cover the run-manifest envelope and its ordered verification report. A report has a revision, verification mode, UTC timestamps, PASS/FAIL overall, exact counts, and rows sorted by (scope, key, code) using closed code/status vocabularies. See Manifest Verification for the read-only service that produces it.

Layer 12 supplies the persisted schema and parser. The Layer 13 repository is bound to exactly one dataset at construction and stages canonical envelope bytes immutably at manifests/<sha256>.json; stage and publish reject a manifest from another dataset. The separate Layer 17 coordinator described below supplies the fenced publication contract for callers that need it.

See Manifest API for the typed envelope, payload, canonicalization, verification report, and packaged-schema loader contracts.

Transactional publication

rfgen.transaction_coordinator.TransactionCoordinator publishes one dataset revision through a fenced, conditional LATEST update. Readers resolve one committed pointer only after every referenced object is immutable and checksum-valid. A failed publication does not change LATEST or any object already visible through an earlier revision.

The coordinator has this closed lifecycle:

NEW -> STAGING -> PREPARED -> COMMITTED
STAGING -> ABORTING -> ABORTED
PREPARED -> ABORTING -> ABORTED
STAGING or PREPARED -> EXPIRED

begin allocates a strictly increasing unsigned-64-bit fencing token per dataset and returns the original intent when the same idempotency digest is replayed. stage accepts only normalized relative keys and immutable objects with lowercase SHA-256 digests. prepare records sorted object digests and the exact pointer bytes before commit conditionally updates LATEST. renew works only for live STAGING or PREPARED leases; abort is terminal.

Leases have a 30-second to 24-hour TTL, defaulting to 10 minutes. Expiry uses provider/server UTC together with process-local monotonic elapsed time. A stale fence raises StaleFencingTokenError; a changed pointer raises CommitConflictError; and an indeterminate pointer write raises CommitOutcomeUnknownError. An unknown outcome must be reconciled with scan and repair before the commit is retried.

Backend

Immutable-object condition

LATEST condition

Local filesystem

Same-filesystem temporary file, file and directory fsync, then atomic replacement

Process-locked read/replace compare-and-swap

Amazon S3

boto3 put_object(..., IfNoneMatch="*")

ETag IfMatch

Google Cloud Storage

upload_from_string(..., if_generation_match=0)

Observed generation match

Azure Blob Storage

upload_blob(..., overwrite=False)

Observed ETag with MatchConditions.IfNotModified

S3 multipart uploads first complete to a unique immutable key because their multipart operations cannot carry this contract’s preconditions. fsspec is limited to transport, listing, and reads; it is not a compare-and-swap implementation.

scan(prefix) returns ordered recovery findings for expired leases, orphan staging objects, incomplete prepared records, pointers without commit records, checksum mismatches, and stale fences. repair supports only the matching closed actions: abort staging, finalize a record, restore a pointer, or quarantine an object. It is mutation-free by default (dry_run=True) and returns the before/after pointer and every applied mutation. Recovery after a crash converges to a committed or aborted revision, never a mixed visible revision.

Transactional annotation overlays

Layer 18 adds append-only annotations without rewriting a canonical record. One overlay row has the authoritative identity (dataset_id, sample_id, annotation_type, run_id). The canonical row envelope contains that key and the row payload, including the annotation template ID when the annotation command writes it. Its idempotency key is the lowercase SHA-256 digest of those RFC 8785 canonical bytes.

Rows and an index are immutable JSON objects. The dataset-scoped pointer datasets/<dataset_id>/LATEST resolves the one authoritative annotation index, not an individual row. The index maps each canonical overlay key to its row digest and its original committed revision. A newly introduced entry uses a current-pointer marker because that pointer’s digest cannot refer to itself; the next index advance materializes its predecessor revision. An immediate replay still returns the current pointer revision. Both append_annotation_row and append_batch read and update this same index, so no separate single-row pointer can diverge from batch visibility.

The overlay publication state is one of STAGED, COMMITTED, CONFLICT, or ABORTED. An identical canonical replay returns its original committed revision with replayed=True and performs no new visible write. A different canonical row with the same key raises AnnotationConflictError before changing the visible index. The expected revision, when supplied, must equal the observed dataset-index revision before a new row is appended.

append_batch validates and deduplicates every input before reserving a transaction. It sorts rows by (dataset_id, sample_id, annotation_type, run_id), rejects a batch spanning datasets, and commits its newly introduced row objects plus one replacement index through exactly one Layer 17 transaction. Therefore a failed batch leaves the prior index visible, and a completed batch never exposes only a prefix of its rows. Checkpoints contain only rows whose publication returned a committed revision, so resume replays persisted rows instead of refetching or rewriting them.

Concurrent writers reread the dataset index after a conditional-publication conflict. Identical writers converge on one visible row and revision; divergent writers conflict. On a lost pointer-write response, the adapter asks the Layer 17 coordinator to scan and repair the transaction before it retries. A staged row or index object is not visible without the committed dataset pointer.

The shipped annotation command enables this transaction path only for a local filesystem path or a file:// URI. It creates the Layer 17 local backend and journal beside the store. Nonlocal store URIs are rejected until a matching provider transaction backend is configured; this local backend is not a cloud conditional-write substitute.

Zarr

Canonical random-access layout:

<dataset>.zarr/
├── zarr.json
├── samples/
│   └── <sample_id>/
│       ├── zarr.json
│       ├── canonical_cbor         # uint8 array
│       ├── scene_cbor             # uint8 array
│       ├── emitters_cbor          # uint8 array
│       ├── bboxes_cbor            # uint8 array
│       ├── iq                     # complex64 [N] or [num_rx, N]
│       ├── seg_mask               # optional int16 [F, T] or uint8 [C, F, T]
│       └── text_cbor              # optional uint8 array
├── shards/
│   └── <shard_id>/
│       ├── zarr.json
│       └── _shard_complete        # completion marker in the backing filesystem
├── annotations/
│   └── <sample_id>/
│       └── text_cbor              # uint8 array; Phase 2 annotation overlay
└── assets/
    └── <kind>/
        └── <content_hash>/
            ├── manifest.cbor
            └── <bundle files...>

Root attrs

Attr

Value

schema_version

integer schema version

compression

"blosc" or "none"

chunk_samples

IQ chunk length along the sample axis

record_axis

per_rx or joint

Per-sample attrs

Attr

Value

schema_version

record schema version

scene_id

record.scene.scene_id

Arrays

Path

dtype

Notes

samples/<sample_id>/iq

complex64

Chunks along the trailing sample axis.

samples/<sample_id>/seg_mask

int16 or uint8

Present only when record.seg_mask exists.

samples/<sample_id>/*_cbor

uint8

Raw deterministic CBOR bytes.

WebDataset

Tar-sharded layout with an rfgen index sidecar:

<dataset>/
├── _rfgen_index.cbor
├── samples/
│   └── <sample_id>.tar
├── annotations/
│   └── <sha256(sample_id)>.cbor   # Phase 2 annotation overlay sidecar
├── shards/
│   ├── <shard_id>.tar
│   └── <shard_id>.tar.SHA256
└── assets/
    └── <kind>/<content_hash>/
        ├── manifest.cbor
        └── <bundle files...>

Each single-record tar under samples/ or multi-record tar under shards/ contains one member set per sample:

Member

Contents

<sample_id>.canonical_cbor

Canonical record bytes

<sample_id>.record_cbor

Scene, emitters, bboxes, optional text, schema version

<sample_id>.iq_npy

NumPy .npy payload containing complex64 IQ

<sample_id>.seg_npy

Optional NumPy .npy payload for segmentation

_rfgen_index.cbor tracks schema version, compression, record axis, per-sample paths, per-shard paths, and deterministic iteration order.

HDF5

Local HDF5 layout plus TorchSig helper groups:

<dataset>/
├── data.h5
└── assets/
    └── <kind>/<content_hash>/
        ├── manifest.cbor
        └── <bundle files...>

Inside data.h5:

/
├── attrs={schema_version, compression, chunk_samples, record_axis}
├── /samples
│   └── /samples/<sample_id>
│       ├── canonical_cbor         # uint8 dataset
│       ├── scene_cbor             # uint8 dataset
│       ├── emitters_cbor          # uint8 dataset
│       ├── bboxes_cbor            # uint8 dataset
│       ├── seg_mask               # optional int16 [F, T] or uint8 [C, F, T]
│       └── text_cbor              # optional uint8 dataset
│       attrs={schema_version, scene_id, torchsig_index}
├── /shards
│   └── /shards/<shard_id>
│       attrs={sample_ids, complete}
├── /annotations
│   └── /annotations/<sample_id>
│       └── text_cbor              # Phase 2 annotation overlay
└── /torchsig
    ├── /data
    ├── /metadata
    ├── /index
    └── /component_signals

The /torchsig subtree is written through TorchSig’s shipped HDF5 helper surface. Component lower_freq, upper_freq, and center_freq values are stored in absolute RF Hz, not receiver-relative offsets.

SigMF

Local SigMF capture layout plus rfgen sidecars:

<dataset>/
├── _rfgen_index.cbor
├── samples/
│   └── <sample_id>/
│       ├── <sample_id>.sigmf-data
│       ├── <sample_id>.sigmf-meta
│       ├── <sample_id>.record.cbor
│       ├── <sample_id>.canonical.cbor
│       └── <sample_id>.seg.npy    # optional
├── annotations/
│   └── <sha256(sample_id)>.cbor   # Phase 2 annotation overlay sidecar
├── shards/
│   ├── <shard_id>.sigmf-data
│   └── <shard_id>.sigmf-meta
└── assets/
    └── <kind>/<content_hash>/
        ├── manifest.cbor
        └── <bundle files...>

The SigMF global block carries sample rate and the persisted IQ shape. Captures store receiver center frequency. Annotations store absolute RF bbox edges under core:freq_lower_edge and core:freq_upper_edge.

Asset manifest

assets/<kind>/<content_hash>/manifest.cbor is a deterministic CBOR asset bundle manifest. It is distinct from the RFC 8785 JSON run manifest above: it describes one content-addressed asset bundle rather than a generation run and has no run self-hash or forward-compatible envelope. It contains these required fields:

Field

Meaning

kind

Asset kind enum value

content_hash

sha256:<hex> bundle hash

entrypoint

Bundle-relative entrypoint path

files

List of {path, sha256, size_bytes, media_type}

source_uri

Original source URI

generator

Provenance label such as manual-mitsuba or antenna-pattern

generator_version

Optional generator version

created_at_iso

ISO-8601 creation timestamp

license

Optional license string

Naming and shard completion

  • sample_id is stable content addressing for a single record.

  • shard_id is caller-supplied by the orchestration layer.

  • has_shard(shard_id) stays False until the backend-specific completion marker is written:

    • Zarr: shard group plus _shard_complete

    • WebDataset: .tar plus .tar.SHA256

    • HDF5: shard group with complete=True

    • SigMF: .sigmf-meta plus .sigmf-data

See Also