Validation Strategy

How we know the dataset is correct, the framework is reproducible, and the synthetic data is grounded in physical reality.

Three layers

Layer

What it verifies

Tools

Schema validation

Every sample matches the documented schema; cross-modality consistency holds

Pydantic, label-layer validator (raises LabelInconsistencyError)

Statistical audit

Dataset-level distributions are within tolerance of expected (SNR, class balance, density, channel-statistics)

rfgen.validation.audit, runs over all shards

Sim-to-real (HIL)

Synthesized signals statistically match real-radio captures

USRP/HackRF playback harness, automated re-labeler, channel-statistics comparator

Each layer gates the next: if schema validation fails on > 0.1 % of samples, the run is rejected; if the statistical audit fails, the dataset version is not promoted; if HIL fails, the channel/fingerprint configuration is revised.

Schema validation

Runs at write time, per sample. Four invariants (full list in Reference / Label Schema § Cross-modality consistency):

  1. IQ ↔ bbox bounds and YOLO derivation

  2. Bbox ↔ segmentation cell occupancy (±1 STFT hop tolerance)

  3. Bbox ↔ per_emitter field-by-field equality

  4. per_emitter ↔ scene aggregates (count, histograms, SNR distribution)

Failures raise; the orchestrator excludes the sample and continues. > 0.1 % failure rate fails the run.

Statistical audit

Runs offline over a completed dataset (rfgen inspect <path> distribution). Checks:

  • SNR distribution. Expected uniform/log-uniform within tolerance.

  • Class balance. No class < 50 % or > 200 % of expected weight.

  • Density. Realized emitters/MHz matches scene.density.* config.

  • Channel statistics. Delay spread, K-factor, Doppler distributions match the channel backend’s specification.

  • Spectral occupancy. Total bandwidth utilization within configured bounds.

Output is stats.json written next to manifest.json. Failures emit a structured report; the dataset’s version is not promoted to the published prefix until the report is clean.

Sim-to-real (HIL)

The production validation program calls for a 1K-sample HIL subset for every dataset version and replay through real radios:

  1. Playback. Write IQ to a SigMF file, transmit via USRP B210 or HackRF One.

  2. Receive. Capture via a second USRP/HackRF in a cabled loopback (clean baseline) and an OTA setup (real-world impairments).

  3. Re-label. Automated detector (TorchSig-trained YOLO) re-labels the captured signal; ground truth comes from the original synthetic metadata.

  4. Compare. Channel statistics (delay spread, K-factor, Doppler) of captured vs. simulated; per-class detection accuracy delta.

Program acceptance gates (per Roadmap v1.5):

  • Sim-to-real accuracy delta ≤ 5 percentage points

  • Channel-statistics KL-divergence ≤ 0.1

  • Multi-RX presets reproducible by seed

HIL execution evidence contract

The implemented HIL runner is a fail-closed qualification boundary, not a simulator that can manufacture a hardware pass. A deployment provides the site-specific radio backend; the framework records and verifies the evidence that the backend returns. The local implementation does not itself stand in for a USRP, HackRF, UHD, GNU Radio, a calibration fixture, or an approved radiated-test policy.

Before acquisition, HILPlanV1 binds the device model and serial, driver and firmware versions, calibration-path and hardware-profile hashes, waveform hash, clock and time sources, physical power limits, threshold profile, interlock, and an operator authorization. The frozen, extra-forbidden plan is self-hashed using RFC 8785 canonical JSON. The authorization must cover the deadline supplied to every public phase, before the runner invokes either a replay result or the physical backend. The plan also rejects a fixture power budget where maximum_conducted_dbm - attenuator_db exceeds the fixture limit.

The lifecycle is deliberately narrow:

NEW -> PREPARED -> SAFE -> TRANSMITTING -> CAPTURING -> ANALYZING
                                                     -> CLEANING -> SUCCEEDED | FAILED

Failures from every acquired state go through CLEANING; only CLEANING may enter either terminal state. The configured prepare, clock-lock, transmit/capture, analysis, and cleanup budgets are respectively bounded by 60 s, 30 s, 600 s, 600 s, and 30 s. The runner uses monotonic global and phase timers. Expiry immediately enters cleanup and a cleanup timeout blocks success.

UNAVAILABLE is a distinct pre-acquisition outcome: it is legal only from NEW, has no device identity, artifacts, or cleanup record, and is never a pass. Once acquisition has begun, loss of a device is a failure that must retain cleanup evidence. A hardware-backed pass additionally requires a pinned site trust root and an exact root-signed hardware attestation embedded in the canonical calibration document. The attestation binds the plan, device identity, firmware and FPGA versions, capability and calibration-path hashes, profile hash, and calibration evidence hash. Missing trust material, a wrong signer, or a fake adapter therefore fails closed rather than producing PASS.

The required evidence lives at artifacts/hil/<plan_hash>/<run_id>/ and has this normative artifact order: PLAN, CALIBRATION, TX_IQ, CAPTURE_SIGMF, METRICS, and CLEANUP_LOG. Every kind must occur exactly once for a pass, its stored hash must verify, and its semantic document must bind the same plan and run. The captured SigMF document must agree with the plan’s sample rate, center frequency, and capture sample count. Safety evidence covers interlock, attenuation, power limit, clock lock, clipping, dropped samples, and RF-disabled state. The metric evidence must contain exactly the five named threshold rows: frequency error (ppm), EVM (dB), occupied-bandwidth error (%), timing error (samples), and power error (dB). Defaults are 1 ppm, -25 dB, 3%, 1 sample, and 0.5 dB; fixtures may only tighten them, with a more-negative EVM limit being tighter.

Cleanup always attempts STOP_TX, STOP_RX, FLUSH, RELEASE_DEVICE, and VERIFY_RF_DISABLED in that order, even after an earlier cleanup failure. A missing action, failed action, false RF-disabled result, or cleanup deadline failure makes cleanup fail and makes the terminal report FAILED. PASS requires SUCCEEDED, all six verified artifacts, all safety and metric rows passing, and passing cleanup. Qualification evidence requires at least 20 such passes for each of two distinct device serials. Local tests validate the contract’s fail-closed behavior; only an approved lab run can supply the attested physical-hardware evidence needed for qualification.

Cross-device transfer qualification

HIL establishes hardware behavior. Cross-device transfer qualification asks a different question: whether a downstream classifier trained on one data provenance generalizes to the other. The implemented qualify_transfer path uses separately constructed injected classifier instances for SYNTHETIC-to-CAPTURED and CAPTURED-to-SYNTHETIC evaluation. It does not claim that the framework ships, trains, or endorses a classifier architecture.

Before inference, the runner canonicalizes and revalidates the self-hashed dataset, chronological split, classifier specification, and validation-fitted threshold artifact. Rows are grouped by site, device, and capture session; an identity group cannot cross TRAIN, VALIDATION, and TEST. The deterministic split requires strict chronological boundaries and every class/domain in each partition. Thresholds bind that split hash and may be fitted only from VALIDATION, so TEST evaluation cannot tune an acceptance gate.

The injected classifier receives source-domain TRAIN labels only. It predicts target-domain TEST IQ in deterministic sample-ID batches, with target labels withheld until prediction completes. Its reported factory/config/code identity and sorted classes must match the pinned classifier specification; malformed probabilities, reused direction instances, target-batch mutation, or identity mismatch raises a classifier-contract error rather than creating a report.

The output is an immutable two-direction report with RFC 8785 hashes. It includes spectral RMSE, KS, Wasserstein, JSD, macro F1, per-class F1, and identity-group bootstrap intervals. A PASS needs every gate in both directions to pass. A completed threshold miss is a FAIL report with failure codes derived from its failed rows; invalid input, artifact, time, split, or threshold evidence fails before inference with no report. This gate measures transfer performance, not physical transmitter compliance, and therefore does not replace HIL evidence.

PAES: text annotation quality

Caption / QA / reasoning quality is graded by PAES (Physical Attribute Extraction Score): a structured metric that asks the LLM-under-test to extract specific physical attributes from the annotation (modulation, SNR, BW, channel) and scores against ground truth.

Two variants:

  • PAES-bulk. Runs on every annotation; gates the bulk annotator’s quality.

  • Hallucination Count. Runs on the verifier subset (5 %); counts statements not supported by metadata.

Acceptance: PAES ≥ 0.80 on all 5 annotation types; Hallucination Count ≤ 0.5 / sample on verifier subset.

Reproducibility

Every dataset is reproducible from manifest.json:

rfgen generate --config-name=$(jq -r .config_name manifest.json) \
              run.seed=$(jq -r .global_seed manifest.json) \
              storage.path=./reproduction

The reproduction reproduces the entire dataset bit-identical (per content-hashed shard naming and per-shard deterministic seeding). The CI smoke test runs this on the smallest preset (narrowband_classifier_baseline_md, 100K samples) on every release and gates the release on byte-exact match.