Metrics

Note

This reference contains both implemented metric contracts and proposed contracts. The cross-device transfer qualification contract below is implemented in rfgen.transfer_qualification; other sections retain their own maturity statements.

Definitions for every metric the framework computes: schema-validation invariants, statistical-audit tests, annotation quality (PAES, Hallucination Count, Prompt Leakage Rate), and sim-to-real gates.

This page is normative: implementations must compute these metrics as specified, and acceptance gates listed here are what rfgen validate and CI release blocks check.

Three measurement layers

Layer

Metric family

When

Effect on release

Schema validation

Cross-modality consistency

Per-sample, write-time

> 0.1 % failure rejects the run

Statistical audit

SNR / class-balance / density / channel-stats / occupancy

Post-run, dataset-level

Outside tolerance blocks version promotion

Annotation quality

PAES, Hallucination Count, Prompt Leakage Rate

Post-Phase-2, on verifier subset

Below threshold blocks release

Sim-to-real

Per-class accuracy delta + channel-stat KL

v1.5 milestone gate

Above bound forces channel/fingerprint config revision

Schema validation

Per-sample, computed by the labeler before returning. Failures raise LabelInconsistencyError. The four invariants:

IQ ↔ bbox

For every bbox b:

  • 0 ≤ b.start_sample and b.start_sample + b.duration_samples ≤ record.iq.shape[-1]

  • b.low_freq_hz < b.high_freq_hz, with both edges in the receiver capture window when the record carries receiver-window extras

  • b.to_yolo(...) returns t_center, f_center, dt, df ∈ [0, 1] and the rectangle stays within [0, 1]²

  • b.to_yolo(...) derives from b.start_sample, b.duration_samples, b.low_freq_hz, and b.high_freq_hz to within 1 ULP of float32

bbox ↔ segmentation (±1 STFT hop tolerance)

For every bbox b and the corresponding class_id in the segmentation mask: the set of cells marked with class_id must lie within the time-frequency rectangle defined by b, expanded by ±1 STFT hop in time and ±1 freq bin in frequency. The ±1 tolerance accounts for the binary-at-0.5-occupancy rule at boundaries.

For multi-label masks: per-class plane.

bbox ↔ emitters

For every b with b.emitter_index = i:

  • 0 ≤ i < len(emitters)

  • emitters[i].start_sample == b.start_sample

  • emitters[i].duration_samples == b.duration_samples

  • emitters[i].realized_carrier_hz, emitters[i].bandwidth_hz, and the active receiver’s center_freq_hz reproduce the bbox frequency extent within the receiver frame; when center_freq_hz is not stored directly on the record, the validator must source it from the receiver that produced the record or convert both sides into one explicit scene frame before comparing them

emitters ↔ scene

  • len(emitters) == scene.realized_emitter_count

  • scene.realized_class_histogram == Counter(emitters[i].class_name for i in ...)

  • scene.realized_snr_db_stats.{min,max,mean,p10,p50,p90} reproduced from emitters[i].snr_db

Acceptance. > 0.1 % per-sample failure rate fails the entire run.

Statistical audit

Run offline over a completed dataset (rfgen inspect <path> distribution).

Audit helper contracts

The helper signatures below use small audit-time records, not the full runtime config objects. They are defined here so the metric contracts are complete:

from collections.abc import Mapping
from dataclasses import dataclass
import numpy as np

@dataclass(frozen=True)
class AuditResult:
    metric: str
    value: float | None = None
    threshold: float | None = None
    passed: bool = True
    failures: tuple[object, ...] = ()
    details: str | None = None


@dataclass(frozen=True)
class SnrDistributionConfig:
    snr_distribution: str                  # currently "log_uniform" or "uniform"
    snr_db_range: tuple[float, float]     # inclusive audit target in dB


@dataclass(frozen=True)
class ChannelStatsArray:
    solver_backend: str                   # one of sionna-uma/umi/rma/tdl/cdl
    values: Mapping[str, np.ndarray]      # 1-D float arrays keyed by metric name
    profile: str | None = None            # TDL/CDL model letter only; None for UMa/UMi/RMa

ChannelStatsArray.values is keyed by the summary metrics the current shipped records actually persist under SignalMetadata.extras["statistical_channel"]. Today those are num_paths, dominant_path_delay_s, and dominant_path_gain_linear, each stored as a one-dimensional NumPy array of length N_records after audit extraction. No shared record-time scene RF anchor is assumed; every audit either reads persisted realized fields from StoredRecord / SceneMetadata or takes explicit generation-config input.

Field notes:

  • AuditResult.metric is the stable check name written into stats.json.

  • AuditResult.value is the realized scalar metric when one exists; checks such as class balance may leave it None and report only failures.

  • AuditResult.threshold is the pass/fail cutoff for scalar metrics; None means “see failures”.

  • AuditResult.failures is a tuple of structured failure payloads, not free text.

  • SnrDistributionConfig.snr_distribution accepts only "log_uniform" and "uniform" in the current shipped audit contract.

  • SnrDistributionConfig.snr_db_range is the inclusive configured target range in dB, (low_db, high_db) with low_db < high_db.

  • ChannelStatsArray.solver_backend must be one of "sionna-uma", "sionna-umi", "sionna-rma", "sionna-tdl", or "sionna-cdl".

  • ChannelStatsArray.profile is required only for TDL/CDL, where it mirrors StatisticalSolverConfig.model ("A" through "E"). It must be None for UMa/UMi/RMa; those backends do not use a letter profile.

  • ChannelStatsArray.values[name] must be a one-dimensional finite float array of equal length across all keys, one element per audited record.

SNR distribution test

Compare realized per-emitter snr_db distribution to the configured snr_distribution:

def snr_distribution_test(realized: np.ndarray, config: SnrDistributionConfig) -> AuditResult:
    """KS test against the configured distribution."""
    def db_to_power_ratio(x_db: np.ndarray | float) -> np.ndarray | float:
        return 10.0 ** (np.asarray(x_db) / 10.0)

    if config.snr_distribution == "log_uniform":
        # Log-uniform in linear power becomes uniform in dB.
        target = scipy.stats.uniform(
            loc=config.snr_db_range[0],
            scale=config.snr_db_range[1] - config.snr_db_range[0],
        )
        sample = realized
    elif config.snr_distribution == "uniform":
        low_lin = db_to_power_ratio(config.snr_db_range[0])
        high_lin = db_to_power_ratio(config.snr_db_range[1])
        target = scipy.stats.uniform(loc=low_lin, scale=high_lin - low_lin)
        sample = db_to_power_ratio(realized)
    else:
        raise ValueError(
            "snr_distribution_test supports only the currently documented "
            "'log_uniform' and 'uniform' audit targets."
        )

    ks_stat, p_value = scipy.stats.kstest(sample, target.cdf)
    return AuditResult(
        metric="snr_distribution_ks",
        value=ks_stat,
        threshold=0.05,         # ≤ 5% KS distance
        passed=ks_stat <= 0.05,
    )

Threshold: KS statistic ≤ 0.05.

Behavior contract:

  • realized is a one-dimensional array of finite realized per-emitter SNR values in dB.

  • Returns an AuditResult(metric="snr_distribution_ks", value=<ks_stat>, threshold=0.05, passed=...).

  • Raises ValueError when config.snr_distribution is not one of the two documented distribution names.

Class balance test

For each class, the realized fraction must be within [0.5×, 2.0×] of the configured emitter-zoo weight (after normalization).

def class_balance_test(realized: Counter, config: EmitterZooConfig) -> AuditResult:
    expected = _normalize_weights(config)
    total = sum(realized.values())
    failures = []
    for cls, weight in expected.items():
        expected_fraction = weight
        realized_fraction = realized.get(cls, 0) / total if total > 0 else 0.0
        ratio = realized_fraction / expected_fraction if expected_fraction > 0 else float("inf")
        if ratio < 0.5 or ratio > 2.0:
            failures.append((cls, ratio))
    for cls in realized:
        if cls not in expected:
            failures.append((cls, "missing_from_config"))
    return AuditResult(
        metric="class_balance",
        passed=len(failures) == 0,
        failures=tuple(failures),
    )

_normalize_weights(config) must normalize only the classes named in EmitterZooConfig.families. Any realized class absent from the configured weight map is an audit failure, not an implicit zero-weight class. A configured class missing from the realized histogram is likewise a failure because its realized fraction is 0.0. The denominator is sum(realized.values()), including counts for unknown classes; those unknown classes therefore both depress the realized fraction of known classes and emit their own ("class_name", "missing_from_config") failure entries.

Behavior contract:

  • realized is a Counter[str] over realized class labels.

  • Returns AuditResult(metric="class_balance", passed=..., failures=...).

  • Does not special-case unknown classes into an implicit zero-weight bucket.

Density test

def density_test(realized_emitters_per_scene: np.ndarray, config: SceneConfig) -> AuditResult:
    expected_density = _expected_density(config)  # mean = density × bandwidth_MHz
    realized_mean = realized_emitters_per_scene.mean()
    relative_err = abs(realized_mean - expected_density) / expected_density
    return AuditResult(
        metric="density",
        value=relative_err,
        threshold=0.05,         # ≤ 5% relative error
        passed=relative_err <= 0.05,
    )

Behavior contract:

  • realized_emitters_per_scene is a one-dimensional numeric array of realized emitter counts, one value per scene.

  • Returns AuditResult(metric="density", value=<relative_err>, threshold=0.05, passed=...).

  • _expected_density(config) must produce the audit target in the same “emitters per scene” units as realized_emitters_per_scene.mean().

Channel-statistics test

For the five statistical Sionna backends that persist extras["statistical_channel"] (sionna-uma, sionna-umi, sionna-rma, sionna-tdl, sionna-cdl), verify the realized summary statistics that are actually present today match the configured profile. The current shipped summary fields are num_paths, dominant_path_delay_s, and dominant_path_gain_linear.

This audit is:

  • in scope for SionnaUMa, SionnaUMi, SionnaRMa, SionnaTDL, and SionnaCDL;

  • skipped for SionnaRT, whose rt_channel extras have a different schema and are not reduced to the same summary fields;

  • rejected as not applicable for AWGNChannel and any non-Sionna propagation backend.

Backend-family semantics:

  • For sionna-tdl and sionna-cdl, realized.profile is required and must be the persisted StatisticalSolverConfig.model letter ("A"-"E").

  • For sionna-uma, sionna-umi, and sionna-rma, realized.profile must be None. The current shipped summary extras do not persist a separate letter-profile discriminator for those system-level backends, so the audit keys off solver_backend only. If a future implementation wants LOS/NLOS bucketed audits, it must add an explicit extracted field rather than overloading profile.

def channel_stats_test(realized: ChannelStatsArray) -> AuditResult:
    if realized.solver_backend in {"sionna-tdl", "sionna-cdl"}:
        if realized.profile is None:
            raise ValueError(
                "channel_stats_test requires ChannelStatsArray.profile for "
                "the TDL/CDL backends."
            )
        spec = _3gpp_channel_spec(realized.solver_backend, model=realized.profile)
    elif realized.solver_backend in {"sionna-uma", "sionna-umi", "sionna-rma"}:
        spec = _3gpp_channel_spec(realized.solver_backend)
    else:
        raise ValueError(
            "channel_stats_test applies only to the statistical Sionna backends "
            "that emit extras['statistical_channel']."
        )
    kl = _empirical_kl(realized.values, spec)
    return AuditResult(
        metric="channel_stats_kl",
        value=kl,
        threshold=0.10,
        passed=kl <= 0.10,
    )

Custom logic provenance and verification. The target distributions come from Sionna-backed backends and 3GPP channel profiles; rfgen owns the audit glue that extracts persisted summary fields into ChannelStatsArray, constructs empirical histograms, and computes the KL divergence. Keep that logic minimal and pin it with golden fixtures for extracted arrays plus numerical tests for histogram edges, smoothing, backend/profile dispatch, and known-answer KL cases.

Threshold: KL-divergence ≤ 0.10 against the channel-profile spec.

Spectral occupancy test

Total bandwidth utilization within configured bounds. At write time the stored SceneMetadata.realized_spectral_occupancy field already records the realized union-of-bands fraction. When recomputing the metric from raw records, use the generation-time scene config explicitly: clip each per-emitter occupied interval to [scene_cfg.center_hz - scene_cfg.bandwidth_hz / 2, scene_cfg.center_hz + scene_cfg.bandwidth_hz / 2] and divide by scene_cfg.bandwidth_hz. Stored records alone do not carry a shared scene RF anchor.

Custom logic provenance and verification. The occupancy target comes from the configured scene contract, but rfgen owns the recomputation from realized emitter intervals, scene-band clipping, and union-of-bands measurement. Pin the recompute path with golden fixtures and numerical tests covering empty scenes, touching or overlapping intervals, out-of-band clipping, and exact-boundary cases.

Threshold: within [0.8×, 1.2×] of the configured target occupancy.

Audit output

Failures emit a structured report (stats.json):

{
  "summary": "FAIL",
  "checks": [
    {"name": "snr_distribution_ks", "value": 0.034, "threshold": 0.05, "passed": true},
    {"name": "class_balance",       "value": null,   "passed": true},
    {"name": "density",             "value": 0.018, "threshold": 0.05, "passed": true},
    {"name": "channel_stats_kl",    "value": 0.124, "threshold": 0.10, "passed": false,
     "details": "TDL-C delay_spread_rms_s realized 412 ns, expected 300 ns"},
    {"name": "spectral_occupancy",  "value": 0.94,  "passed": true}
  ]
}

The dataset’s version is not promoted to the published prefix until the report is clean. A failed check forces config revision.

Cross-device transfer qualification

Cross-device transfer qualification measures whether a classifier trained in one provenance domain transfers to the other. It uses payload-free, self-authenticating sample and dataset records. Each sample records its synthetic or captured domain, class, site, device, capture session, UTC capture time, IQ object URI, and IQ SHA-256 digest. The dataset and every nested sample use RFC 8785 canonical JSON with a SHA-256 self-hash; IQ payload bytes are loaded only after their declared digest verifies.

Chronological split and frozen threshold boundary

build_calibration_split(dataset) first groups rows by (site_id, device_id, capture_session_id). A group is indivisible, so the same capture identity cannot leak across TRAIN, VALIDATION, and TEST. Candidate boundaries require strict time ordering and every class/domain in each partition. The selected split is deterministic, uses seed 1337, records every sample assignment and identity digest, and is itself RFC 8785 self-hashed. Qualification revalidates the supplied split against the dataset before any factory call.

TransferThresholdsV1 is another frozen self-hashed artifact. It contains one global threshold for each metric and one minimum-F1 threshold for every dataset class, is bound to the calibration split hash, and is revalidated before TEST inference. Threshold fitting belongs to VALIDATION only. A caller cannot change a committed threshold while evaluating TEST, and a missing, mismatched, duplicate, or malformed row raises TransferInputError before a report is returned.

Injected classifier boundary

The framework does not select or claim a trained classifier implementation. Instead, qualify_transfer(dataset, split, classifier_spec, classifier_factory, thresholds, *, as_of) accepts an injected factory. ClassifierSpecV1 pins the factory identifier, model-config digest, code digest, sorted class order, and batch size. The factory supplies two distinct objects, one per direction; each object must expose matching classifier_identity_sha256 and classes_ values. Identity or class-order mismatch raises ClassifierContractError before fitting.

For synthetic-to-captured transfer, fit consumes only SYNTHETIC/TRAIN batches and labels and predicts only CAPTURED/TEST batches. Captured-to-synthetic reverses those domains. Batches are ascending by sample ID, use complex64 IQ with int64 original lengths and zero padding, and retain a smaller final batch. Prediction must return finite floating [B, C] probabilities in [0, 1] with each row summing to one within absolute tolerance 1e-6. The runner snapshots each target batch before prediction and rejects mutation. Target labels are not passed to either classifier before target predictions are complete.

Metric gates and report evidence

The transfer report evaluates spectral RMSE in dB, the two-sample KS statistic, Wasserstein distance, Jensen-Shannon divergence (JSD), macro F1, and per-class F1. Spectral RMSE compares mean power spectra using sqrt(mean((10 log10(P_syn + 1e-12) - 10 log10(P_real + 1e-12))^2)). KS and Wasserstein use the SciPy statistics implementations. JSD adds 1e-12 to each nonnegative power-vector entry, normalizes in float64, and uses squared scipy.spatial.distance.jensenshannon(..., base=None); valid results are in [0, ln(2)]. Macro and per-class F1 use scikit-learn with the pinned class order and zero_division=0.

Each global and per-class result includes its value, threshold, comparator, status, bootstrap confidence interval, and evidence digest. Bootstrap intervals use 10,000 seed-1337 resamples of whole capture-identity groups, not independent rows. Equality passes either LE or GE threshold. Nonfinite inputs, invalid probability rows, unreadable or hash-mismatched IQ, and incompatible input artifacts fail closed.

TransferReportV1 is frozen, extra-forbidden, RFC 8785 self-hashed evidence for both directions in enum order. Its PASS status requires every global and per-class gate in both directions to pass. A completed gate failure returns a FAIL report whose sorted failure_codes must derive exactly from the failing metric and per-class rows; forged, omitted, or invented codes are rejected by the report model. Input, hash, time, split, and threshold violations instead raise TransferInputError and return no report.

Annotation quality

PAES: Physical Attribute Extraction Score

Adopted from RF-Analyzer (arXiv:2605.04676). Recall-style metric measuring how much of the ground truth survives the bulk inference pass’s rewrite.

Formal definition

Let \(A_{gt}\) be the ground-truth attribute set derived from whitelist-filtered metadata. Let \(T\) be the generated text. Let \(A_{rec}(T)\) be the set of attributes recovered from \(T\) by an extractor model under tolerance bands. Then:

\[ \mathrm{PAES}(T) = \frac{|A_{rec}(T) \cap A_{gt}|}{|A_{gt}|} \]

Attribute set construction

\(A_{gt}\) is custom framework logic, not a library primitive. That is deliberate: no external library owns rfgen’s whitelist-filtered metadata schema or the canonical (key, value) attribute inventory used for text-audit scoring. The implementation must therefore keep the logic minimal, deterministic, and externally checkable.

Verifiability note. The canonicalization code is acceptable only if the same StoredRecord always yields the same A_gt, the emitted key set is documented and versioned, and golden tests pin both omission rules (None / nan / opaque non-scalars are dropped) and canonicalized shapes for structured extras. Any schema change to the whitelist surface or key naming is a versioned PAES-contract change and must update the golden fixtures.

The deterministic construction is:

import math
from typing import Any

import numpy as np
import torch

def canonicalize_paes_value(value: Any) -> Any | None:
    if value is None:
        return None
    if isinstance(value, np.generic):
        value = value.item()
    elif isinstance(value, torch.Tensor):
        if value.numel() != 1:
            return None
        value = value.detach().cpu().item()
    if isinstance(value, float) and not math.isfinite(value):
        return None
    if isinstance(value, dict):
        items = []
        for key, child in sorted(value.items(), key=lambda kv: str(kv[0])):
            child_value = canonicalize_paes_value(child)
            if child_value is not None:
                items.append((str(key), child_value))
        return tuple(items)
    if isinstance(value, (list, tuple)):
        items = []
        for child in value:
            child_value = canonicalize_paes_value(child)
            if child_value is not None:
                items.append(child_value)
        return tuple(items)
    if isinstance(value, (str, bool, int, float)):
        return value
    return None


def construct_a_gt(record: StoredRecord) -> set[tuple[str, Any]]:
    raw_pairs = [
        ("scene.duration_ms", record.scene.duration_s * 1000),
        ("scene.num_emitters", record.scene.realized_emitter_count),
        ("scene.realized_cochannel_overlap_rate", record.scene.realized_cochannel_overlap_rate),
        ("scene.realized_spectral_occupancy", record.scene.realized_spectral_occupancy),
    ]
    for key, value in record.scene.realized_snr_db_stats.items():
        raw_pairs.append((f"scene.realized_snr_db_stats.{key}", value))
    for class_name, count in record.scene.realized_class_histogram.items():
        raw_pairs.append((f"scene.realized_class_histogram.{class_name}", count))
    for i, em in enumerate(record.emitters):
        raw_pairs.extend([
            (f"emitter[{i}].class_name", em.class_name),
            (f"emitter[{i}].family", em.family),
            (f"emitter[{i}].generator_name", em.generator_name),
            (f"emitter[{i}].realized_carrier_ghz", em.realized_carrier_hz / 1e9),
            (f"emitter[{i}].bandwidth_mhz", em.bandwidth_hz / 1e6),
            (f"emitter[{i}].snr_db", em.snr_db),
            (f"emitter[{i}].duration_ms", em.duration_samples / em.sample_rate_hz * 1000),
            (f"emitter[{i}].start_sample", em.start_sample),
        ])
        for key, value in em.extras.items():
            raw_pairs.append((f"emitter[{i}].extras.{key}", value))
    for i, bbox in enumerate(record.bboxes):
        raw_pairs.extend([
            (f"bbox[{i}].start_sample", bbox.start_sample),
            (f"bbox[{i}].duration_samples", bbox.duration_samples),
            (f"bbox[{i}].low_freq_hz", bbox.low_freq_hz),
            (f"bbox[{i}].high_freq_hz", bbox.high_freq_hz),
            (f"bbox[{i}].class_id", bbox.class_id),
            (f"bbox[{i}].emitter_index", bbox.emitter_index),
        ])
    out: set[tuple[str, Any]] = set()
    for key, value in raw_pairs:
        canonical = canonicalize_paes_value(value)
        if canonical is not None:
            out.add((key, canonical))
    return out

Unavailable values are omitted before computing the PAES denominator. This includes None, nan, inf, and -inf, so a value that the generation path could not determine does not make the score fail by construction. Extras that are structured but JSON-like, such as fingerprint_params, are canonicalized to hashable tuples before the set is built; opaque non-scalar values are omitted. Because this canonicalization is custom rfgen logic, its golden tests must cover at least scalar coercion, stable dict-key sorting, tuple/list normalization, tensor-to-scalar conversion, and omission of unsupported values.

Helper behavior:

  • canonicalize_paes_value(value) returns a hashable scalar or tuple form of value, or None when the value is intentionally omitted from the PAES denominator. Unsupported values are dropped, not raised.

  • construct_a_gt(record) returns the canonical ground-truth attribute set A_gt for one stored record by collecting whitelisted raw pairs and running every value through canonicalize_paes_value.

  • construct_a_gt assumes record already satisfies the StoredRecord schema; missing optional attributes are omitted rather than treated as failures.

Canonical units (all keys use these; extractor recovers in same units):

  • Frequency: GHz for centers, MHz for bandwidths, hertz for bbox edges

  • Power: dB or dBm as labeled

  • Time: ms, µs (per attribute as specified by key suffix)

  • Counts: integer (exact match required)

Tolerance bands

Attribute

Tolerance

SNR (dB)

±2 dB

Bandwidth

±5 % relative or ±100 kHz, whichever is larger

Center frequency

±0.1 % relative or ±50 kHz, whichever is larger

Duration

±5 % relative

PRI (radar)

±2 % relative

Pulse width

±5 % relative

Counts (num_emitters, num_pulses)

exact

Class names

exact match against canonical taxonomy leaf

Channel profile

exact (substring permitted: "TDL-C" matches "tdl-c-300")

Doppler (Hz)

±10 % relative or ±5 Hz, whichever is larger

AoA (degrees)

±2°

Path loss (dB)

±2 dB

The tolerance table lives at rfgen/eval/paes.py::TOLERANCES and is versioned with the metric. Changes are a major bump on paes.schema_version.

Extractor

The extractor is a constrained inference call:

class ExtractedAttribute(BaseModel):
    key: str       # canonical key, e.g. "emitter[0].snr_db"
    value: str     # stringified; we parse per attribute key
    unit: str | None

class ExtractorOutput(BaseModel):
    attributes: list[ExtractedAttribute]

Field notes:

  • ExtractedAttribute.key must be one of the canonical PAES keys exposed to the extractor prompt.

  • ExtractedAttribute.value is always a string at extraction time; numeric parsing happens downstream by key-specific logic.

  • ExtractedAttribute.unit is optional and advisory. Tolerance matching still uses the canonical key’s expected unit.

  • ExtractorOutput.attributes may be empty; an omitted key means “not recovered”, not an extractor error.

The extractor prompt provides the canonical-key list and asks the model to populate every key for which it can find evidence in \(T\). Empty values are allowed; missing keys are equivalent to “not recovered.”

We use the bulk model (gemini-3.1-flash-lite) as the extractor. Extraction is mechanical, and using a different model for extraction than for generation breaks the easiest failure mode (a model that writes prose with the same attribute confusions consistently extracts them too).

Aggregation

Aggregate

Definition

Per-record PAES

Scalar in [0, 1] per (sample_id, annotation_type)

Per-template PAES

Mean across records using a given template_id

Per-class PAES

Mean across records whose dominant taxonomy_path[0] is that class

Dataset PAES

Mean across the verifier subset; reported as manifest.annotation_audit.dataset_paes

Acceptance thresholds

Level

Threshold

Dataset PAES

≥ 0.90

Per-template PAES

≥ 0.85

Per-class PAES

≥ 0.80 (no class falls below)

Below threshold, the release is blocked pending template fix or explicit threshold-relaxation sign-off.

Hallucination Count

Per record: the number of attributes recovered from \(T\) that are not in \(A_{gt}\) and that the extractor flagged with confidence ≥ threshold (default 0.7).

hallucinated_attributes = A_rec(T) - A_gt
hallucination_count = len(hallucinated_attributes)

Common failure modes the count catches:

  • LLM invents noise_floor_dbm not stated in FACTS

  • LLM invents an emitter not in metadata (rare with whitelist injection)

  • LLM generalizes “TDL-C” to “TDL-C-300 with 30 km/h Doppler” when only “TDL-C” was stated

  • LLM names a specific Wi-Fi rate (“12 Mbps”) when only “11g” was stated

Thresholds:

Level

Max

Per-template Hallucination Count

≤ 0.5 / sample (mean)

Dataset Hallucination Count

≤ 0.3 / sample (mean)

Violations block release.

Prompt Leakage Rate

The fraction of generated outputs that contain tokens unique to the system prompt or FACTS scaffolding. These tokens never belong in fluent prose; their presence indicates the LLM regurgitated prompt scaffolding rather than rewriting it.

LEAKAGE_TOKENS = {
    "FACTS", "OUTPUT SCHEMA", "ALLOWED VOCABULARY",
    "channel_profile:", "snr_db:", "bandwidth_hz:",
    "emitter_offset_hz:", "start_sample:", "duration_samples:",
    "noise_floor_dbm:", "extras:", "class_name:",
    "TASK", "SYSTEM", "USER",
    # ... full list in rfgen/eval/leakage.py
}

def prompt_leakage_test(text: str) -> bool:
    return any(tok in text for tok in LEAKAGE_TOKENS)

Threshold: Per-template leakage rate ≤ 1 %. Dataset-level ≤ 0.5 %.

A leak triggers re-annotation of that record (one retry with temperature × 0.5); persistent leaks flag the template for manual review.

Sim-to-real metrics (v1.5 gate)

Per-class accuracy delta

A TorchSig-trained YOLO detector (or comparable head) classifies both:

  • \(D_{\text{sim}}\), the synthetic 1K-sample HIL subset

  • \(D_{\text{capture}}\), the captured replay through USRP/HackRF cabled loopback + OTA

Per-class top-1 accuracy gap:

\[ \Delta_{\text{acc}}(c) = \mathrm{acc}(c \mid D_{\text{sim}}) - \mathrm{acc}(c \mid D_{\text{capture}}) \]

Threshold: \(|\Delta_{\text{acc}}(c)| \leq 5\) percentage points for every class. Worst class > 5 pp triggers a fingerprint or channel-config revision.

Channel-statistics KL

Per-channel-statistic, a Kullback-Leibler divergence between simulated and captured distributions:

\[ \mathrm{KL}(\hat p_{\text{sim}} \| \hat p_{\text{capture}}) \leq 0.1 \]

Computed for: delay-spread RMS, K-factor, peak Doppler. Histograms over 32 bins, Laplace-smoothed.

Multi-RX preset reproducibility

Same scene seed + same RX geometry produces byte-identical IQ on re-run. (Not a tolerance; a strict equality, with the GPU-non-determinism caveat in Reference / Determinism.)

Reproducibility CI smoke test

On every release, the CI runs:

rfgen generate --config-name $(jq -r .config_name manifest.json) \
              run.seed=$(jq -r .global_seed manifest.json) \
              storage.path=./reproduction
diff -r ./out/release-baseline ./reproduction

Smallest preset (narrowband_classifier_baseline_md, 100 K samples) on every release. Any byte-level diff in IQ or labels gates the release.

See Also