Metrics

This page records research and audit vocabulary. It is not a public CLI dashboard, a semantic verifier for hosted-model prose, or a release-gating policy. The supported read-only operational report is:

rfgen inspect STORE_URI

Current executable contracts validate configuration, storage structure, annotation overlay identity, and schema shape. They do not promise a distribution audit, PAES threshold, hallucination score, sim-to-real metric, or automatic release decision. Treat proposed metrics in historical research notes as design context until an implemented API and test explicitly expose them.

Physical Attribute Extraction Score (PAES)

PAES (Physical Attribute Extraction Score) scores one piece of hosted-model text, such as an annotation caption, against physical attributes recovered locally from the stored record it describes. compute_paes(record, text, *, model=...) in rfgen.inspection.audit is the shipped implementation; see rfgen.inspection for its parameters and return type. The ground truth never leaves the process: only text is sent to the extractor, and the recovered attributes are compared against the record locally. PAES is a recall score, the fraction of locally known ground-truth attributes the extractor recovered correctly from text:

PAES = matches / len(A_gt) if A_gt else 1.0

A record with no available ground-truth fields scores 1.0 instead of dividing by zero.

Ground truth: constructing A_gt

A_gt is the set of (path, value) pairs read straight from the stored record. The pseudocode below mirrors the private _ground_truth() helper in rfgen.inspection.audit; the real implementation also walks per-emitter tx_pose and radar/pulse extras fields the same way, one add() call per field.

def construct_a_gt(record):
    """Build the local ground-truth attribute set A_gt for one stored record."""
    pairs = []

    def add(path, value):
        if _available(value):
            pairs.append((path, value))

    add("scene.duration_s", record.scene.duration_s)
    add("scene.num_emitters", record.scene.num_emitters)
    add("scene.realized_emitter_count", record.scene.realized_emitter_count)
    for key, value in record.scene.realized_snr_db_stats.items():
        add(f"scene.realized_snr_db_stats.{key}", value)
    for key, value in record.scene.realized_class_histogram.items():
        add(f"scene.realized_class_histogram.{key}", value)
    add(
        "scene.realized_cochannel_overlap_rate",
        record.scene.realized_cochannel_overlap_rate,
    )
    add("scene.realized_spectral_occupancy", record.scene.realized_spectral_occupancy)

    for index, emitter in enumerate(record.emitters):
        add(f"emitters[{index}].class_name", emitter.class_name)
        add(f"emitters[{index}].realized_carrier_hz", emitter.realized_carrier_hz)
        add(f"emitters[{index}].bandwidth_hz", emitter.bandwidth_hz)
        add(f"emitters[{index}].snr_db", emitter.snr_db)
        # ... remaining emitter, tx_pose, and extras fields follow the same pattern

    for index, bbox in enumerate(record.bboxes):
        add(f"bboxes[{index}].low_freq_hz", bbox.low_freq_hz)
        add(f"bboxes[{index}].high_freq_hz", bbox.high_freq_hz)
        add(f"bboxes[{index}].class_id", bbox.class_id)
        # ... remaining bbox fields follow the same pattern

    return pairs

Unavailable values

Unavailable values are omitted before computing the PAES denominator. A stored field counts as unavailable when it is None, or when it is numeric and not finite: nan, inf, or -inf. _available() in rfgen.inspection.audit implements this rule, and construct_a_gt calls it before adding any pair, so an unavailable field never enters A_gt and never appears in the denominator len(A_gt). The same helper gates a recovered attribute on the extractor side: an extracted value that is itself None or non-finite never counts as a match, even when its path matches a ground-truth path exactly.

Two frequency frames share one vocabulary

A_gt walks both emitter fields and box fields, and their hertz are not in the same frame. emitters[i].realized_carrier_hz is absolute RF; bboxes[i].low_freq_hz and high_freq_hz are offsets from the receiver’s tuned centre. One record can therefore hold 2.4005e9 and 450000.0 side by side, describing the same emitter.

Nothing in the scoring reconciles them, and nothing needs to: an extractor is compared against the stored value at each path, so each path is scored in the frame it was stored in. What breaks is a reader — human or model — that sees one *_hz family and converts between them. When interpreting PAES output, read the box paths in the receiver-baseband frame and the emitter paths in absolute RF.

Canonical units

All keys use these, and the extractor recovers in the same units:

Path suffix	Canonical unit	Extractor-recoverable unit tokens
bandwidth_hz, realized_carrier_hz, doppler_hz	hertz (absolute RF)	hz, khz (x1e3), mhz (x1e6), ghz (x1e9)
snr_db, path_loss_db	decibels	db
duration_s, pri_s, pulse_width_s	seconds	s, ms (x1e-3), us (x1e-6), ns (x1e-9)
aoa_deg	degrees	deg, rad (x180/pi)
velocity_mps	meters per second	m/s, km/h (x1/3.6)

The suffixes are matched exactly, not as a *_hz family. Several walked ground-truth fields therefore fall outside this table entirely: sample_rate_hz, and every box field — bboxes[i].low_freq_hz, high_freq_hz, start_sample, duration_samples, class_id, and emitter_index. None has a unit-token entry, so a string answer such as "20 mhz" never matches; none has a dedicated tolerance branch, so a numeric answer must equal the stored value exactly. That is strict, and deliberately so for the identifier columns; for the continuous box edges it means a correct-but-rounded answer scores as a miss.

paes_attribute_matches(path, expected, actual) performs the unit conversion and the per-attribute tolerance check; see rfgen.inspection for the full comparison contract.