`rfgen.benchmark_qualification`¶

rfgen.benchmark_qualification turns five retained benchmark observations per metric into a self-authenticating release-qualification report. It is an offline policy boundary: callers collect timings, memory observations, and execution evidence; this module computes statistics, applies fixed gates, and binds the result to the release support inventory and a signed baseline set.

All public records are strict, frozen Pydantic v2 models with extra="forbid". Each self-hash is SHA-256 over RFC 8785 canonical JSON after removing only its own hash field. The module uses Python’s statistics module for median and inclusive p95 and Layer 14 SigstoreDSSEAdapter for DSSE signing. It does not run a benchmark, contact a signing service directly, or reimplement Layer 11 evidence validation.

Public API¶

Symbol	Purpose
`BenchmarkName`, `BenchmarkMetric`, `BenchmarkStatus`, `BenchmarkErrorCode`	Closed benchmark vocabulary.
`BenchmarkBaselineV1`, `BenchmarkBaselineSetV1`	Signed, canonical reference-host baselines.
`BenchmarkResultV1`, `BenchmarkQualificationReportV1`	Per-benchmark result and aggregate release record.
`BenchmarkExecutionContextV1`	Typed Layer 11 Chapter 7 evidence input.
`benchmark_statistics`, `qualify_benchmarks`	Fixed five-observation statistics and qualification.
`attest_benchmark_baseline_set`	Layer 14 DSSE baseline update boundary.
`seal_benchmark_wave`	PASS-only candidate Wave 5 sealing boundary.
`BenchmarkQualificationError`	No-report failure for malformed inputs or evidence.

Closed values and records¶

BenchmarkName is CLEAN_WHEEL_GOLDEN, SPARK_PLAN_100K, or AUDIT_1M. BenchmarkMetric is ELAPSED_SECONDS or PEAK_BYTES; BenchmarkStatus is PASS or FAIL. BenchmarkErrorCode is BASELINE_HASH, REFERENCE_HOST, SAMPLE_COUNT, ABSOLUTE_TIME, ABSOLUTE_MEMORY, TIME_REGRESSION, MEMORY_REGRESSION, or NONFINITE.

class BenchmarkBaselineV1(BaseModel):
    name: BenchmarkName
    reference_host_sha256: Sha256
    elapsed_sample_count: int
    elapsed_median_s: float
    elapsed_p95_s: float
    memory_sample_count: int
    memory_median_bytes: float
    memory_p95_bytes: float
    created_at: datetime
    source_revision: GitSha
    signature_evidence_id: str
    baseline_sha256: Sha256

class BenchmarkBaselineSetV1(BaseModel):
    schema_version: Literal[1]
    reference_host_sha256: Sha256
    rows: tuple[BenchmarkBaselineV1, ...]
    created_at: datetime
    source_revision: GitSha
    baseline_set_sha256: Sha256

Each baseline has exactly five elapsed and five memory samples, finite nonnegative metrics, p95 no lower than its median, a UTC timestamp, and a self-hash. The set contains exactly one row per BenchmarkName in enum order; all rows share the set host and source revision. Use .create(**fields) to calculate a self-hash; direct parsing verifies it.

Lossless memory p95 representation¶

The original plan called the persisted memory statistics integers while also requiring Python’s inclusive p95. Those requirements conflict: for integer observations (1, 2, 3, 4, 5), inclusive p95 is 4.8. The implementation therefore stores memory_median_bytes and memory_p95_bytes as float and retains 4.8 exactly as the defined statistic. It does not truncate, round, ceil, or reject a valid fractional quantile. Raw memory observations remain nonnegative integers measured in bytes.

class BenchmarkResultV1(BaseModel):
    name: BenchmarkName
    reference_host_sha256: Sha256
    elapsed_observations_s: tuple[float, ...]
    memory_observations_bytes: tuple[int, ...]
    elapsed_median_s: float
    elapsed_p95_s: float
    memory_median_bytes: float
    memory_p95_bytes: float
    baseline_sha256: Sha256
    time_delta_pct: float
    memory_delta_pct: float
    absolute_time_limit_s: float
    absolute_memory_limit_bytes: int
    status: BenchmarkStatus
    error_codes: tuple[BenchmarkErrorCode, ...]
    result_sha256: Sha256

Results require five finite elapsed observations and five exact nonnegative integer byte observations. Their stored statistics must equal recomputation; error codes are unique and sorted; PASS means no error code.

class BenchmarkQualificationReportV1(BaseModel):
    schema_version: Literal[1]
    support_inventory_sha256: Sha256
    baseline_set_sha256: Sha256
    reference_host_sha256: Sha256
    started_at: datetime
    finished_at: datetime
    rows: tuple[BenchmarkResultV1, ...]
    overall: BenchmarkStatus
    causes: tuple[BenchmarkErrorCode, ...]
    report_sha256: Sha256

The report has one host-consistent row per name in enum order. causes is the sorted union of row errors, and overall is PASS exactly when that union is empty. Its timestamps are UTC and cannot run backward.

Statistics and thresholds¶

def benchmark_statistics(values: tuple[float, ...]) -> tuple[float, float]: ...

The function accepts exactly five finite values and returns statistics.median(values) plus statistics.quantiles(values, n=100, method="inclusive")[94].

Each benchmark requires candidate median time delta at most 15 percent and memory delta at most 10 percent relative to a nonzero baseline median. The absolute gates are:

Name	Time	Memory
`CLEAN_WHEEL_GOLDEN`	120 seconds	2 GiB
`SPARK_PLAN_100K`	10 seconds	1 GiB
`AUDIT_1M`	120 seconds	32 MiB traced

Evidence, qualification, and sealing¶

def qualify_benchmarks(
    support: ReleaseSupportInventoryV1,
    baselines: BenchmarkBaselineSetV1,
    observations: dict[BenchmarkName, tuple[tuple[float, ...], tuple[int, ...]]],
    *, started_at: datetime, finished_at: datetime,
    execution_context: BenchmarkExecutionContextV1,
    validator: ExecutionEvidenceValidatorV1,
) -> BenchmarkQualificationReportV1: ...

BenchmarkExecutionContextV1 supplies the Chapter 7 manifest, chapter gate, module statuses, scientific results, and selector inventory. Qualification calls only Layer 11 validate_workstream_manifest and validate_chapter_gate. Wave 4 must exist and its base_commit must equal support.source_revision. The one BENCHMARKS check must be PASS, contain nonmock evidence with valid independent artifact SHA-256 values, and carry an environment_sha256 equal to the baseline reference-host hash. Artifact hashes identify retained evidence; they are not required to equal the host fingerprint.

Malformed observations, baseline errors, or Layer 11 evidence failures raise BenchmarkQualificationError and return no report. A valid performance regression returns a self-hashed FAIL report containing every row error.

def seal_benchmark_wave(
    context: BenchmarkExecutionContextV1,
    report: BenchmarkQualificationReportV1,
    wave_five: WaveCommitV1,
    *, support: ReleaseSupportInventoryV1,
    baselines: BenchmarkBaselineSetV1,
    validator: ExecutionEvidenceValidatorV1,
) -> WorkstreamExecutionManifestV1: ...

Sealing first reparses the report, including its self-hash and invariants, so a forged model_copy cannot turn a failed report into PASS. The verified PASS report must bind the supplied support inventory hash, baseline-set hash, and reference-host hash. Wave 4 must be unsealed and bind the support revision; the supplied WaveCommitV1 must be Wave 5, descend from Wave 4’s merge, and have a UTC commit time. Layer 11’s reseal_workstream_manifest produces the rehash candidate, then validate_workstream_manifest remains the sole authority for repository, ancestry, ordering, timestamp, and manifest semantics. Any violation raises BenchmarkQualificationError.

Baseline attestation¶

def attest_benchmark_baseline_set(
    baselines: BenchmarkBaselineSetV1, *, support: ReleaseSupportInventoryV1,
    adapter: SigstoreDSSEAdapter, policy: AttestationPolicyV1,
    candidate_execution: bool = False,
) -> Bundle: ...

This is the only baseline-update path. It rejects candidate execution, requires each signature_evidence_id to resolve to PASS evidence in the supplied support inventory, and requires an exact Layer 14 BENCHMARK_BASELINE_V1 policy with subject name benchmark-baselines.json and the baseline-set hash. It signs a canonical in-toto statement through the adapter; it does not update a baseline while qualifying a candidate.