rfgen.storage¶
Storage turns generated scene records into a published, indexable collection. The generic lifecycle lets generation and inspection select a format without importing its physical implementation.
ProducedRecord is the generic executor result handed to DatasetPublicationService, which
applies the pipeline-bound publication profile before writing source or derived datasets.
Storage extension contract¶
Storage extends abc.ABC. Implementations must provide all four abstract
methods — the three lifecycle methods and compare:
Static Transcript — not runnable
Audit pending; do not treat this block as a runnable example.
def write_shard(
self, records: Sequence[SceneRecord], destination: str, *,
shard_index: int, first_ordinal: int, scene_ids: Sequence[str],
work_id: str | None = None, attempt: int = 0,
) -> StoredShard: ...
def publish_shards(
self, shards: Sequence[StoredShard], destination: str, *,
dataset_id: str, snapshot_id: str,
provenance: Mapping[str, str] | None = None,
) -> RecordCollection: ...
def open(self, destination: str) -> RecordCollection: ...
def compare(self, left: RecordCollection, right: RecordCollection) -> bool: ...
write_shard writes unpublished records. scene_ids has one item per record;
first_ordinal is the inclusive collection index and StoredShard.stop_ordinal
is exclusive. Implementations must preserve that interval and return a
ShardPublication(format: str, payload: dict[str, object]) containing only
JSON-compatible, implementation-owned reopening data. work_id and attempt
identify retried distributed work; they do not change record identity.
publish_shards validates implementation format, non-overlapping contiguous
ordinal intervals, and the complete shard set before making the destination
visible. Publication is create-only: an existing destination raises
rfgen.errors.StorageCollisionError. dataset_id names the logical collection;
snapshot_id names this immutable publication. Empty publications are invalid.
provenance is opaque string-to-string metadata about how the run was
produced, recorded once at the collection root beside dataset_id and
snapshot_id. Core owns the keys; an implementation must persist the mapping
verbatim and retrievably, or raise if its format cannot carry root metadata.
Silently dropping it is the failure the parameter exists to prevent: a corpus
that cannot say which environment built it cannot be told apart from one built
by a different environment. None means the caller did not establish the facts,
which is not the same as establishing that there are none — an implementation
must be able to render that distinction rather than collapsing both to empty.
The keys core currently supplies are in rfgen.storage.provenance.
For bounded candidate execution, waveform_plan_root_provenance stores the
invariant candidate, selection, implementation, accounting, grid, resource, RNG,
and replay authority once at the dataset root.
validate_waveform_plan_root_provenance rejects missing, forged, cross-plan, or
record-shaped copies. These facts are deliberately absent from individual model
records.
open returns a RecordCollection protocol with __len__() -> int and
__getitem__(index: int) -> object. Missing, incomplete, corrupt, or wrong-format
destinations raise the implementation’s storage error. Generation calls only
the three lifecycle methods.
compare reports whether two collections hold identical persisted content.
rfgen.inspection turns a true result into a published determinism claim, so
it must be exact: true only when no persisted value, value type, name,
ordering, or metadata differs. There is no neutral default, because the base
class cannot inspect a record type it does not define. A backend whose open
yields rfgen.graph.SceneRecord instances delegates to
rfgen.storage.compare_record_collections; one that yields its own record type
owns an exact comparison of that type, as the built-in SDS backend does.
Comparing under two bars¶
compare is the exact bar and always has been. Byte identity is the wrong bar
for a corpus whose graph contains an environment-bound node: a work-stealing
compute backend sums each contribution as its thread finishes and
floating-point addition is not associative, so identical rays under an
identical seed differ in their last bits. That sits downstream of randomness
entirely and is not a seeding question.
compare_physics(left, right, *, waveform_rtol) is the two-bar comparison and
returns a rfgen.storage.ComparisonReport. Ground truth is exact and
waveforms are held to a relative tolerance. Exactly one thing is relaxed: the
payload values of fields that core itself pins to the unit V, whose persisted
unit token agrees, and whose payload is of inexact type. Core pins units only
for the field names it produces, so a use-case field cannot obtain the weaker
bar by annotating itself as volts. A non-finite deviation always fails, because
nan > tol is False and a bare threshold would pass it.
Record count and ordering, record ids, record metadata, the
field-name set, every field’s axes, metadata, dtype and shape, and every other
payload are still compared exactly, whatever waveform_rtol is set to.
ComparisonReport carries the verdict and the measurement behind it: equal,
labels_equal, waveforms_equal, waveform_rtol, max_waveform_deviation
(the largest relative L2 deviation observed), first_difference, and
measured — which is False when the neutral default ran, so the deviation
figure is not evidence of anything.
compare_physics is a concrete method rather than an abstract one, because
adding a required keyword to an abstract method breaks every out-of-tree
implementation. A backend that does not override it inherits a default that
defers to compare and reports the exact verdict, so it is stricter than
asked and never looser.
The SDS backend implements it through compare_sds_collection_physics in
rfgen.storage.sds.validation, which returns a CollectionComparison and keys
the waveform classification on the module’s WAVEFORM_UNIT token ("V"). Call
compare_sds_collection_physics directly when comparing two SDS datasets
outside the storage port; compare_sds_collections remains the exact-only
boolean.
Storage also carries two members that inspection consumes and an
implementation may override: validate(collection), whose neutral default only
proves every ordinal is reachable, and the class attribute
supports_structured_inspection: bool, which selects a rich per-record audit
over a record-count summary.
Third-party backends can run the full contract as a test suite by subclassing
StorageContractSuite from tests/unit/storage/conformance.py and overriding
make_storage.
Minimal external implementation¶
Static Transcript — not runnable
Audit pending; do not treat this block as a runnable example.
from rfgen.errors import StorageCollisionError
from rfgen.storage import (
ShardPublication, Storage, StoredShard, compare_record_collections,
)
class MemoryStorage(Storage):
def __init__(self):
self.objects = {}
def write_shard(self, records, destination, *, shard_index,
first_ordinal, scene_ids, work_id=None, attempt=0):
if len(records) != len(scene_ids):
raise ValueError("scene_ids must align with records")
key = f"{destination}.shard-{shard_index}"
self.objects[key] = tuple(records)
payload = {"key": key, "scene_ids": list(scene_ids)}
return StoredShard(first_ordinal, first_ordinal + len(records),
ShardPublication("memory", payload))
def publish_shards(self, shards, destination, *, dataset_id, snapshot_id,
provenance=None):
if destination in self.objects:
raise StorageCollisionError("destination already exists")
ordered = sorted(shards, key=lambda item: item.first_ordinal)
if not ordered or ordered[0].first_ordinal != 0:
raise ValueError("publication must begin at ordinal zero")
records, stop = [], 0
for shard in ordered:
if shard.publication.format != "memory" or shard.first_ordinal != stop:
raise ValueError("wrong format or noncontiguous shards")
records.extend(self.objects[shard.publication.payload["key"]])
stop = shard.stop_ordinal
self.objects[destination] = tuple(records)
return self.objects[destination]
def open(self, destination):
return self.objects[destination]
def compare(self, left, right):
return compare_record_collections(left, right)
Register the class in the distribution metadata:
Static Transcript — not runnable
Audit pending; do not treat this block as a runnable example.
[project.entry-points."rfgen.storage"]
memory = "my_package.storage:MemoryStorage"
get_storage("memory", **params) loads the class, verifies it subclasses
Storage, and passes params to its constructor.
Built-in SDS¶
SdsStorage(Storage) is selected by the default configuration:
Static Transcript — not runnable
Audit pending; do not treat this block as a runnable example.
storage:
format: sds
destination: ./rfgen-output.sds
params: {}
It owns Signal Dataset encoding, atomic publication, reopening, physical
validation, and deterministic comparison under rfgen.storage.sds.
rfgen.storage.sds.accessors is the read side of that record contract, paired
with the write side in rfgen.storage.sds.record. Its functions — waveform,
detection_boxes, all_detection_boxes, box_families, segmentation_mask,
and yolo_boxes — each take an sds.Record and return a validated projection
of it. They exist so a reader of a published collection does not have to know
the physical field and axis layout; they are read-only views for inspection and
annotation, not training loaders, samplers, or collators.
entity_group_frame(record) is the canonical reader for framed emitter
identity. It accepts at most one scalar EntityGroupFrame projection and
reconciles it with the exact enum catalogs embedded in flat ragged SNR rows.
Unknown versions or indexed roles, partial frame columns, duplicate scalar
catalogs, and crossed catalog order refuse. SdsEntityGroupFrame.group_name
resolves a checked code; it never defaults an absent group to zero. Detection
accessors likewise require emitter_group_id and the frame to be present
together. A record with no frame follows the exact historical bare-index
schema. Discovery is namespace-first: any scalar
*.emitter_group_frame.* leaf or direct flat-row emitter_group_* leaf opts
into this validation, so an unknown or misspelled leaf cannot disappear into
the legacy fallback.
entity_group_membership(record) decodes the optional immutable v1 family
authority joined from dataset-root metadata into
SdsEntityGroupMembership, SdsEntityGroupFamilyMembership, and
SdsEntityGroupItemRange values. record_with_entity_group_membership checks
the exact indexed-family census, dense range coverage, frame codes, and any
physical group-ID columns, then reconstructs missing group IDs only in the
logical reader view. Raw SDS records and signal arrays remain unchanged; an
older dataset with no authority keeps its historical unframed behavior.
declared(entity, name, *, ordinal, group) looks up one well-known declaration
inside a single decoded entity mapping. entity maps
relative member paths to their values; name is the final member token to
find. ordinal and group identify that entity in any diagnostic. The lookup
accepts either an exact path or a nested path whose final segment is name, so
both bandwidth_hz and source.bandwidth_hz satisfy
declared(entity, "bandwidth_hz", ordinal=0, group="emitters"). It returns the
stored value, including a falsey value, or None when no declaration exists.
If two member paths end in the requested token, it raises
rfgen.errors.ValidationError rather than guessing which declaration the
caller meant. Callers must resolve any choice among multiple entity groups
before this lookup; declared never chooses a group.
A record may carry more than one family of detection boxes, so the three that
read boxes ask three different questions. box_families returns the family
prefixes this record actually carries. detection_boxes reads one of them,
defaulting to ground_truth.boxes. all_detection_boxes reads every family,
which is what “the boxes in this record” means and what yolo_boxes exports.
Reading the default prefix alone answers a narrower question, and on a corpus
whose emitters are not all produced the same way it answers it wrongly; see
the label schema.
RFGen supports Signal Dataset >=0.2.3,<0.3; see the
Signal Dataset project.
Signal Atlas training datasets¶
DigitizedRFObservation is the public model-input concept. Its concrete,
versioned source profiles are CommunicationsSignalProfile,
FMCWRadarSignalProfile, and the storage-only future
PulsedRadarSignalProfile. DigitizedRecord, communications_record, and
fmcw_radar_record enforce direct signed-int16 adc_iq storage and the exact
communication [rx,time,I_Q] or FMCW radar [chirp,rx,fast_time,I_Q] axes.
Graph identities, path evidence, processing products, and realized simulation
facts are deliberately absent from the source record.
RecordDatasetWriter is the source-publication seam;
SDS021RecordDatasetWriter is its SDS 0.2.1 implementation. It delegates to
native write_shard and publish, with SourceReceipt, PublicationError,
select_source_attempts, and compact_root_metadata defining create-only
attempt selection and a root whose size does not scale with record count.
ReceiptSequence is the durable SQLite-backed coordinator input to those native
SDS APIs. It streams canonical receipts through the standard Sequence
contract, survives coordinator restart, and avoids retaining shard receipts in
RAM; it does not replace or extend an SDS API.
SignalAtlasPublicationService and execute_dataset_plan coordinate the
bounded stock path. RecordExecutionProducer pages records into target-sized
shards using the preflight ADC byte model; qualification_rows projects typed
gate evidence. A production URI without its publication provider fails before
any propagation solve or record allocation.
ExecutionOutputStaging defines write_attempt, select_winners, read,
mark_annotated, and cleanup. BoundedObjectStoreStaging stores versioned,
framed native SDS 0.2.1 SafeTensors annotation payloads through the public SDS
ObjectStore. A fresh coordinator can reconstruct a selected original payload
from its receipt and object store; its immutable value
types are StagedOriginalOutputBatch, StageReceipt, StagedSelection, and
AnnotatedSelection, and StagingState is the closed
OPEN -> SELECTED -> ANNOTATED -> CLEANED lifecycle. The SDS API has no delete
operation, so cleanup releases the bounded local spool only after annotation
readback; remote expiry is an object-store lifecycle concern.
Attempt uniqueness is held in a durable SQLite coordinator table keyed by run,
source record, work item, and attempt—not an in-memory set or fixed 10,000-item
cap. The preflight-computed shard count is the configured bound. Staged payloads
decode to OriginalExecutionOutput, whose dense audit record is the complete
typed propagation-through-processing restart payload.
scientific_audit_annotation, evidence_fields,
publish_original_scientific_audit, supervision_annotation,
recomputed_scientific_audit_annotation,
publish_recomputed_scientific_audit, derived_products_annotation, and
publish_aligned_annotations and publish_streamed_annotations build native,
source-aligned SDS annotations. Every source ordinal is SUCCESS, SKIPPED,
or FAILED; tensor evidence stays in fields and bounded descriptors stay in
values/provenance. Radar auxiliary products are recomputed from published ADC
and the dataset-level acquisition/calibration contract before staging cleanup.
derived_products_annotation is owned separately by
rfgen.storage.sds.derived_radar_annotations: it emits dense range-Doppler,
detection, and angle fields, or an explicit aligned SKIPPED row when no
auxiliary product exists.
ScientificCatalog stores the canonical relationship, entity, geometry,
material, interaction, partition, component, limitation, emitter, and
modulation vocabularies once. Per-record kinematics, multi-object path rows,
receiver calibration, component powers, metrics, RNG seeds, and conditioning
arrays are dense SafeTensors fields referencing those integer IDs. Offset and
catalog bounds are validated before staging and again after SDS readback; the
object/path/trace rows are not repeated as JSON.
The encoder accepts QualificationEvidenceValue, ExecutionTraceValue, and
ReceiverEvidenceValue structural contracts. They expose only the validated
fields needed for SDS projection, so storage does not import the generation
executor that produced transient evidence. Runtime protocol checks reject
unrelated objects before dense fields are emitted.
The dense path authority additionally retains clock/carrier/epoch, states,
array elements, trajectories, solver budgets and seed, while bounded metadata
retains world/scene/material/geometry provenance, endpoint/port/relationship
identities, roles, patterns, polarization, limitations, and RNG key/provider.
Communication and radar source records carry realized carrier frequency because
it is required to interpret sampled observations; they still carry no graph ID.
Signal Atlas publication and scientific audit helpers¶
The graph-to-publication boundary remains typed rather than passing loose
dictionaries. GraphRecordExecutionProducer executes a realized graph record
and returns a ProducedSignalAtlasRecord; iter_produced_batches pages those
values according to the preflight shard bound. array_field is the shared
constructor for an axis-labelled SDS tensor field. acquisition_contract
extracts the dataset-level clock, ADC, carrier, array, and FMCW calibration
authority, while annotation_equal performs the exact aligned-annotation
readback check. reprocessed_product_batches streams product annotations from
published ADC; reprocess_published_radar_record is the single-record radar
decoder/range-Doppler/CFAR/angle replay used by that stream.
catalog_from_params creates the bounded ScientificCatalog from declared
pipeline entities before records are generated.
WaveformCatalogEntry is the dataset-level immutable parent/variant/selector/support
vocabulary row. catalog_with_observed_waveforms validates runtime selections against that
authored vocabulary; waveform_selection_annotation encodes the aligned compact integer
sidecar, resolve_waveform_selection decodes it without use-case imports,
validate_waveform_selection_dataset checks record identity and emitter-slot order, and
compact_qualification_audit removes transient string selection rows before publication.
Waveform-selection corpora declare READINESS_KEY in the source root.
publication_readiness_declaration names the mandatory aligned publication,
readiness_receipt reconstructs its exact dataset/snapshot/count authority,
publish_readiness creates the completion receipt only after sidecar readback, and
require_ready_dataset makes RFGen readers refuse a declared root until that receipt matches.
The dense scientific audit is split by ownership:
path_authority_fieldsandpath_authority_metadataencode native-solve and coherent-conversion authority;reconstruct_path_authorityproves the dense fields and bounded metadata reconstruct the original typed authority.receiver_evidence_fieldsencodes receiver, converter, ADC, trace, provider, and RNG measurements, andradar_truth_fieldsencodes kinematics, interactions, component powers, SNR/SIR/SINR, and processing limits.validate_dense_auditis the public closed-schema, catalog-index, shape, and ragged-offset validator. It is re-exported fromsds_annotations; both names identify the same validation boundary rather than two implementations.
Source and staging receipts have closed canonical codecs:
encode_source_receipt / decode_source_receipt and
encode_stage_receipt / decode_stage_receipt. Decoders validate the complete
typed value and do not treat receipt JSON as trusted merely because it parses.
Create-only artifacts and durable receipt sequences¶
canonical_artifact_path fixes an absolute lexical identity once, and
validate_create_only_target rejects unsafe path components. open_parent
performs descriptor-relative, no-follow traversal. reserve_artifact returns a
ReservedArtifact holding the retained parent and leaf descriptors plus its
ArtifactReservationAuthority; publication verifies inode, device, link count,
and visible leaf before and after writing. create_only_file is the bounded
one-call form. Collisions, symlinks, races, and terminal partial reservations
raise CreateOnlyFileError; no helper reopens a validated path by name.
MappedReceiptSequence is the read-only mapped view over a sealed durable
receipt sequence. It preserves canonical shard order and Sequence semantics
without materializing receipt objects in memory.
Qualification artifacts and remote object closure¶
LoaderQualificationArtifact combines the selected closed profile, complete
measured result, runtime observation/identity, fixture authority, and verdict.
create_qualification_artifact derives the verdict from those values;
write_qualification_artifact uses create-only local publication and
read_qualification_artifact performs closed typed decoding.
validate_qualification_artifact re-evaluates profile, environment, metrics,
trace, and thresholds. validate_remote_owner_authority additionally binds the
remote run marker, workload, backend, and evidence destination, while
validate_inventory_bounds enforces the registered unique-object and creation
event ceilings before accepting remote evidence.
CreatedObjectReference is one native URI/generation pair.
InventoryTrackingObjectStore records successful native creates and CAS
updates without recomputing payload digests. ClosedObjectInventory is the
canonical, segment-bounded, generation-checked final listing plus its
post-comparison seal time; duplicate, escaped, missing, replaced, or excessive
objects fail.
Remote qualification planning and run ownership¶
RemoteLoaderQualificationConfig is the closed operator input, and
plan_remote_loader_qualification produces a zero-write
RemoteLoaderQualificationPlan. RemoteLoaderQualificationService executes
that plan using a CloudQualificationEnvironment; production uses
GoogleCloudQualificationEnvironment. Their resulting
CloudQualificationAuthority binds account, project number, bucket ownership,
runtime workload, source, profile, and native generations.
RemoteQualificationRequestAuthority is the canonical run request shared by
preflight, marker, command, and artifact. OwnerMarkerPayload contains facts
known before its create; OwnerMarkerAuthority pairs that payload with the
native returned marker URI/generation. create_owner_marker is create-only and
assert_owner_marker_only proves no other run-prefix object exists at that
point. These owner-marker types are re-exported from both
remote_run_authority and remote_loader_qualification as one contract.
remote_profile binds the registered dynamic GCS template to the attested
provider environment and measured result; it does not mutate the portable
registry template.
Cloud Run provider-document primitives¶
The following exported values make the closed Cloud Run v2 decoding contract inspectable to deployment and evidence tooling:
CLOUD_RUN_CONDITION_FIELDS,CLOUD_RUN_CONDITION_STATES,CLOUD_RUN_CONDITION_SEVERITIES,CLOUD_RUN_CONDITION_REASON_ENUMS, andCLOUD_RUN_SUCCESSFUL_RETRY_CONDITIONSenumerate the accepted provider Condition surface and the narrow successfulRetryexception.CLOUD_RUN_INT32_MAX,provider_count, andprovider_false_or_booleanstrictly decode proto-JSON counters and omitted false booleans without Python bool/int coercion.latest_execution_resourceandexecution_job_resourcenormalize only the documented short relationship names against the expected full resource segments; the full names remain stored authority.PROVIDER_TIMESTAMP_PATTERNis re-exported by bothruntime_authorityandruntime_timestamps.validate_condition_timestampapplies the same lossless, timezone-qualified RFC3339/nanosecond contract used by all provider chronology evidence.
LoaderBenchmarkProfile, LoaderBenchmarkResult,
RegisteredLoaderBenchmarkProfiles, run_loader_benchmark, and
evaluate_loader_benchmark pin and execute source-only, the real SDS 0.2.1
iter_record_metadata path, requested-annotation, byte-amplification, latency,
throughput, and peak-memory gates. CountingShardStore measures actual indexed
payload reads; callers cannot submit fabricated counters to the runner.
RuntimeEnvironmentAuthority is collected by the runner rather than copied
from dataset metadata. For GCS, run_loader_benchmark requires the immutable
image digest already obtained from RuntimeAttestationProvider, passes it into
that independent runtime collection, and validates the resulting identity
against the bound profile. Omitting the digest still fails with
LOADER_GCS_TRUSTED_IMAGE_REQUIRED; no environment variable substitutes for
provider authority. Its canonical identity records backend attestation,
OS/machine, Python ABI, normalized distribution locations/build identities and
source head; its raw observation separately retains absolute executable,
environment and module paths. Fresh offline installs at different temporary
paths therefore compare equal only when their substantive identities match.
Local qualification is explicitly a local_process with no invented image.
GCS requires AttestedGCSObjectStore, a gs:// root, generation-aware SDS
operations, an actual Google client transport and trusted container digest;
protocol doubles are contract_only and cannot emit qualified evidence.
The runner records requested workers separately from worker overlap and from
active/timestamped reads around the real shard-store delegate. The local profile
is sequential; a parallel profile must observe matching storage-read overlap.
Local cache authority says only that a fresh dataset/client with no application
cache layer was used: OS page cache is explicitly uncontrolled and backend
cache is unknown, so this is not called a cold-cache latency result. A cached
profile remains unavailable until its store exposes real cache counters.
Concurrency evidence retains worker-task intervals separately from delegate
storage-read intervals. Both use successful half-open [start, end) intervals:
an end and start at the same timestamp do not overlap. Artifact validation
derives each maximum from its own intervals and rejects inflated or deflated
summaries.
The registered remote qualification uses SDS 0.2.1 directly against GCS. Its
operator command requires an explicit runtime service account, project, bucket,
owned prefix, unique run ID, Cloud Run location/job, expected container image
digest, and disjoint same-bucket evidence URI.
GoogleServiceAccount and validate_google_service_account define that runtime
principal precisely: either a standard custom
NAME@PROJECT.iam.gserviceaccount.com identity or the Compute Engine default
PROJECT_NUMBER-compute@developer.gserviceaccount.com identity. Human operator
accounts are validated separately and are not runtime service accounts.
Cloud Run workload variables locate the current job/execution but do not assert
its image: Resource Manager and Cloud Run v2 control-plane responses must bind
the project number, execution and digest-pinned job image. GCS’s authoritative
bucket project number must match. Preflight verifies the refreshed ADC runtime
service-account principal/project, permissions, and empty run namespace before
fixture creation. The owner marker captures the complete runtime,
project, prefix, source and native object-generation authority. Existing or
partially written run prefixes are terminal: recovery is ABANDONED -> NEW RUN ID; no automatic deletion or same-run resume is claimed.
Source authenticity begins from a clean checkout at an explicit reviewed
40-hex head. A sanitized fresh Git context is uploaded, and the Docker build
must derive the same head from its retained objects while both Git diffs and
the untracked inventory are empty. Cloud Run Job and Execution labels, the
verified image revision file, runtime authority, and artifact source_head
must agree. CloudRunQualificationResources and
remote_cloud_run_resources close the one-task, no-retry, 2 CPU, 4 GiB,
7,200-second, generation-2, no-volume resource envelope.
CloudRunQualificationDeployment and cloud_run_job_manifest emit a true JSON
argument array. No comma-delimited gcloud --args or reconstructed shell
command is an authority. The deployment model rejects any contradiction between
its project, region, job, service account, source head, or immutable image
digest and the nested QualificationCommandAuthority; the command remains the
single authority for run/profile/dataset/evidence fields that are not duplicated.
validate_cloud_run_v1_job_manifest enforces the
provider’s Job → execution template → task template nesting and binds every
field to the typed deployment/command/resource authority: exact metadata,
singleton container, digest image, command/argv, service account, retry,
timeout, counts, and resource limits. Unknown keys and type substitutions fail.
Labels and annotations live on supported Job/execution metadata, while the inner
task template contains only its spec. Runtime attestation consumes the Cloud Run
v2 response shapes—Job execution template plus nested task template, and the
Execution’s direct task template—rather than assuming the v1 YAML layout is
returned unchanged.
QualificationCommandAuthority, build_qualification_command,
qualification_argv, and command_annotations are the single canonical
builder/serialization path. commands_match and
semantic_authority_matches require the provider-returned Job and current
Execution to match that command, argv, labels and annotations exactly.
The GCS store wrapper records every successful native create/create-file/CAS generation. After all source and annotation writes—and before measurement or artifact emission—the exact run-prefix listing must equal the closed URI and latest-generation inventory. Extra, missing, replaced or escaped objects fail; the bounded inventory and marker generation remain in the typed artifact. The registered remote profile caps unique objects and creation events. A provisional check follows publication, then the authoritative final listing occurs after the complete read trace and immediately before artifact construction; its seal timestamp must fall between benchmark completion and artifact generation.
After the benchmark and final dataset seal, the validated artifact is created with GCS’s native generation-zero precondition at the separate evidence URI. The secret-safe receipt binds artifact schema, source, profile, fixture, run, URI and returned generation. Exact-generation readback is validated before success; collisions never overwrite. See GCS loader qualification.
The durable evidence surface is QualificationArtifactReceipt,
gcs_object_parts, validate_evidence_destination,
serialize_qualification_artifact, publish_qualification_artifact, and
validate_qualification_artifact_receipt. Startup and final in-process
workload evidence uses RuntimeAttestationProvider,
CloudRunRuntimeAttestation, RuntimeWorkloadAuthority,
JobConfigurationAuthority, ExecutionConfigurationAuthority,
ProviderResourceRevision, ExecutionResourceRevision,
RuntimeFinalCheckpoint, and RuntimeTerminalAuthority; each binds native
Cloud Run names, UUIDs, etags, generations, observed generations, and provider
times. job_resource_revision, execution_resource_revision, and
resources_match are the strict provider-document decoders/comparator.
RuntimeStabilizationPolicy bounds startup control-plane convergence to eight
observations with inspectable 0.25-second exponential backoff capped at two
seconds. Startup proceeds only after two identical active-phase reads: the Job
is current and non-reconciling, while the Execution is reconciling with exactly
one successful each of Started, ContainerReady, and ResourcesAvailable,
one reconciling Completed, one running task, zero
terminal counters/time, and at most the qualified auxiliary Retry. The
in-process final checkpoint requires the same active phase. Only the external
terminal collector accepts a non-reconciling Execution with successful
all four core conditions (including Completed), terminal counters, and
completion time. Unknown/duplicate core types and reason or error fields on a
supporting success fail closed. Provider proto JSON short names in Execution.job and
Job.latestCreatedExecution.name are normalized only when they exactly match
the corresponding full resource-name segments; full resource names remain the
stored authority. Omitted/null/false reconciling means false, while other
types fail, and omitted counters mean zero without accepting coercions.
Execution counters are bounded to the provider’s nonnegative int32 range.
active_execution_conditions recognizes the active condition cardinality;
runtime_documents_converged combines it with relationship, generation,
counter, and lifecycle checks.
provider_time is the single required provider-timestamp decoder;
provider_optional_time admits only omission/null or the same timezone-qualified
RFC3339 string. Create, update, start, completion, and condition-transition
times share this contract, so falsey placeholders and Python datetime objects
are not accepted as provider JSON. ProviderTimestamp serializes a canonical
UTC instant and signed epoch nanoseconds; instant_nanoseconds and
datetime_nanoseconds perform ordering without float or microsecond truncation.
timestamp_components is the shared exact parser used to bind both serialized
fields, so a forged text/integer pair fails reconstruction. Both fields are
strict (canonical_utc text and epoch_nanoseconds real integer): Pydantic
string, boolean, float, or decimal coercion is rejected. Offset normalization
at the supported calendar boundary fails with the named provider-time error.
Cloud Run updateTime is resource/configuration chronology and may remain at
creation time when status reaches terminal success. Resource authority still
requires createTime <= updateTime; terminal status authority separately
requires start time ≤ completion time ≤ the Completed transition ≤ collector
observation. It never orders status against updateTime.
Revision, observed-generation, latest-execution, and
running-state convergence may retry; configuration, semantic, resource, or
relationship drift and any terminal state fail immediately. Cloud Run v2 marks
CONDITION_FAILED terminal for every condition type, not only Completed;
error severity and fatal common/revision/execution/instance reason-union values
also fail immediately. The decoder uses closed official state, severity, and
per-union-field enum sets, rejects malformed or cross-field enum values and
more than one populated reason field, rejects fields outside the official
nine-field Condition document, and requires timezone-qualified RFC3339
lastTransitionTime values. Common reason UNKNOWN is a failure.
validate_cloud_run_condition, FATAL_CLOUD_RUN_CONDITION_REASONS,
and cloud_run_condition_reports_failure keep that schema visible.
One provider-observed exception is closed explicitly: Cloud Run Jobs may emit
Retry / CONDITION_SUCCEEDED / WAITING_FOR_OPERATION / INFO to say that retry
scheduling succeeded. That exact tuple is valid but nonterminal; no other
successful condition/reason combination is inferred from it.
Tests inject sleep
and policy values, so no wall-clock wait is needed to verify the boundary.
job_configuration_authority, execution_configuration_authority,
configuration_fields, immutable_resource_document, canonical_json, and
provider_time produce the
closed immutable/mutable split from provider documents;
provider_generation strictly decodes Cloud Run’s positive REST int64 decimal
strings. terminal_conditions_succeeded validates the actual provider
Condition fields without inventing condition-level generation metadata.
CLOUD_RUN_CONDITION_REASON_FIELDS lists the complete official reason union so
successful conditions cannot conceal a contradictory instance-level reason.
QualificationTerminalReceipt, TerminalAttestationProvider,
collect_qualification_terminal_receipt, and serialize_terminal_receipt
form the separate read-only post-execution success gate. The terminal authority
stores the configured task count and, during deserialization, requires zero
running/failed/cancelled tasks and exactly that many successes. Its receipt
also binds the terminal Execution name, UID, and terminal etag to the nested
artifact receipt. Canonical projection validation makes every typed
Job/Execution field prove itself against its own immutable provider document,
requires the two documents’ task templates and semantic metadata to agree, and
binds terminal task cardinality to both documents. Empty, malformed,
noncanonical, asymmetric, or projection-drifted documents fail during
deserialization. The terminal authority also retains the closed Cloud Run
resource envelope and verifies both documents’ retries, timeout, execution
environment, volumes, CPU, memory, task count, and parallelism against it.
Provider counts are strict nonnegative int32 values: booleans, floats, strings,
explicit null, and omitted required counts are rejected. The measured Cloud
Run v2 documents omit volumes for the native-GCS profile, so omission is the
only accepted no-volume representation; explicit null, [], and other falsy
substitutes are not treated as absence. Timeout, execution environment, CPU, and memory are
strict strings with the exact registered values.
Fixture ownership is separate:
ArtifactReceiptLogQuery, ReceiptLogProvider,
GoogleCloudReceiptLogProvider, cloud_logging_filter,
cloud_logging_string_literal, receipt_log_event, and
collect_artifact_receipt_from_log recover the exact
artifact generation from one bounded, exact-Execution Cloud Logging event.
RECEIPT_LOG_EVENT and RECEIPT_LOG_SCHEMA are its stable structured-log
discriminators.
GCS_OBJECT_SEGMENT_PATTERN is the closed operational character policy shared
by qualification evidence object paths.
RemoteAnnotationSequence, build_remote_fixture, and
publish_remote_annotations construct only the bounded benchmark fixture.
qualify_local_sds021_loader.py builds the closed fixture and emits a
typed artifact whose verdict is re-derived from its complete result and profile
thresholds.
The registry ships local_sds021@1 as the executable qualified profile and
gcs_remote@1 as available_unqualified. That entry is intentionally a
dynamic production template: each run must bind its provider-attested image,
source, runtime, and measured result before its selected profile becomes
qualified. The corrected m1-gcs-20260828-03 result, exact-generation
artifact receipt, and successful terminal receipt are retained under
.agent-state/deliveries/m1-gcs-sds021-loader-*. This measured authority does
not turn the unbound registry template into a portable qualification claim.
The local result uses a real non-editable SDS 0.2.1 wheel and a 64-record
int16[2,16,2] fixture with 16 selected 4 KiB annotations; the bounded measured
result is retained in .agent-state/deliveries/m1-local-sds021-loader-qualification.json.
Resource profiles also cap native SDS source and annotation shard counts using a
measured 1 KiB manifest/receipt allowance and 2x safety factor, because SDS
0.2.1 internally materializes its publication manifest sequence.
Runtime provider documents are owned by canonical_configuration_document,
configuration_projection, configuration_documents_match,
configuration_resource_documents_match, configuration_resources_match,
configuration_task_count, exact_resource_limits, required_provider_count, and
validate_configuration_projection; these functions compare exact provider authority without
reintroducing a monolithic runtime-authority module.
SDS dataset invariants and emitter SNR¶
Waveform execution provenance¶
WAVEFORM_ROOT_KEYS names the invariant H3 candidate/provider/runtime/RNG/H2
metadata stored once at the dataset root. waveform_root_provenance produces
that closed mapping and validate_waveform_root_provenance rejects missing,
forged, cross-provider, or per-record copies. Its replay scope requires exact
configuration and ground truth while explicitly making no whole-environment
waveform byte-identity claim.
DATASET_INVARIANTS_KEY names immutable root metadata promoted by
dataset_invariant_metadata. dataset_invariants validates that root view;
to_sds_varying_record removes the configured invariant fields from the hot
record, and record_with_invariants joins them for logical reads.
DatasetInvariantView is the typed root/record split. Emitter SNR readers use
emitter_snr_by_entity_receiver for the complete identity key and
emitter_snr_by_receiver only when emitter identity is unambiguous.