Pipeline authoring and staged compiler IR
RFGen job authoring separates scientific intent from compiler wiring. A production job names
a stable installed PipelineDefinition and DatasetRecipe; the definition builds a canonical
rfgen.staged/v1 graph without executing physics. Workers receive the resulting immutable
ExecutionPlan, not authoring Python.
Radar plans include a typed RadarProcessingPreflight that bounds per-record
FFT, dense range-Doppler, CFAR, detection, angle, and raw-capture workspace
before execution. It records the simultaneous live-byte subtotal, exact dense
column and FFT-scratch assumptions, and the allocator safety factor separately.
job.yaml -> pipeline/recipe resolution -> strict Params -> deterministic build
-> canonical staged IR -> validation -> ExecutionPlan -> workers
Public authoring surfaces
PipelineDefinition[P] owns a stable identity, ParamsSchema[P], closed StageProfile, and
build(GraphBuilder, P). ParamsSchema is the genuine third-party codec seam: decoding rejects
unknown or missing keys and coercion; encoding produces finite canonical JSON. The built-in
PydanticParamsSchema implements that contract for strict Pydantic models.
Each recipe-addressable path also has a SampleableField: its Python type, unit, vector
dimension, allowed distributions, and unit-bearing bound keys are schema-owned rather than
guessed from YAML literals.
The other authoring records are JobConfig, JobProviderOverrides, DatasetRecipe, QualificationPlan,
PipelineResolution, and ExecutionPlan; compile_job and pipeline_definitions form the public
composition boundary.
DatasetRecipeDefinition is independently registered through dataset_recipes. The core
InlineDatasetRecipe strictly decodes InlineRecipeParams into per-stratum allocation, Stratum,
CoverageGate, and AnnotationRetention. plan_recipe emits a compact RecipePlan with
RecordAllocationPlan/StratumAllocation identity formulas and one ScopeSamplingTemplate per
authored field. Dataset, stratum, scene, device-bundle, and record scopes are all lazy; no
SamplingOperation or PlannedRecord table grows with realization count. Workers page records
through record_at/iter_records and facts through realize_at/iter_operations using the same
stable key regardless of retry or traversal order. WorkAllocation similarly pages shard IDs.
Templates are canonicalized by field path and then closed scope precedence: dataset, stratum,
scene realization, device-bundle realization, record. The same field may intentionally appear at
different scopes; realization applies all facts least-to-most specific. A duplicate scope/path is
rejected because it would create two authorities at one precedence level. Within-record process
variation is not a scalar PLAN override.
Every realized operation carries a discriminated typed value specification, typed value,
SamplingKeyRecord, provider identity, SamplingFieldDescriptor, and operation identity.
Compilation proves the joint finite domain corners after stratum overrides, including
schema-owned wavelength/array and acquisition-Nyquist checks. Workers reapply lazy concrete facts
and decode Params again as a defense at execution time.
realize_record_params is that worker-side lazy realization and validation entry point.
Public ExecutionPlan.realize_record returns a typed RecordRealizationAuthority containing the
validated Params and fact provenance. The future RecordAuthorityExecutor seam rejects raw plans
or templates; non-dry-run execution remains intentionally unavailable until later waves.
An executor binds one exact plan snapshot and accepts an ordinal. It recomputes the expected
authority, compares the complete canonical transport value (plan/graph/pipeline/schema/record
identities, immutable canonical Params JSON, and ordered facts/keys/providers), then independently
decodes Params immediately before future dispatch. Deserialized authority is closed but untrusted;
cross-plan, stale, missing, extra, or mutated authority never dispatches.
ResourcePreflightProfile, ResourcePreflightProfileDefinition, ResourceEstimateModel,
ResourceShapeBounds, ResourcePreflight, and
estimate_resources provide the pinned closed ADC/evidence/annotation/solver preflight.
Concise jobs omit these operational constants: the compiler deterministically resolves the
registered, versioned communications or radar profile and records its identity, rationale, and
complete typed assumptions in RecipePlan.resource_preflight. An explicit job may select a
registered profile by storage.resource_profile; arbitrary inline models are rejected. Profiles
include SafeTensors and metadata overhead,
retention density, maximum paths/interactions, target shards, and explicit total-byte budgets.
RetentionCountEstimate marks pilot retention as exact. Hash-mod corpus retention is explicitly a
conservative_upper_bound: without enumerating record hashes, storage assumes every eligible
record could be retained rather than using an unsafe expected fraction.
resolve_resource_preflight_profile and resolve_resource_preflight_definition are the
fail-closed catalog lookups used by compilation.
Sizing runs before record paging and uses the maximum declared stratum/sample bound, every native
relationship, authored
max_depth, and a solve for each distinct scene/device bundle or path-affecting record fact.
The installed domain profile derives that maximum as typed ResourceShapeBounds before
estimation.
set_semantic_path applies ID-addressed facts without ordinal coupling. An oversized record,
total budget, or uint64 publication estimate fails compilation.
The compiler uses merge_mapping to apply semantic raw overrides before complete decoding and
estimate_record_bytes to derive the mandatory ADC payload bound.
Record and dataset publication policy is also installed by exact pipeline identity. A
RecordProjector receives only immutable RecordProjectionContext values and converts transient
execution outputs into the pipeline’s source-record shape. record_projectors and
resolve_record_projector discover and validate that binding. A DatasetPublicationProfile
owns source axes, qualification-row projection, and optional derived-product policy;
dataset_publication_profiles and resolve_dataset_publication_profile reject absent,
malformed, duplicate, or cross-pipeline bindings.
StagedGraphRuntime executes the generic staged graph, producing a ProducedRecord; the
DatasetPublicationService then applies the exact installed publication profile.
Prepublication gates implement the registered DatasetQualification ABC. QualificationPlan
contains typed QualificationGatePlan entries; QualificationEvidence is evaluated into an
immutable QualificationResult. qualification_gates currently registers strict typed
categorical-count, geometry-grid, and radar-grid implementations.
The JSON plan stores no runtime class: it carries registered identity, Params schema version,
origin, and concrete typed Params. Reconstruction resolves and verifies the implementation.
Selectors and per_group counts are applied independently against the expected stratum/category
universe, so an absent group or enum/choice value has an explicit zero count and fails.
Graph construction requires an entered stage, explicit local semantic ID, and typed handles for
every runtime input. Repeated components use GraphBuilder.scope. Instance identity is derived
from stage, scope, and local ID, so inserting an unrelated node cannot rename or reseed an existing
one. Configuration mappings are accepted only as operator Params; they are never auto-lifted into
runtime ports.
NodeDescriptor.type_id is exactly the canonical operator use value
rfgen.<physical_category>.<operator>@<major>. instance_id is the separate graph location and
RNG identity. This distinction makes a compiled graph auditable without confusing an operator’s
public meaning with one occurrence of it.
The staged module also exposes CompiledGraphArtifact, GraphHandle, canonical_type_id, and
portable_id. The planning module’s CompilationPlanningContext and PredeclaredPlanFact carry immutable recipe facts
into PLAN compilation; builders bind them but never allocate records or draw RNG.
OperatorCatalog resolves each OperatorDescriptor, validates its Params and exact port kinds,
and selects an explicit or sole-default ProviderDescriptor. CapabilityRequirement,
CapabilityOffer, and persisted ProviderResolution make exact/range satisfaction auditable.
provider_resolution_mapping serializes that decision into compiled provenance.
validate_staged_ir validates the closed raw grammar, preceding references, publications, joins,
projects, foreach bodies, bindings, stages, and catalog identities.
Human job shape
The closed rfgen.job/v1 root contains only version, pipeline, recipe, run, and storage.
Users write semantic Params, recipe coverage/allocation, run identity/count/seed, and destination
budgets. They do not write nodes, ports, Python module paths, sampled IDs, or work IDs.
Stable production definitions are discovered from rfgen.pipeline_definitions. A directly
registered or module-loaded development definition may be compiled with production checks disabled,
but cannot authorize publication because it lacks installed distribution/entry-point provenance.
Compilation performs two independent clean builds and rejects observable differences.
The installed Signal Atlas communications and radar distributions register
MultiEmitterCommunications and FMCWMultiRadar through the standard pipeline
entry-point group. Their stable selectors, deeply typed Params, visible stages,
and exact node/use/edge oracles compile through stock rfgen validate and
rfgen inspect. Core discovers the entry points but does not import either
use-case package.
Staged IR
rfgen.staged/v1 is generated audit material. The root contains ID-keyed resources and ordered
stages. Every descriptor retains the canonical operator type and explicit instance identity.
References may point only to typed handles already returned by a resource or preceding node.
Stage profiles reject unknown, nested, omitted, empty, or disallowed stages and operators.
LegacyGraphSpecAdapter is compatibility-only. A checked mapping must name every legacy
role/selector and its exact input/output ports; unknown selectors or semantic port differences fail
closed. Existing GraphSpec remains the default legacy API during migration.
Each reviewed table row is a LegacyOperatorMapping; compile_graphspec_default is the explicit
mapping-table route. retained_legacy_selectors is the checked retained identity census, and
compile_retained_graphspec_default is the GraphSpec/stock-CLI default audit route. It preserves
old execution while unknown role/selectors fail closed.
is_retained_graphspec(graph) compares the graph’s canonical configured identity with the
generated census of exact checked-in YAML graphs. Stock commands apply the frozen compatibility
audit only when it returns true; inline experiments and installed third-party graphs still receive
normal typed catalog validation without being misrepresented as retained programme artifacts.
The retained artifact uses canonical role/selector operator identities and the materialized
GraphSpec’s exact typed port surfaces; stock generation runs this audit before dispatch.
Deterministic planning randomness
PhiloxSamplingProvider implements rfgen.random.philox@1 with Random123 Philox4x32-10. Its key
contains provider identity, run seed, sample ordinal, stable instance identity, stream, and canonical
field path. Supported M1 value specs are fixed, unweighted choice, continuous uniform, and
uniform_box. Choice is nonempty and uses unbiased inclusive integer sampling; continuous bounds
are finite, half-open, and rejected when binary64 subtraction overflows. The provider never mutates
NumPy, Torch, Sionna, or process-global RNG state.
WaveformGenerationPreflight is the finite candidate-table maximum included in
the existing resource preflight. It reports the maximum native/output complex64
bytes, peak work bytes, and provider-specific dimensions and rejects the job
before candidate execution when a configured bound is exceeded.
aggregate_waveform_preflight computes that closed maximum from the immutable
candidate resource table and enforces the configured output and peak-work caps.
Exact world membership
MembershipDeclaration separates radio endpoints from geometry objects. Each endpoint binds a
native array and exactly one static pose or trajectory, and ONE_WAY requests require
TX_SOURCE -> RX_SINK; a monostatic entity may hold both endpoint roles. ActualMembership
holds the compiler’s independent census. Validation rejects duplicate typed IDs before exact
endpoint, entity, port, native-array, pose-or-trajectory, relationship, geometry-object, and
entity-state comparisons. Its other closed records are EndpointDeclaration, EndpointRole,
RelationshipRequest, and StaticPose; validate_exact_membership is the comparison boundary.
Executable operator and plugin authority
NodeOperator is the public execution seam. An implementation declares its canonical
OperatorDescriptor, typed parameter model, PlanInputs/PlanOutputs, and evaluates
RealizedInputs into RealizedOutputs with an immutable ExecutionContext. The context carries
the typed OperatorExecutionState, ExecutionPlanAuthority, RecordAuthority,
RecordProjectionContext, and RecordProjectorContract; CapabilityRequirement declares a
logical capability without confusing it with an instance ID. OperatorExecutionContext remains
the descriptive alias for this contract, and operator_descriptor constructs the canonical
instance-independent descriptor. ExecutionTraceEntry records the executed instance and canonical
operator identity. YAML use, descriptor type_id, and the compiled node’s
type_id must agree exactly. validate_compiled_operator_identities enforces
that agreement for resources, nodes, joins, and record projections before execution.
PluginBinding records the selector, distribution name/version, entry-point group/name/target,
implementation identity, and contract version selected by the compiler. The four-registry
PluginAuthorityRegistries is resolved once by resolve_plugin_binding; workers call
verify_plugin_binding and reject any drift before execution.
Directly cataloged node classes carry the parallel ImplementationBinding authority.
resolve_implementation_binding records their installed distribution and version at compilation;
verify_implementation_binding re-resolves that authority at the worker and rejects class or
distribution substitution. The staged-contract validators finite_json and require_id close the
portable parameter and identifier boundaries used by GraphBuilder.
The split planning records PipelineSelector, JobStorage, and JobRun retain job ownership;
derive_work_allocation produces the immutable allocation. GraphPipeline, build_twice, and
compilation_planning_context own graph compilation and its determinism check. The qualification
boundary exposes qualification_expected_categories for the closed expected-category set.
Candidate qualification before publication
qualify_candidate consumes generated source records, the selected original
staged attempts, and typed qualification evidence before an SDS root can be
created. Its immutable CandidateQualificationResult contains only bounded
aggregate gate results; record IDs and realized per-record facts never enter
dataset-root metadata.
qualify_staged_candidate is the streaming form used by Signal Atlas
publication. It reads only the winning StagedSelection, reconciles each
original execution payload with its source receipt and expected work ordinal,
and evaluates the same schema, conservation, coverage, and resource gates.
The result is still bounded aggregate evidence; staged per-record authority is
never copied into the SDS root.