Resume or repair a partial run¶
Warning
Pre-implementation. Commands describe the target CLI surface.
Resume a generation or annotation run after worker failure, quota exhaustion, manual cancellation, or transient storage errors.
When to use this¶
Use this when a run stops before all shards or annotation columns are complete. Do not mutate generated shard contents in place; preserve idempotency by re-running the same resolved config or creating a new dataset version.
Prerequisites¶
Find the original command, resolved config hash, output path, and run manifest.
rfgen inspect gs://rf-fm-datasets-synth/wideband/v0.5.0 summary
rfgen inspect gs://rf-fm-datasets-synth/wideband/v0.5.0 errors
Minimal command path¶
Resume Phase 1:
rfgen generate +preset=wideband_detection_baseline \
storage.path=gs://rf-fm-datasets-synth/wideband/v0.5.0 \
run.seed=1234
Resume Phase 2:
rfgen annotate gs://rf-fm-datasets-synth/wideband/v0.5.0 \
annotator=full_suite \
annotator.run_id=gemini-flash-v1
The framework should skip already-complete shards and already-complete
(sample_id, annotation_type, template_id, run_id) outputs.
Reconcile a controlled run¶
Layer 19 makes the run-control protocol executable through the programmatic
InMemoryRunController reference implementation. It is not a durable control
plane and does not make the target CLI commands above available. In particular,
its state, event snapshots, and pagination-token signing secret are local to
one controller process. A process restart cannot resume that reference
controller; persistence adapters are a later roadmap responsibility.
For a controller that remains available, keep the returned RunHandleV1 and
use the same handle for status, event, and reconciliation calls. Submit uses a
caller-provided idempotency key: reusing that key with the same canonical
RunSpecV1 returns the original handle, while reusing it for a different
specification fails. Every control operation also has a UTC deadline. An
expired deadline, a foreign handle, or a compare-and-swap state mismatch must
be treated as a failed operation, then investigated from a fresh status read.
The run lifecycle is PENDING, RUNNING, PAUSING, PAUSED, then a terminal
SUCCEEDED, FAILED, or CANCELLED state. A pause or cancellation request is
not completion: wait for the provider-facing progression to record PAUSED or
CANCELLED before issuing the next action. Terminal runs have no active lease,
and a run can reach SUCCEEDED only after every unit has succeeded.
For an incomplete run, collect its append-only event page before making an operator decision. The first event-page request fixes a 24-hour snapshot; pass the returned token unchanged to retrieve the next page with the same limit. The controller rejects a tampered, foreign, expired, or unknown token. Record the returned page hash and snapshot bounds with the incident evidence rather than reconstructing events from a later live view.
Reconciliation compares the controller’s unit state with provider observations
and returns a hash-bound ReconciliationReportV1. It can adopt an observed
successful unit, mark an observed failed unit, requeue a missing running unit,
or cancel an observed orphan through an explicitly configured provider
callback. It does not transition the enclosing run lifecycle. Review every
row, including its before and after unit state and evidence digest, before
continuing work. If orphan cancellation has no configured provider boundary,
reconciliation fails instead of silently claiming that the orphan was stopped.
Recover a publication transaction¶
Layer 17 provides the transaction and recovery contract used to publish a
manifest or object revision. A staged object is immutable and remains
invisible until the transaction compares and swaps the dataset-bound
LATEST pointer. The transaction records the expected prior revision, a
strictly increasing fencing token, a lease, and the hashes of every staged
object.
Do not retry an indeterminate pointer write blindly. First scan the recovery inventory and retain the returned finding and report with the run evidence. The scan is ordered by dataset, key, and finding code, so a paged scan can be resumed without changing repair order. The current release exposes this workflow through the Python transaction API; the commands in this guide remain the target CLI surface.
Run every proposed repair with dry_run=True first. A dry run makes no
mutation, leaves LATEST unchanged, and returns a NOOP report. Apply the
same finding with dry_run=False only after reviewing the report and before
its deadline. The coordinator rescans before a mutating repair; a finding that
is no longer current fails rather than applying a stale instruction.
Recovery finding |
Safe action |
Result |
|---|---|---|
Expired lease or stale fence |
Abort the stage |
Marks only the active transaction aborted; it does not change |
Prepared record without a pointer |
Finalize the record or abort the expired stage |
Finalization requires the current fence and checksum-valid immutable objects. |
Pointer without a commit record |
Finalize the record or restore the pointer |
Reconciles only when the observed pointer is the expected prior revision or the prepared revision. |
Hash mismatch |
Quarantine |
Keeps corrupt staged content out of a future publication. |
Orphaned stage |
Abort the stage or quarantine |
Removes the unreferenced staged object from the publication path. |
An invalid finding-action pair raises InvalidRepairActionError. A repair that
encounters a newer pointer, a stale fence, or a changed object fails and
preserves the visible revision for a fresh scan and reconciliation. Local recovery includes staged
object inventory and quarantine. Provider-backed recovery must use the
configured backend’s conditional-write and inventory capabilities; do not
apply local filesystem repair assumptions to an object store.
For a normal publication conflict, re-read LATEST and begin a new logical
attempt with the observed revision as expected_latest. If another writer
advanced the pointer first, CommitConflictError preserves the visible revision and
requires reconciliation rather than overwrite. The local-path or file://
manifest publication interface is provided by
ManifestRepository.
Repair patterns¶
Problem |
Action |
|---|---|
Missing shard files |
Re-submit the same generation command. |
Transient storage errors |
Re-submit after storage service recovers. |
Bad config |
Create a new dataset version; do not patch old shards. |
Failed annotation JSON |
Re-run the same annotation job; persistent failures remain in |
Prompt template bug |
Bump |
Verify¶
rfgen inspect gs://rf-fm-datasets-synth/wideband/v0.5.0 summary
rfgen inspect gs://rf-fm-datasets-synth/wideband/v0.5.0 errors
rfgen inspect gs://rf-fm-datasets-synth/wideband/v0.5.0 distribution
The run is repaired when missing shard count is zero and remaining errors are within the configured exclusion threshold.
Layer 43: retain disaster-recovery release evidence¶
Before approving a release after a recovery exercise, retain an immutable evidence bundle that identifies the dataset revision, recovery runbook revision, backend and region, exercise start and end times, and the operator responsible. Include the recovery-inventory finding/report hashes, the reconciliation-report hash, the restored manifest verification report, and the measured recovery-point and recovery-time outcomes. Record a failed or partial exercise as such; a successful repair of one run is not evidence that another backend, region, or dataset revision can be recovered.
Resume a failed release rehearsal¶
Do not resume a failed rehearsal from a partial RehearsalReportV1. Preserve
the report, its ordered causes, and every immutable evidence ID, then correct
the failed dependency or deadline condition and start a new rehearsal from the
same self-hashed RehearsalSpecV1. A failed step makes descendants SKIPPED;
ROLLBACK is still required, so inspect its cleanup status before re-running.
For GLOBAL_TIMEOUT, reduce the selected work or extend the caller deadline
under the approved release procedure. Do not edit the stored report or reuse
its PASS evidence. For a provider failure, repair the required provider
selector and retain the new PROCESS evidence. Wave 6 remains sealed only after
a new PASS report and a Layer 11-validated post-run REHEARSAL gate receipt.
Verify a published manifest¶
Layer 15 ships a programmatic, read-only verifier; it does not yet ship an
rfgen verify CLI command. Construct
ManifestVerifier
with a repository reader and choose the least invasive mode needed:
Need |
Mode |
Required capability |
Effect |
|---|---|---|---|
Check the manifest is readable and bound to its revision |
|
Repository reader |
Validates schema and self-hash; object work is explicitly skipped. |
Check every published object is present and unmodified |
|
Repository and object readers |
Streams each object and reports every missing or digest-mismatched key. |
Establish bounded regeneration evidence |
|
Repository, object, and fresh-process reproduction capabilities |
Adds deterministic recreation of 32 selected samples and exact/toleranced comparisons. |
Inspect the returned VerificationReportV1; a requested capability that is
unavailable is a FAIL, not a skip. Verification never advances LATEST,
writes objects, signs a statement, or repairs a manifest. Reconcile a failed
report by preserving its revision and creating a new run or using the local
compare-and-swap publication workflow above; do not patch published data in
place.
Troubleshoot¶
Symptom |
Fix |
|---|---|
Re-run starts from scratch |
Confirm the resolved config hash and storage path match the original run. |
Same shard fails repeatedly |
Inspect the shard error; it may be deterministic invalid config, not transient failure. |
Annotation overwrite risk |
Use a new |