Signal Dataset annotation on Dataproc Serverless (2026-08-13)¶

The first real Serverless batch of the signal_dataset annotation route: a six-record gs:// snapshot annotated by a Dataproc batch running the image pinned on 2026-08-13, publishing a dense annotation set beside the snapshot.

It took three submissions to get there, and the two that failed are the reason this record exists. A note at the end records a backend that was qualified and then removed before merge, and the three provider defects that run found. Everything below was found by running the route, not by reading it.

What was run¶

Project rf-foundation-models, region us-central1, snapshot gs://rfgen-e2e-409806686028-8a95942a34de/annotation-sds-dataproc/dataset.sds (6 records, snapshot_id: snap-1). Image us-central1-docker.pkg.dev/rf-foundation-models/rfgen/rfgen@sha256:d455e286…, published as 0.0.6-20260813-annotation-sds-r2, which was the maintained pin on that date. The pin has since moved to sha256:2d104c1e…, so this run does not qualify the current image. The 2026-08-14 gemini_batch run does: see Signal Dataset annotation with backend: gemini_batch. backend: local_concurrent, execution_mode: dataproc_serverless, provider gemini, credential resolved from Secret Manager.

Run	`run_id`	Model	Batch	Result
1	`dataproc-live-v1`	`gemini-2.5-flash-lite`	SUCCEEDED	0 succeeded, 6 failed (`vertex_unknown`)
2	`dataproc-live-v2`	`gemini-2.5-flash-lite`	SUCCEEDED	0 succeeded, 6 failed (`provider_error`)
3	`dataproc-live-v3`	`gemini-3.1-flash-lite`	SUCCEEDED	6 succeeded, 0 failed

Run 3’s wait exited 0 with status: complete, succeeded: 6, failed: 0, and a re-submission of the same configuration adopted the same batch id rather than starting a second one, so no work was re-billed.

Provider prose from run 3, verbatim:

The scene contains a single strong signal that is continuous and occupies the entire channel. This emitter is consistent with qpsk modulation.

Note

“Occupies the entire channel” is a fact-projection artifact. The caption facts normalize frequency to the union of the record’s bounding boxes, so a single-emitter scene reads as full-band regardless of its real occupancy. This run evidences that the route completed, not that the sentence is true.

What the failures found¶

Run 1: the credential resolver was never called. The Secret Manager resolver existed, was unit-tested, and validated its configuration field, but the runner still read only the environment: an edit had been applied to the module and not to its one call site. A Dataproc driver cannot read the submitting machine’s environment, so the batch reached Gemini with no credential and failed every row. The driver log’s silence was the evidence: the resolver’s own log line was simply absent.

Run 2: the model was retired. With the credential resolving (the log line now present), every call returned 404 NOT_FOUND: This model models/gemini-2.5-flash-lite is no longer available. It is still listed by models.list, which is why listing is not a substitute for calling it. gemini-3.1-flash-lite works. A published retirement date is a lower bound, not a guarantee: this model was withdrawn two months before its published date. Probe by calling the exact model you intend to use.

Both runs published. A dense set with six failed rows, each carrying its detail_status, is what made the diagnosis possible: the batch reported complete_with_errors and exited 0, because a published set is a finished batch whatever its rows say.

What this qualifies¶

The batch runs the annotation loop, rather than re-entering the submit path.
The staged configuration and PySpark wrapper are written under <dataset_uri>/rfgen/annotations/<type>/<template>/<run_id>/work-<digest>/ and read back by the driver at their pinned generations. The observed layout matched the documented tree.
An API key reaches the driver from Secret Manager, under the batch’s own identity, with no key in the configuration, the batch resource, or any log.
submit → handle → wait reports succeeded/failed from the published set’s rfgen.row_counts, without reading a row.
Re-submitting an unchanged configuration adopts the running batch.

Boundaries¶

Six records, one model, one region, one account. This is a correctness qualification, not a throughput or cost measurement.
backend: gemini_batch under dataproc_serverless was not run here. It was qualified separately on 2026-08-14. See signal-dataset-gemini-batch-20260814.md.
Resume from a partially staged batch was not exercised: no run was killed mid-flight. What was shown is adoption of a live batch and the idempotent replay of a published set.
A batch runs the image’s rfgen. This qualifies the pin recorded above; an older pin runs whatever it contains.
The snapshot and its annotation sets were deleted after these results were read, so the URIs above no longer resolve.

Withdrawn: Vertex Batch over a snapshot¶

This record carried an addendum for backend: vertex_snapshot_batch, a synchronous Vertex batch backend qualified live at 6/6 on 2026-08-13 with execution_mode: local.

The backend was removed before merge. It matched gemini_batch on price and throughput and differed only in platform and credential, so it was a third way to do one thing. gemini_batch reaches the same batch pricing without a second platform, a bucket for staged requests, or a vertex block.

Three defects that run found are worth keeping, because they are properties of the provider rather than of the deleted code, and the next person to reach for Vertex Batch will meet all three:

google.genai’s JobState is a str Enum with no __str__ override, so str(state) is "JobState.JOB_STATE_SUCCEEDED" and matches no terminal state. A finished job polls to its deadline.
Vertex does not write to the destination it is given. A job given .../output wrote .../output/prediction-model-<ISO8601>/predictions.jsonl.
A GCP label value must begin with a lowercase letter, and Vertex echoes a label back with a different type than it was sent: {"k": 1} for a value sent as "1".