Annotation¶

Annotation adds text supervision to a dataset rfgen generate already wrote. It never rewrites I/Q, structured labels, or source metadata.

Start here¶

rfgen generate writes a Signal Dataset snapshot by default, and annotation reads that source by default. If you generated with stock settings, this is your whole configuration:

# annotation-config/caption.yaml, whose name is the --config-name
execution_mode: local
annotation_type: caption
template_id: caption.declared.v1
run_id: caption-v1
model: gemini-3.1-flash-lite
backend: local_concurrent
signal_dataset: {dataset_uri: /abs/path/rfgen-output}

uv pip install -e '.[gemini]'   # from the repository root
export GEMINI_API_KEY=...
rfgen annotate submit --config-dir ./annotation-config --config-name caption

Exit	Meaning
`0`	`complete`, every record the run attempted succeeded. A subset run’s `skipped` rows do not affect this.
`2`	`complete_with_errors`, some records failed; the rest are written
`1`	Stopped before any terminal result; a JSON `error_code` on stderr

Read the set back through the signal_dataset package, against the same root:

import signal_dataset as sd

annotations = sd.open("/abs/path/rfgen-output").annotations["caption-v1"]
print(len(annotations), dict(annotations.metadata["rfgen.row_counts"]))
# 6 {'success': 6, 'failed': 0, 'skipped': 0}

Three places report counts, in three vocabularies¶

Worth knowing before you write a check around any of them:

Where	Shape
A local `submit`’s `terminal_counts`	`succeeded`, `skipped_existing`, `skipped_outside_subset`, and one key per failure kind: `provider_error`, `annotation_schema_invalid`. There is no `failed` key; a run failed nothing only when every failure key is zero.
A `dataproc_serverless` `wait` result	flat `succeeded` and `failed`, and nothing else. `failures` is `null` on this route.
The published set’s `rfgen.row_counts`	`success`, `failed`, `skipped`, the Signal Dataset format’s own words

So succeeded and success are the same records under two names, and the submit output is the only one that makes you add the failure kinds up yourself. Filter a row on status == "success", never "succeeded".

Omitting source_kind is deliberate: it defaults to signal_dataset, so generating and then annotating needs no source selection in between. Follow Annotate an existing dataset for the full task.

The four annotation modes¶

Two settings decide a run, and they are independent. execution_mode says where the loop runs; backend says how the model is called. Every combination writes the same annotation set. They differ in what the run costs, how long it takes, and whether your shell stays busy.

#	Mode	`execution_mode`	`backend`	Use it for
1	Smoke test	`local`	`local_concurrent`	Ten records. Errors arrive as they happen.
2	Local dataset	`local`	`gemini_batch`	A dataset small enough to sit and wait for. Roughly half the price.
3	Managed, per record	`dataproc_serverless`	`local_concurrent`	A corpus you cannot host. Full price, shell free.
4	Managed, batched	`dataproc_serverless`	`gemini_batch`	Half price and shell free. The choice for a corpus.

All four have a recorded live run. See Where annotation runs for the qualification detail and what it does not cover.

1. Smoke test¶

The Start here configuration above, unchanged. Nothing to add.

2. Local dataset¶

One line on top of it:

backend: gemini_batch

3 and 4. On Dataproc Serverless¶

Moving the loop is not one line: a batch cannot read your disk or your shell. Seven keys across four blocks, on top of the Start here configuration. Three change values you already set; four are new.

Two prerequisites first, both of which fail after you have paid for a batch if you skip them.

The snapshot has to be in a bucket. A batch cannot read your disk, and a snapshot cannot be relocated with gsutil cp: its pointers pin object generations, so a copy breaks the root. Generate to the bucket directly by setting the storage path to a gs:// URI:

# In your generation config, not the annotation config.
storage: {backend: signal_dataset, path: gs://my-bucket/rfgen-output}

The same key works as a command-line override, which is the shorter route when you are pointing an existing generation config at a bucket:

rfgen generate --config-dir ./generation-config --config-name config \
  --num-samples 24 storage.path=gs://my-bucket/rfgen-output

rfgen generate prints nothing on success. Confirm the write with gcloud storage ls gs://my-bucket/rfgen-output, which should list root.json beside the objects/, manifests/, and snapshots/ prefixes.

Annotation writes into the snapshot root, and there is no key to redirect it. Scope bucket permissions accordingly: the identity that submits, and the Dataproc runtime service account for a remote run, both need write access to <dataset_uri>. A run creates exactly these, all create-only:

Path	Written by	When
`<dataset_uri>/annotations/sets/<run_id>/`	the publisher	on publish
`<dataset_uri>/annotations/catalog.json` and `catalogs/<sha>.json`	the publisher	on publish
`<dataset_uri>/rfgen/annotations/<type>/<template>/<run_id>/work-<digest>/`	`submit`	before the batch starts

The last one holds the staged config.json and PySpark wrapper, and sits outside annotations/ because the Signal Dataset format reserves that namespace. dataproc.deps_bucket is somewhere else entirely: Dataproc’s own staging area, required by the schema, and normal for rfgen to leave empty.

A snapshot already on your disk cannot be moved to a bucket at all. There is no relocation command, and copying the tree yields CorruptDatasetError: GCS snapshot reference is missing a generation. Generate again with the storage path set to the bucket. Reuse the same configuration, seed, and --shard-size and the regenerated snapshot carries the same sample_id values as the local one; change the shard size and the ids change with it.

uv pip install -e '.[gemini,gcs]'   # gcs extra, and gcloud on PATH
gcloud auth application-default login
rfgen generate --config-dir ./generation-config --config-name my-scene

The API key has to travel as a secret. Your shell’s GEMINI_API_KEY does not reach a Dataproc worker.

Look before you create. On a shared project the secret usually exists already under a name nobody wrote down, and creating a second one leaves two keys to rotate:

gcloud secrets list --project my-project

If one is there, use its name in inference.api_key_secret below and skip to the IAM binding. gemini-api-key here is an example name, not a convention the tooling expects: api_key_secret takes whatever resource name you point it at.

printf %s "$GEMINI_API_KEY" | gcloud secrets create gemini-api-key \
  --project my-project --replication-policy=automatic --data-file=-
# Rotating later: add a version instead of recreating.
#   printf %s "$NEW_KEY" | gcloud secrets versions add gemini-api-key --data-file=-
gcloud secrets add-iam-policy-binding gemini-api-key --project my-project \
  --member "serviceAccount:$(gcloud projects describe my-project \
    --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
  --role roles/secretmanager.secretAccessor

That member is the project’s Dataproc runtime service account, which is what the job runs as. Then the configuration:

execution_mode: dataproc_serverless
backend: gemini_batch                        # or local_concurrent for mode 3
signal_dataset:
  dataset_uri: gs://my-bucket/rfgen-output   # a batch cannot read your disk
inference:
  # Your shell's GEMINI_API_KEY does not travel. Omitting this is not refused
  # at validation; it fails every row of a batch you have paid for.
  api_key_secret: projects/my-project/secrets/gemini-api-key/versions/latest
dataproc:
  project: my-project
  region: us-central1              # a Dataproc region; `global` is refused here
  deps_bucket: gs://my-deps/staging

submit returns a durable handle and exits. wait advances one polling cycle, so call it in a loop:

rfgen annotate submit --config-dir ./annotation-config --config-name caption | tee submit.json
HANDLE=$(jq -c .handle submit.json)
while :; do
  rfgen annotate wait --handle-json "$HANDLE" > wait.json; rc=$?
  jq -e 'has("result")' wait.json >/dev/null && break
  # rc 1 means the batch finished without publishing: there is no terminal
  # result and no amount of further polling will produce one.
  [ "$rc" -eq 1 ] && { echo "batch published nothing; see the driver log" >&2; break; }
  sleep 15
done
# The terminal payload differs by route, so read it defensively rather than
# assuming one shape.
jq -re '.result.status // .result.result.status // empty' wait.json || echo "no result"

Local annotation

For commands, recovery, and qualification limits, see Generate, then annotate.