Remote Spark Execution

Use Dataproc Serverless when the remote, immutable GCS Signal Dataset boundary fits the dataset workflow. Set executor.name: dataproc_serverless, validate the job, then run rfgen generate; it submits an asynchronous PySpark batch rather than generating records locally. Submission returns after the batch is created. Use the printed provider status and cancellation commands to manage the run.

Before configuring the job, set PROJECT_ID to the target GCP project and run this submitter preflight:

gcloud auth login
gcloud config set project "$PROJECT_ID"
gcloud services enable dataproc.googleapis.com --project="$PROJECT_ID"
gcloud auth list

Runtime image

You do not choose an image. Remote generation runs against a maintained image pinned by digest in rfgen.remote_image.DEFAULT_DATAPROC_IMAGE, and a configuration that leaves executor.dataproc.image_uri unset submits against that pin.

Two access requirements remain, and they fall on different identities. The worker service account named in executor.dataproc.service_account needs Artifact Registry Reader on the maintained repository so it can pull the image at run time; that repository is private and lives in the rf-foundation-models project, so for any other project this is a cross-project grant a maintainer there has to make. The submitter needs that role only to list or resolve tags, which the default path never does.

Set image_uri only to run against a different image, for instance one built from a change that has not merged. See Runtime image for how to select or build one, and for when a rebuild is genuinely required.

The submitter login is distinct from the Dataproc worker service_account. Set executor.dataproc.project and region explicitly rather than relying on a default. Enabling the API requires the appropriate project permission; ask a project administrator if it is unavailable.

Add this supported boundary to a materialized baseline, using your pre-created GCS locations and worker service account:

storage:
  backend: signal_dataset
  path: gs://my-output-bucket/datasets/example
executor:
  name: dataproc_serverless
  dataproc:
    project: YOUR_PROJECT_ID
    region: YOUR_REGION
    staging_uri: gs://my-staging-bucket/rfgen
    service_account: rfgen-worker@example-project.iam.gserviceaccount.com

image_uri is deliberately absent: the batch runs against the pinned default. Adding it with a placeholder digest would pass rfgen validate, because the field is only checked for being a non-blank string, and then fail inside Dataproc on an image pull.

rfgen validate --config-dir .
rfgen generate --config-dir .

Submission prints a batch ID plus exact status and cancellation commands; it does not wait for shard publication. To observe or cancel a submitted batch, use that ID with the configured project and region:

gcloud dataproc batches describe BATCH_ID --region=REGION --project=PROJECT
gcloud dataproc batches cancel BATCH_ID --region=REGION --project=PROJECT --quiet

After completion, the configured GCS output is a root-last Signal Dataset snapshot. Open the root by URI; do not address worker shards directly.

For the full supported setup and submission recipe, see Run a remote Dataproc job.

Remote execution boundary

Remote execution has these boundaries:

• rfgen stages the resolved configuration and canonical shard specifications.

• Workers publish independent immutable Signal Dataset shards to GCS. The coordinator publishes the ordinal index after collecting attempt receipts.

• storage.backend must be signal_dataset (or a compatible native dataset store plugin) and the output URI must use gs://. Before submission, provide a staging URI, worker service account, active gcloud login, and Dataproc and staging/output IAM permissions, including Artifact Registry Reader for the worker on the maintained image repository. rfgen creates neither those resources nor IAM bindings; workers write shards after submission.

Availability and qualification

Dataproc Serverless is available with manual live end-to-end evidence, but it is not production-qualified and its guarded live test is excluded from normal CI. Use it only when its remote, immutable GCS Signal Dataset boundary fits the dataset workflow; otherwise begin with Local execution.

Dataproc Serverless is authoritative for prerequisites, configuration fields, runtime-image requirements, and lifecycle commands. Storage layouts defines the published-root contract.