Run a Remote Dataproc Job

Use this path only when the dataset should be generated remotely as immutable GCS Signal Dataset snapshot. It is separate from the local dataset workflow, and rfgen generate returns after submitting a Dataproc Serverless batch.

Read Remote Spark execution for the compute-mode boundary, or return to Generate and Configure to choose between this remote path and the default local dataset path.

Before you submit

The local rfgen installation supplies the CLI only. Remote execution also needs a pre-existing GCS staging URI, GCS output URI, Dataproc worker service account, and active gcloud login. You do not need to supply an image: the batch runs against a maintained image pinned by digest, and the worker service account needs Artifact Registry Reader on that repository to pull it. Follow the Dataproc Serverless prerequisites before editing a configuration.

Submit and observe the job

Start from a materialized baseline, then replace the executor, storage, and run blocks using the complete remote configuration and lifecycle example. The required boundary is executor.name: dataproc_serverless, storage.backend: signal_dataset, and a gs:// storage path.

If the configuration selects a node from a use-case package, the pinned image does not contain it. Build that package into a wheel (uv build --wheel <use-case-directory> -o dist) and list the local path under executor.dataproc.extra_packages; it travels with the submission and needs no image rebuild. Installing it into the submitting environment too (uv pip install -e <use-case-directory>, or the same built wheel) is still required: rfgen generate compiles the whole graph locally before it submits anything, so the package has to be importable on the submitting machine independently of whatever extra_packages ships to the worker. See Shipping plugin packages.

A configuration that already targets a gs:// root — a shipped use-case configuration, for instance — needs only an executor block added; leave its storage and run blocks alone rather than replacing them with the example’s.

Remote publication is create-only, like local publication: the first successful run claims the output root permanently, and a batch that fails partway leaves objects that must be deleted before the root can be used again. Point storage.path at a root you intend to claim, and treat the submission as the release rather than a trial.

Run rfgen validate before rfgen generate. Submission prints the batch ID and exact gcloud status and cancellation commands. After every expected shard succeeds, the coordinator publishes the root. Open that root through SignalDatasetStore.

This path is manually validated and available, but it is not one of the production-qualified local Golden Paths.