Compute Environment

Warning

Pre-implementation. This page describes proposed contracts. Class signatures, parameter types, config field names, and behavior are subject to change before code lands. Once implementation exists, content here will be regenerated from docstrings or sourced from running tests.

Where to run the framework. The framework’s core has no cloud dependencies; concrete cloud backends are optional extras (see Reference / Cloud Backends). The sections below show worked examples for local development and for GCP. AWS and Azure are sketched at the end of the page; the framework treats all three clouds equally at the abstraction level.

Decision matrix

Pick by workload, not by habit:

Workload

Recommended environment

GPU?

Cost order

Develop configs, write a custom emitter, debug

Local laptop

No

Free

Quickstart (100 samples, AWGN)

Local laptop

No

Free

First Scene example (1K samples, TDL channels)

Local laptop

No

Free

Iterate on a custom emitter / channel + visualize

Notebook with optional GPU (Vertex AI Workbench, SageMaker Studio, Azure ML)

Optional

\(0.10 to \)1/hr

Generate _md (100K) preset, no RT

Local workstation or any Spark cluster

No

Free or ~$15

Generate _md with Sionna RT

GPU notebook or single GPU VM

Yes

~\(5 to \)20/hr

Generate _lg (1M)

Spark cluster (Managed Spark Serverless on GCP, EMR on AWS, Synapse on Azure, Databricks anywhere)

Optional

~$140

Generate _xl (10M+) with RT

Spark cluster + GPU pool

Yes

~$1000+

Phase 2 annotation (any size)

Batch annotation orchestrator (Vertex Batch on GCP, SageMaker Batch on AWS, Anthropic/OpenAI Batch APIs anywhere)

No

~$0.001/sample

HIL validation harness

Lab workstation with USRP/HackRF

No

Hardware-bound

Option 1: Local laptop (development)

For. Iteration on configs, custom emitters, debugging, small-scale verification.

Spec.

Resource

Minimum

Recommended

CPU

4 cores

8+ cores (M-series Mac, Ryzen 9, i9)

RAM

16 GB

32 GB

Disk

20 GB free

100 GB SSD

Python

3.11+

3.11

GPU

Not required

Optional; accelerates Sionna RT when available

Setup.

git clone https://github.com/superpose-labs/rf-data-generation.git
cd rf-data-generation
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[torchsig,sionna]" # Sionna CPU works; GPU strongly recommended for RT

rfgen --version

What works.

  • All comms emitters (TorchSig)

  • All radar / drone / IoT / ADS-B / cellular emitters (no GPU required for the synthesis itself)

  • AWGN, TorchSigImpairments, SionnaTDL, SionnaCDL (CPU mode)

  • Bbox + segmentation + per-emitter labels

  • Phase 2 annotation (drives LLM APIs over network)

  • Spark dry runs (validates configs without submitting)

What doesn’t.

  • Large Sionna RT scenes on laptops. CPU fallback works through Dr.Jit’s LLVM backend, but ray tracing is too slow for routine dataset generation; use a stochastic config such as channel=sionna_phy_tdl_c for development or a GPU VM for RT.

  • Real Spark execution (need GCP).

  • Bulk generation > ~1M samples (memory/time prohibitive; use Spark).

Option 2: Vertex AI Workbench (interactive notebook)

For. Interactive exploration, visualization, ad-hoc dataset analysis, prototyping with Sionna RT on a single GPU, demoing the framework.

Vertex AI Workbench gives you a managed JupyterLab environment with optional GPU, GCS-mounted, and persistent state.

Setup

gcloud auth login
gcloud config set project $GCP_PROJECT
gcloud services enable notebooks.googleapis.com aiplatform.googleapis.com

gcloud workbench instances create rfgen-dev \
    --location=us-central1-a \
    --machine-type=n1-standard-8 \
    --accelerator-type=NVIDIA_TESLA_T4 \
    --accelerator-core-count=1 \
    --vm-image-family=tf-latest-gpu \
    --boot-disk-size-gb=100 \
    --metadata=startup-script-url=gs://rfgen-startup/install.sh

The startup script (gs://rfgen-startup/install.sh):

#!/bin/bash
set -e
pip install --upgrade pip
pip install "rfgen[sionna,torchsig,openai-batch,anthropic-batch,gcp]"
echo "rfgen $(rfgen --version) installed"

Recommended machine types:

Machine

GPU

RAM

Per-hour

When

n2-standard-8

none

32 GB

~$0.39

Notebook + Phase 2 driving + non-RT scenes

n1-standard-8 + T4

NVIDIA T4 (16 GB)

30 GB

~$0.74

Sionna RT prototyping (small scenes)

n1-standard-8 + L4

NVIDIA L4 (24 GB)

30 GB

~$0.86

Sionna RT, larger scenes

n1-highmem-8 + V100

NVIDIA V100 (16 GB)

52 GB

~$2.84

Long-duration RT scenes, multi-RX

a2-highgpu-1g

NVIDIA A100 (40 GB)

85 GB

~$3.67

Production-ish RT batches

Always idle-stop. Workbench instances bill hourly even when you’re not using them. Configure auto-stop in the instance settings (the --idle-shutdown-timeout 3600 flag stops after 1 hour of inactivity).

# Manual stop
gcloud workbench instances stop rfgen-dev --location=us-central1-a

# Resume
gcloud workbench instances start rfgen-dev --location=us-central1-a

Use case: interactive RT scene exploration

# Inside the notebook
from rfgen.cli import generate
from rfgen.viz import spectrogram, overlay_bboxes
import matplotlib.pyplot as plt

# Generate 10 RT samples
generate(
    config_overrides=[
        "+preset=dense_urban_2_4ghz",
        "size=xs",
        "channel=sionna_rt_munich",
        "run.num_samples=10",
        "storage.path=/tmp/rt-test",
    ],
)

# Visualize
from rfgen.storage import open_store
with open_store("/tmp/rt-test") as store:
    sample = store.read_random()
spec = spectrogram(sample.iq, n_fft=1024)
fig, ax = plt.subplots(figsize=(14, 7))
ax.imshow(spec, aspect="auto", origin="lower", cmap="viridis")
overlay_bboxes(ax, sample.bboxes, sample.scene)
plt.show()

Cost example. 1 hour on n1-standard-8 + T4 ≈ \(0.74. A full afternoon of debugging on a T4 ≈ \)5–10. Use the GPU only when you need it.

Option 3: GCE GPU instance (ad-hoc batch with full control)

For. Batch generation of _md-scale RT datasets without Spark complexity. SSH-driven workflow. Custom Docker images. When you need root access on a GPU box.

Setup

gcloud compute instances create rfgen-gpu \
    --zone=us-central1-a \
    --machine-type=g2-standard-8 \
    --accelerator=type=nvidia-l4,count=1 \
    --image-family=common-cu123-debian-12 \
    --image-project=deeplearning-platform-release \
    --maintenance-policy=TERMINATE \
    --provisioning-model=SPOT \
    --boot-disk-size=200GB \
    --boot-disk-type=pd-ssd \
    --metadata=install-nvidia-driver=True

gcloud compute ssh rfgen-gpu --zone=us-central1-a --command "
    pip install rfgen[sionna,torchsig,openai-batch,anthropic-batch,gcp]
    nvidia-smi
"

Recommended machine types (us-central1, spot prices are ~70 % off):

Machine

GPU

RAM

On-demand $/hr

Spot $/hr

When

g2-standard-8

L4 (24 GB)

32 GB

~$0.86

~$0.26

Default RT generation

g2-standard-12

L4 (24 GB)

48 GB

~$1.05

~$0.32

Larger scenes / multi-RX

a2-highgpu-1g

A100 (40 GB)

85 GB

~$3.67

~$1.10

Heavy batch RT, multi-emitter

a3-highgpu-1g

H100 (80 GB)

234 GB

~$10.50

~$3.15

Largest RT scenes, fastest turn

Spot VMs

The Spark and GCE workflows both default to spot instances (preemptible, 60–90 % cheaper, can be terminated with 30 sec notice). Phase 1 generation is shard-idempotent; a spot termination loses at most one in-flight shard, which the orchestrator re-runs.

Cost example. Generating wideband_baseline_md (100K samples, RT) on a spot g2-standard-8:

  • Wall-clock: ~6 hours (RT-bound)

  • Cost: 6 × \(0.26 = **~\)1.50**

vs. on-demand: ~$5.20. Use spot for production batches.

# Stop when done (don't pay for idle)
gcloud compute instances stop rfgen-gpu --zone=us-central1-a

# Or delete entirely
gcloud compute instances delete rfgen-gpu --zone=us-central1-a

Use case: overnight RT batch

gcloud compute ssh rfgen-gpu --zone=us-central1-a --command "
    rfgen generate \
        +preset=dense_urban_2_4ghz \
        size=md \
        channel=sionna_rt_munich \
        storage.path=gs://rf-fm-datasets-synth/dev/dense_urban_md/v0.5.0 \
        > generation.log 2>&1 &
    echo \$! > rfgen.pid
    disown
"
# Job runs in background. Check status periodically:
gcloud compute ssh rfgen-gpu --zone=us-central1-a --command "tail -100 generation.log"

Option 4: Managed Spark serverless (production Phase 1)

For. All _lg and _xl preset variants. Datasets > 1M samples. Production-grade idempotency and autoscaling.

Already covered in detail in How-to / Run on Spark and Reference / CLI § rfgen generate. Quick recap:

# configs/orchestration/spark_serverless.yaml
backend: spark_serverless
spark:
  project_id: ${oc.env:GCP_PROJECT}
  region: us-central1
  service_account: rfgen-runner@${oc.env:GCP_PROJECT}.iam.gserviceaccount.com
  machine_type: n2-standard-8
  max_executors: 32
  image_uri: us-central1-docker.pkg.dev/${oc.env:GCP_PROJECT}/rfgen/spark:latest

rfgen generate +preset=dense_urban_2_4ghz size=lg orchestration=spark_serverless

GPU executor pools for RT scenes split CPU and GPU pools:

orchestration:
  spark:
    executor_pool:
      cpu:
        machine_type: n2-standard-8
        max_executors: 32
      gpu:
        machine_type: g2-standard-8
        max_executors: 8
        accelerator: nvidia-l4

The framework routes RT-channel shards to the GPU pool; comms-only shards stay on CPU.

Cost order of magnitude:

Preset

Wall-clock

Cost (spot)

wideband_baseline_md (100K, no RT)

~30 min

~$15

wideband_baseline_lg (1M, no RT)

~4 hrs

~$140

dense_urban_2_4ghz_lg (1M, RT)

~4 hrs

~$420

dense_urban_2_4ghz_xl (10M, RT)

~24 hrs

~$3500

Option 5: Vertex AI Batch Prediction (Phase 2 annotation)

For. Generating annotations for any-size dataset. Driven from anywhere (laptop, Workbench, Cloud Shell).

Already covered in How-to / Annotate an existing dataset and Reference / Annotation Templates § Vertex AI Batch Prediction integration. Quick recap:

rfgen annotate gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0 \
    annotator=full_suite \
    orchestration=vertex_batch

Vertex Batch handles:

  • Rate limiting per provider (Gemini / Anthropic / OpenAI)

  • Retry on transient failures

  • Output JSONL collection back to GCS

  • Cost capping (annotator.cost_cap_usd=N)

You can run this from a laptop with no compute load on your machine, just network I/O while polling.

Cost order of magnitude (1M samples, full 5-type annotation suite, Gemini gemini-3.1-flash-lite bulk + Sonnet 4.6 verifier @ 5 %): ~$80.

Option 6: HIL validation lab

For. Sim-to-real validation harness (Background / Validation Strategy v1.5 milestone). Replay synthetic IQ through real radios.

Hardware required:

Component

Recommended

TX SDR

Ettus USRP B210 (or HackRF One for budget)

RX SDR

Ettus USRP B210 (cabled loopback) + B210 with antenna (OTA)

Cabled loopback

SMA cable + 30 dB attenuator

Antennas (OTA)

VERT2450 or similar 2.4/5 GHz dipole

Workstation

x86 Linux, Ubuntu 22.04+, USB 3.0+

Software:

sudo apt install libuhd-dev uhd-host
sudo /usr/lib/uhd/utils/uhd_images_downloader.py
pip install rfgen[hil,torchsig]

uhd_find_devices  # verify USRP detection

Use case (replay one synthetic sample):

rfgen hil play \
    --input-store gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0 \
    --sample-id f9a3b7c2-... \
    --tx-device serial=B2102Y4 \
    --rx-device serial=B2102Y5 \
    --mode cabled_loopback \
    --output-capture ./out/hil/f9a3b7c2.sigmf

The captured SigMF file can then be re-labeled (rfgen hil relabel) and the framework computes per-class accuracy delta + channel-statistics KL (Reference / Metrics § Sim-to-real metrics).

AWS examples (stubs)

The framework’s AWS backends ship in pip install rfgen[aws]. The same workflow patterns apply; only the backend names differ.

Notebook for interactive work

Use SageMaker Studio Notebook with an ml.g5.xlarge (NVIDIA A10) or ml.g6.xlarge (NVIDIA L4) instance. Comparable cost and capability to GCP Vertex AI Workbench.

Single GPU VM for ad-hoc batch

EC2 spot instance, g6.2xlarge (L4) or g5.2xlarge (A10):

# Sketch; AWS-specific instance launch
aws ec2 run-instances \
    --instance-market-options MarketType=spot \
    --instance-type g6.2xlarge \
    --image-id ami-... \
    ...

Spark cluster: EMR

# configs/executor/emr.yaml
name: emr
parallelism: 1
params:
  cluster_id: ${oc.env:EMR_CLUSTER_ID}
  release_label: emr-7.0.0
  instance_type: r6i.2xlarge
  max_executors: 32

rfgen generate +preset=wideband_baseline_lg executor=emr

Annotation today: synchronous AnnotatorConfig

SageMaker Batch Transform annotation orchestration is not shipped in Layer 8. Do not add an annotation_orchestrator block to the root config yet; GenerationConfig rejects that key until a runtime batch-orchestrator consumer ships. Use the synchronous annotator block instead.

# configs/annotator/caption_with_verifier.yaml
enabled: true
types: [caption]
bulk_llm:
  provider: anthropic
  model: claude-haiku-3.5
verifier_llm:
  provider: anthropic
  model: claude-sonnet-4-6
verifier_subset_pct: 10.0

Run it with the normal generation config, or instantiate MetadataAnnotator.from_config(...) directly when annotating an existing store. AWS users who need remote batch inference before that surface ships should call the provider’s own batch API outside rfgen.

Azure examples (stubs)

The framework’s Azure backends ship in pip install rfgen[azure]. Feature-complete from v2.

Notebook

Azure Machine Learning Notebook with a Standard_NC4as_T4_v3 (T4 GPU) or Standard_NC24ads_A100_v4 (A100). Comparable to GCP Vertex AI Workbench.

Single GPU VM

az vm create --size Standard_NC4as_T4_v3 --priority Spot --eviction-policy Deallocate ...

Spark cluster: Synapse

# configs/executor/synapse.yaml
name: synapse
parallelism: 1
params:
  workspace_name: my-synapse-ws
  spark_pool: rfgen-pool
  node_size: Medium
  max_nodes: 32

Annotation today: synchronous AnnotatorConfig

Azure Batch / Azure ML annotation orchestration is not shipped in Layer 8. Do not add an annotation_orchestrator block to the root config yet; GenerationConfig rejects that key until a runtime batch-orchestrator consumer ships. Use the synchronous annotator block instead.

# configs/annotator/caption_with_verifier.yaml
enabled: true
types: [caption]
bulk_llm:
  provider: openai
  model: gpt-4o-mini
verifier_llm:
  provider: openai
  model: gpt-4o
verifier_subset_pct: 10.0

Cloud-agnostic alternatives

Two combinations work on any cloud (or no cloud) without committing to a single vendor:

Need

Cloud-agnostic backend

Distributed Spark

databricks (multi-cloud)

Distributed compute (no Spark)

dask or ray

Batch inference annotation

anthropic_batch or openai_batch

These pull no cloud SDKs; they hit each provider’s own batch APIs. Useful when GCP/AWS/Azure lock-in is undesirable, or when a workflow has to run inside a corporate network with restricted cloud access.

Routing your workflow

Concrete example: you want to develop a custom radar emitter, validate it on a small dataset, then generate 1M samples for downstream training.

1. Develop locally
   $ pip install -e ".[torchsig,test]"
   $ vim src/rfgen/emitters/radar/my_radar.py
   $ pytest tests/contract/test_base_emitter_contract.py -k my_radar

2. Visual sanity-check on Vertex AI Workbench (no GPU)
   - Open notebook
   - rfgen generate +my_radar_test ... → 100 samples
   - Plot spectrograms; verify bboxes overlay correctly

3. Small RT validation on a GCE GPU spot instance
   - gcloud compute instances create rfgen-gpu --provisioning-model=SPOT ...
   - rfgen generate +my_radar_test channel=sionna_rt size=xs run.num_samples=1000
   - Inspect, verify, terminate the instance

4. Production 1M run on Spark
   - rfgen generate +preset=my_radar_lg orchestration=spark_serverless
   - ~4 hours, ~$140

5. Phase 2 annotation
   - rfgen annotate gs://.../my_radar_lg ... → ~$80, ~2 hours wall-clock

Total cost for that workflow: development free + ~\(1 of Workbench + ~\)2 of GCE spot + ~\(220 production. Compares favorably to provisioning a permanent GPU box (~\)700/month idle).

GCP project bootstrap

A one-time setup for using any of the GCP options above.

# Set project + region
export GCP_PROJECT=my-rf-fm-project
gcloud config set project $GCP_PROJECT
gcloud config set compute/region us-central1

# Enable APIs
gcloud services enable \
    aiplatform.googleapis.com \
    dataproc.googleapis.com \
    compute.googleapis.com \
    notebooks.googleapis.com \
    storage.googleapis.com \
    artifactregistry.googleapis.com

# Create GCS buckets
gsutil mb -l us-central1 gs://rf-fm-datasets-synth
gsutil mb -l us-central1 gs://rfgen-startup
gsutil mb -l us-central1 gs://rfgen-ci-golden

# Service account for the framework
gcloud iam service-accounts create rfgen-runner \
    --description="Framework runner SA"

for role in \
    roles/dataproc.worker \
    roles/storage.admin \
    roles/aiplatform.user \
    roles/artifactregistry.reader; do
  gcloud projects add-iam-policy-binding $GCP_PROJECT \
      --member="serviceAccount:rfgen-runner@$GCP_PROJECT.iam.gserviceaccount.com" \
      --role=$role
done

# Push the Docker image (one-time; CI does this for tagged releases)
docker build -f docker/Dockerfile.spark -t rfgen-spark:0.5.0 .
gcloud artifacts repositories create rfgen \
    --repository-format=docker --location=us-central1
docker tag rfgen-spark:0.5.0 \
    us-central1-docker.pkg.dev/$GCP_PROJECT/rfgen/spark:0.5.0
docker push us-central1-docker.pkg.dev/$GCP_PROJECT/rfgen/spark:0.5.0

Quotas to request

For production-grade Spark / GPU usage, request quota increases ahead of time:

Quota

Default (us-central1)

Recommended

Dataproc Serverless Batches: max concurrent

5

20

CPUs (n2)

24

256

NVIDIA L4 GPUs

1

8

NVIDIA A100 GPUs

0

2–4

Vertex AI Batch Prediction concurrent jobs

4

10

Persistent Disk SSD GB

500

5000

Submit at https://console.cloud.google.com/iam-admin/quotas.

Cost summary

For a complete v1 run end-to-end (dense_urban_2_4ghz_lg, 1M samples, RT, full annotation suite):

Component

Cost

Phase 1 generation (Spark + GPU pool, spot)

~$420

Phase 2 annotation (Vertex Batch)

~$80

GCS storage (8 TB × 1 month)

~$160

Total

~$660

For a _md (100K) variant: ~$50–70 total.

For development-only (laptop + occasional Workbench): essentially free.

See Also