Generate a dense urban dataset¶

Run the dense_urban_2_4ghz preset end to end: 1 M samples of dense 2.4 GHz ISM with full annotations, distributed on GCP Managed Spark.

Warning

Pre-implementation. Commands target v1+. Steps that interact with GCP resources will incur cost when implementation lands.

What you’ll get¶

1 M samples × 100 ms × 200 Msps wideband recordings (samples.zarr/ ≈ 8 TB)
~50 emitters per scene drawn from comms / IoT (Wi-Fi, BLE, Zigbee, LoRa)
Sionna-RT ray-traced multipath over the Munich downtown OSM scene
Joint labels (bbox + segmentation + per-emitter + scene metadata)
Full annotation suite (caption + Q&A + reasoning + scene_report + contrastive pairs) via Gemini Flash bulk + Sonnet 4.6 verifier subset

Prerequisites¶

pip install "rfgen[sionna,torchsig,annotator,spark]"

# GCP auth
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json
export GCP_PROJECT=my-rf-fm-project

You’ll need:

A GCS bucket (e.g., gs://rf-fm-datasets-synth)
Vertex AI API enabled in your project
Managed Spark serverless API enabled
Sionna’s Munich OSM scene downloaded to gs://your-bucket/sionna-scenes/munich/

1. Inspect the preset¶

rfgen show-preset dense_urban_2_4ghz

Confirm parameters look right (especially run.num_samples, storage.path, orchestration.spark.project_id).

2. Validate first (free, ~5 seconds)¶

rfgen validate +preset=dense_urban_2_4ghz \
    storage.path=gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0 \
    orchestration.spark.project_id=$GCP_PROJECT

Output:

Note

Illustrative. Specific values below (config hash, cost estimates, timestamps, audit metrics) show the target output shape, not measured behavior. Real numbers come from real runs once v0 lands.

✓ emitter_zoo (4 families, 22 classes)
✓ channel: sionna_rt (scene: munich.xml, 100 paths max)
✓ scene: dense (sample_rate=200e6, bw=100e6, density=poisson(50))
✓ label: joint (bbox + seg + metadata)
✓ annotator: full_suite (bulk: gemini-3.1-flash-lite, verifier: claude-sonnet-4-6 @ 5%)
✓ storage: zarr_gcs
✓ orchestration: spark_serverless
Resolved config hash: sha256:<digest>
Estimated cost: ~\$<compute> compute + ~\$<llm> LLM = ~\$<total> total

3. Phase 1: generate IQ + structured labels¶

rfgen generate +preset=dense_urban_2_4ghz \
    storage.path=gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0 \
    orchestration.spark.project_id=$GCP_PROJECT

Output:

Submitting Spark batch job: rfgen-dense-urban-<timestamp>
  region: us-central1
  template: managed-spark-rfgen-v0.5.0
  executors: autoscale 8–32 × n2-standard-8
  
Streaming job logs ... (Ctrl-C to detach; job continues)

[<t0>] Worker pool warming up ...
[<t1>] Shard 0 / 1000 (sample 0 / 1_000_000) started
...
[<tN>] All shards complete
[<tN+1>] Cross-modality consistency: <pct> (<excluded> samples excluded)
[<tN+2>] Statistics audit: PASS
[<tN+3>] Phase 1 complete. Wrote <kept> of 1000 shards.

gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0/
  manifest.json
  samples.zarr/        <size>
  shards/              <size>
  stats.json
  errors.jsonl         (<excluded> records; see audit report)

Wall-clock: ~4 hours on autoscaling Spark with 8–32 n2-standard-8 executors. Cost: ~$.

4. Phase 2: annotate¶

rfgen annotate gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0 \
    annotator=full_suite \
    orchestration=vertex_batch

Output:

Submitting Vertex AI Batch Prediction job: rfgen-annotate-<timestamp>
  bulk model: gemini-3.1-flash-lite
  verifier: claude-sonnet-4-6 @ 5% subset
  total samples: 1_000_000
  total annotation calls: 5_000_000 (5 types × bulk) + 250_000 (verifier)

[<t0>] Templating Phase-1 records → JSONL ...
[<t1>] Submitting bulk batch (gemini-3.1-flash-lite, 5 sub-jobs) ...
[<t2>] Bulk done. PAES estimate (verifier subset): <value>
[<t3>] Submitting verifier batch (claude-sonnet-4-6, 5 sub-jobs) ...
[<t4>] Verifier done. PAES: <value>. Hallucination Count: <value> / sample.
[<t5>] Appending annotations to Zarr ...
[<t6>] Phase 2 complete.

Wall-clock: ~1.5 hours via Vertex Batch (rate-limited but parallelized across sub-jobs). Cost: ~$.

5. Verify¶

rfgen inspect gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0 distribution

Audit checks (target shape; specific values come from a real run):

✓ Class balance: every class within 50–200% of expected weight
✓ SNR distribution: log-uniform within 0.05 KL-divergence
✓ Density: realized within tolerance of target
✓ Delay-spread RMS: within tolerance of target
✓ Annotation PAES: ≥ 0.80 per class (per-template ≥ 0.85, dataset ≥ 0.90)
✓ Hallucination Count: ≤ 0.5 / sample on verifier subset

If any audit fails, the dataset version is not promoted to the published prefix.

6. Train¶

The dataset is now ready to consume in any of three ways:

# WebDataset (recommended for GPU pretraining)
from torch.utils.data import DataLoader
import webdataset as wds

ds = (
    wds.WebDataset("gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0/shards/shard-{000000..000999}.tar")
    .decode()
    .to_tuple("iq.npy", "bbox.json", "caption.txt")
)
loader = DataLoader(ds, batch_size=8, num_workers=4)

# Zarr (recommended for analysis / random access)
from rfgen.storage import open_store

with open_store("gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0") as store:
    for sample in store.iter_random(n=1000, seed=0):
        ...

# HDF5 (recommended for TorchSig-trained model interop)
from torchsig.utils.dataset import StaticTorchSigDataset

ds = StaticTorchSigDataset(root="gs://rf-fm-datasets-synth/dense_urban_2_4ghz/v0.5.0/hdf5/")

Generate a dense urban dataset¶

What you’ll get¶

Prerequisites¶

1. Inspect the preset¶

2. Validate first (free, ~5 seconds)¶

3. Phase 1: generate IQ + structured labels¶

4. Phase 2: annotate¶

5. Verify¶

6. Train¶

Tips¶