Dataset layout#

data/generated/<dataset>/
|-- dataset_manifest.json
|-- LICENSE.md                     data licences and credits, rebuilt when a run ends
|-- index.parquet                  one row per complete radio map, rebuilt when a run ends
`-- scene_001_<label>/
    |-- scene_info.json
    |-- provenance.json
    |-- source_buildings.json
    |-- source_antennas.json       only when the scene takes its transmitters from antennas
    |-- scene.xml
    |-- mesh/
    |-- terrain/                   only when the scene has an environment grid
    |-- measurement_surfaces/
    |   |-- surface_<cell>m_rx_<height>m_agl.ply, .json, .npz
    |   `-- surface_<cell>m_rx_<height>m_agl_buildings.npz
    |-- vox_slices/
    |-- vox_depth/
    |-- 2D_Building_Height_Map.npy
    `-- radio_maps/
        `-- radio_map_0001_<pattern>_<n>tx_<frequency>ghz/   (..._registered/ with antennas)
            |-- radio_map_info.json
            |-- arrays/
            |   `-- path_gain/path_gain.npz  per-transmitter path gain, packed as uint16 dB
            `-- images/                      only with output.previews
                |-- path_gain_db.png, path_gain_db_masked.png
                |-- rss_db.png, rss_db_masked.png
                |-- sinr.png, sinr_masked.png
                `-- association.png

<cell> and <height> are the grid resolution and the receiver height in metres, written without a decimal point when whole and otherwise with the shortest decimal that reads back as the value, so two grids never share a file name.

Manifest#

dataset_manifest.json holds the dataset format (schema_version), the index_revision (below), the validated request, the seed derivation, the software version, the execution_contracts (the Sionna radio-map contract version and the RF material profile version), the array conventions (below), the data licences (below), summary counts, one record per scene, and a generation_runs list with the software version, status and the initial, generated and failed radio map counts of each run. Neither the manifest nor the records hold the times of a run. Once a scene’s assets are ready, its manifest record is the same as its scene_info.json: specification, provenance, shared assets, a status and the counts radio_maps: {planned, completed, failed}. status is pending (not reached in the current run; the record may predate the scene’s scene_info.json), running, completed, completed_with_errors or failed. Neither file lists radio maps. Each radio map’s expanded parameters, seed and status are in its own radio_map_info.json, which is the authoritative record. scene_info.json and every radio_map_info.json carry the same schema_version. This page describes dataset format 2.

scene_info.json is written when the scene’s assets are ready, after each of its radio maps and when the scene ends; the manifest at the same points and at run start and run end. When a run stops on an error, the run and the scene whose radio maps were running are recorded as failed, with completed counts taken from the radio map records. A pending record carries the counts of the resume scan. The scan cannot count scenes whose transmitters come from registered antennas; for those the previous run’s count is kept until the scene is reached. The counts can therefore lag behind the radio map directories, for example after a killed run; resume re-reads each radio map’s record and artifacts instead (Resume and extend).

Conventions#

The manifest’s conventions block states the row order, units, no-data value and data type of the array products it lists, where each applies, and how RSS, SINR and the serving transmitter are derived from the stored path gain. The values come from openworld_radio_twin.conventions, the same module the product writers use, so each product’s own metadata states the same values.

Key

Contents

column_order, horizontal_unit

Columns run west to east in every product; lengths are metres.

height_reference

value is terrain AGL; applies_to names the transmitter altitude_m and receiver_height_m, which are both above ground.

cell_size

Cells are 2 * radius_m / ceil(2 * radius_m / resolution_m) wide, so the cell size can be smaller than the requested resolution. The absolute column_step_m and row_step_m recorded with each grid are authoritative.

path_gain

path_gain.npz, key path_gain: little-endian uint16, axes (transmitter, south_to_north_row, west_to_east_column), with CF-style packing attributes units (dB), scale_factor (0.01), add_offset (-500), _FillValue (0) and valid_range (1 to 65535). A positive linear gain is stored as round((10 * log10(gain) + 500) / 0.01), rounding half to even, so the stored dB value is at most 0.005 dB off the engine’s. RSS shares this bound; SINR can be off by up to twice it, because signal and interference are rounded independently. 0 means that the transmitter contributes nothing to the cell. The range covers -499.99 to 155.35 dB; a gain outside it fails the radio map instead of being clipped. The float32 linear gain that load_radio_map returns keeps the 0.005 dB bound down to -379.3 dB, the smallest normal float32; below it float32 holds fewer digits, and stored values of -451.55 dB and below load as 0.

derived

RSS in W is tx_power_w[i] * path_gain[i]; SINR is rss[i] / (sum over j != i of rss[j] + noise_power_w); the serving transmitter (int32, axes (south_to_north_row, west_to_east_column)) is the 0-based index into the request’s transmitters of the largest value under association_metric, the first on a tie, and -1 where no transmitter has a positive value. The inputs are the record’s radio block; load_radio_map computes them (below).

images

The PNG previews, written only with output.previews and never packaged, are north-up (the arrays flipped), in dB or dBm per metric, transparent where there is no data.

measurement_surfaces

Rows south to north.

building_rasters

<surface>_buildings.npz next to each measurement surface, on the cell centres of that surface and so of its radio arrays, rows south to north: footprint_mask (bool, the cell centre lies in the vertical projection of a building), roof_height_agl_m (float32 roof height above ground in metres, as in 2D_Building_Height_Map.npy, NaN where there is no building) and receiver_in_building_mask (bool, the receiver point lies inside a building).

height_map

2D_Building_Height_Map.npy: float32 roof height above ground in metres at each cell centre, rows north to south, 0 where there is no building. Its pixel size is recorded as height_map.pixel_size_m in vox_slices/scene_info.json.

voxel_depth

float32 metres, rows north to south, NaN where no voxel is occupied: the highest solid voxel level at each cell centre. Cells are 2 * radius_m / ceil(2 * radius_m / voxel_pitch_m) wide; the grid block that vox_slices/scene_info.json and vox_depth/scene_info.json both carry records them, with the vertical rule (Scene layout).

terrain

Rows south to north. Elevation is sampled on grid nodes and surface classes on cells, so the elevation arrays have one more row and column. The surface class ids are listed.

Licences#

A scene’s provenance in scene_info.json and in its manifest record has a licences list with one entry per data layer and source: layer (buildings, terrain, surfaces or antennas), source, release, license, license_url and attribution. Overture layers add one entry per upstream dataset that Overture names, such as OpenStreetMap, via Overture Maps, with via set to Overture Maps and the licence Overture names for it (an SPDX identifier such as ODbL-1.0 is written as ODbL 1.0); these data reach the scene through the Overture release, so the entry is a credit. OpenStreetMap and ESA WorldCover have their own credit lines; the other upstream datasets are credited As listed at docs.overturemaps.org/attribution. Mapzen terrain has the terms Per upstream source for the tile region, the full upstream credit list and the tiles it used. Flat ground and layers without data have no entry. The values are kept in source_buildings.json, terrain/scene_info.json and source_antennas.json (Scene layout), so a resumed run records the same entries (Resume and extend). A licence that is not recorded is an empty string, in these entries and in the upstream datasets those files list.

The manifest’s data_licences is the union of the scenes’ entries, each once; Mapzen entries that differ only in their tiles are merged. When a run ends, also when it stops on an error, the generator rebuilds data_licences and writes LICENSE.md from the manifest’s scene records. A scene whose run stopped before its record was written is left out until a later run writes it. When any layer is under the ODbL 1.0, it opens with the ODbL notice for the Overture or OpenStreetMap data and offers the dataset as a derivative database under the ODbL 1.0, with the radio arrays and images produced from it; otherwise it says that no single licence covers the dataset. Scenes with a CC BY-SA 4.0 layer, such as OpenCelliD antennas, are named and left out of the ODbL offer, because share-alike data cannot be offered under the ODbL. Scenes with a layer whose licence is not recorded or is not known to allow redistribution, such as a local file, are named with that layer and the reason; the upstream datasets Overture names are not checked this way. The notice then lists each source with its release, licence or terms and licence page, credit, Mapzen tiles, and the scenes that use it. The licences a release may redistribute are listed in openworld_radio_twin.licences, and packaging checks every scene against them (Packaging for release).

Scene directory#

Each scene directory is a complete compiled scene (Scene layout) plus the derived assets that its radio map space needs: one measurement surface per requested cell size and receiver height with its building rasters, voxel slices and depth at the requested pitch. shared_assets.measurement_surfaces in scene_info.json lists each surface’s path, requested cell size (resolution_m), receiver height, cell shape, height reference and building_rasters path. A later run keeps a surface when its files match the requested grid and the scene terrain, and keeps its building rasters when they equal the rasters computed from the scene’s current buildings and terrain. Missing or mismatching files are written through temporary files, so an interrupted run leaves no partial surface file. Its provenance.json carries the same compilation block as a compiled scene, and the cache key there is the one rt.load_scene uses for the same geometry inputs. A scene with an environment grid (include_terrain=True, or flat ground with the itu material profile) writes the terrain/ arrays and the provider warnings raised while fetching them; resume reloads both without a new provider query, so the compilation block rewritten by a resumed run keeps the same warnings. A scene of buildings on flat ground (include_terrain=False and the uniform profile, Planning and generation) has no environment grid: no terrain/ directory, concrete ground at height 0, and the other files of the tree above.

Radio map directory#

radio_map_info.json is mandatory. A completed radio map’s record holds:

Key

Contents

schema_version, name, status, seed

Dataset format, directory name, completed, and the solver seed.

scene

Path of the scene’s scene_info.json, which records the provenance of buildings, terrain and surfaces.

request

The expanded simulation request, with every transmitter’s position and parameters.

generation_run

The run that wrote the record: its one-based position in the manifest’s generation_runs.

data_provenance

Scene name and shared_geometry path, building provider, exported building count, the requested and resolved building source with its implementation version (and its configuration identity as text, for a source that has one) when buildings are included, and the antenna provider, source, release, licence, attribution and count when the scene takes transmitters from antennas.

array_grid_mapping

The grid of the arrays: origin, the ENU centre of the first sample (the south-west cell), column_step_m, row_step_m (positive: rows run north), sample layout, WGS84 corners, horizontal datum and local axis order.

radio_layers

Per metric, the label, image unit, value range, display range and reduction name (best_transmitter_per_cell: the strongest transmitter per cell), taken from the engine’s own arrays.

array_contract

Array shape (n_tx, rows, columns) and axis order, and under path_gain the path, NPZ key, dtype, byte_order and packing attributes of the stored array.

radio

What the derived arrays need: tx_power_w, the transmit power in W of each transmitter in request order, 10 ** ((power_dbm - 30) / 10) computed in float64 (Sionna applies it in float32); noise_power_w, the engine’s thermal noise power; tx_positions_enu_m, each transmitter’s east, north and up position in the scene frame where the engine placed it (Sionna at the terrain elevation plus the height above ground, the preview at that height above flat ground); association_metric; and the scene-relative paths of the measurement_surface and its building_rasters.

image_products

Only with output.previews: image row order, source grid and rendered size, pixels per cell, spatial resampling, no-data and building-mask rendering, value source, and per metric its unit, colour map and display range. In a package the previews this block describes are left out.

engine_details, warnings

Solver details, including the noise bandwidth, temperature and thermal noise power, and for Sionna the radio-map contract version and the RF material profile version; and warnings.

A running record holds the keys up to generation_run; a failed record adds error. A completed radio map stores one array: path_gain.npz, the path gain of every transmitter with shape (n_tx, rows, columns), rows south to north, packed as above. RSS, SINR and the serving transmitter are not stored; they follow from the path gain and the radio block under the engine’s definition, within the rounding bounds given for path_gain above. A near tie between two transmitters can therefore change the serving transmitter.

With output.previews (Planning and generation), the radio map also writes PNG previews, rendered from the engine’s in-memory float32 arrays. They are north-up, with a whole number of pixels per cell recorded as pixels_per_solver_cell, transparent no-data and no interpolation; the _masked variants hide the cells of the scene’s receiver_in_building_mask for the same cell size and receiver height. association.png is rendered from the serving-transmitter indices.

Metric

Loaded unit

Preview scale

path_gain

linear ratio

dB

rss

W

dBm

sinr

linear ratio

dB

association

transmitter index (int32)

categorical

Reading a dataset#

load_radio_map takes a radio map directory or its radio_map_info.json and returns a RadioMap: the radio map directory, the record, the array grid as grid, the linear float32 arrays path_gain, rss (W) and sinr with shape (n_tx, rows, columns), and the int32 association with shape (rows, columns). path_gain_db, rss_dbm and sinr_db give the same values in dB and dBm, -inf where the linear value is 0. The sums of the SINR run in float64. The loader refuses a record of another dataset format, a record that is not completed, that lacks the array_contract or that records another packing, a stored array whose shape is not the recorded three-axis shape, and a radio block that lacks a key, names an unknown association metric, or holds a transmit power that is negative or not finite or a noise power that is not finite and positive.

import numpy as np
from pathlib import Path
from openworld_radio_twin.dataset import load_radio_map

scene = Path("data/generated/my-radio-dataset/scene_001_amsterdam")
radio_map = load_radio_map(scene / "radio_maps/radio_map_0001_sector_2tx_3p5ghz")
best_dbm = radio_map.rss_dbm.max(axis=0)                     # strongest transmitter per cell
serving = radio_map.association                              # -1 where no transmitter serves

grid = radio_map.grid                                        # row 0 is the southern row
rows, columns = radio_map.path_gain.shape[1:]
north_m = grid.first_sample_north_m + grid.row_step_m * np.arange(rows)
east_m = grid.first_sample_east_m + grid.column_step_m * np.arange(columns)

buildings = np.load(scene / radio_map.record["radio"]["building_rasters"])
indoor = buildings["receiver_in_building_mask"]              # same rows and columns

local_elevation = np.load(scene / "terrain/local_elevation_m.npy")
surface_classes = np.load(scene / "terrain/surface_classes.npy")

Without the package, the stored array unpacks with NumPy alone:

directory = scene / "radio_maps/radio_map_0001_sector_2tx_3p5ghz"
packed = np.load(directory / "arrays/path_gain/path_gain.npz")["path_gain"]
path_gain_db = np.where(packed == 0, -np.inf, packed * 0.01 - 500.0)

Elevation arrays are grid nodes and surface classes are cells, so their shapes differ by one in each dimension. Terrain and radio arrays run south to north; flip them to align with the north-up previews, and use the coordinates stored with each product for the alignment. For the radio arrays these are array_grid_mapping; north_m and east_m above are the ENU cell centres of the rows and columns. PyTorch reads these arrays as tensors on the radio grid.

Finding radio maps#

index.parquet at the dataset root is a table with one row per complete radio map, by the completeness check of resume (Resume and extend), in scene order and then radio map order. It is built from the scene and radio map records and repeats a few scene values in each row, so that radio maps can be selected without opening their records; the other scene details stay in scene_info.json. Paths are relative to the dataset root and use /. A package has an index of the same form (Packaging for release).

Column

Type

Contents

radio_map_id

string

<scene>/<name>, unique in the dataset

scene, name, label, site

string

Scene directory, radio map directory, scene label and the optional site (null when not set)

split, archive

string

In a package, the split of the scene and the path of its archive; null in a generated dataset

latitude, longitude, radius_m

double

Scene centre in degrees and half-side of the scene square in metres

building_source

string

The resolved building source, such as 3dbag or global; none when the scene’s building inputs are disabled. A scene whose source returned no buildings keeps that source; data_provenance.building_count in radio_map_info.json gives the count

terrain_provider

string

The terrain provider of the scene; disabled without an environment grid

material_profile

string

itu or uniform; uniform with terrain_provider disabled and a building_source other than none marks buildings on flat ground (Planning and generation)

engine, association_metric

string

Solver and the metric that picks the serving transmitter

seed, samples_per_tx

int64

Solver seed and ray samples per transmitter

max_depth

int32

Maximum number of interactions per path

frequency_hz, bandwidth_hz, noise_power_w

double

Carrier frequency shared by the transmitters and noise bandwidth, both in whole hertz, and thermal noise power

n_tx, rows, columns

int32

Shape of the path-gain array

resolution_m, row_step_m, column_step_m

double

Requested cell size and the cell steps of the array grid

receiver_height_m

double

Receiver height above ground

tx_east_m, tx_north_m, tx_up_m

list of double

Transmitter positions in the scene frame, as in radio.tx_positions_enu_m

tx_height_agl_m, tx_power_w, tx_power_dbm, tx_azimuth_deg, tx_downtilt_deg

list of double

Per transmitter, in request order: height above ground, transmit power in W and dBm, azimuth and downtilt

antenna_pattern

list of string

Antenna pattern per transmitter

scene_record, record

string

Paths of scene_info.json and radio_map_info.json

path_gain, measurement_surface, building_rasters

string

Paths of the path-gain array, the measurement surface’s .ply (its .npz and .json sit next to it with the same stem) and its building rasters

Only site, split and archive may be null. The Parquet footer records the dataset format as owrt.schema_version, the owrt.index_revision it was built for, and the OpenWorld Radio Twin and pyarrow versions that wrote it; the same inputs with the same versions give the same bytes. The index of a package also records owrt.source_manifest_sha256, the SHA-256 checksum of the manifest it was packaged from.

The index is derived and never read by generation or resume. Every run advances the manifest’s index_revision before it writes a radio map and rebuilds the index for that revision when it ends, also when it stops on an error. While a run is in progress, and after a run that is killed, the index is built for an earlier revision. read_index refuses such an index, and a missing one, with the instruction to rebuild it once no run is active; owrt dataset index (or build_index) rebuilds it from the records:

owrt dataset index data/generated/my-radio-dataset

The rebuild takes the generation lock exclusively, so it refuses while a run, a packaging job or another rebuild holds it, and it requires the version that created the dataset. It refuses a package, whose index is written by packaging.

read_index returns a pyarrow.Table. Selecting the radio maps of one site at 3.5 GHz with two transmitters:

import pyarrow.compute as pc
from openworld_radio_twin.dataset import load_radio_map, read_index

root = "data/generated/my-radio-dataset"
index = read_index(root)
selected = index.filter(
    (pc.field("site") == "amsterdam")
    & (pc.field("frequency_hz") == 3.5e9)
    & (pc.field("n_tx") == 2)
)
radio_maps = [load_radio_map(f"{root}/{path}") for path in selected.column("record").to_pylist()]

Any Parquet reader opens the file; these do not check the revision. With pandas:

import pandas as pd

index = pd.read_parquet("data/generated/my-radio-dataset/index.parquet")
selected = index.query("site == 'amsterdam' and frequency_hz == 3.5e9 and n_tx == 2")

With DuckDB, list columns take list functions:

SELECT radio_map_id, path_gain, list_max(tx_height_agl_m) AS highest_tx_m
FROM 'data/generated/my-radio-dataset/index.parquet'
WHERE site = 'amsterdam' AND frequency_hz = 3.5e9 AND n_tx = 2
ORDER BY radio_map_id;