Dataset layout#
data/generated/<dataset>/
|-- dataset_manifest.json
|-- LICENSE.md data licences and credits, rebuilt when a run ends
|-- index.parquet one row per complete radio map, rebuilt when a run ends
`-- scene_001_<label>/
|-- scene_info.json
|-- provenance.json
|-- source_buildings.json
|-- source_antennas.json only when the scene takes its transmitters from antennas
|-- scene.xml
|-- mesh/
|-- terrain/ only when the scene has an environment grid
|-- measurement_surfaces/
| |-- surface_<cell>m_rx_<height>m_agl.ply, .json, .npz
| `-- surface_<cell>m_rx_<height>m_agl_buildings.npz
|-- vox_slices/
|-- vox_depth/
|-- 2D_Building_Height_Map.npy
`-- radio_maps/
`-- radio_map_0001_<pattern>_<n>tx_<frequency>ghz/ (..._registered/ with antennas)
|-- radio_map_info.json
|-- arrays/
| `-- path_gain/path_gain.npz per-transmitter path gain, packed as uint16 dB
`-- images/ only with output.previews
|-- path_gain_db.png, path_gain_db_masked.png
|-- rss_db.png, rss_db_masked.png
|-- sinr.png, sinr_masked.png
`-- association.png
<cell> and <height> are the grid resolution and the receiver height in metres, written
without a decimal point when whole and otherwise with the shortest decimal that reads back as
the value, so two grids never share a file name.
Manifest#
dataset_manifest.json holds the dataset format (schema_version), the index_revision
(below), the validated request, the seed derivation, the software version, the
execution_contracts (the Sionna radio-map contract version and the RF material profile
version), the array conventions (below), the data licences (below), summary counts, one record
per scene, and a generation_runs list with the software version, status and the initial,
generated and failed radio map counts of each run. Neither the manifest nor the records hold
the times of a run. Once a scene’s assets are ready, its manifest record is the same as its
scene_info.json: specification, provenance, shared assets, a status and the counts
radio_maps: {planned, completed, failed}. status is pending (not reached in the current
run; the record may predate the scene’s scene_info.json), running, completed,
completed_with_errors or failed. Neither file lists radio maps. Each radio map’s expanded
parameters, seed and status are in its own radio_map_info.json, which is the authoritative
record. scene_info.json and every radio_map_info.json carry the same schema_version.
This page describes dataset format 2.
scene_info.json is written when the scene’s assets are ready, after each of its radio maps
and when the scene ends; the manifest at the same points and at run start and run end. When a
run stops on an error, the run and the scene whose radio maps were running are recorded as
failed, with completed counts taken from the radio map records. A pending record carries
the counts of the resume scan. The scan cannot count scenes whose transmitters come from
registered antennas; for those the previous run’s count is kept until the scene is reached.
The counts can therefore lag behind the radio map directories, for example after a killed run;
resume re-reads each radio map’s record and artifacts instead (Resume and extend).
Conventions#
The manifest’s conventions block states the row order, units, no-data value and data type
of the array products it lists, where each applies, and how RSS, SINR and the serving
transmitter are derived from the stored path gain. The values come from
openworld_radio_twin.conventions, the same module the product writers use, so each
product’s own metadata states the same values.
Key |
Contents |
|---|---|
|
Columns run west to east in every product; lengths are metres. |
|
|
|
Cells are |
|
|
|
RSS in W is |
|
The PNG previews, written only with |
|
Rows south to north. |
|
|
|
|
|
|
|
Rows south to north. Elevation is sampled on grid nodes and surface classes on cells, so the elevation arrays have one more row and column. The surface class ids are listed. |
Licences#
A scene’s provenance in scene_info.json and in its manifest record has a licences list
with one entry per data layer and source: layer (buildings, terrain, surfaces or
antennas), source, release, license, license_url and attribution. Overture layers
add one entry per upstream dataset that Overture names, such as OpenStreetMap, via Overture Maps, with via set to Overture Maps and the licence Overture names for it (an SPDX
identifier such as ODbL-1.0 is written as ODbL 1.0); these data reach the scene through
the Overture release, so the entry is a credit. OpenStreetMap and ESA WorldCover have their
own credit lines; the other upstream datasets are credited As listed at docs.overturemaps.org/attribution. Mapzen terrain has the terms Per upstream source for the tile region, the full upstream credit list and the tiles it used. Flat ground and layers
without data have no entry. The values are kept in source_buildings.json,
terrain/scene_info.json and source_antennas.json (Scene layout), so a resumed
run records the same entries (Resume and extend). A licence that is not recorded is an empty
string, in these entries and in the upstream datasets those files list.
The manifest’s data_licences is the union of the scenes’ entries, each once; Mapzen entries
that differ only in their tiles are merged. When a run ends, also when it stops on an error,
the generator rebuilds data_licences and writes LICENSE.md from the manifest’s scene
records. A scene whose run stopped before its record was written is left out until a later run
writes it. When any layer is under the ODbL 1.0, it opens with the ODbL notice for the
Overture or OpenStreetMap data and offers the dataset as a derivative database under the ODbL
1.0, with the radio arrays and images produced from it; otherwise it says that no single
licence covers the dataset. Scenes with a CC BY-SA 4.0 layer, such as OpenCelliD antennas, are
named and left out of the ODbL offer, because share-alike data cannot be offered under the
ODbL. Scenes with a layer whose licence is not recorded or is not known to allow
redistribution, such as a local file, are named with that layer and the reason; the upstream
datasets Overture names are not checked this way. The notice then lists each source with its
release, licence or terms and licence page, credit, Mapzen tiles, and the scenes that use it.
The licences a release may redistribute are listed in openworld_radio_twin.licences, and
packaging checks every scene against them (Packaging for release).
Scene directory#
Each scene directory is a complete compiled scene (Scene layout) plus the derived
assets that its radio map space needs: one measurement surface per requested cell
size and receiver height with its building rasters, voxel slices and depth at the requested
pitch. shared_assets.measurement_surfaces in scene_info.json lists each surface’s path,
requested cell size (resolution_m), receiver height, cell shape, height reference and
building_rasters path. A later run keeps a surface when its files match the requested grid
and the scene terrain, and keeps its building rasters when they equal the rasters computed
from the scene’s current buildings and terrain. Missing or mismatching files are written
through temporary files, so an interrupted run leaves no partial surface file. Its
provenance.json carries the same compilation block as a compiled scene, and the cache key
there is the one rt.load_scene uses for the same geometry inputs. A scene with an
environment grid (include_terrain=True, or flat ground with the itu material profile)
writes the terrain/ arrays and the provider warnings raised while fetching them; resume
reloads both without a new provider query, so the compilation block rewritten by a resumed
run keeps the same warnings. A scene of buildings on flat ground (include_terrain=False and
the uniform profile, Planning and generation) has no environment grid: no terrain/ directory,
concrete ground at height 0, and the other files of the tree above.
Radio map directory#
radio_map_info.json is mandatory. A completed radio map’s record holds:
Key |
Contents |
|---|---|
|
Dataset format, directory name, |
|
Path of the scene’s |
|
The expanded simulation request, with every transmitter’s position and parameters. |
|
The run that wrote the record: its one-based position in the manifest’s |
|
Scene name and |
|
The grid of the arrays: origin, the ENU centre of the first sample (the south-west cell), |
|
Per metric, the label, image unit, value range, display range and reduction name ( |
|
Array shape |
|
What the derived arrays need: |
|
Only with |
|
Solver details, including the noise bandwidth, temperature and thermal noise power, and for Sionna the radio-map contract version and the RF material profile version; and warnings. |
A running record holds the keys up to generation_run; a failed record adds error. A completed radio map stores one array: path_gain.npz, the path gain of every
transmitter with shape (n_tx, rows, columns), rows south to north, packed as above. RSS,
SINR and the serving transmitter are not stored; they follow from the path gain and the
radio block under the engine’s definition, within the rounding bounds given for
path_gain above. A near tie between two transmitters can therefore change the serving
transmitter.
With output.previews (Planning and generation), the radio map also writes PNG previews,
rendered from the engine’s in-memory float32 arrays. They are north-up, with a whole
number of pixels per cell recorded as pixels_per_solver_cell, transparent no-data and no
interpolation; the _masked variants hide the cells of the scene’s
receiver_in_building_mask for the same cell size and receiver height. association.png
is rendered from the serving-transmitter indices.
Metric |
Loaded unit |
Preview scale |
|---|---|---|
|
linear ratio |
dB |
|
W |
dBm |
|
linear ratio |
dB |
association |
transmitter index ( |
categorical |
Reading a dataset#
load_radio_map takes a radio map directory or its radio_map_info.json and returns a
RadioMap: the radio map directory, the record, the array grid as grid, the linear
float32 arrays path_gain, rss (W) and sinr with shape (n_tx, rows, columns), and
the int32 association with shape (rows, columns). path_gain_db, rss_dbm and
sinr_db give the same values in dB and dBm, -inf where the linear value is 0. The sums of
the SINR run in float64. The loader refuses a record of another dataset format, a record
that is not completed, that lacks the array_contract or that records another packing, a
stored array whose shape is not the recorded three-axis shape, and a radio block that
lacks a key, names an unknown association metric, or holds a transmit power that is
negative or not finite or a noise power that is not finite and positive.
import numpy as np
from pathlib import Path
from openworld_radio_twin.dataset import load_radio_map
scene = Path("data/generated/my-radio-dataset/scene_001_amsterdam")
radio_map = load_radio_map(scene / "radio_maps/radio_map_0001_sector_2tx_3p5ghz")
best_dbm = radio_map.rss_dbm.max(axis=0) # strongest transmitter per cell
serving = radio_map.association # -1 where no transmitter serves
grid = radio_map.grid # row 0 is the southern row
rows, columns = radio_map.path_gain.shape[1:]
north_m = grid.first_sample_north_m + grid.row_step_m * np.arange(rows)
east_m = grid.first_sample_east_m + grid.column_step_m * np.arange(columns)
buildings = np.load(scene / radio_map.record["radio"]["building_rasters"])
indoor = buildings["receiver_in_building_mask"] # same rows and columns
local_elevation = np.load(scene / "terrain/local_elevation_m.npy")
surface_classes = np.load(scene / "terrain/surface_classes.npy")
Without the package, the stored array unpacks with NumPy alone:
directory = scene / "radio_maps/radio_map_0001_sector_2tx_3p5ghz"
packed = np.load(directory / "arrays/path_gain/path_gain.npz")["path_gain"]
path_gain_db = np.where(packed == 0, -np.inf, packed * 0.01 - 500.0)
Elevation arrays are grid nodes and surface classes are cells, so their shapes differ by one
in each dimension. Terrain and radio arrays run south to north; flip them to align with the
north-up previews, and use the coordinates stored with each product for the alignment. For
the radio arrays these are array_grid_mapping; north_m and east_m above are the ENU
cell centres of the rows and columns. PyTorch reads these arrays as tensors on the
radio grid.
Finding radio maps#
index.parquet at the dataset root is a table with one row per complete radio map, by the
completeness check of resume (Resume and extend), in scene order and then radio map order. It is
built from the scene and radio map records and repeats a few scene values in each row, so
that radio maps can be selected without opening their records; the other scene details stay
in scene_info.json. Paths are relative to the dataset root and use /. A package has an
index of the same form (Packaging for release).
Column |
Type |
Contents |
|---|---|---|
|
string |
|
|
string |
Scene directory, radio map directory, scene label and the optional |
|
string |
In a package, the split of the scene and the path of its archive; null in a generated dataset |
|
double |
Scene centre in degrees and half-side of the scene square in metres |
|
string |
The resolved building source, such as |
|
string |
The terrain provider of the scene; |
|
string |
|
|
string |
Solver and the metric that picks the serving transmitter |
|
int64 |
Solver seed and ray samples per transmitter |
|
int32 |
Maximum number of interactions per path |
|
double |
Carrier frequency shared by the transmitters and noise bandwidth, both in whole hertz, and thermal noise power |
|
int32 |
Shape of the path-gain array |
|
double |
Requested cell size and the cell steps of the array grid |
|
double |
Receiver height above ground |
|
list of double |
Transmitter positions in the scene frame, as in |
|
list of double |
Per transmitter, in request order: height above ground, transmit power in W and dBm, azimuth and downtilt |
|
list of string |
Antenna pattern per transmitter |
|
string |
Paths of |
|
string |
Paths of the path-gain array, the measurement surface’s |
Only site, split and archive may be null. The Parquet footer records the dataset format
as owrt.schema_version, the owrt.index_revision it was built for, and the OpenWorld Radio
Twin and pyarrow versions that wrote it; the same inputs with the same versions give the
same bytes. The index of a package also records owrt.source_manifest_sha256, the SHA-256
checksum of the manifest it was packaged from.
The index is derived and never read by generation or resume. Every run advances the manifest’s
index_revision before it writes a radio map and rebuilds the index for that revision when it
ends, also when it stops on an error. While a run is in progress, and after a run that is
killed, the index is built for an earlier revision. read_index refuses such an index, and a
missing one, with the instruction to rebuild it once no run is active; owrt dataset index
(or build_index) rebuilds it from the records:
owrt dataset index data/generated/my-radio-dataset
The rebuild takes the generation lock exclusively, so it refuses while a run, a packaging job or another rebuild holds it, and it requires the version that created the dataset. It refuses a package, whose index is written by packaging.
read_index returns a pyarrow.Table. Selecting the radio maps of one site at 3.5 GHz with
two transmitters:
import pyarrow.compute as pc
from openworld_radio_twin.dataset import load_radio_map, read_index
root = "data/generated/my-radio-dataset"
index = read_index(root)
selected = index.filter(
(pc.field("site") == "amsterdam")
& (pc.field("frequency_hz") == 3.5e9)
& (pc.field("n_tx") == 2)
)
radio_maps = [load_radio_map(f"{root}/{path}") for path in selected.column("record").to_pylist()]
Any Parquet reader opens the file; these do not check the revision. With pandas:
import pandas as pd
index = pd.read_parquet("data/generated/my-radio-dataset/index.parquet")
selected = index.query("site == 'amsterdam' and frequency_hz == 3.5e9 and n_tx == 2")
With DuckDB, list columns take list functions:
SELECT radio_map_id, path_gain, list_max(tx_height_agl_m) AS highest_tx_m
FROM 'data/generated/my-radio-dataset/index.parquet'
WHERE site = 'amsterdam' AND frequency_hz = 3.5e9 AND n_tx = 2
ORDER BY radio_map_id;