Scene compile performance#
Loading buildings and compiling a scene are the slowest steps of load_scene, the explorer
and dataset generation. This page records what the compile path costs, how it is organised,
and where the time goes. All numbers come from one shared node with two AMD EPYC 7282 CPUs
(32 cores, 64 hardware threads) running OpenWorld Radio Twin 0.3.0, Python 3.11, NumPy 2.4,
Shapely 2.1 and pyarrow 25; the network figures also depend on the remote services at the
time of the run.
Summary#
Step |
Time |
|---|---|
Compile Bergen, 3,507 footprint buildings, r = 750 m (offline) |
6.5 s |
Compile Amsterdam, 2,282 3DBAG LoD2.2 shells, r = 500 m (offline) |
13.2 s |
Compile Los Angeles, 1,115 footprint buildings, r = 450 m (offline) |
2.7 s |
1 m measurement surface, Bergen (2.25 million cells) |
16.7 s |
Voxels at 3 m, Amsterdam shells |
2.8 s |
Provider queries, Bergen (Overture buildings, Mapzen terrain, Overture surfaces) |
10.2 s |
Provider queries, Los Angeles (Overture, USGS 3DEP, Overture surfaces) |
7.9 s |
Provider queries, Amsterdam (3DBAG API, Mapzen, Overture surfaces) |
56 s |
Cold |
18.4 s |
Cold |
69 s |
Explorer read-back of the cached Amsterdam scene (load and |
5.0 s |
Offline figures are medians of three runs of scripts/benchmarks/benchmark_scene_compile.py;
network figures are medians of the runs listed under
Network measurements.
How the compile path is organised#
Compile step (simulation/scene_builder.py, simulation/native_geometry.py):
Terrain and measurement-surface meshes are built with array indexing, and the triangle areas come from one
numpy.linalg.normcall over all triangles.The footing of a building tests all candidate terrain triangles with one vectorized Shapely
intersectscall.Footprints are sampled at cell centres with one
contains_xycall over the cells of their bounding box. The height map, the voxel products, the radio building mask and the building rasters use this sampler (building_columns).Ray-shell intersections are evaluated for all (triangle, grid point) pairs of a building at once, in chunks of at most two million pairs. Columns are paired in one pass; the few columns with crossings closer than the tolerance keep the sequential rule. Shell components are found by label propagation.
provenance.jsonandsource_buildings.jsonare written with the native JSON encoder, one record per line, so the files stay readable. Placement records are built without deep copies, and the scene XML is indented withElementTree.PLY files are written from NumPy buffers.
Native shells are split into roof and wall faces with a boolean mask; footprint roofs are triangulated once and oriented by their signed area.
Acquisition (compiler.py, providers/):
Buildings and the environment grid are queried at the same time, and the first failure is raised.
providers/overture_reader.pykeeps one S3 client per process and the STAC index of each release in memory and on disk (OWRT_OVERTURE_INDEX_ROOT), retries the index download briefly when the server throttles, and projects each read to the columns the providers decode. Objects are selected through the index only: when it cannot be fetched, the Overture query fails, buildings fall back to OpenStreetMap, and the surface query fails.The Overture building and building-part objects, and the four surface themes, are read concurrently and assembled in a fixed order.
The USGS 3DEP export and catalog lookup run together.
The 3DBAG provider requests 100 items per page, the service maximum. When the service reports
numberMatched, the remaining pages up to the building limit are fetched byoffset, four at a time, and decoded in offset order; the last page’snextlink is then followed. Decoding a CityJSON face orients all of its triangles in one array operation.
Service (compiler.py, scenes.py, main.py, providers/three_d_bag.py):
Decoding 3DBAG pages, building a scene, and, for the scene endpoints, reading a compiled scene back and building its response run in worker threads, so the service keeps answering other requests while a large view request is decoded.
Read-back (scene_builder.load_scene_assets, scenes.py, main.py, models.py):
Building mesh triangle counts come from the placement records the compiler writes; the PLY headers are read only for caches without these records.
The explorer and the simulation response build native display meshes from the placed buildings of the scene.
A cache hit in
compile_cached_scenevalidates the scene and compares its inputs without loading it.BuildingGeometryvalidates triangle indices with NumPy.
Equivalence tests#
tests/test_scene_builder_vectorized.pycompares each vectorized function with a sequential reference algorithm on random inputs: grid mesh, cell selection, PLY bytes, terrain footing, ray-shell intervals including the near-duplicate rule, and triangle orientation.tests/test_cityjson_vectorized.pydoes the same for the triangulation of CityJSON faces.tests/test_jsonio.py,tests/test_compiler_acquisition.pyandtests/test_overture_reader.pycover the row writer, concurrent acquisition and its failures, index caching, retries and column projection.
Reproducing the measurements#
conda activate owrt
# offline compile, derived assets and read-back, from the records of cached scenes
python scripts/benchmarks/benchmark_scene_compile.py .cache/scenes/scene_* --repeats 3
# provider queries only (network)
python scripts/benchmarks/benchmark_acquisition.py --site bergen --repeats 3
# cold load_scene into an empty cache (network + compile)
python scripts/benchmarks/benchmark_end_to_end.py --site amsterdam --repeats 2
Offline results by scene (median seconds):
Scene |
compile |
load assets |
1 m surface |
voxels 3 m |
read-back |
|---|---|---|---|---|---|
Los Angeles |
2.65 |
0.21 |
3.43 |
0.43 |
0.22 |
Bergen |
6.52 |
0.59 |
16.71 |
1.38 |
1.00 |
Amsterdam 3DBAG |
13.15 |
1.92 |
7.60 |
2.78 |
5.01 |
“read-back” is load_compiled_scene plus scene_response, the two calls behind the
explorer’s scene endpoint.
Network measurements#
Runs one after another from the same node. “Provider queries” is _acquire_inputs with a
fresh HTTP client and provider each time; “cold compile” is compile_cached_scene into an
empty cache.
Measurement |
Runs (s) |
Median (s) |
|---|---|---|
Provider queries, Bergen |
12.5, 10.2, 9.4 |
10.2 |
Provider queries, Los Angeles |
15.5, 6.8, 7.9 |
7.9 |
Provider queries, Amsterdam (3DBAG) |
57.4, 54.8 |
56.1 |
Cold compile, Bergen |
19.8, 17.1 |
18.4 |
Cold compile, Amsterdam (3DBAG) |
71.2, 66.9 |
69.1 |
The Bergen and Los Angeles scenes take the global route: Overture buildings, Overture surface classes, and Mapzen or USGS terrain. Amsterdam takes the 3DBAG route: its buildings arrive in 49 pages of 100 items, four pages in flight at a time, and decoding their LoD2.2 faces takes most of that time.
The STAC index server (stac.overturemaps.org) rate-limits bursts of requests. The reader
fetches the index once per release, keeps it on disk and retries briefly on a 429.
Where the time goes#
Amsterdam’s compile spends about three seconds in
localize_buildings(mostly the ground projection of each shell and the footing test), three seconds in the height map (column_intervals), and about three and a half seconds encoding and reading back the 18 MBprovenance.jsonand the 22 MBsource_buildings.jsonof native geometry. The compiler readsprovenance.jsonback to add the compilation record and again to validate the scene. Decoding these JSON files is also most of the explorer’s read-back.Every building becomes two PLY files, as the scene layout requires; writing the 7,019 files of the Bergen scene takes about one second.
The 1 m measurement surface of Bergen spends about 12 s building its 4.5 million triangles with their areas and cell centres, and about 3.5 s compressing its arrays.
The provider queries are bound by the remote services and, for 3DBAG, by decoding. The 3DBAG API serves a page in about 0.3 s; decoding the roughly 200,000 LoD2.2 faces of the Amsterdam scene takes about 50 s, most of it in their triangulation (
providers/cityjson.py).sionna.rt.load_sceneon the compiled Bergen scene takes 6 to 7 s withmerge_shapes=Trueand about 3 s without, with the CPU (LLVM) variant; that time is Mitsuba’s scene loading.