Resume and extend#

Dataset generation supports interruption recovery and deterministic extension. Generate 100 radio maps, raise the same scene’s radio_maps.count to 200 or add scenes after the existing ones later, and resume the existing dataset:

result = generate_dataset_sync(request, output_root="data/generated", resume=True)

The batch page exposes the same behaviour through Resume / extend, and the HTTP API through POST /api/batches?resume=true.

How resume decides what to run#

Resume reads each radio map’s record and arrays; the counts in the manifest can lag behind the radio map directories (Dataset layout). A radio map is complete when its radio_map_info.json has status completed and its path-gain array exists, with the PNG previews when output.previews is set; failed, interrupted, corrupt or incomplete radio maps are generated again. Because each radio map has an independent seed derived from the dataset seed and its scene and radio map indices, existing radio maps keep their parameters when the target count grows.

Compatibility checks#

Resume accepts only a dataset in the current dataset format and a request whose following parts are unchanged:

  • the dataset name and seed,

  • the definition of every existing scene, including its site, its solver settings and parameter ranges, and the resolved building and antenna sources with their versions,

  • the output.previews option,

  • the software version and, for Sionna radio maps, the radio-map contract version and the RF material profile version.

Previews are fixed for the lifetime of a dataset: a request that changes output.previews is refused with a message that says so. Per-scene radio map counts may stay the same or increase, and new scenes may follow the existing ones (below). A refused scene change names the scene and the fields that changed. A refused request leaves the dataset as it was. Persisted building, terrain and antenna inputs rebuild the runtime scene, so a new provider release is never mixed into an existing dataset.

A dataset in another dataset format, and Sionna data with another radio-map contract version or RF material profile version, are refused with a message that names the recorded version. Such a dataset is continued with the generator that created it, or generated again in a new output directory.

Each run appends a record to generation_runs in the manifest with its software version, status, and initial, generated and failed radio map counts, so the history of a dataset stays visible. Each radio map record names the run that wrote it in generation_run; the run records no times.

Growing a dataset#

A dataset grows by resuming it with a larger request. The request may:

  • raise a scene’s radio_maps.count: the existing radio maps keep their seeds, records and arrays, and the new ones are generated;

  • append scenes after the existing ones: an appended scene takes the next scene number, as in a new dataset generated from the whole request. Its measurement surfaces and building rasters are written with it, for any receiver height and grid.

Everything else about an existing scene is fixed, its site included: the site decides which scenes share a split when the dataset is packaged (Packaging for release), so a new site could move a scene into another split. Removing or reordering scenes, lowering a count and changing output.previews are refused as well. An appended scene is fixed from the first run that includes it, also when that run stops before reaching it, so try a new scene in a small separate dataset first.

When the run ends, also when it stops on an error, the manifest records the run’s status, and its data_licences, LICENSE.md and index.parquet are rebuilt from the records of the grown dataset. The error that stopped the run reaches the caller unchanged. Stopping a run with Ctrl+C lets the running solver step finish, records the run as failed with the error interrupted, and leaves the radio map in progress to the next run. A killed run cannot do this.

Package the grown dataset into a new directory, with the earlier package as --previous, so that the earlier scenes keep their split and only new scene groups are assigned (Packaging for release):

owrt dataset package data/generated/my-radio-dataset releases/my-radio-dataset-v2 \
    --previous releases/my-radio-dataset

The index after a stopped run#

Each run advances the manifest’s index_revision before it writes a radio map and rebuilds index.parquet when it ends, also when it stops on an error (Dataset layout). A run that is killed leaves the index of the previous revision, which read_index refuses, as it does while a run is in progress. Resume does not read the index; the next run rebuilds it, and so does

owrt dataset index data/generated/my-radio-dataset

This command rebuilds only the index. LICENSE.md can miss the sources of scenes the killed run reached first; the next run rebuilds it when it ends.

A resume whose request is refused, such as one with a changed seed, leaves the index and its revision as they were. A run stopped later by a check of a scene’s saved inputs ends as a failed run and rebuilds the index.