Packaging for release#

package_dataset turns a generated dataset into a release: one zstd-compressed tar archive per scene, train, validation and test splits by scene group, the licence notice and checksums. The command line runs the same function.

from openworld_radio_twin.dataset import package_dataset

package_dataset(
    "data/generated/my-radio-dataset",
    "releases/my-radio-dataset",
    fractions={"train": 0.8, "validation": 0.1, "test": 0.1},
    seed=0,
)
owrt dataset package data/generated/my-radio-dataset releases/my-radio-dataset \
    --split train=0.8,validation=0.1,test=0.1 --seed 0

The package directory must not exist or must be an empty directory, and must lie outside the dataset. Each package is a new snapshot; a dataset that has grown is packaged again into a new directory. The dataset is read under its generation lock, so it cannot be packaged while a run writes it or owrt dataset index rebuilds its index, and a generation or resume run started during packaging is refused with the message that another run, or a packaging or indexing job, holds the lock. As for resume, the dataset must have been generated with the running version, because the plan that names the radio maps may change between versions. The fractions and seed shown are the defaults.

Contents#

releases/my-radio-dataset/
|-- scenes/
|   `-- scene_001_<label>.tar.zst
|-- dataset_manifest.json
|-- LICENSE.md
|-- splits.json
|-- index.parquet
|-- README.md
`-- SHA256SUMS

File

Contents

scenes/<scene>.tar.zst

The scene directory (Dataset layout) with its packaged radio maps, without their PNG previews, as a zstd-compressed tar. A record’s image_products block describes previews that are not packaged.

dataset_manifest.json

The dataset’s manifest, unchanged. It describes the whole generated dataset, including scenes and radio maps left out; LICENSE.md, splits.json and index.parquet describe the package. Its data_licences may name layers of scenes left out; LICENSE.md lists the licences of the package. Its conventions block states the row order, units, no-data values and data types of the arrays.

LICENSE.md

The licence notice the generator writes (Dataset layout), built from the manifest records of the packaged scenes. When every scene is packaged, it is the text the generator writes from the same manifest.

splits.json

The fractions, the seed, the groups of scenes with their id and sites, the ids of the groups that kept the split of a previous package under kept_from_previous, and the split of each scene under assignment.

index.parquet

One row per packaged radio map, with the columns of the dataset index (Dataset layout), rebuilt for the packaged radio maps. split holds the split of the scene and archive the path of its archive. The paths of records and arrays resolve in the package directory once the archives are extracted there (below). The footer records the manifest’s index_revision and the manifest’s SHA-256 checksum as owrt.source_manifest_sha256; read_index reads the index from the package directory. owrt dataset index refuses a package; packaging the dataset again rebuilds its index.

README.md

The layout, the index revision and manifest checksum of the snapshot, the groups, scenes and radio maps per split, the compressed size of each scene archive, what was left out and why, and the NumPy, zstandard, zstd library and pyarrow versions that wrote the package.

SHA256SUMS

The SHA-256 checksum of every other file, in the format sha256sum -c SHA256SUMS checks.

SHA256SUMS is written last and moved into place whole, so a package without it is incomplete.

Extracting#

Each archive is extracted once, into the package directory. In that directory:

sha256sum -c SHA256SUMS
for archive in scenes/*.tar.zst; do tar --zstd -xf "$archive"; done

tar --zstd needs GNU tar 1.31 or later with the zstd program, or libarchive’s bsdtar. With an older GNU tar, tar --use-compress-program=zstd -xf (short form -I zstd) does the same. Python can extract the archives with the zstandard package, which OpenWorld Radio Twin depends on:

import tarfile
from pathlib import Path

import zstandard

for path in sorted(Path("scenes").glob("*.tar.zst")):
    with path.open("rb") as handle, zstandard.ZstdDecompressor().stream_reader(handle) as reader:
        with tarfile.open(fileobj=reader, mode="r|") as archive:
            archive.extractall(".", filter="data")

The filter argument of extractall needs Python 3.10.12, 3.11.4 or later. Each archive yields <scene>/ next to index.parquet, and the paths in the index resolve in the package directory, as they do in a generated dataset. load_radio_map(package / row["record"]) then loads a radio map of the index, and RadioMapDataset reads the package directory (PyTorch).

What is packaged#

A radio map is packaged when its radio_map_info.json has status completed and its path-gain array exists, as resume checks (Resume and extend); its PNG previews are not needed, because they are never packaged. Failed, interrupted and incomplete radio maps, and planned ones not generated yet, stay out of the archive and of index.parquet; the manifest’s counts still include the failed ones. A scene without a complete radio map is left out. The archives also leave out the PNG previews of the radio maps, hidden files and directories, such as locks and temporary directories, and files ending in .tmp.

Splits#

Radio maps of one scene share its geometry, so the split is made by scene groups, never by radio map. Two scenes are in one group when their squares (centre and radius_m) overlap, or when they carry the same site. The optional site field of a scene (Planning and generation) joins scenes that do not overlap, such as distant parts of one city. It is fixed once the scene exists (Resume and extend), so growing a dataset never moves an earlier scene out of its group.

Groups are formed over all scenes of the dataset, also those left out of the package, so a scene left out still joins the scenes it overlaps. A group’s id is the name of its earliest scene, packaged or not. Groups follow the scene order of the request, which is the order of the scene names up to 999 scenes, and are shuffled with NumPy’s PCG64 from seed. The number of groups per split follows the fractions by the largest remainder method. When there are at least as many groups as splits with a positive fraction, each of those splits gets at least one group; with fewer groups, README.md names the splits left empty. splits.json is the authoritative record of the assignment.

Keeping the splits of an earlier package#

When a dataset grows and is packaged again, previous (--previous) takes the earlier package directory or its splits.json, so that scenes keep their split:

owrt dataset package data/generated/my-radio-dataset releases/my-radio-dataset-v2 \
    --previous releases/my-radio-dataset

A group with scenes of the earlier package keeps their split and is listed under kept_from_previous. The groups without such a scene are shuffled with seed and fill what the kept groups leave of each split’s count for all groups of the package, divided by the largest remainder method. The splits therefore stay near the fractions as the dataset grows. A split whose kept groups already reach its count gets no new group, and a split can stay empty when the kept groups of other splits exceed their counts; README.md then names the empty splits. A new scene that joins groups of different earlier splits is refused with an error that names the scenes and their splits, because its geometry would be shared between splits. A scene of the earlier package that is not a scene of the dataset, or an earlier split that is not among the fractions, is refused as well.

Licences#

Before anything is written, each scene’s licences list in the manifest is checked with openworld_radio_twin.licences. A scene is refused when a layer has no recorded licence or a licence that is not known to allow redistribution, such as a local file or the ACMA register, or when a CC BY-SA 4.0 layer and an ODbL 1.0 layer are in the same scene, such as OpenCelliD antennas beside Overture buildings. A scene record without a licences list is refused as well. Mapzen terrain passes with its credit list. The error names each scene and layer. With skip_unreleasable=True (--skip-unreleasable) these scenes are left out, and README.md lists them with the reason.

Determinism#

The archives are GNU tar streams whose members are sorted by name and carry modification time 0, owner and group 0 with empty names, and mode 0644 for files and 0755 for directories. Each stream is compressed with zstd level 3 in a single thread, into one frame with a content checksum. The index is written by pyarrow, whose footer records its version, and NumPy shuffles the new split groups. Packaging the same dataset again with the same fractions, seed, previous package and options, and the same NumPy, zstandard, zstd library and pyarrow versions, therefore gives the same bytes in every file, including SHA256SUMS. The package’s README.md records these versions. A dataset generated again differs, because each radio_map_info.json records when its radio map ran.