Everything is a dataset

Scenes, ground truth, prompts, labels and scores are all the same kind of thing: a named, versioned, immutable set of rows whose ancestry is recorded. Treating them that way is what turns “which labels came from which prompt against which truth” into a query rather than an archaeology project.

The concept

A dataset is (name, version). Resolving it yields a set of rows and the lineage that produced them.

name        labelrefinery.labels.site_seed1.round2
version     0.3.0
kind        labels
rows        2 377
parents     labelrefinery.labels.site_seed1.round1 @ 0.2.0
            labelrefinery.prompts.construction @ 1.1.0
produced_by improve_offboard_model  (git 6f49927, params {...})
snapshot    4612849022315…          ← the Iceberg snapshot this pins

Three properties, and each one earns its keep.

Named. A dataset is referred to by name, not by path. --labels runs/a/r2_labels.csv is how the workflows address data today. It is fine for one machine and hopeless for two — nothing about that string says which scene, which round, or which filter produced it.

Versioned and immutable. A version is never rewritten. A workflow that re-runs produces a new version, and anything that consumed the old one still resolves to exactly the rows it saw. This is what keeps a training run reproducible after a labeling bug is fixed upstream: the old labels do not change underneath it, they simply stop being the newest.

Lineage-bearing. Every version records its parents and the run that produced it. “Which labels are downstream of prompt v1.0.0” becomes a graph walk over recorded edges rather than a guess from timestamps.

Five kinds

Scores are datasets too. A metric is a claim about two other datasets and is meaningless without knowing which versions of them it compared. Storing it as a first-class dataset with both parents recorded is the only way an old number stays interpretable.

Why ground truth is one dataset with three facets

tracks is what the detection and tracking scorers read; views is per-camera 2D boxes measured from instance masks, which is what prompt validation reads; points is per-point instance ids for segmentation. They share one identity space — instance_id and class mean the same thing in all three, because the same pass wrote them.

Keeping them one dataset rather than standing up a separate per-view golden set is deliberate. A separate set can drift from the tracks, and then two evaluations of the same run disagree about what was present.

Two schema choices that carry findings

num_lidar_points lives in the truth, not in an eval config. It is what lets a scorer say “recall on objects with ≥ 5 returns” instead of hard-coding --exclude grade_stake. The stake is not a special case; it is the tail of a distribution, and a schema that admits this stops the exclusion list from growing.

Prompts are one row per phrase, not per prompt string, because maps_to_class is per phrase. That mapping is what turns a detector's raw output into an ontology class without string-matching at the call site — the mapping becomes data, versioned alongside the prompt that needs it.

Layout on magmalake

Storage is Apache Iceberg on magmalake, Parquet underneath. Nothing here needs a bespoke store — the properties wanted are exactly the ones a table format already provides.

magmalake                              catalog
└── labelrefinery                      namespace
    ├── datasets                       the registry
    ├── scenes
    ├── ground_truth_tracks            partitioned by (dataset_name, day(t))
    ├── ground_truth_views             partitioned by (dataset_name, camera)
    ├── ground_truth_points            partitioned by (dataset_name)
    ├── prompts
    ├── labels                         partitioned by (dataset_name)
    └── evaluations

The registry is the only new concept: it maps (name, version) to a table and a pinned snapshot id, alongside the parents, the workflow, its git sha and its params. Resolving a dataset is one lookup and one snapshot-pinned scan.

A snapshot id rather than a timestamp, because Iceberg snapshots are immutable by construction: a pinned read returns the same rows forever, even after compaction, schema evolution or a later append to the same table. A timestamp does not survive a rewrite.

One table per kind rather than one per dataset, because a table per dataset means thousands of tables and a catalog nobody can list. Partitioning by dataset_name gives the same pruning with one schema to evolve, and the registry restores named-thing addressing on top.

Browsing them

Two tables are published and readable in the browser on the demo site. Nothing is downloaded up front and there is no server in the path: the browser reads the Iceberg metadata, walks the manifests and pulls only the byte ranges of Parquet it needs, straight from where the tables are published — the same trick Foxglove uses on the recording.

Browse the tables on the demo →

The viewer there is icebird, about 43 KB of pure JavaScript with no WebAssembly. It resolves the table's current snapshot from version-hint.text, so a row is pinned to a snapshot id rather than to whatever happens to be latest.

Querying them instead

For anything beyond looking, the tables are ordinary Iceberg and every engine reads them. These work today against the published URLs:

duckdb -c "INSTALL iceberg; LOAD iceberg;
  SELECT class, count(*), median(num_lidar_points)
  FROM iceberg_scan('https://samples.magmalake.org/datasets/v0.2.0/ground_truth_tracks')
  GROUP BY 1"

Status

The registry and the writer are real Mojo now — `refinery.datasets` holds the single definition of every schema, and publishing a label set registers (name, version) pinned to the snapshot it wrote, so a version resolves to the same rows forever. Two of the five kinds are populated, from the artefacts that were already public rather than from a fresh run — ground_truth_tracks (9 000 rows) and labels (6 463 rows, the three rounds). That is migration steps 1 and 3 of the four below. scene, prompt and evaluation are still design only, and no pipeline writes Iceberg yet — these tables were built by reading the published MCAPs.

Two honest gaps in what is published. labels.class is null for every row: the published labels file is a visualisation artefact and carries geometry and a track id but no class name, so there was nothing truthful to put there — cls_conf and cls_source are null for the same reason. And num_lidar_points is known on 1 800 of 9 000 rows, because the per-point oracle is published at 2 Hz while boxes are at 10 Hz; where a sweep exists a value of 0 means seen and got no returns, and null elsewhere means unknown. The difference matters: 0 is what makes “recall on objects with ≥ 5 returns” answerable.

That column immediately earns its keep. Across the whole scene the six grade stakes never exceed 3 returns, while workers sit at a median of 12 and trucks at 143. The stake is not a special case to be excluded by name; it is the bottom of a distribution, and a threshold expresses that without a hard-coded exception.

The remaining gap is smaller than it looks: the current CSVs already carry the right columns and are missing identity. What is left is a sitegen views export from the instance masks, real class names plumbed through to labels, and the run manifest promoted to the registry.

That last one is the one to notice. Every workflow already records its inputs, outputs, params and hashes per step into a JSON file per run — that is what makes re-running a finished workflow cost a second. A dataset version is that record with a name attached. This design does not add provenance tracking; it moves provenance that already exists somewhere queryable.