The pipelines

Three named workflows, each resumable and each recording what it did. They are wiring, not algorithms — every stage is a component with its own repository. They also run as a durable service that can stop and ask a human.

1 · geo_kinetic_discovery

No model, no checkpoint, no class list — and no class names out either. What it produces is instances: a discrete something is here, this is its box, this is its trajectory.

proprioception self-mask   forward kinematics from /ego/joint_states
  → terrain classification   Stone.mojo — connectivity, not height
  → voxel clustering         26-neighbourhood connected components
  → oriented box fit         PCA heading in the ground plane
  → gated association        Hungarian.mojo — solve_gated
  → RTS smoothing            Kalman.mojo — smooth_track
  → motion filter            the stage that decides whether the loop compounds

What geometry and motion can actually settle:

• Terrain against not-terrain — free and reliable.
• Instances — connected clusters standing on that terrain.
• Motion attributes — the strongest signal here by some distance, and the one the whole loop turns on.
• A size bucket — machine-sized and person-sized boxes differ by two and a half orders of magnitude, and a 5 m³ split separates them 97.5% of the time.
• The ego's own parts, exactly — kinematics knows which rigid body is the boom. The one place geometry hands over true named classes, free.

What it can never settle: excavator against dozer; worker against any person-shaped object; stockpile against spoil pile, where the discriminator is intent and simply is not present in a point cloud.

Why terrain is a connectivity question

Ground removal by height fails on a stockpile: a pile at its 34° angle of repose climbs 0.67 m across a one-metre cell, so the bottom is called ground and the top third survives as slivers that cluster into vehicle-shaped objects. Tightening the tolerance only trades those false positives for missed objects near slopes.

So the ground surface is grown, not thresholded — seeded from the cells the machine drove through, flooding outward one repose-limited step at a time. A pile is continuous with the grade beneath it; a truck is a three-metre jump. Height stops mattering, continuity starts mattering.

2 · bootstrap_new_classes

Gives those instances names, from two sources that fail in opposite directions.

select views          where each instance projects largest and is in frame
  → detect            Grounding DINO, prompt "excavator . haul truck . worker ."
  → associate         detection ↔ projected 3D box, by IoU
  → vote              aggregate across views, confidence-weighted
  → size prior        for what the detector could not name

The detector names machines and cannot see people. Over ten unobstructed views, nine returned haul truck with the correct label; a worker visible in three of them was never detected once. Geometry finds people and cannot name machines. So the detector is asked only where it has a chance, and the size prior covers the rest.

Every name carries cls_conf and cls_source, because the two paths are not equally trustworthy — see the numbers.

3 · improve_offboard_model

Distil a detector from existing labels, then use it to label better. Run it repeatedly and it is a self-training loop.

filter labels        → the load-bearing parameter, not a knob
  → training set     NPZ in the layout CenterPillars already reads
  → train student    CenterPillars.py
  → infer            CenterPillars.mojo
  → track            the same association and smoothing as round 0
  → score            against the held-out oracle

Two further stages are implemented and evaluated but not in the default path — OfflinePoly.mojo (Offline-Poly) and LabelFormer.mojo (LabelFormer). What holds each back is on the papers page.

Nothing human-labelled enters at any point. The output is the same schema as the input, so feeding one round's labels into the next is the loop.

Running them

python -m workflows geo_kinetic_discovery \
    --scene site.mcap --work runs/a --truth runs/a/truth.csv

python -m workflows bootstrap_new_classes \
    --scene site.mcap --work runs/a --labels runs/a/labels.csv

python -m workflows improve_offboard_model \
    --scene site.mcap --work runs/a --labels runs/a/labels.csv --round r1

Every stage declares its inputs, outputs and parameters; the ledger hashes them and skips any stage a previous run already produced. Re-running a finished workflow costs a second and touches nothing.

The loop as a service

The commands above are the pipeline run by hand. The same stages also run as a Restate service, which is what makes the loop able to stop and wait for a person — for days if it needs to, holding no process while it waits — and to survive being killed mid-stage.

One call advances a run by one stage: ask the router what is runnable, record the decision, run it, record the outcome. A stage runs one of three ways, and the timeline says which:

Which stage runs next is a decision, and it is recorded. Today the router picks at random from whatever is legal — a stage whose inputs exist and whose work the ledger has not already done — which is deliberately the least interesting policy that still exercises the machinery a model would need. Every choice is written down with the candidates it was choosing between, so replacing the policy changes what gets picked and nothing else.

Two guarantees that sound alike and are not. Restate's journal makes a crashed run resume without redoing finished work within one invocation. The content-hash ledger skips work across separate runs. A stage is runnable when its inputs exist and the ledger lacks its key — not when its outputs are missing, which would make a stage whose inputs changed permanently ineligible and quietly serve stale results.

Watch it run →