Training, bake-off and promote

Training setup over the train routes (captured from Cropwright, a frontend for this backend)

The file protocol

/curation/projects/{project}/train/* is a control plane over files on the shared /jobs volume; the API never calls the trainer. Routes in this page are project-scoped; the prefix is omitted. Each project's job files live in its own directory, /jobs/projects/<slug>/, and the one trainer serves every project (it scans each project's directory for jobs). A run trains on, and is promoted from, one project's export.

File	Written by	Meaning
<job_id>.job.json	API (POST /train/start)	The job spec; the trainer claims it
<job_id>.cancel	API (POST /train/cancel/{job_id})	Cancel sentinel
<job_id>.status.json	Trainer, every ~5 s	State, epochs, metrics, the eval block
<job_id>.run.log	Trainer	Log, served by GET /train/log/tail/{job_id}
<job_id>.manifest.json	Trainer, at the end	Lineage: dataset SHA, code versions, class remap, eval
.trainer_capabilities.json	Trainer, every ~30 s	Heartbeat used by preflight

The shipped trainer (docker/trainer/, service curation-trainer) runs jobs through Ultralytics. Any trainer that speaks this protocol can replace it.

Running a training job

docker compose --profile training up -d          # trainer + MLflow
T="$API/curation/projects/$P/train"
curl -s -X POST "$T/preflight" -H 'content-type: application/json' -d '{}'
curl -s "$T/profiles"                            # hyperparameter tables per family/size/profile
curl -s -X POST "$T/start" -H 'content-type: application/json' -d '{
  "model_family": "yolo26", "model_size": "s", "profile": "small",
  "cuda_visible_devices": "0",
  "hyperparameters": {"epochs": 70, "imgsz": 640, "batch": 16, "optimizer": "MuSGD"}
}'
curl -s "$T/status"

• Profiles are not merged into the request. GET /train/profiles returns a table for a UI to render; posting "profile": "small" alone does not pull those values in. Copy every value you want into hyperparameters. A top-level epochs (outside hyperparameters) is a 422.
• dataset_export_dir defaults to the current export. With no export at all, preflight returns one clear blocking check telling you to run POST /export/yolo first.
• Subsets: hyperparameters.include_classes trains on a subset of a multi-class export; GET /train/presets lists named subsets (OP_TRAIN_PRESETS_PATH adds your own).
• Preflight returns blocking checks and warnings (empty train or val split, a trained class missing from a split, unlabeled objects on exported images, trainer reachability, GPU availability). ?force=true on /train/start and /train/start_campaign bypasses blocking size and balance checks, except that only one run can be active at a time and an empty validation split is never bypassable: it answers 422 empty_val_split (the trainer cannot validate on an empty val folder). Re-export with enough non-holdout validated items instead.
• GPUs: cuda_visible_devices names host GPU ids, restricted to OP_GPU_ALLOWED_IDS; GET /train/gpus lists the options. An id outside that list, or a GPU the trainer container is not attached to (preflight check trainer_gpus, blocking), fails preflight before any job is written.
• Campaigns: POST /train/start_campaign runs a progressive, multi-size sequence; POST /train/cancel_campaign/{campaign_id} cancels every non-terminal run in it.
• Offline hosts: unless hyperparameters.model points at a local file, the trainer downloads the base checkpoint for the requested family/size from Ultralytics' release assets at run start. Pre-download it and set hyperparameters.model on an air-gapped host.
• MLflow (curation-mlflow, port 4609) is optional; set OP_MLFLOW_PUBLIC_URL so served run links resolve in a browser.

GET /train/runs, GET /train/status/{job_id}, GET /train/manifest/{job_id} and GET /train/artifacts/{job_id}/{name} expose past runs, their evaluation on the frozen test split and their lineage.

Runs in MLflow

With curation-mlflow running (--profile training, port 4609), the trainer logs every run to the experiment named by MLFLOW_EXPERIMENT_NAME (default openprocessor): the full Ultralytics argument set as parameters, per-epoch losses, learning rates and mAP as metrics, lineage tags (dataset_sha, frozen_test_sha, test_label_sha, profile, seed), the training plots and run_metadata.json as artifacts, and a model version registered under OP_MLFLOW_MODEL_NAME (default openprocessor_detector).

The openprocessor experiment with the runs the trainer logged (public-sample-data stack)

One run: lineage tags, the registered model version, 112 parameters and 20 metrics

Per-epoch metric charts for the same run

Two runs compared with Show diff only: epochs and the per-class mAP differ. These are short runs on the small public sample, so the mAP values are low

Bake-off: comparing models

Bake-off setup over the bakeoff routes (captured from Cropwright, a frontend for this backend)

The evaluator (curation-evaluator, profile evaluator) scores trained runs and baseline models per class on the test split of any of the project's exports, or on frozen external sets under the project's bakeoff_eval directory ({OP_PROJECTS_DATA_ROOT}/{slug}/bakeoff_eval/, with curated, public and sample subdirectories). Each project queues its own jobs; the evaluator watches every project's queue.

1. GET /bakeoff/eval_datasets lists what can be scored (exports with a labeled test split and external sets), with class lists and SHAs.
2. GET /bakeoff/trained_models, GET /bakeoff/baseline_models and GET /bakeoff/profiles list candidates and scoring profiles (the generic profile scores every class present in the eval split).
3. POST /bakeoff/run queues a job file for the evaluator, in the project's own queue.
4. GET /bakeoff/status/{job_id}, /results/{job_id} and /matrix/{job_id} return the per-class comparison and the model x dataset matrix (schema v2; older result files answer 409).

Classes a model doesn't cover are shown, not hidden. A queued bake-off also claims its GPU through the arbiter, like training.

Promote

POST /curation/projects/{project}/train/promote/{job_id} takes a finished run's ONNX export, writes it into the Triton model repository (models/<name>/1/model.onnx plus a YOLO26 single-output config.pbtxt and a labels.txt), and asks Triton to load it, then runs two warm-up inferences so the TensorRT engine is built before the first real request.

That takes 2-3 minutes, so by default the call validates the request (gate, class remap, name) and returns 202 with a promote_id. Poll GET .../train/promote/{job_id}/jobs/{promote_id}: status moves through queued, exporting (copy and config), loading (Triton load), building (first warm-up, where the engine compiles), warming (max-batch warm-up) and ends at done (with result, the same body a synchronous promote returns) or failed (with error and error_status, the HTTP status a synchronous call would have returned). The latest promote also appears as promote in GET .../train/status/{job_id}. A second promote of the same run while one is active returns that job with 200 (409 promote_in_progress if it asked for a different triton_name); a job whose API process died reads failed. Use the 202 job and polling for UIs and for anything behind a reverse proxy: proxies time out near 120 seconds (an nginx in front of the API answers 504 at 120 s), well before a promote ends. ?wait=true blocks and returns the 200 body directly; use it only from scripts that call the API directly, with a client timeout of at least 300 seconds. If a wait=true connection is cut (a proxy 504, a client timeout), the promote is not cancelled: it keeps running and the model finishes loading and warming, so check that the model reports ready in the project's model list instead of re-submitting.

Promote is gated on the run's metrics (mAP50, per-class precision and support). To bypass the gate, send "force": true in the JSON body or add ?force=true to the URL; either works (the 422 override text names both). A forced promote reports force_used in the response and the run manifest.

The Triton model is named {slug}__<triton_name> (the default project's models have no prefix), so two projects can promote the same name without colliding. The name you request is not the served name: the final one is returned as triton_name in the promote result and in the promote job status, and that is the name to use for /detect?model_name= and DELETE .../models/{model_name}. A request that names the bare <triton_name> fails with unknown model or not a model owned by this project. A requested name may not itself contain __ (422 triton_name_reserved_separator). A promoted model belongs to its project: GET .../models/{model_name}/class_mapping shows how its classes map onto a project's classes by name, and PUT .../models/{model_name}/sharing opts it into use by other projects (for example as a region detector).

PUT .../models/{model_name}/sharing takes {"shared": bool, "expected_revision": int}; expected_revision is the model's current sharing_revision (from GET .../models/status), and a stale one is 409 revision_conflict with current_revision. Unsharing while another project's active detector uses the model is 409 in_use with projects and used_by; the query parameter ?force=true bypasses that refusal (logged). GET .../models/status?include_other_projects=true also lists other projects' models that their owners shared (one row per model name; your own models and the core models come first).

When a promoted model was trained on a single class, labels.txt carries that class's own name from the project registry; a run that collapsed several classes into one keeps the trainer's name for it.

The promote response's cold_start_expected_on_first_inference is false only after the promote ran a warm-up inference at batch 1 and at the model's max_batch_size; a request with a batch size outside what the engine has seen makes the TensorRT accelerator rebuild (about 90 s), so both are warmed.

It is gated on the run's eval block:

Gate	Threshold
mAP50	at least 0.65
Per-class precision	at least 0.50 for every class
Per-class support	at least 5 for every class

Each failure has a code (no_eval_block, map50_below_floor, per_class_not_list, and the per-class precision and support codes), for example map50_below_floor when mAP50 is under 0.65. A failing gate is a 422 whose body lists structured failures and says whether force=true can bypass that particular failure (a run that isn't finished can't be forced). Other outcomes: 404 (no job or no ONNX yet), 409 (a model with that name exists; pass overwrite=true), 502 (Triton refused the load).

For a subset or single-class run, promote resolves the class remap (from the manifest's lineage, then the weights directory) before writing labels.txt, so a subset model never serves the full registry's names.

Triton runs in explicit model-control mode, so a restart forgets promoted models that aren't in its --load-model list. POST /train/reload_promoted (or make reload-promoted, which make restart-triton runs for you) loads them again.

Promoted models and ingest

A promoted YOLO26 engine has one [300, 6] output. It serves fine through POST /detect?model_name=<promoted> (use the prefixed Triton name), but doesn't yet fit either ingest detector contract, so don't point OP_INGEST_PRIMARY_DETECTOR_MODEL or OP_INGEST_SECONDARY_DETECTOR_MODEL at it.

Closing the loop

POST /curation/projects/{project}/probe/run runs the active-learning probe over the item pool and stores its prediction, confidence and entropy per item. The model_disagreements review queue then shows items where the model and the human label disagree, and uncertainty sorts by entropy. A disagreement is offered as a one-click fix only when the probe's confidence is at least OP_PROBE_ACTIONABLE_MIN_CONFIDENCE (default 0.5). Those corrections feed the next export.