Skip to main content

Curation workflow, end to end

The whole active-learning loop over the /curation API, in order. Every step works from curl; a frontend such as Cropwright drives the same routes. API=http://localhost:4603 below.

This page walks the full loop. You do not have to run all of it: detection only, detection with embeddings and clustering, and VLM-assisted labeling are different settings of the same stack. See Pipeline modes for which workers to start, what to switch off per project, and what the code cannot yet switch off.

0. Prerequisites​

  1. Core stack up and the PE-Core encoders built (make export-pe). Without pe_image_encoder, ingest writes no pe_embedding, and semantic search, near-duplicate detection and clustering have nothing to work on.
  2. OpenSearch indexes are created automatically, per project, when the project is created.
  3. You bring the models: an item detector (a stock COCO YOLO works to start), and optionally a VLM, a region detector and a segmenter. See Models you must supply.

1. Project​

All curation data lives in a project. A fresh stack has the default project; create one per dataset:

P=my-dataset
curl -s -X POST "$API/curation/projects" -H 'content-type: application/json' \
  -d "{\"slug\": \"$P\", \"display_name\": \"My dataset\"}"

Every route below is under $API/curation/projects/$P/; the section text omits that prefix. A project can copy another's settings at creation (clone_settings_from), be archived (read-only) and be deleted (not default). Projects share the models, workers and GPU, not their data.

The projects page: one row per project with its counts (captured from Cropwright against the public COCO sample projects)

2. Classes​

Each project has its own class registry, a JSON file under its data directory. Start from zero with POST /classes (see the Quick start), copy data/class_registry.example.json (a small warehouse example) and edit it, seed every class a detector knows with scripts/curation/seed_class_registry.py --project $P --model <detector.onnx> so class ids can't drift from the model's order, or let a dataset import create them. Names must match ^[a-z0-9_]+$. A class is identified by its name: datasets, combined projects and promoted models are matched by name, never by id.

3. Ingest or import​

SourceTool
Images on storage the API container mountsPOST /ingest/image, or scripts/curation/bulk_ingest.py --folder DIR --project $P (or a --manifest list) for a resumable bulk run with progress, cancel and failure reporting
Images only on your machinescripts/curation/ingest_upload.py --project $P (uploads bytes to POST /ingest/upload; resuming relies on server-side content dedup)
Images only (no labels) from a labeled datasetscripts/curation/import_labeled_dataset.py --images-only (skips every class)
An already-labeled YOLO / COCO dataset or an OpenProcessor exportPOST /datasets/preview, then POST /datasets/imports (or scripts/curation/import_labeled_dataset.py); classes map by name, the import can be resumed and undone

Ingest runs the item detector (OP_INGEST_PRIMARY_*), crops each detection into an item, embeds it with PE-Core, rejects duplicates and low-quality crops, and indexes items in bulk. Human labels already on an item survive re-ingest. With a region profile active, each new item is also seeded pending_detection for the detection worker.

4. Cluster​

Cluster cards from GET /clusters (captured from Cropwright, a frontend for this backend)
docker compose --profile curation up -d
curl -s -X POST "$API/curation/projects/$P/pipeline/auto_label/start?train_clusters=true"
curl -s "$API/curation/projects/$P/pipeline/auto_label/status"     # poll until terminal
curl -s "$API/curation/projects/$P/clusters" | jq '.clusters[0]'

Clustering groups items by embedding (residual clustering reduces pe_embedding by default). Cluster ids have fixed ranges: below 0 is unassigned/noise, 0..N mirrors class ids, and ids from 10000 up are candidate clusters that still need a class. Each card carries cluster_kind, dominant class, purity, validated count, sub-clusters and representatives; POST /clusters/refine/{cluster_id} splits one further, and POST /clusters/auto_promote validates classifier-labelled items in high-purity clusters. Promotion is gated on the accuracy audit (section 5): every class a run would promote needs at least min_per_class audited crops and an audited detector precision of at least promote_min_precision (default 0.95), else the call answers 409 audit_required or 409 audit_precision_low with the classes listed and writes nothing. force=true bypasses the gate and is logged; a dry run is never gated. curation-cluster-refresh calls auto-promote as each project's pool grows, so it promotes nothing until the audit clears the classes.

:::caution Known issue: a gated promote stops the retrain The daemon treats the 409 audit_required answer as a failure. It logs auto_promote failed and returns before it queues the centroid retrain, so a project with an unaudited promotable class stops refreshing its clusters until the audit passes. (issue #280). See Pipeline modes. :::

5. Label: VLM suggestions and review queues​

Review queue from GET /review/{tab} (captured from Cropwright, a frontend for this backend)
  • VLM assist. A project runs its active VLM endpoint and calls no VLM until it activates one (the env endpoint from OP_VLM_URL/OP_VLM_MODEL included), so nothing waits for a VLM. Switch an active project off with POST /vlm/endpoints/deactivate (see VLM selection), or run the VLM for one job only with vlm=<name> (see the selective recipe). Run auto_label/start?run_vlm=true, label a cluster, a sub-cluster or a sample with POST /assist/runs, or let curation-vlm-worker suggest classes continuously. Register and activate other endpoints as in VLM selection; ?vlm=<name> picks one for a single run. VLM answers that name no registry class surface as new-class proposals rather than being forced into a class.
  • Prompts. The VLM's prompts are a prompt pack; test a draft on stored crops with POST /prompt_packs/test before you activate it (Test on a crop).
  • Review queues. GET /review/tabs lists them; the main ones are all, mismatches (VLM answer didn't match the registry), vlm_low_conf, detector_disagreements (the VLM's class differs from the detector's own), outliers, uncertainty and model_disagreements (both fed by the probe), primary_low_conf, classifier_blind_spots, new_class_proposals, imported and regions.
  • Accuracy audit. Nothing else measures how often the detector or the VLM is right. POST /audit/start draws a stratified sample of machine-labelled, unvalidated, non-holdout crops that carry the detector's answer (min_per_class per detector class, sample_size in total), GET /audit/queue lists them, and you label them through the normal label routes. Each human label stamps whether the detector, the VLM, both or neither was right. GET /audit/report returns per-class precision with Wilson 95% intervals, the {detector class: {human class: n}} confusion matrix and an insufficient_sample flag for a class with too few audited crops. Audited crops that a human labelled are ordinary human labels, so they are holdout-eligible.
  • Label writes. PUT /crops/{crop_id}/label, PUT /crops/batch_label, POST /crops/move, exclude/discard, and undo for each. Every write is recorded in GET /crops/{crop_id}/history. A human label is locked: no worker, VLM pass, re-ingest or reprocess overwrites it.
  • Keyboard. Each project has its own keymap.
  • Semantic search (GET /search/text, with OP_SEMANTIC_SEARCH_ENABLED=1) finds items by description, e.g. "blurry close-up".

6. Regions of interest (optional)​

Region review: box, detector chain and text reading (captured from Cropwright, a frontend for this backend)

A region profile describes a sub-region to find inside each item (a wheel on a car, a label, a defect) and which models find, verify and read it. Create or clone one, check it on a stored crop, then activate it:

curl -s -X POST "$API/curation/projects/$P/region_profiles/test" ...   # one stored crop, writes nothing
curl -s "$API/curation/projects/$P/region_profiles/active/impact"      # what activation affects
curl -s -X POST "$API/curation/projects/$P/region_profiles/vehicle_wheel/activate"

The detection worker's cascade selects up to max_regions_per_item candidate boxes per item (region detector, then the segmenter leg), sends one numbered overlay per item to the VLM, and writes a state, verdict and optional text per box. Every write records its provenance in region_detector_chain. Humans accept, reject or mark boxes false positive one box at a time (PATCH /crops/{crop_id}/regions/{box_id}, POST /regions/batch_box_state) or set a whole-item status (POST /regions/batch_status). Watch progress on GET /ingest/region_drain, and see Health and stall reasons when it stops moving. The wheel example walks this on public data and Multi-box regions explains the box model.

Region review: several numbered boxes on one item, each with its own state (captured from Cropwright against the public COCO sample projects)

After changing a profile, pack or VLM, re-run existing items with POST /reprocess (see Health and stall reasons). Locked items and boxes are skipped.

7. Freeze, export, train, compare, promote​

  1. POST /test_holdout/freeze once.
  2. POST /export/yolo (or /export/single_class). See Exports and retention.
  3. POST /train/preflight, then POST /train/start.
  4. POST /bakeoff/run to compare against earlier runs and baselines.
  5. POST /train/promote/{job_id} to serve the winner from Triton.
  6. POST /probe/run, then work model_disagreements, and loop back to step 4 of this page.

Details: Training, bake-off and promote.

8. Combine projects​

To merge several projects into one new project, use POST /curation/projects/combine/preview, then POST /curation/projects/combine. Sources are only read; the target is built beside them. See Combine projects.

Models you must supply​

Nothing in the curation subsystem ships a pretrained region detector, VLM or trained dataset. A deployment provides:

WhatRequired?How
PE-Core-L14-336 image and text encodersYesmake export-pe; the Triton names pe_image_encoder / pe_text_encoder are fixed
Item detectorYes, for ingestAny end2end Triton model via OP_INGEST_PRIMARY_DETECTOR_MODEL
Secondary detectorNoRaw-output model via OP_INGEST_SECONDARY_DETECTOR_MODEL; also the source of backbone_embedding when exported with make export-dual-head
Region profile + region detectorNoA stored profile activated per project (Region profiles), or the deployment default from OP_REGION_PROFILE_PATH / OP_REGION_PROFILE; see advanced configuration
OCR recognition modelOnly if regions carry textThe profile's ocr_rec_model
SegmenterNo--profile segmenter (SAM 3, needs HF_TOKEN) or any service speaking the same protocol at OP_SEGMENTER_URL
VLMNo--profile vlm, a VLM on the host (http://host.docker.internal:<port>/v1), or anywhere on the network; registered as an endpoint (VLM selection)
Training data and base weightsFor trainingYour exports; the trainer fetches base checkpoints unless hyperparameters.model is a local path