Pipeline modes

OpenProcessor has no single "mode" switch. What runs is the sum of five things you control: which compose services are up, the project's ingest policy, the project's VLM activation, the project's VLM policy, and one worker setting. This page names four common ways to run it, gives the exact commands for each, and says plainly where the code cannot do what the mode name suggests.

API=http://localhost:4603 and project cars are assumed. In a checkout the API port is whatever API_PORT says in .env.

Mode	What runs	How you select it	What you cannot turn off today
(a) Detection only	Core stack. Ingest runs the item detector and stores boxes.	Ingest policy embedding.mode=none and clustering.automatic=false.	Nothing: no vector of any kind, no cluster placement, no VLM.
(b) Detection, embedding, clustering	Core stack and the curation workers. Every detection embedded.	The default.	A cluster-refresh retrain stalls on an audit-gate 409. (issue #280)
(c) VLM-assisted labeling of a large batch	Mode (b) plus a VLM.	vlm profile or a remote endpoint, project activation, VLM policy.	Nothing: a project calls no VLM until it activates one.
Selective VLM	Mode (b), then the VLM for one cluster, batch or filter.	The recipe below.	Nothing runs the VLM on its own.
(d) Iterative cluster-sampled labeling	Not shipped.	Not selectable.	Planned. (issue #286)

:::note How the commands on this page were checked Blocks marked # ran were run on the local stack against the public project coco-bench-2k, and the output shown is from that run. Blocks marked # not run were checked against the code but not executed, because they change a running stack or a project. Treat those as code-verified, not field-tested. :::

The knobs that exist

Knob	Where	What it does
Compose profiles	COMPOSE_PROFILES in .env, or --profile on the command line	Which services exist. See Compose profiles.
GPU profile	./openprocessor profile, GPU_PROFILE	Triton instance counts and batch sizes only. It does not choose models. See Deployment and GPU sizing.
Ingest policy	GET and PUT /curation/projects/{project}/ingest/policy	Which detections are stored, which get a vector (embedding.mode: all, selected, lazy, none) and whether clustering runs on its own (clustering.automatic). See Ingest policy.
VLM activation	POST /curation/projects/{project}/vlm/endpoints/{name}/activate and .../vlm/endpoints/deactivate	Which VLM a project calls. A new project starts with none (off).
VLM policy	GET and PUT /curation/projects/{project}/vlm/policy	scope (all, uncertain, representatives, off), per_cluster, max_crops_per_day, sample_frac. See VLM selection.
Region profile and prompt pack	config store	What the detection worker's cascade does per item. See Region profiles.
OP_CLUSTER_REFRESH_*	.env, read by curation-cluster-refresh	Interval, growth threshold, deferral, retry backoff and the retrain job of the refresh daemon. See mode (b), step 3.
OP_IVF_RETRAIN_CHECK_S	.env, read by curation-auto-label-worker	0 disables that worker's idle centroid retrain for every project. A project turns it (and the refresh daemon) off with clustering.automatic=false.
OP_REGION_EMBED_ENABLED	.env	false skips the per-box region embeddings.
VLM CLI	./openprocessor vlm list|status|use|apply|probe|key set	Switch the in-compose VLM model.

(a) Detection only

You want boxes and classes from the detector, stored in a project, with as little embedding and clustering work as the code allows.

What you get. The API runs the item detector at ingest and stores one item per detection. With embedding.mode=none nothing is embedded at ingest, the whole-frame vector of each image included, and with clustering.automatic=false no item is placed in a cluster and no background retrain touches the project. No worker is needed, and no VLM runs unless the project activated one.

What stays on. An item that carries a human or imported label (a dataset import) still gets a vector, whatever the mode. On the GPU decode path the ensemble computes the whole-frame vector in its single pass and the API drops it.

1. Stateless detection, no storage

This needs only Triton and the API. Nothing is written to OpenSearch.

# ran
curl -s -X POST $API/detect -F "image=@000000000139.jpg" | head -c 300

{"detections":[{"x1":0.0089874267578125,"y1":0.3906983435153961,"x2":0.241455078125,"y2":0.6137030720710754,"confidence":0.923828125,"class":62,"class_name":"tv"},{"x1":0.457763671875,"y1":0.5102699398994446,"x2":0.5498046875,"y2":0.7501466870307922,"confidence":0.865234375,"class":56,"class_name":"chair"},{"x1":0.5654296875,"y1":0.5154049396514893,"x2":0.6484375,"y2":0.7325410842895508,

Batches: POST /detect/batch, up to 64 images.

2. Stored detections, no vectors, no clustering

Set the project's policy before you ingest. none stores every detection as deferred with no vector; clustering.automatic=false keeps ingest and the background retrains from clustering:

# not run
curl -s -X PUT $API/curation/projects/cars/ingest/policy \
  -H 'content-type: application/json' \
  -d '{"expected_revision": 0, "embedding": {"mode": "none"},
       "clustering": {"automatic": false}}'

Then ingest with Ingest a large folder. The policy and its limits are in Ingest policy. No curation profile is needed: the core stack (Triton, API, OpenSearch) is enough.

3. Embed and cluster later, on demand

Nothing was embedded, so the project can be embedded and clustered whenever you decide. One auto-label job does both (embed_missing embeds the deferred items first; the clustering is the job's own, so clustering.automatic=false does not stop it):

# not run
curl -s -X POST "$API/curation/projects/cars/pipeline/auto_label/start?embed_missing=true&train_clusters=true"

Switch the project back to automatic clustering (and a normal embedding.mode) with another PUT of the policy when you want ingest to cluster on its own again. The policy never rewrites stored data, so items already stored keep their state. The embed scope of Reprocess embeds without clustering.

The workers you run decide the rest. With only some of them running, start them by name. There is no CLI for a subset: ./openprocessor curation up and make curation-up start all five curation services. (issue #285) With only some workers running, start them by name:

./openprocessor curation up curation-detection-worker
make curation-up SERVICES="curation-detection-worker curation-auto-label-worker"

curation up with no names starts the four curation workers. curation-evaluator (a bake-off watcher with no restart policy that needs the NVIDIA runtime) has its own evaluator profile and starts only when you name it, or with docker compose run --rm curation-evaluator.

curation-detection-worker runs the region cascade. Without a region profile it idles, so for plain item detection you do not need it at all.

(b) Detection, embedding and clustering

This is the default once the curation profile is up: every detection is embedded (embedding.mode=all), clustering refreshes on its own, and a VLM labels suggestions if one is configured.

1. Start the workers

# not run
./openprocessor curation up
./openprocessor curation status

curation up starts the four curation workers. curation-evaluator is not among them: it needs the NVIDIA runtime and is a bake-off watcher with no restart policy, so it has its own evaluator profile (./openprocessor curation up curation-evaluator).

2. Decide about the VLM now

A project calls no VLM until it activates one. OP_VLM_URL defines an endpoint (env) the project can opt into, nothing more, so curation-vlm-worker and the auto-label VLM stage skip a project that has not. On a new project the API says so:

# not run  (code-verified; the stack this page was checked on predates the change)
curl -s $API/curation/projects/cars/vlm/endpoints/active

{"axis":"vlm","active":{"name":null,"revision":null},"source":"off","activated_at":null,...}

To run mode (b) with a VLM, activate an endpoint for the project (mode (c), step 3). To keep the VLM off, do nothing. To switch it off again later:

# not run  (the project then answers 409 vlm_not_configured to VLM calls that name no endpoint)
curl -s -X POST $API/curation/projects/cars/vlm/endpoints/deactivate \
  -H 'content-type: application/json' -d '{"expected_active": {"name": "env", "revision": null}}'

# not run  (VLM stays active but no automated writer labels; revision from the GET)
curl -s -X PUT $API/curation/projects/cars/vlm/policy \
  -H 'content-type: application/json' -d '{"expected_revision": 0, "scope": "off"}'

With the first, every VLM call that does not name an endpoint is refused. With the second, only the two automated writers stop (the VLM worker and the auto-label VLM stage); explicit requests still work.

:::note Behavior change in 0.6.0 Earlier releases let every project that never activated an endpoint inherit env. A project created by an earlier release is pinned once, at the first start of the new API, to what it inherited: env when OP_VLM_URL was set, else off. Its behavior does not change, and GET .../vlm/endpoints/active now shows "source":"env" with an activated_at for it. A default project created by a fresh install is pinned the same way on its first start. Every project created after that starts off. :::

Settings, Models: a project with its VLM off (public sample project)

3. What the cluster-refresh daemon does

curation-cluster-refresh (scripts/curation/cluster_refresh_daemon.py) polls every project's item count. When a project has grown by the threshold since its last refresh it does two things in order:

1. Posts POST /curation/projects/{project}/clusters/auto_promote.
2. Queues the auto-label job for the retrain (POST .../pipeline/auto_label/start with no VLM stage), which the auto-label worker runs.

It waits for ingest to go quiet first, up to a deferral limit, and retries a failed retrain after a backoff. Set its behavior in .env (the service reads .env, so no compose change is needed) and recreate it with ./openprocessor restart curation-cluster-refresh. A command-line flag overrides the variable:

Variable	Flag	Default
OP_CLUSTER_REFRESH_INTERVAL_S	--interval-seconds	120
OP_CLUSTER_REFRESH_GROWTH_THRESHOLD	--growth-threshold	200 new items
OP_CLUSTER_REFRESH_MAX_DEFERRAL_S	--max-deferral-seconds	1800
OP_CLUSTER_REFRESH_RETRY_BACKOFF_S	--retry-backoff-seconds	600
OP_CLUSTER_REFRESH_AUTO_LABEL	--auto-label / --no-auto-label	true (queue the retrain). false runs auto_promote only.
	--project	every active project
	--once	one pass, then exit

Per project off switch. A project whose ingest policy has clustering.automatic false is skipped by this daemon entirely, by ingest-time placement and by the auto-label worker's idle retrain (mode (a), step 2). It is read at each poll, so the change applies without a restart. Explicit clustering (/clusters/*, an auto-label job you start) always runs. To change a flag, replace the service command in docker-compose.local.yml, which ./openprocessor and make load after the stock files. The merged result was checked with docker compose config:

# docker-compose.local.yml
services:
  curation-cluster-refresh:
    command:
      - python
      - scripts/curation/cluster_refresh_daemon.py
      - --api=http://api:8000
      - --opensearch=http://opensearch:9200
      - --growth-threshold=1000

./openprocessor curation up curation-cluster-refresh

The second trigger lives in the auto-label worker. While idle it checks each project every OP_IVF_RETRAIN_CHECK_S seconds (default 1800). It retrains the centroids when the pool has grown to OP_IVF_RETRAIN_GROWTH times (default 1.5) the trained size and at least OP_IVF_RETRAIN_MIN_INTERVAL_S seconds (default 86400) have passed. Neither trigger runs the VLM unless the project activated one and the retrain job asks for it (the daemon's job has no VLM stage).

4. Known issue: the audit gate stops the retrain

auto_promote is gated on the accuracy audit. When a class it would promote has too few audited crops or too low a detector precision, the route answers 409 audit_required or 409 audit_precision_low. The daemon treats any HTTP error as a failure: it logs auto_promote failed, returns, and does not queue the retrain. The project's clusters then stop refreshing on every poll until the audit passes. Run the audit for that project, or switch the daemon to --no-auto-label and queue retrains yourself. (issue #280)

(c) VLM-assisted labeling of a large batch

You have many unlabeled items and want a VLM to propose classes, under a budget.

The VLM needs embeddings: it only looks at items that have a pe_embedding. If the project's policy was lazy, embed first (step 5).

1. Provide a VLM

Either run one in the stack or point at one elsewhere.

# not run  (installed directory; adds the vlm tier)
./setup-openprocessor.sh --tiers core,curation,vlm

# not run  (compose; the profile can also be listed in COMPOSE_PROFILES)
docker compose -p <project> --profile vlm up -d vlm

For a server on another host, register it as an endpoint instead. See VLM selection.

2. Choose the model

For the in-compose VLM, pick from the catalog. This is real output:

# ran
./openprocessor vlm list

id                   licence    vram_gb  status    
gemma-4-e4b          Apache-2.0 23       tested    
qwen3-vl-8b-fp8      Apache-2.0 19       to_verify 
gemma-4-e2b          Apache-2.0 17       to_verify 
qwen3-vl-4b          Apache-2.0 17       tested    
qwen2.5-vl-7b-awq    Apache-2.0 15       to_verify 
qwen3-vl-8b          Apache-2.0 26       to_verify 

# not run
./openprocessor vlm use qwen3-vl-4b
./openprocessor vlm probe

vlm use needs OP_LOCAL_VLM_ENDPOINT, which the installer sets when you add the vlm tier. Details, including enabling it on an existing install, are in VLM selection.

3. Activate it for the project

A new project has no VLM. Activate the built-in env endpoint, or one you registered:

# not run
curl -s -X POST $API/curation/projects/cars/vlm/endpoints/env/activate \
  -H 'content-type: application/json' \
  -d '{"revision": null, "acknowledge_external": false, "force": false, "expected_active": null}'

Use .../vlm/endpoints/host-vlm/activate for a registered endpoint named host-vlm.

4. Set scope and budget before anything runs

The default scope is all, which labels every eligible crop. On a large project set a budget first. representatives labels the per_cluster crops nearest each cluster centre, plus every unassigned crop:

# not run
curl -s $API/curation/projects/cars/vlm/policy
curl -s -X PUT $API/curation/projects/cars/vlm/policy \
  -H 'content-type: application/json' \
  -d '{"expected_revision": 0, "scope": "representatives", "per_cluster": 5,
       "max_crops_per_day": 2000}'

The continuous worker re-reads the policy about every 30 seconds. The fields are in VLM selection.

5. Run it

Two ways. Use either or both.

• Continuous. With the curation profile up and a VLM active, curation-vlm-worker labels embedded, unvalidated items on its own, inside the policy.
• One run. Queue an auto-label job with the VLM stage on. Add embed_missing=true if the policy was lazy, so deferred items are embedded first:

# not run
curl -s -X POST "$API/curation/projects/cars/pipeline/auto_label/start?run_vlm=true&embed_missing=true"
curl -s "$API/curation/projects/cars/pipeline/auto_label/status"

A project with no VLM answers the first call with 409 vlm_not_configured unless the request names an endpoint with vlm=<name> (see the recipe below). VLM labels stay unvalidated suggestions (class_source: "vlm"). Nothing reaches an export until a person validates it.

To collect the pack's item attributes in the same run, add vlm_attributes=true. The VLM stage then uses the pack's combined assist call (class and attributes in one request) and stores the attributes in each item's assist slot, apart from the class. It needs run_vlm=true (422 invalid_request otherwise) and a pack that declares item attributes and fills the assist prompts (422 pack_declares_no_attributes). The region worker's combined call does the same for the items it classifies, when the active pack declares item attributes.

Recipe: ingest without the VLM, cluster, name by hand, then run the VLM for one thing

The goal: ingest images, detect cars and people, cluster, name clusters yourself (for example spoiler, a class the detector does not have), and call the VLM only for the part you choose. Nothing in this recipe waits for a VLM, and nothing runs one by itself.

:::note How this was run Run on 2026-10-15 against the public project data of coco-bench-2k (40 COCO images copied into a throwaway project, deleted afterwards). The API on that stack predates the "no VLM until activated" change, so step 0 deactivates the project's VLM, which is the state every new project now starts in. Steps 1 to 5 are unchanged code. That stack's VLM endpoint was unreachable, so the VLM calls in step 6 show the refusal and the failure report, not labels. Where a command was only checked in the code, it says # not run. :::

API=http://localhost:4603/curation/projects/cars

0. The project has no VLM.

# ran  (needed only on a release before this change; new projects start here)
curl -s -X POST $API/vlm/endpoints/deactivate -H 'content-type: application/json' \
  -d '{"expected_active": null}'

{"axis":"vlm","active":{"name":null,"revision":null},"source":"off", ...}

1. Keep only cars and people. The detector knows 80 classes; this stores two.

# ran
curl -s -X PUT $API/ingest/policy -H 'content-type: application/json' \
  -d '{"expected_revision":0,"detect":{"classes":["car","person"]},"embedding":{"mode":"all"}}'

2. Ingest. Upload shown; a server folder uses ingest/batch (see Ingest sources).

# ran
curl -s -X POST $API/ingest/upload -F images=@a.jpg -F images=@b.jpg   # 40 files in the run

{"status":"success","summary":{"successful":40,"duplicates":0,"failed":0,"crops_indexed":162,
 "n_embedded":162,"n_not_embedded":0,"n_embed_failed":0,"n_filtered":155}, ...}

162 boxes stored, 155 other detections dropped by the policy, and the VLM was not called.

3. Cluster, with no VLM stage. run_vlm defaults to false.

# ran
curl -s -X POST "$API/pipeline/auto_label/start?train_clusters=true&run_vlm=false"
curl -s $API/pipeline/auto_label/status      # status "completed"; stages.vlm.skipped: true
curl -s "$API/clusters?limit=20"

On 162 crops the pass made one cluster (10000). Split it into sub-clusters:

# ran
curl -s -X POST "$API/clusters/refine/10000?distance_threshold=0.6"

{"cluster_id":10000,"n_items":162,"n_subclusters":10,...,"action":"refined"}

4. Name a cluster by hand. Create the class, then label the members of a sub-cluster.

# ran
curl -s -X POST $API/classes -H 'content-type: application/json' \
  -d '{"name":"spoiler","group":"car_part"}'
curl -s "$API/crops?cluster_id=10000&cluster_subid=10000b&limit=200"   # the ids of sub-cluster 10000b
curl -s -X PUT $API/crops/batch_label -H 'content-type: application/json' \
  -d '{"crop_ids":["050ad3a2...","05ad8ec2..."],"class_id":0,"label_source":"human"}'

{"updated":28,"updated_ids":["050ad3a212d9be5304089683dc26c6a6", ...]}

Those 28 are now human-validated. No automated writer, the VLM included, overwrites them. In Cropwright, open the cluster, filter to the crops and use Label all matching….

5. Run the VLM for one thing only. Without an endpoint the project refuses, which is the point:

An assist run plans first (dry_run is the default) and writes nothing:

# not run  (the refusal is code-verified by tests/curation/test_assist_run_classify.py)
curl -s -X POST "$API/assist/runs" -H 'content-type: application/json' \
  -d '{"target":{"kind":"cluster","cluster_id":10000},"actions":["classify"]}'

409 {"detail":{"error":"vlm_not_configured","message":"this project's VLM is off or not configured; activate an endpoint first"}}

Name the endpoint for this run, then start it. The project's own setting stays off:

# not run
curl -s -X POST "$API/assist/runs" -H 'content-type: application/json' \
  -d '{"target":{"kind":"cluster","cluster_id":10000},"actions":["classify"],"vlm":"env","dry_run":false}'
curl -s "$API/assist/runs/RUN_ID"

The run sends the members of cluster 10000 that are not in the frozen holdout and not ignored, including the 28 you named: they are sent for a suggestion, and their class is never written (locked_suggest_only in the result). Other ways to pick "one thing", all # not run and all taking vlm=<name>:

Scope	Request
One cluster, one sub-cluster, a sample of either	POST $API/assist/runs with target {"kind":"cluster","cluster_id":10000,"cluster_subid":"10000b"} and sample {"mode":"representative"}
Chosen crops, or one crop	POST $API/assist/runs with target {"kind":"crops","crop_ids":[...]}
A class, a confidence band or a size band	POST $API/pipeline/auto_label/start?run_vlm=true&vlm=env&class_id=3&conf_max=0.5 (the same item filter as the crop list: class_name, exclude_class_name, conf_min, conf_max, min_area, max_area, max_rank, origin, embedding_state, review_status)
A budget	add max_vlm_crops=200 to any auto_label/start
Only representatives	vlm_scope=representatives on auto_label/start, or the project's VLM policy

A run over a cluster or sub-cluster returns a proposal: the class most sampled answers agree on, the vote distribution, the new-class slugs the VLM proposed and a summary of the attributes the pack declares. Read it from the run, and apply it only on purpose:

# not run  (code-verified by tests/curation/test_assist_proposal.py)
curl -s "$API/assist/runs?cluster_id=10000&cluster_subid=10000b"
curl -s -X POST "$API/assist/runs/RUN_ID/apply" -H 'content-type: application/json' \
  -d '{"class_name":"spoiler","dry_run":true}'

dry_run counts the members that would be labelled and the locked ones left alone; send "dry_run": false to label them as a human would. After a re-refine the run's scope_now.changed is true, and apply refuses with scope_changed until you run again (or pass allow_changed_scope).

The sample modes, attributes and segmenter verification are described in Selective VLM and segmenter assist; in Cropwright it is the Assist… button on a cluster page. To make the VLM the project's own default instead, activate it (mode (c), step 3).

6. Check what the run did. A VLM run that could label nothing now fails instead of reporting completed with predicted: 0. Found by this run: the auto-label VLM stage swallowed an unreachable endpoint and reported success. The stage now reports failed (requests that failed) and unreadable (crops whose image the worker could not read) next to predicted, and the job fails with those counts when every selected crop failed:

# not run  (code-verified by tests/curation/test_vlm_stage_nothing_labeled.py)
status: failed
error: the VLM stage labeled nothing: 134 request(s) failed and 0 crop image(s) could
       not be read, of 134 selected. Check that the VLM endpoint is reachable and the
       source images are mounted in the worker.

Labels for parts the detector does not have (a wheel, a spoiler). Two routes besides naming clusters by hand. A region profile makes the segmenter propose the part inside each detected car, text-prompted (wheel), with the VLM verifying each box: see Worked example: wheels on cars and Region profiles. The VLM can also label crops against an open-vocabulary prompt set: see Open vocabulary. Both call the VLM, so activate one for the project first.

(d) Iterative cluster-sampled auto-labeling (planned)

:::info Planned This mode is not shipped. There is no route, flag or setting for it, and no design document yet. (issue #286), related to #97 and #218.

The idea is a loop: sample crops from each cluster and sub-cluster, label them with the VLM, measure how well the labels agree, propagate the label to the whole cluster when agreement passes a gate, and split the cluster and repeat when it does not. The parts exist as single steps today: the seeded, stratified sample, the proposal with its agreement gate and the explicit apply of an assist run (step 5 of the recipe above). The loop that drives them is what is not shipped. :::

What exists today, and can be combined by hand:

Piece	Where
Label only a few crops per cluster	VLM policy scope=representatives with per_cluster (mode (c), step 4)
Promote whole clusters of classified items, behind the audit gate	POST /curation/projects/{project}/clusters/auto_promote
Split a cluster into sub-clusters	POST /curation/projects/{project}/clusters/refine/{cluster_id}

A cluster page (public sample project)

Each step is manual and one-shot. Nothing measures agreement or repeats.

Troubleshooting

• A worker is up but nothing happens. Check Health and stall reasons and Workers and heartbeats.
• 409 vlm_not_configured. The project's VLM is off or no endpoint exists. Activate one (mode (c), step 3).
• The VLM worker labels nothing. The items have no pe_embedding (policy lazy), or the VLM policy scope excludes them, or a daily budget is used up.
• Clusters stopped refreshing. See mode (b), step 4.