Workers and heartbeats

The synchronous curation API (browse, label, cluster, export) works with no workers at all. The workers keep datasets moving without a person driving every step. They run the same openprocessor image as the API with a different entrypoint, so none of them needs an image rebuild.

One set of workers serves every project. A worker is not bound to one project: each cycle it lists the active projects, skips paused ones, and binds each project only around that project's own work, so a project's queries and writes only ever touch its own indexes. --project <slug> narrows a worker to one project. The detection worker splits each fetch between projects round-robin (equal quota, leftover capacity to projects that filled theirs), caps each project's in-flight items so no project starves the others, and backs idle projects off from 5 s to 60 s.

./openprocessor curation up                    # or: make curation-up
make curation-status                           # which worker containers are up
make curation-logs                             # follow all worker logs
make curation-down

The workers

Service	Entrypoint	What it does
curation-detection-worker	python -m scripts.curation.worker --continuous	Pulls pending_detection / pending_verification items and runs each project's active region profile's cascade: detector, optional segmenter leg, VLM verification, per-box text reading. Idles for a project with no active profile. Follows a project's config at quiesce points: a newly activated profile, prompt pack or VLM endpoint takes effect once the queues drain.
curation-vlm-worker	python -m scripts.curation.vlm_worker --continuous	Continuously asks each project's active VLM for class suggestions on unvalidated items that have a pe_embedding, on its own schedule.
curation-auto-label-worker	scripts.curation.auto_label_worker	Claims jobs queued by POST /curation/projects/{project}/pipeline/auto_label/start from the shared /jobs volume and runs them one at a time across projects, oldest first: clustering, and VLM labeling only when the job says run_vlm=true. Can also trigger a centroid retrain when a project's pool has grown.
curation-cluster-refresh	scripts/curation/cluster_refresh_daemon.py	Polls each project's items index (every 120 s by default). Once a project has grown by 200 items it calls that project's POST /curation/projects/{project}/clusters/auto_promote, then queues the auto-label job (no VLM stage) as the centroid retrain. The retrain is on by default (--no-auto-label turns it off). The flags are command-line only; see Pipeline modes.
curation-evaluator	its own image	Watches each project's bakeoff_jobs directory and runs model comparisons. It has no restart policy and its own evaluator profile, so --profile curation up -d and ./openprocessor curation up do not start it. Start it on purpose with ./openprocessor curation up curation-evaluator or docker compose run --rm curation-evaluator. It needs the NVIDIA runtime and the card named by EVALUATOR_GPU_ID.

api, curation-detection-worker and curation-auto-label-worker must mount the same source directory at the same container path (OP_SOURCE_ROOT_HOST → /data/source, read-only). If a worker lacks it, ingest succeeds but that worker's reads fail with detection_failed / image_unavailable. After changing OP_SOURCE_ROOT_HOST, recreate all three with ./openprocessor start. See Ingest a large folder.

Which worker needs what

Worker	Needs Triton	Needs a VLM	Needs a GPU of its own
curation-detection-worker	Yes (depends_on: triton-server), for the region detector	Only if the region profile uses VLM verification	No
curation-vlm-worker	No	Yes. It idles when the project has no VLM.	No
curation-auto-label-worker	Only for embed_missing=true jobs, which call the PE encoder on triton-server (no compose dependency)	Only for jobs with run_vlm=true	No
curation-cluster-refresh	No	No	No
curation-evaluator	No	No	Yes (EVALUATOR_GPU_ID)

The detection worker verifies regions (it sends numbered overlays to the VLM). The VLM worker labels classes (it suggests a class for each unvalidated item). A project whose VLM is off does neither VLM step.

Which workers to run

Not every mode needs every worker. Pipeline modes says which to start, and how to stop the two clustering triggers (the cluster-refresh daemon and the auto-label worker's idle centroid retrain). Turn both off for one project with the ingest policy field clustering.automatic=false, or for all projects with OP_IVF_RETRAIN_CHECK_S=0 and by not running the daemon.

Heartbeats instead of pgrep

A process can look alive to pgrep while its event loop is deadlocked. Every worker therefore runs a heartbeat task next to its real work (src/services/curation/worker_liveness.py) that writes {ts, pid, tasks} to $OP_HEARTBEAT_DIR/<name>.json on a fixed interval, including while idle. tasks is a liveness map of the worker's own sub-tasks (producer, consumers, writer); a sub-task that has died fails the check even when the file is fresh.

The compose healthcheck runs, inside the same container:

python /app/src/services/curation/worker_liveness.py check detection_worker --max-age 120

exiting 0 (healthy) or 1 (unhealthy) and printing the reason. The names are detection_worker, vlm_worker, auto_label_worker and cluster_refresh. OP_HEARTBEAT_DIR defaults to a container-local path, so no volume is needed.

docker compose ps shows the result as healthy / unhealthy; a worker that is unhealthy is stuck, not merely idle.

The trainer has a separate heartbeat, .trainer_capabilities.json on the shared /jobs volume, which the API reads for training preflight (see Health and stall reasons).

Pausing

There are two independent pauses.

• Per project. POST /curation/projects/{project}/pause makes every worker skip that project's work (the flag is <project state dir>/pipeline_paused.flag); POST .../resume clears it and GET .../pause reports paused and paused_by. Other projects keep running.
• Global. The VLM worker idles while its pause sentinel file exists (OP_PAUSE_SENTINEL / OP_WORKER_PAUSE_SENTINEL, default /var/lib/openprocessor/vlm_worker/pause.sentinel). The GPU arbiter creates it before a single-GPU training run so the worker doesn't compete with the trainer, and openprocessor vlm use creates it while it swaps the model. For a multi-GPU claim the arbiter stops configured containers instead (OP_GPU_ARBITER_CONTAINERS). You can create and remove the file by hand to pause and resume everything without stopping the container.

Multi-process state

api runs many uvicorn worker processes, and any of them can answer a given request. Every piece of job state that must look the same from all of them lives in files on the shared /jobs volume, not in process memory:

Job	Directory (default)
Training	/jobs (OP_TRAIN_JOBS_DIR), nested per project
Auto-label	/jobs/auto_label, nested per project
Probe	/jobs/probe
Scores	/jobs/scores
Diverse selection	/jobs/select
Visualization	/jobs/viz
Region drain streak	/jobs/region_drain
Reprocess	/jobs/reprocess
Dataset import	/jobs/imports
Combine	/jobs/combine

Events work the same way: with OP_EVENT_BUS=file (the default) every uvicorn process tails one shared, rotated JSONL log, so a client on GET /curation/events sees events published by any process, including the detection worker's POST /curation/projects/{project}/events/publish calls. Events are stamped with their project: a project's stream delivers only its own, and GET /curation/events delivers the global ones.

Worker metrics

The detection worker serves its own Prometheus /metrics on OP_REGION_WORKER_METRICS_PORT. Stage concurrency is set by OP_REGION_WORKER_VLM_CONCURRENCY and OP_REGION_WORKER_VLM_VISIBLE_CONCURRENCY.