VLM selection

The vision-language model (VLM) that verifies regions, reads text and proposes labels is a runtime choice. Endpoints are registered once for the whole deployment, and each project activates one of them. Any OpenAI-compatible /v1/chat/completions server with vision input works: a vLLM container in the stack, a server on the host, a server elsewhere on the network, or a hosted API.

API=http://localhost:4603 and project cars are assumed.

Concepts

• Endpoint. A named, versioned record: base_url, model and request limits. The API key is never stored. An endpoint only holds api_key_ref, a pointer to where the key lives.
• Built-in env endpoint. Built from OP_VLM_URL, OP_VLM_MODEL and OP_VLM_API_KEY. It is read-only. Clone it to get an editable copy. It is an endpoint a project opts into like any other: a project calls no VLM until it activates one, so OP_VLM_URL alone makes curation-vlm-worker and the auto-label VLM stage do nothing for a project. Activate it with POST .../vlm/endpoints/env/activate; the active endpoint then shows "source": "env". A run can still name any endpoint without changing the project's setting (vlm=<name>, see below).
• Off. What a new project starts in: active.name is null and source is off. POST .../vlm/endpoints/deactivate returns an active project to it. A VLM call that names no endpoint answers 409 vlm_not_configured.
• Projects from before 0.6.0. They used to inherit env without activating it. The first start of the new API pins each one to what it inherited (env if OP_VLM_URL was set, else off), so nothing changes for them. They show "source": "env" with an activated_at from then on.
• Key references. secret:<slug> names a file <slug> in the secrets directory (OP_VLM_SECRETS_DIR, a read-only mount of secrets/vlm). The reference env:OP_VLM_API_KEY is valid only on the built-in endpoint. Write a secret with openprocessor vlm key set <slug>, never through the API.
• Locality. The API classifies where base_url points: compose, host, private, external or unknown. An unresolvable host counts as external. Endpoints outside the deployment send crops to a third party.
• External policy. OP_VLM_EXTERNAL_POLICY is ack (default) or deny. With ack, an external endpoint needs allow_external: true when registered and acknowledge_external: true when activated. With deny it is refused outright.
• URL policy. Some addresses are always refused: this stack's own non-VLM services, link-local and cloud-metadata addresses. Neither flag bypasses this.
• Probe. A test call with synthetic images. It records whether the model is listed, vision works, JSON mode works, the multi-image limit holds, and the context length.
• Local catalog. A list of models the in-compose vLLM can serve, with VRAM needs and a fits flag for the GPU. It is shipped in examples/vlm/catalog.tsv.
• Activation axis. The active VLM is one of three activation axes (prompt_pack, detection_profile, vlm). Activation passes a gate that checks the VLM against the active pack and profile.

Routes

Global registry (not project scoped):

Call	Purpose
GET /curation/vlm/endpoints	List endpoints, usage, policy and secret refs
POST /curation/vlm/endpoints	Register an endpoint
GET /curation/vlm/endpoints/schema	Field specs for a form
POST /curation/vlm/endpoints/validate	Validate a draft without saving
GET /curation/vlm/endpoints/{name}	One endpoint with its last probe
PUT /curation/vlm/endpoints/{name}	Save a new revision
DELETE /curation/vlm/endpoints/{name}	Delete (refused while in use)
POST /curation/vlm/endpoints/{name}/clone	Copy to a new name
POST /curation/vlm/endpoints/{name}/probe	Probe and record the result
GET /curation/vlm/endpoints/{name}/revisions	Revision history
GET /curation/vlm/endpoints/{name}/revisions/{revision}	One past revision
GET /curation/vlm/catalog	Local catalog with fits, serving, desired
GET /curation/vlm/local	Local VLM status
POST /curation/vlm/local/select	Record the desired local model
DELETE /curation/vlm/local/select	Clear that request

Per project:

Call	Purpose
GET /curation/projects/{project}/vlm/endpoints/active	Active endpoint and what workers applied
POST /curation/projects/{project}/vlm/endpoints/{name}/activate	Activate
POST /curation/projects/{project}/vlm/endpoints/deactivate	VLM off for this project
POST /curation/projects/{project}/vlm/endpoints/active/rollback	Back to the previous endpoint

Register an endpoint

Draft first. validate never writes and always answers 200. With probe=true it also contacts the endpoint.

curl -s -X POST "$API/curation/vlm/endpoints/validate?probe=true" \
  -H 'Content-Type: application/json' \
  -d '{
        "body": {
          "base_url": "http://host.docker.internal:8012/v1",
          "model": "my-vision-model",
          "api_key_ref": "secret:my_server",
          "max_images_per_call": 8,
          "open_images_per_call": 3,
          "timeout_s": 240,
          "requests_per_second": 500,
          "json_mode": "auto",
          "allow_external": false,
          "catalog_id": null
        }
      }'

The response has validation (ok, errors, warnings, force_allowed), locality, sends_images_externally and probe.

Body fields:

Field	Default	Notes
base_url	required	http or https, no credentials, query or fragment in the URL
model	required	Model id the server expects
api_key_ref	null	secret:<slug> (slug is 1 to 64 of a-z0-9_)
max_images_per_call	8	1 to 64. Must not exceed the server's own per-prompt image limit
open_images_per_call	3	1 to 64. Chunk size for open-vocabulary labeling, clamped to the cap
timeout_s	240	5 to 900
requests_per_second	500	0.1 to 10000
json_mode	auto	auto, on or off
allow_external	false	Required for endpoints outside the deployment
catalog_id	null	Links the endpoint to a local catalog entry

Names match ^[a-z0-9][a-z0-9_.-]{1,63}$. These are reserved: env, off, none, default, local, active, schema, validate, deactivate. A name starting with an underscore is also reserved.

Register it:

curl -s -X POST "$API/curation/vlm/endpoints" \
  -H 'Content-Type: application/json' \
  -d '{
        "name": "host-vlm",
        "description": "vLLM on the host",
        "body": {"base_url": "http://host.docker.internal:8012/v1", "model": "my-vision-model"}
      }'

The answer is 201 with the endpoint document: body, revision, etag, locality, sends_images_externally, warning, last_probe, validation, active_in (projects running it).

Edit with the revision you read. This creates a new revision and changes nothing a project runs until it is activated again:

curl -s -X PUT "$API/curation/vlm/endpoints/host-vlm" \
  -H 'Content-Type: application/json' \
  -d '{"expected_revision": 1, "description": "vLLM on the host",
       "body": {"base_url": "http://host.docker.internal:8012/v1",
                "model": "my-vision-model", "max_images_per_call": 4}}'

Clone, list revisions, and delete:

curl -s -X POST "$API/curation/vlm/endpoints/env/clone" \
  -H 'Content-Type: application/json' -d '{"new_name": "env-copy"}'
curl -s "$API/curation/vlm/endpoints/host-vlm/revisions"
curl -s -X DELETE "$API/curation/vlm/endpoints/host-vlm?expected_revision=2"

Clone accepts an optional revision and description. Cloning the built-in endpoint drops its env:OP_VLM_API_KEY reference so a copy cannot send the deployment key elsewhere.

Probe

curl -s -X POST "$API/curation/vlm/endpoints/host-vlm/probe"

The response has ok, latency_ms, model_listed, models_listed, root, max_model_len, vision_ok, json_mode_supported, reasoning_channel, max_images_ok, image_tokens and issues. The result is stored with the endpoint and does not change its revision. At most a few probes run at once; extra calls get 429 probe_busy.

Activate for a project

curl -s -X POST "$API/curation/projects/cars/vlm/endpoints/host-vlm/activate" \
  -H 'Content-Type: application/json' \
  -d '{"revision": null, "acknowledge_external": false, "force": false, "expected_active": null}'

revision null means the latest. expected_active is {"name": ..., "revision": ...} and makes the call fail with 409 if the project's active endpoint changed in the meantime. force bypasses errors the gate marks as bypassable, such as an endpoint that has not been probed. The response (also returned by the other activation calls) has:

Field	Content
active	{name, revision}; name null means off
source	stored, env or off
previous	The endpoint a rollback would return to
activated_at, config_revision	When, and the config revision
applied	One entry per worker: which pack, profile and VLM it runs, and lagging
validation	The gate report

Workers swap to a newly activated endpoint at a quiet point, so applied can lag briefly.

Turn the VLM off for a project

deactivate stores the explicit off state. After it, a call that needs the project's VLM and does not name an endpoint is refused with 409 vlm_not_configured, and the VLM worker idles for that project. It does not touch the registry or other projects. To turn it back on, activate an endpoint (env for the built-in one).

To keep the VLM available for explicit runs but stop the automated writers, set the VLM policy to scope: off instead (see Scope and budget). When to use which, and the other pipeline settings, are in Pipeline modes.

curl -s "$API/curation/projects/cars/vlm/endpoints/active"

curl -s -X POST "$API/curation/projects/cars/vlm/endpoints/deactivate" \
  -H 'Content-Type: application/json' -d '{"expected_active": null}'

curl -s -X POST "$API/curation/projects/cars/vlm/endpoints/active/rollback" \
  -H 'Content-Type: application/json' -d '{"expected_active": null}'

Several labeling routes (POST /curation/projects/{project}/vlm/label_batch, POST /curation/projects/{project}/pipeline/auto_label and others) accept a per-run vlm query parameter, a name or name@revision. It overrides the project default for that run only and is never stored.

Local models

When the stack runs its own vLLM container, the catalog lists what it can serve.

curl -s "$API/curation/vlm/catalog"
curl -s "$API/curation/vlm/local"

Each catalog entry has id, hf_repo, license, gated, vram_gb, context_max, max_images, status (tested or to_verify), fits, serving and desired. The local block has configured, endpoint, served, desired, restart_required, can_restart_from_api and gpu_total_gb.

tested entries are verified with the prompt contract; to_verify entries are served with the label "Unverified (experimental)". They stay in the catalog but are never auto-selected: the installer picks only a tested entry (the default is the first tested entry that fits), and an unverified one runs only when you name it with --vlm-model-id, with a warning. After choosing one, run openprocessor vlm probe and read the pairing warnings.

The API cannot restart containers. Selecting a model records the request; the host applies it:

curl -s -X POST "$API/curation/vlm/local/select" \
  -H 'Content-Type: application/json' \
  -d '{"catalog_id": "<id from the catalog>", "force": false}'

The answer is 202 with restart_required: true and desired.command. Then run the command on the host (see the CLI below). Clear the request with DELETE /curation/vlm/local/select.

CLI

Run in the install directory (./openprocessor), or in a checkout as ./scripts/openprocessor.sh.

Command	Action
openprocessor vlm list	Print the catalog: id, licence, VRAM, status
openprocessor vlm status	Show the model in .env, the endpoint, what is served now and what is desired
openprocessor vlm use <id> [--force] [--yes]	Switch the in-compose VLM to a catalog entry
openprocessor vlm apply [--force] [--yes]	Apply the model requested through the API
openprocessor vlm probe	Probe the local endpoint named by OP_LOCAL_VLM_ENDPOINT
openprocessor vlm key set <slug>	Read a key from the terminal into secrets/vlm/<slug> (mode 600)

vlm use checks that the model fits the GPU (override with --force), that a gated model's Hugging Face token can see it, and that no training run holds the GPUs. It pauses the workers, rewrites .env after saving a backup, recreates the vlm container, waits until the new model is served, probes the endpoint, clears the desired request and unpauses. On any failure it restores the previous .env and container. --yes skips the confirmation prompt. Both commands need OP_LOCAL_VLM_ENDPOINT to be set, which the installer does when it sets up an in-compose VLM.

Enabling the in-compose VLM on an existing install

An install without the vlm tier has no OP_LOCAL_VLM_ENDPOINT, and vlm use / vlm probe stop with a message that points here. Either:

• re-run the installer in the install directory with the tier added: ./setup-openprocessor.sh --tiers core,curation,vlm (list your installed tiers plus vlm). It picks a catalog model that fits your GPU and writes VLM_CATALOG_ID, VLM_MODEL, OP_VLM_URL and OP_LOCAL_VLM_ENDPOINT=env; or
• by hand: set OP_LOCAL_VLM_ENDPOINT=env in the install's .env, keep vlm in COMPOSE_PROFILES, and start the container with docker compose --profile vlm up -d vlm (on a checkout, openprocessor update pulls and rebuilds first). Set the VLM_* model values yourself in this case, or run openprocessor vlm use <id> once the container is up.

vlm key set needs a terminal. The slug matches [a-z0-9][a-z0-9_-]{0,63} and the endpoint refers to it as secret:<slug>.

Errors

Status	error	When
404	not_found	Unknown endpoint name
404	unknown_revision	The endpoint has no such revision
403	read_only	Edit or delete of the built-in env endpoint
409	name_conflict	Name already used
409	revision_conflict	expected_revision is stale
409	in_use	Delete while a project runs it; projects lists them
409	active_conflict	Another writer activated first; current is returned
409	no_previous, previous_deleted	Nothing to roll back to
409	no_local_vlm	Local select on a stack without an in-compose VLM
422	validation_failed	Blocking validation errors; report lists them
422	vlm_external_not_acknowledged	External endpoint without acknowledgement; activate_via says where to give it
422	unknown_catalog_id	Not in the catalog; valid_ids is returned
422	vlm_catalog_does_not_fit	Model needs more VRAM than the card has
429	probe_busy	Too many probes running

Validation issue codes start with vlm_. Examples: vlm_url_invalid, vlm_url_denied_internal_service, vlm_url_denied_address, vlm_api_key_unresolved, vlm_external_denied, vlm_unreachable, vlm_timeout, vlm_auth_failed, vlm_model_not_listed, vlm_no_vision, vlm_json_mode_unsupported, vlm_max_images_exceeds_server, vlm_not_probed, vlm_context_too_small, vlm_multi_box_unverified and vlm_reads_text_unverified.

Scope and budget

Which VLM is used and which crops it labels are separate choices. Two automated writers label crops: the always-on curation-vlm-worker and the auto_label VLM stage (run_vlm=true). By default both label every eligible crop of the project, which on a large project is a long and, for a hosted endpoint, paid run. A per-project scope policy limits them. VLM labels stay unvalidated suggestions (class_source: "vlm"): they are excluded from export and from the test holdout until a human validates them.

curl -s "$API/curation/projects/cars/vlm/policy"
curl -s -X PUT "$API/curation/projects/cars/vlm/policy" \
  -H 'Content-Type: application/json' \
  -d '{"expected_revision": 0, "scope": "representatives", "per_cluster": 5,
       "max_crops_per_day": 2000}'

Field	Default	Notes
scope	all	all, uncertain, representatives or off
conf_max	0.80	uncertain only: crops with a detector confidence below this, or none recorded
per_cluster	5	representatives only: crops nearest each cluster centre; every unassigned crop is also kept. 1 to 100
max_crops_per_day	0	VLM class attempts per UTC day, answered or empty. 0 is unlimited
region_verify	vlm	How the region worker verifies boxes: vlm (the active VLM, today's behaviour) or none (accepted without a VLM). Not a crop scope. A per-item region_run_verify from an assist segment run wins over it
sample_frac	1.0	Above 0 up to 1. Keeps a stable hash-chosen fraction of the crops (the same crops on every poll)

off stops both automated writers for the project. The write needs the expected_revision you read and answers 409 revision_conflict when it is stale. The worker re-reads the policy about every 30 seconds; the next auto_label run uses it at once, and a run can override the scope with vlm_scope=<scope>. The policy is cloned with the project.

sample_frac hashes the crop id, so the sample is only as uniform as the ids: hash-like ids give a proportional sample, sequential or patterned ids may not.

POST /pipeline/auto_label/start takes vlm_attributes=true to store the pack's item attributes in the same run (see the pipeline modes guide, mode (c)).

An explicit request is never limited by the policy: an assist run (POST /assist/runs) over a cluster, a sub-cluster or named crops, an auto_label run with cluster_id, and labelling named crops. A human-validated crop is never selected under any scope. When a run selects nothing because of the policy, its VLM stage reports why (skipped with the reason).

Assist runs

POST /curation/projects/{project}/assist/runs sends a crop, a selection, a cluster, a sub-cluster or a filter to the VLM, optionally as a seeded representative sample, with vlm=<name> and prompt_pack for that run only. It is a dry run by default. It needs an active endpoint or a vlm name (409 vlm_not_configured otherwise). It is not limited by the VLM policy, and ignores the daily budget. See Selective VLM and segmenter assist for the sample modes, attributes and segmenter verification, Pipeline modes for the recipe and the API contract (docs/design/curation_api_contract.md) for the request and result.

Cropwright

A project with its VLM deactivated: no active endpoint (public sample project)

The endpoint registry (locality, status, last probe) and the model choices table (captured from Cropwright against the public COCO sample projects)

See also

• Prompt packs
• Region profiles
• Pipeline modes
• Multi-box regions
• Settings and configuration
• Use your own domain