Monitoring: Prometheus, Grafana and Loki

Monitoring is opt-in. The monitoring compose profile starts seven services next to the core stack; none of them changes how inference runs.

docker compose --profile monitoring up -d
# or
make up-monitoring

Service	Port (default)	Env override	What it does
prometheus	4604	PROMETHEUS_PORT	Scrapes metrics and evaluates the alert rules in monitoring/alerts/.
grafana	4605	GRAFANA_PORT	Dashboards over Prometheus and Loki, provisioned from monitoring/.
loki	4606	LOKI_PORT	Stores container logs.
alloy	none		Grafana Alloy: discovers the stack's containers on the Docker socket and ships their logs to Loki.
node-exporter	none		Host CPU, memory and load metrics.
dcgm-exporter	none		NVIDIA DCGM GPU metrics (utilization, memory, power, temperature). It reads every GPU on the host.
opensearch-dashboards	4608	OPENSEARCH_DASHBOARDS_PORT	Browse the OpenSearch indexes (see below).

Change the Grafana admin password

Grafana's admin password is set by GF_SECURITY_ADMIN_PASSWORD in docker-compose.yml. Override it before anyone else can reach port 4605, and keep every monitoring port off the public internet: Prometheus, Loki and OpenSearch Dashboards run without authentication, like the API.

Prometheus targets

monitoring/prometheus.yml scrapes these jobs:

Job	Target	Metrics
triton	triton-server:8002	Triton's nv_* series: per-model request, queue and compute time, batch execution, pending requests, GPU utilization and memory.
api	api:8000	The API's own /metrics.
node	node-exporter:9100	Host CPU, memory, load.
dcgm	dcgm-exporter:9400	Per-GPU DCGM_FI_DEV_* series.
loki	loki:3100	Loki's own health.
curation-detection-worker	curation-detection-worker:4609	The detection worker's own op_* series (segmenter calls, stage timings, items written, VLM calls).
curation-vlm-worker	curation-vlm-worker:4610	The VLM worker's op_* series (VLM tokens and latency, heartbeat).
curation-auto-label-worker	curation-auto-label-worker:4611	The auto-label worker's heartbeat series.

The three worker jobs belong to the curation compose profile; each target shows as down whenever its worker is not running. Override the ports with OP_REGION_WORKER_METRICS_PORT, OP_VLM_WORKER_METRICS_PORT and OP_AUTO_LABEL_WORKER_METRICS_PORT (0 turns the endpoint off; if you change a port, change the matching job in prometheus.yml).

Prometheus target health on a public-sample-data stack: all five jobs up

Open http://localhost:4604/targets to check them. A target that stays down usually means its service is not running (for example, dcgm on a host without the NVIDIA container toolkit).

Alert rules

monitoring/alerts/triton-alerts.yml defines the rules Prometheus evaluates: HighInferenceLatency and CriticalInferenceLatency, HighInferenceFailureRate, HighGPUUtilization (above 95% for 5 minutes), HighGPUMemoryUsage (above 90%), HighQueueDepth (more than 50 pending requests) and TritonServerDown. They show on Prometheus's Alerts page (http://localhost:4604/alerts).

monitoring/grafana-alerting.yml is provisioned into Grafana as a second, separate rule set under the OpenProcessor Alerts folder: API error rate and slow responses, Triton down, latency and queue, GPU utilization, memory and temperature (DCGM), and host memory, CPU and load. They show at http://localhost:4605/alerting/list.

Neither set notifies anyone out of the box. Prometheus has no Alertmanager. Grafana's two contact points are named unconfigured-default and unconfigured-critical and post to http://localhost:9999/..., which is the Grafana container itself, where nothing listens. A firing alert shows in Grafana and its delivery fails with a logged error; nobody is told. To be notified, replace the receivers in grafana-alerting.yml with your own webhook, Slack or email and restart Grafana (the file is provisioned read-only, so the Grafana UI cannot change them).

Grafana dashboards

Grafana is provisioned from monitoring/grafana-datasources.yml (Prometheus as the default data source, plus Loki) and monitoring/grafana-dashboards.yml, which loads every JSON file in monitoring/dashboards/. Open Grafana (port 4605) and use the OpenProcessor dashboards link at the top of any dashboard, or search by the openprocessor tag. Every dashboard has a data-source variable, a "Firing alerts" annotation toggle, and short row descriptions.

Dashboard	Use it to	Metrics
Overview (overview.json)	See service health at a glance: scrape targets, API rate/5xx/p95, Triton failures, firing alerts, queue backlog and oldest item, worker liveness, OpenSearch shards and storage per project, GPU and host load	up, http_request_duration_seconds_*, nv_inference_*, ALERTS, DCGM_FI_DEV_*, node_*, op_queue_*, op_worker_*, op_opensearch_*
Ingest pipeline (ingest-pipeline.json)	Find the slow ingest stage; watch decode, prefetch, thumbnail and crop caches, write conflicts; per-project images, items, failures and embedding backlog	op_pipeline_stage_seconds_*, op_pipeline_stage_bytes_total, op_source_image_*, op_thumbnail_cache_*, op_shm_crop_cache_*, op_open_vocab_items_written_total, op_ingest_occ_final_conflict_total, op_ingest_*, op_embedding_state_items
API and labeling (api-curation.json)	Per-route traffic, errors, latency; VLM call mix, parse failures, tokens and latency; segmenter hit ratio and latency; detection worker items	http_request_duration_seconds_*, op_vlm_*, op_region_segmenter_request*, op_detection_worker_items_total
OpenProcessor Triton Dashboard (triton-unified-dashboard.json)	Per-model throughput, latency, batching, queue time	nv_inference_*, nv_gpu_*, node_*
GPUs (gpu-metrics-dashboard.json)	Per-card utilization, VRAM, power, temperature, clocks, codecs, XID errors (gpu variable)	DCGM_FI_DEV_*, nv_gpu_*
Logs (logs.json)	Log volume, error and warning counts, recent errors, search (Loki; job, container, search variables)	Loki streams from Alloy

API metrics are aggregated across workers

The compose file sets PROMETHEUS_MULTIPROC_DIR on the API to a tmpfs, so /metrics sums the op_* and request metrics of every worker and Prometheus scrapes it once. The directory is empty on each container start. Counters of a recycled worker (--limit-max-requests) stay included, so totals never drop. The ingest dashboard stays flat until images are ingested.

Panels without data

The error-ratio and error-log panels are empty when nothing is failing, and the latency percentiles of idle stages are empty because there are no observations. Worker panels (segmenter, detection, VLM, worker liveness) are empty while the curation profile is not running. The segmenter container exports no metrics of its own: its requests are timed by the detection worker, and its liveness is a /health probe made by the API.

Panels only move while the API is serving requests; an idle stack shows flat lines. To add a dashboard, drop its JSON into monitoring/dashboards/ and restart Grafana.

No rule for model readiness or OpenSearch status

The shipped Triton does not export nv_model_ready_state or nv_gpu_temperature, and no OpenSearch exporter ships with the stack, so there is no rule on per-model readiness or on OpenSearch cluster status. Use GET /models/ (below) for model readiness and the GPUs dashboard (DCGM) for temperature; the Grafana temperature rule reads DCGM_FI_DEV_GPU_TEMP. An offline test checks that every metric an alert rule or dashboard panel reads is one the stack exports, so a rule on a missing series cannot come back.

Logs with Loki

Alloy (monitoring/alloy-config.alloy) discovers containers through the Docker socket, keeps this project's triton-server and api services plus the detection, VLM and auto-label workers and the segmenter, labels each stream with container, stream and job (triton, fastapi or worker), and pushes it to Loki. In Grafana's Explore view, pick the Loki data source and filter on those labels. Loki keeps its data in the loki_data volume.

Which containers ship logs

Alloy keeps only containers whose compose project label equals OP_LOG_PROJECT (set from COMPOSE_PROJECT_NAME), so a second stack on the same host does not leak into this Loki. To list the container names that are shipping, run curl -s localhost:4606/loki/api/v1/label/container/values.

Operator metrics reference

These op_* series feed the Overview, Ingest and API dashboards. Counters and histograms are emitted where the work happens (API or worker container). Snapshot gauges are sampled from OpenSearch every OP_METRICS_REFRESH_INTERVAL_S seconds (default 30, 0 disables) by the API; one API worker per interval does the queries and the gauges use multiprocess_mode='livemostrecent', so a scrape shows the latest value rather than a sum. A failed OpenSearch read leaves the previous value in place.

Metric	Type	Labels	Meaning
op_ingest_images_total	counter	project, outcome (ok, failed, skipped)	Images that finished ingest; skipped is a duplicate of an existing image.
op_ingest_items_total	counter	project	Items (crops) indexed by ingest.
op_reprocess_decode_total	counter	route (gpu, cpu_<reason>)	Photos of the embed reprocess pass while the GPU decode is usable: gpu = decoded and embedded by gpu_image_prep, cpu_small = below OP_INGEST_GPU_MIN_MEGAPIXELS, any other cpu_* = declined or failed and decoded in the API.
op_queue_depth	gauge	queue (segment, label, embed), project	Items waiting: segment = pending_detection, label = pending_verification, embed = vector deferred.
op_queue_oldest_item_age_seconds	gauge	queue	Age of the longest-waiting item in that queue, worst project.
op_embedding_state_items	gauge	project, state (pending, embedded, failed)	Items by embedding state; pending is embedding_state=deferred.
op_worker_last_heartbeat_timestamp_seconds	gauge	worker (detection, vlm, curation, segmenter)	Unix time of the worker's last liveness heartbeat (segmenter: last successful /health probe).
op_worker_up	gauge	worker (same values)	1 when every sub-task reported alive on the last heartbeat, else 0.
op_region_segmenter_requests_total	counter	outcome (hit, miss, error)	Crop-stage segmenter requests made by the detection worker.
op_region_segmenter_request_seconds	histogram	outcome	Duration of those requests.
op_detection_worker_items_total	counter	outcome (ok, skipped, failed)	Items the detection worker finished: written, skipped on an OCC conflict, or lost to a bulk-write error.
op_vlm_tokens_total	counter	direction (prompt, completion), model	Tokens from the VLM endpoint's usage block; endpoints that omit it add nothing.
op_vlm_request_seconds	histogram	model, outcome (ok, error)	Duration of VLM chat-completion requests (retries included).
op_opensearch_shards	gauge	project, index_role	Primaries x (1 + replicas) per project index role.
op_opensearch_store_bytes	gauge	project, index_role	On-disk bytes (primaries + replicas) per project index role.

Label cardinality. No label is per item, crop or image. project is the one label that scales with usage, so it is capped: at most OP_METRICS_MAX_PROJECT_LABELS (default 50) projects get their own series, chosen as the first slugs in sorted order from the project registry (every process picks the same set); the rest are folded into project="other". model is the configured VLM model id.

Not emitted. There is no ingest value for op_queue_depth because ingest is a synchronous request with no persisted backlog. op_plan_cost_units_total (proposed in the tracking issue) is not defined: OpenProcessor has no plan or cost-unit accounting for requests to count, and an always-zero series would mislead.

OpenSearch Dashboards

OpenSearch Dashboards (port 4608) is in the same profile. Its Index Management page lists the curation indexes with document counts and sizes, and Discover can browse them once you add an index pattern. Each project owns its own set, named op_prj_<slug>__<role> (images, items, classes, labels_confirmed, umap_state, configs), next to the global op_projects and op_global_configs indexes. Every project adds shards, so watch the shard count against the heap budget (see OpenSearch heap sizing).

OpenSearch Dashboards index management on a public-sample-data stack

Resource links

GET /curation/projects/{project}/settings serves resource_links, the list a frontend renders as its Resources menu. Each entry is {id, label, url, kind, status, hint, reachable}.

• kind: docs (swagger, redoc, openapi_json): url is path-relative (/docs, /redoc, /openapi.json); prefix the origin the API is served from. The pages are self-hosted (vendored bundles at /docs-assets, no CDN, no web fonts), so they work offline. A proxy in front of the API must forward /docs, /redoc, /openapi.json, /docs/oauth2-redirect and /docs-assets/ at the root, because the pages request them by absolute path.
• kind: service (grafana, prometheus, opensearch_dashboards, mlflow): the URL rule, in order:
  1. OP_GRAFANA_URL, OP_PROMETHEUS_URL, OP_DASHBOARDS_URL or OP_MLFLOW_PUBLIC_URL, served verbatim. Use these behind a reverse proxy or a DNS name.
  2. With OP_GATEWAY_SUBPATHS=true (see Gateway mode), the path-relative /grafana/, /prometheus/, /dashboards/ and /mlflow/. A port of 0 still hides the link.
  3. Otherwise <scheme>://<host the client used>:<service port>. The host comes from X-Forwarded-Host (first value) or Host, the scheme from X-Forwarded-Proto or the connection. A LAN client opening http://10.10.10.20:5184 gets http://10.10.10.20:<port>, never localhost. The ports are OP_GRAFANA_PORT (4605), OP_PROMETHEUS_PORT (4604), OP_DASHBOARDS_PORT (4608), and OP_MLFLOW_PORT (4609); 0 disables. make dev-up passes the stack's own *_PORT values.
  4. No request context, or port 0: url: null, status: not_configured.
• The MLflow run link (mlflow_run_url on train status and manifest, and mlflow_public_url on /health) follows the same rule.
• reachable means the service answers from the server: the API probes the compose-internal address (1 s timeout, cached 15 s, background task). It does not mean your browser can reach it. null = not probed yet or no URL; false = the service is not running (for example the monitoring profile is off).

For LAN clients the server must allow inbound TCP on every published service port you want linked (Grafana 4605, Prometheus 4604, Dashboards 4608, MLflow 4609, plus 4602 and 4610 for the raw metrics), and the UI ports must not be bound to 127.0.0.1, or the browser cannot connect. Set OP_UI_BIND_ADDRESS=0.0.0.0 (or a LAN IP) to open only those four UIs; it defaults to OP_BIND_ADDRESS, and DCGM, Loki, Triton metrics, OpenSearch and the model APIs stay on OP_BIND_ADDRESS. The resource-link host is derived from the request host the browser used, with the port from the published port. Security: Grafana ships admin/admin (set GF_SECURITY_ADMIN_PASSWORD before exposing it); MLflow and Prometheus have no auth. Behind a single-origin reverse proxy, set the OP_*_URL overrides to the proxied addresses instead.

Gateway mode (one port)

Instead of publishing a port per UI, publish only Cropwright's port and let its nginx proxy the UIs under sub-paths. The UIs stay on 127.0.0.1.

# .env
COMPOSE_PROFILES=cropwright,monitoring,training
OP_GATEWAY_SUBPATHS=true
GF_SECURITY_ADMIN_PASSWORD=<not admin>
OP_CROPWRIGHT_BIND_ADDRESS=0.0.0.0   # the gateway; leave OP_UI_BIND_ADDRESS unset

One switch does everything: compose starts Grafana, Prometheus and OpenSearch Dashboards serving from their sub-path, Cropwright's nginx adds the routes, and the API serves the path-relative links. MLflow keeps its own root (the gateway strips /mlflow) so the trainer's in-network tracking URI is unchanged, and the gateway sends MLflow its own service name as Host, so its allowed-hosts guard passes whatever name or address the browser used. nginx forwards Host and X-Forwarded-Host (with the port) and X-Forwarded-Proto on every proxied location; Grafana Live (/grafana/api/live/) is upgraded to a websocket.

Fail closed: with the switch on, the four UI services refuse to start if OP_UI_BIND_ADDRESS / OP_BIND_ADDRESS publish them beyond loopback, and Grafana refuses the default or an empty GF_SECURITY_ADMIN_PASSWORD. Prometheus and MLflow have no login: anyone who reaches the gateway port reaches them (see SECURITY.md). The gateway only proxies to bare compose service names (GRAFANA_UPSTREAM, PROMETHEUS_UPSTREAM, DASHBOARDS_UPSTREAM, MLFLOW_UPSTREAM override the defaults; an IP, dotted name or path is refused).

Link audit

Link	Gateway URL	Notes
Swagger UI	/docs	Proxied to the API at the root, with /docs/oauth2-redirect
ReDoc	/redoc	Assets from /docs-assets/
OpenAPI JSON	/openapi.json
Grafana	/grafana/	GF_SERVER_SERVE_FROM_SUB_PATH; websocket on /grafana/api/live/; login required
Prometheus	/prometheus/	--web.external-url and --web.route-prefix /prometheus/; no auth
OpenSearch Dashboards	/dashboards/	server.basePath and server.rewriteBasePath; no auth
MLflow	/mlflow/	Prefix stripped by the gateway; run links are /mlflow/#/experiments/<id>/runs/<run>; no auth
Triton metrics, DCGM	not routed	Stay on their own loopback ports; no resource link when OP_*_PORT is 0

make dev-up starts the dev stack; the monitoring profile is opt-in (make dev-monitoring-up adds it), as DCGM reserves every GPU and Alloy reads the Docker socket.

Model status without Grafana

For a quick check, the API reports each Triton model's state directly:

curl -s http://localhost:4603/models/ | jq -c '.models[] | {name, status, backend, max_batch_size}'

GET /models/ on a public-sample-data stack. Models listed as UNKNOWN are in the model repository but not loaded (Triton runs in explicit model-control mode)

GET /curation/projects/{project}/models/status adds the curation roles (item proposer, region detector, segmenter, one row per VLM endpoint), inference counts and average latency.