Deployment and GPU sizing
Hardware
| Minimum | Recommended | |
|---|---|---|
| GPU | NVIDIA, 8 GB+ VRAM, Ampere or newer | A100 / A6000 / RTX 4090 class, 16 GB+ |
| RAM | 16 GB | 64 GB |
| CPU | 16 cores | 48+ cores |
| Storage | NVMe SSD for image storage | |
| Software | Docker with the NVIDIA Container Toolkit |
GPU profiles
A profile (minimal, standard or full) is a set of Triton model settings. It
does not choose which models exist. scripts/setup.sh and the installer detect your
GPU and pick one from its VRAM.
| Profile | VRAM | Example GPUs | Triton instances per model | Max batch | ocr_pipeline instances | GPU decode written to .env |
|---|---|---|---|---|---|---|
minimal | 6-8 GB | RTX 3060, RTX 4060 | 1 | 16 | 1 | OP_INGEST_GPU_DECODE=0 |
standard | 12-39 GB | RTX 3080, RTX 4090, 32 GB cards | 2 | 32 | 2 | 0 |
full | 40 GB+ | A6000, A100 | 4 | 64 | 4 | 1 |
The installer picks full only when the Triton card has 40 GB or more free: the profile's
models take 17,408 MiB and its two gpu_image_prep instances take 2 x 8,704 MiB more, 34,816 MiB
in all. A 24 or 32 GB Triton card gets standard with the GPU decode off (the Triton models
alone take 12,288 MiB). You can still set GPU_PROFILE=full by hand; ./openprocessor gpu plan
warns when the card cannot hold it.
The values are in config_templates/profiles/*.json. We have not measured a throughput
figure per profile; the measured rows are in Measured performance
below and in docs/PERFORMANCE.md, and they come from one RTX A6000.
What a profile changes, and what it does not:
- It rewrites
models/<name>/config.pbtxtfor the YOLO, SCRFD, ArcFace, MobileCLIP and PaddleOCR models andocr_pipeline: instance count, batch sizes and the GPU id (gpus: [ 0 ]). These are tracked files, so switching a profile in a checkout shows up ingit status. The PE encoders are not touched. Triton reads them when it starts. - It writes
GPU_PROFILEandOP_INGEST_GPU_DECODEto.env: the installer andscripts/setup.shwhen there is no.envyet,./openprocessor profileinto the existing one. The installer usesGPU_PROFILEfor its model setup, and the API readsOP_INGEST_GPU_DECODE. - It does not change the model list. The Triton load list is fixed in
docker-compose.yml. A model that was never exported stays unloaded and the server keeps running (--exit-on-error=false).scripts/setup.sh --profile=minimalexports the essential models only, with no OCR. - It does not set the API worker count or
shm_size. The API runs 32 uvicorn workers and every service hasshm_size: 8gindocker-compose.yml, whatever the profile says. To change either, put the change indocker-compose.local.yml, which./openprocessorandmakeload after the stock files (see Ingest a large folder).
Switching profile later
./openprocessor profile <name> needs a checkout. It regenerates the Triton configs and sets
GPU_PROFILE and OP_INGEST_GPU_DECODE in the existing .env, leaving the other lines alone:
./openprocessor profile standard # rewrites models/*/config.pbtxt, updates .env
./openprocessor restart triton-server
./openprocessor gpu plan # shows the GPU decode state and the VRAM it needs
Leave off the optional second argument (a GPU id): it writes that id into every
config.pbtxt.
A checkout also needs COMPOSE_PROJECT_NAME in .env before ./openprocessor will
run; a .env made by scripts/setup.sh or copied from env.template does not set it.
Add COMPOSE_PROJECT_NAME=openprocessor (or your project name).
Core Triton loadout
Measured on a 48 GB card by unloading each model in turn. The previous
default instance counts came to about 25.2 GB; the shipped
config.pbtxt defaults now use count: 1 for the three over-provisioned
models, bringing the core loadout to roughly 16-17 GB.
| Model | Instances (old, now default) | Approx. VRAM | Notes |
|---|---|---|---|
scrfd_10g_bnkps | 4, now 1 | 6.8 GB, now ~1.7 GB | Face detection |
mobileclip2_s2_image_encoder | 2, now 1 | 3.4 GB, now ~1.7 GB | FP32 build |
arcface_w600k_r50 | 4, now 1 | 2.75 GB, now ~0.7 GB | Face embeddings |
mobileclip2_s2_text_encoder | 1 | 0.7 GB | |
| YOLO11 / PE / OCR (remaining core models) | as shipped | ~11.65 GB combined | No per-model breakdown measured yet |
pe_text_encoder | 1, KIND_CPU | 0 GB | Host RAM only |
Raise instance counts in config.pbtxt (or re-export) on a card with
headroom; hot-path models benefit most. A curation-only deployment that
doesn't need faces can unload those models entirely.
Segmenter
The optional segmenter (--profile segmenter) is the largest curation
VRAM item after Triton.
SEGMENTER_INSTANCES | SEGMENTER_SHARED_WEIGHTS | Approx. VRAM | Notes |
|---|---|---|---|
| 1 | 0 or 1 | ~2 GB | Fits a 12 GB card next to Triton |
| 2 | 1 (recommended) | ~4 GB | One weight copy, per-instance activations |
| 2 | 0 | ~4-6 GB | Each instance loads its own weights; no benefit |
| 4 | 1 | ~7-8 GB | 48 GB-class cards, with room for training too |
Rule of thumb: 1-2 instances on a 12-24 GB card, 2-4 on a 48 GB card,
shared weights either way. Measure with nvidia-smi once the service's
/health reports "loaded":true; these are starting points, and actual
use depends on image resolution and batch size.
VLM
The in-compose vlm service (vLLM) claims VLM_GPU_MEMORY_UTILIZATION
of its GPU (default 0.4). The shipped default model measured about
23 GB in the same end-to-end run that measured the Triton loadout
above. Example budget from env.template: a 48 GB card running Triton
(~10 GB) plus the segmenter (~4 GB) plus the VLM at 0.30 (~14 GB) leaves
headroom; a 12-24 GB card should run a smaller or quantized model
instead.
The local model catalog (examples/vlm/catalog.tsv) lists each model's
VRAM need. openprocessor vlm list shows it, and GET /curation/vlm/catalog
marks which entries fit the card (VLM_GPU_TOTAL_MIB).
openprocessor vlm use <id> refuses a model that does not fit unless you
pass --force. See VLM selection.
Any OpenAI-compatible endpoint works, so a project's VLM can also live on
another host entirely: register it as an endpoint, or point OP_VLM_URL at
it.
Training
A training run holds its GPU(s) for hours, and one trainer serves every
project. curation-trainer is pinned by
OP_TRAIN_GPU_ORDER (default 0); training/bake-off jobs may only target
ids in OP_GPU_ALLOWED_IDS (empty means any). If training shares a card
with GPU-resident services, OP_GPU_ARBITER_CONTAINERS can stop and
restart named containers around a run (requires the gpu-arbiter overlay;
see Security).
GPU ids
Every GPU-using service is pinned independently, so a multi-GPU host can spread the load:
| Variable | Service |
|---|---|
TRITON_GPU_ID | triton-server |
API_GPU_ID | api |
EVALUATOR_GPU_ID | curation-evaluator |
SEGMENTER_GPU_ID | segmenter |
VLM_GPU_ID | vlm |
OP_TRAIN_GPU_ORDER | curation-trainer |
All default to 0. Set each explicitly: .env values are not
shell-expanded, so one can't reference another. The trainer accepts a
single id from the variable; a multi-GPU trainer (OP_TRAIN_GPU_ORDER=0,2)
also needs a docker-compose.local.yml listing the same device_ids in the
same order.
Example layout on a three-GPU host (one card per heavy tenant):
TRITON_GPU_ID=1
API_GPU_ID=1
SEGMENTER_GPU_ID=0
EVALUATOR_GPU_ID=0
VLM_GPU_ID=2
OP_TRAIN_GPU_ORDER=0
OP_GPU_ALLOWED_IDS=0
Measured performance
Single-request latency from the README (one GPU):
| Operation | Time | Throughput |
|---|---|---|
| Object detection | 140-170 ms | ~6-7 RPS |
| Face detection | 100-150 ms | ~7-10 RPS |
| Face recognition | 105-130 ms | ~8-9 RPS |
| Image embedding (CLIP) | 6-8 ms | ~120 RPS |
| Text embedding (CLIP) | 5-17 ms | ~60-200 RPS |
| OCR | 170-350 ms | ~3-6 RPS |
| Full analyze | 280-430 ms | ~2-3 RPS |
| Single-image ingest | 750-950 ms | ~1-1.3 RPS |
| Batch ingest (50 images) | 7.3 s total | ~6.8 images/s |
Batching gives roughly 2-3x the throughput of sequential single-image calls. The planned throughput work is on the Roadmap.