Skip to main content

Reprocess

Reprocess re-runs part of the pipeline on images or items that already exist: after a new region profile or prompt pack is activated, after a detector is promoted, or to retry failures. One route covers every case. It is a dry run by default.

API=http://localhost:4603 and project cars are assumed.

Concepts​

  • Scopes. What to regenerate. A request lists one or more.

    ScopeUnitWhat it regenerates
    detectimageMachine items from the ingest detectors. New proposals are merged with existing items
    open_vocabimageItems from the active open-vocabulary prompt set run on the whole image. See Open-vocabulary detection
    regionitemMachine region boxes
    vlmitemThe VLM's class answer, so the VLM worker selects the item again
    embedimageThe PE vectors: item crop, whole frame, and one per accepted or false-positive region box

    Scopes run in the order detect, open_vocab, region, vlm, embed.

  • Lock rule. Only machine output is regenerated. Human and imported labels and boxes are locked and never written. A locked item that has nothing to regenerate is reported as locked_skipped. detect never writes, replaces or deletes a locked item. A validated region set (set by a person, including a human "no region visible") is not redone by the region scope. embed also covers locked items, because vectors are derived data.

  • Region mode. redetect (default) removes unlocked machine boxes, keeps locked ones and queues the item as pending_detection. reverify turns unlocked rejected boxes back into proposed and queues the item as pending_verification. A detected set cannot be re-verified.

  • Targets. Exactly one of crop_ids, image_ids or filter. Up to 5000 ids. An empty filter is refused, since it would mean every item.

  • Sync or job. region and vlm run inside the call. detect and embed run inside the call for a few images and as a background job above OP_REPROCESS_SYNC_MAX images (default 20). One live job per project. Job state is kept under OP_REPROCESS_JOBS_DIR.

Routes​

CallPurpose
POST /curation/projects/{project}/reprocessUnified reprocess over ids or a filter
GET /curation/projects/{project}/reprocess/jobs/{job_id}Job status
POST /curation/projects/{project}/reprocess/jobs/{job_id}/cancelStop a job after its current chunk
POST /curation/projects/{project}/crops/{crop_id}/reprocessOne item; applies by default
POST /curation/projects/{project}/images/{image_id}/reprocessOne image; applies by default

Filter​

All fields are optional but at least one must be set.

FieldMeaning
region_statusRegion statuses to select (list)
missing_statusItems with no region status at all. Exclusive with region_status
detector, reasonSelect by the detector or rejection reason that produced the status
profile_not, profile_revision_belowItems not produced by that region profile, or by a revision below the number. Given together: not produced by exactly that profile at that revision
include_detectedWiden the region scope to detected items. Only valid with a profile selector
missing_provenanceItems with no recorded detector
region_gate_skippedItems the region gate skipped (region_gate_skip set): the segmenter was never asked, so they are safe to re-run with scope region
import_idItems from one dataset import
source, class_id, dataset_splitNarrow by item source, class or split
all_imagesEvery stored image, including images with no item yet. For the image scopes only; not combined with the item selectors
open_vocab_statusImages whose open-vocabulary state is one of pending, done, skipped_gate, failed. Same restrictions as all_images

The region scope needs region_status, missing_status or a profile selector. With only a profile selector it selects detection_failed, verify_rejected, no_region_box and no_region_visible items (and detected ones with include_detected).

targets.limit with sample (random with a seed, or largest) caps a filter for every scope, region and vlm included: the dry run counts and the apply acts on the same capped items.

Dry run, then apply​

curl -s -X POST "$API/curation/projects/cars/reprocess" \
-H 'Content-Type: application/json' \
-d '{
"targets": {"filter": {"profile_not": "vehicle_wheel", "include_detected": true}},
"scopes": ["region"],
"region_mode": "redetect",
"dry_run": true
}'

The response has dry_run, scopes and job. Each scope result has selected, locked_skipped, queued, not_found, failed, breakdown (rows of detector, reason, count) and detail (scope counters such as no_box).

Run it for real by sending the same body with "dry_run": false.

The region impact route of the active profile returns this request ready to post as suggested_reprocess; see Region profiles.

Other targets:

curl -s -X POST "$API/curation/projects/cars/reprocess" \
-H 'Content-Type: application/json' \
-d '{"targets": {"image_ids": ["IMG_A", "IMG_B"]}, "scopes": ["detect", "embed"], "dry_run": false}'

curl -s -X POST "$API/curation/projects/cars/reprocess" \
-H 'Content-Type: application/json' \
-d '{"targets": {"crop_ids": ["CROP_A"]}, "scopes": ["vlm"], "dry_run": false}'

One item or one image​

The per-target routes use the path as the target and apply by default (dry_run false):

curl -s -X POST "$API/curation/projects/cars/crops/CROP_ID/reprocess" \
-H 'Content-Type: application/json' \
-d '{"scopes": ["region"], "region_mode": "redetect", "dry_run": false}'

curl -s -X POST "$API/curation/projects/cars/images/IMAGE_ID/reprocess" \
-H 'Content-Type: application/json' \
-d '{"scopes": ["detect"]}'

The response is the same shape as above and also carries items: the item or the image's items as written, so a client can adopt them. A locked set with nothing to regenerate answers locked_skipped: 1 and queued: 0.

Jobs​

When detect or embed covers more images than the sync limit, the response carries job.

curl -s "$API/curation/projects/cars/reprocess/jobs/JOB_ID"
curl -s -X POST "$API/curation/projects/cars/reprocess/jobs/JOB_ID/cancel"

A job has job_id, status (queued, running, completed, completed_with_errors, cancelled, failed), scopes, images_total, images_done, images_failed, results (per scope, when finished), started_at, updated_at, finished_at, error and poll_after_s (present while the job is active). Cancel stops the job after its current chunk.

Errors​

StatuserrorWhen
404not_foundUnknown crop or job
404image_not_foundUnknown image
409reprocess_busyA job is already running for this project
422reprocess_targets_invalidNot exactly one target form, an empty id list, more than 5000 ids, an empty filter, a region selection without a status or profile selector, include_detected without a profile selector, detected with reverify, or an unknown region status

Cropwright​

The reprocess dialog after Check what would run: per-scope counts (captured from Cropwright against the public COCO sample projects)

See also​