Source-linked AI summary
WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes
Kishor Datta Gupta, Ahmed Rafi Hasan, Md. Mahfuzur Rahman, Md. Sadman Haque, Mohd Ariful Haque
TL;DR
The paper asks whether identifying one specified object in clutter, including reporting its absence, can be achieved cheaply from frozen world-model representations. WALDO uses a 3.4M-parameter exemplar- and language-conditioned head with synthesized episode supervision, and improves catalogue detection and absence detection, while instance identity remains unresolved.
Problem
The task is to localize one specified object instance from an exemplar and description in clutter and report when that instance is absent, unlike category-level detection.
Method
WALDO reads frozen V-JEPA 2.1 spatial features with a 3.4M-parameter head that fuses exemplar correlation and FiLM conditioning for joint localization and presence prediction.
Results
WALDO reaches 0.461 catalogue AP@50 versus 0.306 for Grounding DINO, while instance-level Success@1 is 0.190 at the 0.190 category-chance floor.
Takeaways & Limitations
World-model features transfer to localization precision and absence detection but not to instance identity in this pipeline and data.
Takeaways & Limitations
The instance setting uses an unaugmented target crop from the test image, and instance discrimination therefore rests on a small exemplar rather than cross-view re-identification.
Abstract
from arXiv · showhide
Locating a specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent, large vision-language models usually address this task. We ask whether the same capability is available far more cheaply, from representations already learned by a world-model pretraining objective. We present WALDO, a one-shot exemplar- and language-conditioned detection head with 3.4M trainable parameters that reads frozen V-JEPA 2.1 features to jointly predict object localization and target presence, with no gradient on the backbone. Because exemplar-conditioned supervision is scarce, we synthesize training episodes from instance annotations, mining exemplars from ground-truth boxes and constructing absence cases that exclude the referenced instance while leaving same-category distractors in view. This is easy to get wrong: in the obvious implementation, crop size alone predicts the label, and a head trained on it reaches 0.9998 absence AUROC without ever consulting the exemplar, and we report the negative controls that close the shortcut. On 35 held-out cluttered scenes, WALDO achieves a 0.461 catalogue AP@50, compared to 0.306 for a prompted Grounding DINO baseline under an identical scorer. Substituting DINOv3 for V-JEPA under a matched 576-token grid drops within-category absence AUROC from 0.880 to 0.726 and instance AP@50 from 0.201 to 0.141, isolating the pretraining objective rather than input resolution as the source of the gain. Instance-level Success@1, however, reaches only 0.190 against a 0.190 category-chance floor: world-model features transfer to localization precision and absence detection but not to instance identity.
1 Introduction
WALDO targets the under-addressed problem of identifying one specific object instance in clutter, including deciding when that instance is absent. It uses a lightweight conditioned head over frozen world-model features and trains localization and presence jointly.
- Problem: Specific-instance localization asks a system to find the referenced object among near-identical distractors or report that it is absent.The task extends open-vocabulary retrieval from category recognition to identity-level matching.
- Evidence: 15.9% of scenes with V-JEPA 2 and 14.3% with DINOv3 place the target above all same-size distractors, near the 11.1% chance level.Frozen generic similarity therefore provides little instance-discriminating signal without learned correspondence and explicit presence supervision.
- Approach: WALDO uses 3.4M trainable parameters over frozen V-JEPA 2.1 spatial tokens, combining exemplar correlation and FiLM conditioning.It jointly predicts per-patch localization and scene-level presence, making absence a trained output.
- Approach: The paper formulates exemplar-conditioned detection with explicit presence determination and mines positive, within-category absent, and cross-category absent episodes from sparse annotations.Region geometry is decoupled from labels to prevent shortcut learning.
2 Related Work
Related work largely treats detection as category recognition, language disambiguation with guaranteed presence, or one-shot correspondence without instance-level absence. WALDO combines exemplar and language conditioning with separately supervised presence and tests whether video-predictive features transfer to this task.
- Existing settings: Open-vocabulary and few-shot detectors answer category-level questions, while referring-expression benchmarks generally guarantee that the referent is present.These settings do not evaluate rejecting a missing specific instance among same-category objects.
- Existing settings: Open-set detection models absence as category novelty, not whether one previously seen object is missing while same-category look-alikes remain.This leaves instance-level absence without a corresponding metric in the cited related work.
- WALDO: WALDO jointly conditions on an exemplar and free text, resolves instance rather than category identity, and adds a separately supervised presence decision.Its negatives are split into same-category and different-category cases.
- WALDO: The study holds the detection head fixed while substituting video-predictive, image-contrastive, and image-self-distilled backbones.This tests transfer from world-model pretraining to a spatial instance-level task.
3 Method
WALDO formulates one-shot detection as jointly localizing an exemplar-conditioned object and deciding whether that instance is present, distinguishing catalogue from instance evaluation. It synthesizes balanced episode types from sparse annotations and uses frozen visual-language features with a lightweight conditioned detection head.
- Problem setting: WALDO returns ranked boxes and a presence probability, while separating catalogue correctness from instance correctness when same-category distractors are present.In the catalogue setting any instance of the exemplar category is correct; in the instance setting only the referenced exemplar’s box is correct.
- Synthesizing exemplar-conditioned episodes: Training episodes mine augmented exemplars from annotated boxes and include positives, within-category absences, and cross-category absences.Absent episodes exclude the referenced instance or use another category, with a realized mixture of 66% positive, 19% within-category absent, and 15% cross-category absent.
- Synthesizing exemplar-conditioned episodes: Region scale is sampled independently of labels because crop geometry can otherwise yield 0.9998 absence AUROC without consulting the exemplar.The shortcut arises when positives are whole-scene regions and absences are sub-crops; positive crop placement uses interval arithmetic to avoid this leak.
- Conditioned detection head: Frozen V-JEPA 2.1 supplies 576 query patch tokens and a mean-pooled exemplar vector, while frozen SigLIP2 encodes the description.The backbone remains frozen, and the query grid is 24 × 24 tokens at 384 px.
- Conditioned detection head: A patch-wise cosine correlation is appended before projection, then exemplar-text features FiLM-condition transformer tokens that feed box, objectness, and presence heads.Concatenating correlation before projection exposes the signal to downstream layers; the description reaches prediction through global feature-wise modulation.
- Training and inference: Attention pooling produces presence from the most responsive patches, while per-patch supervision assigns positives near box centers and protects small objects from losing their only positive patch.The head totals 3.39M parameters and uses focal loss to reduce domination by background patches.
4 Experimental Setup
The experiments isolate pretrained representation effects with frozen backbones, matched spatial grids, and a fixed WALDO head while comparing against frozen open-vocabulary detectors under their expressible catalogue setting. The evaluation uses scene-level splits, single-pass inference, explicit presence scoring, and controls for prompting, parameter-count, and reproducibility differences.
- Backbone arms and baselines: The backbone ablation substitutes DINOv3 or SigLIP2 for V-JEPA 2.1 while keeping the head, sampler, assignment, loss, optimizer, split, and seed identical.This isolates the pretrained representation as the only free variable; the baseline comparison is coarser because complete detectors expose different proposals, conditioning, and supervision.
- Dataset and splits: The dataset contains dense parking-lot and aircraft-bay scenes with 763 instance boxes over 220 scenes, split by scene to prevent evaluation-pixel leakage.Scenes contain a median of 3.5 instances and up to 10, producing 2,638 within-scene same-category pairs for hard instance negatives.
- Dataset and splits: Two categories and two descriptions make language a category switch, so instance discrimination rests on the exemplar mined from the same scene.At test time the exemplar is an unaugmented crop from the image being searched, making this setting easier than cross-view re-identification.
- Backbone arms and baselines: The main model uses frozen V-JEPA 2.1 spatial tokens with a SigLIP2 text tower because V-JEPA has no language pathway.DINOv3 is the clean image-only control, while SigLIP2 is a deliberately confounded contrastively pre-aligned arm.
- Backbone arms and baselines: All backbones emit a matched 24 × 24 = 576-token grid, while DINOv3 receives 23% fewer input pixels and runs away from its pretraining resolution.The control matches token budget rather than pixel budget, so resolution differences are explicitly documented rather than treated as fully eliminated.
- Evaluation and baselines: Grounding DINO and OWLv2 are frozen and evaluated with generic and category prompts, with catalogue retrieval as the only setting their interfaces express.All results share single-pass inference, one scorer, NMS IoU = 0.5, and one detection cap; Success@K and AUROC quantify localization and presence.
5 Results
Across held-out scenes, WALDO retrieves categories and detects absence more reliably than it identifies the referenced instance. V-JEPA features support tighter localization and stronger hard-absence discrimination, while instance selection remains near category-chance and baseline comparisons require protocol caution.
- Evaluation: 35 scenes, 116 instances, and 151 queries were evaluated with one scorer, NMS setting, and detection cap across configurations.All results use the held-out test split and a common evaluation scale.
- Category retrieval succeeds, instance retrieval fails: 0.190 V-JEPA 2.1 instance Success@1 matches its 0.190 category-chance floor, making instance selection statistically indistinguishable from a fair draw.The exact one-sided test reports p = 0.55 for 22 observed versus 22 expected hits.
- Category retrieval succeeds, instance retrieval fails: 68.1% of V-JEPA instance queries have a top-1 box with exactly zero IoU against the correct object, compared with 65.5% for DINOv3 and 63.8% for SigLIP2.The catalogue-minus-instance Success@1 transfer gaps are 0.439, 0.410, and 0.519, respectively.
- Target absence: 0.880 hard-absence AUROC for V-JEPA 2.1 exceeds 0.831 for SigLIP2 and 0.726 for DINOv3, while easy negatives reach 1.000 for every backbone.Pooling easy and hard negatives would conceal this separation; the hard-absence estimate uses 46 episodes.
- Open-vocabulary baselines: 0.461 catalogue AP@50 for V-JEPA exceeds prompted Grounding DINO’s 0.306 under the shared scorer, while baseline protocol differences limit broader detector claims.SigLIP2 reaches 0.481; the prompt protocol changes Grounding DINO more than the detector choice does.
- Scale and box tightness: 0.208 V-JEPA AP@[.5:.95] exceeds 0.189 for SigLIP2 and 0.139 for DINOv3, indicating better box precision despite lower catalogue Success@1 than SigLIP2.V-JEPA falls from 0.461 at AP@50 to 0.168 at AP@75; cars are much harder than aircraft, with median car boxes measuring 67 × 83 px.
6 Conclusion
WALDO shows that frozen representations can support category-level localization and explicit absence detection, while instance identity remains unresolved.
- 0.461 catalogue AP@50 versus 0.306 for the strongest open-vocabulary baseline under one scorer.
- 0.880 within-category absence AUROC versus 0.831 and 0.726 for image-pretrained controls.
- 0.190 instance-level Success@1 matches the category-chance floor, indicating that localization and absence detection do not yield instance identity.
- 68.1% of instance queries return boxes overlapping nothing that counts, despite the model finding the category.