PaperlayerView exact revision on arXiv

Source-linked AI summary

LOCUS: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding with a Box Head and Coordinate-Token Verification

Muhammad Talha

arXiv:2610.35053v1cs.CV

TL;DR

Coordinate-token grounding is precise but sequential, while regression heads are fast but lose high-IoU accuracy. Locus uses a head to draft coordinate tokens for verification by the same backbone; on RefCOCO-family splits, verification improves Locus-2B’s Acc@0.9 from 54.87 to 59.13.

  • Problem

    Coordinate-token grounding takes sequential decoding steps, while regression heads lose precision at high IoU despite predicting in one pass.

    Source: §1 · p. 2

  • Method

    Locus renders a regression head’s box as coordinate tokens and verifies them in parallel with the same backbone over a shared image prefix.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Acc@0.9 rises from 54.87 to 59.13 for Locus-2B on the 8 RefCOCO/+/g splits; typical-acceptance Locus gains +4.52 points over Jag at its size and budget.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The greedy coordinate-token answer is reached in 5.60 sequential passes on average instead of 18.46 decoding steps.

    Source: §1 · p. 2

    Locus is slower than a pure regression head, and at 0.8B verification lowers Acc@0.5, so its benefit at that size is high-IoU accuracy only.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) ground a referring expression in one of two ways. They decode the box as coordinate tokens, which is precise but takes one sequential step per token, or they regress it from a hidden state with a small head, which takes one step but loses accuracy at high IoU. The two fail differently: a head usually finds the right object but places its edges loosely, while coordinate tokens place edges precisely. We therefore use the regression head as a speculative drafter for the model's own coordinate tokens. Locus writes the head's box as coordinate tokens and checks them with the same backbone by Jacobi iteration over a shared image prefix. The fixed point is the model's greedy coordinate-token answer, reached in about 5.60 sequential passes on average instead of 18.46 decoding steps; head and verifier are one set of weights, trained jointly. On the 8 RefCOCO, RefCOCO+ and RefCOCOg splits, verification raises Locus-2B's Acc@0.9 from 54.87 for its own head to 59.13, and the one-pass head is ahead at Acc@0.5 of every published model with at most 3B parameters except Florence-2-L. Against Jag, the strongest open regression-head model, at its own size and training budget, Locus ties at Acc@0.5 and, with typical acceptance, gains +4.52 points at Acc@0.9, with paired significance tests. Exact answers come 1.5x faster than vLLM greedy-decoding the same coordinate tokens with the same weights; a pure regression head remains faster still.

1 Introduction

Locus uses a regression head to draft coordinate tokens that the same model verifies, combining the head’s object selection with coordinate decoding’s tighter edges. Verification improves high-IoU accuracy, while the head remains a fast one-pass predictor.

1 Introduction

  • The head usually finds the right object, while coordinate tokens place its edges more exactly, motivating draft-and-verify grounding.On Locus-2B’s same weights, the head scores 90.01 versus tokens’ 89.30 at Acc@0.5, while tokens score 59.13 versus the head’s 54.87 at Acc@0.9.Source: §1 · p. 2
  • A head drafts its box as the model’s coordinate tokens, which the same backbone checks in parallel through Jacobi iteration over a cached image prefix.The verifier keeps the agreed prefix and corrects the next token until the draft reaches the model’s greedy answer.Source: §1 · p. 2
  • 5.60 sequential verification passes on average reach the greedy coordinate-token answer, versus 18.46 decoding steps.Exact acceptance preserves the greedy answer; typical acceptance trades exactness for fewer passes.Source: §1 · p. 2
  • The head and verifier share one set of weights and are jointly trained, with separate question branches sharing one image prefill.A dual mode preserves the base model’s answers to other image questions.Source: §1 · p. 2
  • Acc@0.9 rises from 54.87 to 59.13 on the 8 RefCOCO/+/g splits, and typical-acceptance Locus gains +4.52 points over Jag at its size and budget.The one-pass 2B head also exceeds every published model with at most 3B parameters at Acc@0.5 except Florence-2-L.Source: §1 · p. 2
  • 46.9% of evaluation rows contain targets appearing in another dataset’s training split, so the paper also reports results on clean rows.No evaluation images are used for training.Source: §1 · p. 2
  • LocateAnything-3B, trained on more data, remains ahead at high IoU; Locus and Jag are trained on the RefCOCO family, and both fall far below the zero-shot base model on Ref-L4 Objects365.The Objects365 targets are smaller than any in RefCOCO.Source: §1 · p. 2

2 Related Work

Related work spans coordinate-token grounding, regression heads, speculative decoding, and shared-prefix computation. Locus combines these ideas by using a regression head to draft the coordinate tokens its own backbone verifies.

Coordinate-token grounding

  • Coordinate-token systems such as Shikra [8], the Qwen-VL family [1] [2], and LocateAnything [25] still produce answers by decoding, which Locus uses as its verifier.Some methods change the vocabulary or emit a whole box per step, but still decode the answer.Source: §2 · p. 2

Regression and decoder heads on VLMs

  • Jag [36] regresses boxes from the last input token, while earlier systems route special-token embeddings to separate task decoders; Locus returns its head output only after coordinate-token agreement.Locus’s drafter follows Jag’s design and also predicts IoU-aware confidence.Source: §2 · p. 2

Speculative and parallel decoding

  • Unlike prior speculative and parallel decoding drafts, Locus renders a regression head’s continuous box into verifier tokens and drafts the whole answer at once.Related approaches draft tokens using a smaller language model, extra token heads, or the target model itself.Source: §2 · p. 3

Shared-prefix computation

  • Locus reuses shared-prefix attention within one request: the image is encoded once, and each question, draft, and verification pass branches over its cache.Earlier shared-prefix work reuses prompt keys and values across requests.Source: §2 · p. 3

3 Method

Locus shares one image prefill across box and verifier branches, jointly training a box head and the backbone’s coordinate-token verifier. Jacobi iteration extends a verified token prefix until the draft reaches the greedy answer.

3.1 Shared prefill and question branches

  • Each expression runs as an isolated branch over cached image states, so multiple questions share one image encoding and prefill.Box, verifier, and decision questions all use this branching arrangement.Source: §3.1 · p. 3

3.2 Box head (drafter)

  • A learned box-query token feeds a small MLP that predicts a normalized center-size box and confidence logit in one branch pass.Training combines L1 and GIoU box losses with IoU-aware binary cross-entropy for confidence.Source: §3.2 · pp. 3–4

3.3 Coordinate-token verification

  • The verifier targets the adapted backbone’s greedy coordinate-token answer, and Jacobi passes predict every position in parallel while preserving the longest agreed prefix.Each pass replaces the incorrect suffix with remaining predictions; iteration stops when the draft no longer changes.Source: §3.3 · p. 4
  • 5.60 passes at 2B reach the fixed point on average, versus 18.46 sequential greedy steps; exact fp32 verification matched greedy boxes on all 505 RefCOCO val questions.The iteration is capped at eight passes, with remaining rows completed by greedy decoding from the verified prefix.Source: §3.3 · p. 4
  • In bf16, verification differs from greedy decoding at a similar rate because greedy decoding itself varies with batch shape and computation settings.On the accuracy path, 35.4% of 10,834 validation questions differ, with a median two-unit coordinate-grid change.Source: §3.3 · p. 4

3.4 Training one model for both roles

  • Joint training uses shared weights and one image prefill for head and verifier branches, while dual mode uses base weights for other image questions.Mixed box and non-box requests require a second prefill in dual mode.Source: §3.4 · p. 4

4 Experimental Setup

Experiments evaluate Locus on RefCOCO-family benchmarks, with comparisons designed to control training budget, dataset overlap, and inference settings. The setup also includes Ref-L4 and timed comparisons against regression and coordinate-token systems.

Backbones

  • Locus uses Qwen3-VL-2B as its main backbone and Qwen3.5-0.8B, Jag’s backbone, for equal-size comparisons.The 0.8B backbone combines Gated DeltaNet linear-attention and full-attention layers.Source: §4 · p. 4

Training data

  • Training excludes images appearing in any evaluation split, and results are reported on all rows and on clean rows whose targets appear in no training split.The clean-row evaluation addresses cross-dataset target overlap in the RefCOCO-family splits.Source: §4 · pp. 4–5

Training budget

  • The fully fine-tuned 0.8B and 2B models train for 36k steps, matching Jag’s budget; the 2B LoRA ablation uses 12k steps.A separate 54k-step run uses more training than Jag.Source: §4 · p. 5

Evaluation

  • Accuracy is measured at Acc@0.5, Acc@0.75, and Acc@0.9 across eight RefCOCO, RefCOCO+, and RefCOCOg validation and test splits.Thresholds are selected on RefCOCO val and applied unchanged elsewhere; the models are evaluated at their training resolution and each other’s resolution.Source: §4 · p. 5

Outside RefCOCO

  • Ref-L4 evaluation includes Objects365 images unseen in training and separately reports COCO rows on images absent from RefCOCO-family training splits.Models receive no Ref-L4 tuning.Source: §4 · p. 5

Baselines and latency

  • Released baselines are evaluated with their model-card or native prompts, while latency comparisons use batch-one timing on one RTX A6000 GPU.Jag and Locus share CUDA graphs and GPU preprocessing; coordinate-token baselines run in vLLM for the timed comparisons.Source: §4 · p. 5

5 Results

Across the RefCOCO-family results, verification improves high-IoU accuracy while the head remains competitive at Acc@0.5. Comparisons also show how results vary with model size, clean evaluation rows, latency, and training choices.

5.1 Comparison with published models

  • 90.1 Acc@0.5 puts the one-pass Locus-2B head above published models with at most 3B parameters except Florence-2-L at 91.1.Table 1 averages the five test splits; training data differ across models.Source: §5.1 · p. 5, Table 1 · p. 6

5.2 Accuracy across IoU thresholds

  • Acc@0.9 rises from 54.87 for the Locus-2B head to 59.13 for its exact verified answer across eight splits.The head is more accurate at Acc@0.5; LocateAnything-3B, trained on far more grounding data, leads at high IoU, and Locus-2B reaches 6.2 Acc@0.5 on Objects365’s smallest-target quarter.Source: §5.2 · pp. 5–7, Figure 3 · p. 7

5.3 Comparison at equal size and budget

  • At matched size and training budget, Locus-0.8B ties Jag at Acc@0.5 and typical acceptance adds 4.52 points at Acc@0.9.The Acc@0.9 gain has a 95% image-bootstrap interval of [+3.86, +5.35] and p = 7.2 × 10−125; verification costs 1.11 points at Acc@0.5 at this size.Source: §5.3 · p. 7

5.4 Evaluation on clean rows

  • Every margin over Jag grows on evaluation rows neither model was trained on, compared with all rows.46.9% of evaluation rows involve objects present in another RefCOCO-family training split; the clean rows exclude targets seen by either model.Source: Figure 4 · p. 7, §5.4 · p. 7

5.5 Latency

  • For like-for-like coordinate-token decoding, Locus-2B exact verification takes 102.2 ms versus 151.6 ms for its own weights served by vLLM, making it 1.5× faster.The vLLM run decodes exactly 18.3 coordinate tokens and returns the same greedy box; a pure regression head is faster.Source: §5.5 · p. 8

5.6 Ablations

  • At 0.8B, aLoRA reduces head Acc@0.9 to 33.58 versus 45.72 in dual mode, while verified accuracy differs by only a few points across variants.The aLoRA draft also agrees less with the verifier, requiring 7.98 versus 7.22 passes.Source: §5.6 · p. 9
  • At 2B, full fine-tuning raises head Acc@0.9 from 44.80 to 54.87 and reduces passes from 6.86 to 5.60, while verified accuracy changes little.A separate head and verifier are no more accurate than joint training and require two prefills; typical acceptance trades some exactness for fewer passes.Source: §5.6 · p. 9

6 Limitations and Conclusion

Locus uses a regression head to draft the model’s own coordinate-token answer, recovering high-IoU accuracy with fewer sequential passes. Its benefits are bounded by latency, model size, and the RefCOCO-family training scope.

Limitations

  • Locus is slower than an equally engineered pure regression head, and verification adds passes as draft errors increase.At 0.8B, verification lowers Acc@0.5, so its benefit at that size is limited to high-IoU accuracy.Source: §6 · p. 9
  • Training is limited to RefCOCO-family data, and Locus-2B reaches Acc@0.5 47.81 on Ref-L4 Objects365 images versus 74.45 for its zero-shot base model.Half the Objects365 targets are smaller than every RefCOCO-family training target; whether adding small-target training data closes the gap is untested, and the largest backbone is 2B.Source: §6 · p. 9

Conclusion

  • Acc@0.9 rises from 54.87 to 59.13 for Locus-2B on the same weights, with typical acceptance adding 4.52 points over Jag at its size and budget.The verified answer is the model’s greedy box, reached in 5.60 passes rather than 18.46 decoding steps.Source: §6 · p. 10

A.1 Additional tables

The appendix tables cover coordinate-token decoding, exactness, clean evaluation rows, and the typical-acceptance threshold; the supplied captions describe scope but do not report table-cell results.

A.1 Additional tables

  • Table 7 covers coordinate-token decoding without grounding fine-tuning on the first 3,000 RefCOCO val, testA, and testB rows, scored on identical rows.The Qwen models’ pretraining includes RefCOCO training data.Source: Table 7 · p. 12
  • Table 8 reports Locus-2B exactness on RefCOCO val, including the proportion of boxes with any coordinate difference, large changes, and mean verifier passes.The comparison baseline is 18.44 greedy steps.Source: Table 8 · p. 12
  • Table 9 restricts evaluation to rows whose target object is absent from every RefCOCO-family training split, excluding 46.9% of rows.Results are pooled across the eight splits with row weighting and image-bootstrap confidence intervals.Source: Table 9 · p. 12
  • Table 10 reports the effect of typical-acceptance threshold α for Locus-2B, averaged over the eight splits.Source: Table 10 · p. 12

A.2 Negative results

The appendix reports limited gains from a learned selector and little change from drafter distillation; accompanying figures show selected comparisons with Jag and qualitative examples without ground-truth scoring.

A.2 Negative results

  • At 2B, the learned selector matches the agreement rule on held-out splits; at 0.8B, it improves Acc@0.5 over the rule but remains below Jag.The authors retain the rule because the signals make it hard to predict which of the draft and verified boxes is better.Source: §A.2 · p. 12
  • Distilling the 2B head on verifier-greedy boxes changes RefCOCO val passes only from 6.85 to 6.76, with essentially unchanged accuracy.Across eight splits, 0.02% of drafts match all four greedy integers and 9.0% match at least two, so nearly every question still needs correction.Source: §A.2 · p. 12
  • Figure 6 selects three eligible questions where Locus-2B’s IoU exceeds 0.9 and Jag’s is below 0.6, plus one reverse case.Eligible expressions use only words appearing at least 100 times in the training expressions.Source: Figure 6 · p. 13
  • Figure 7 shows 12 prewritten expressions on licensed photos without ground truth, so it makes no accuracy claim.The models mostly agree, with several tighter Locus boxes, one disagreement, and one expression for which both return the whole person.Source: Figure 7 · p. 13

Related papers

Loading 2610.35053v1…