Jag: Direct Box Prediction for Efficient Visual Grounding

• Autoregressive coordinate generation followed by parsing adds output computation to the image–text processing already needed for localization.Source: §1 · p. 1
• Joint adaptation lets box supervision shape both multimodal representations and their mapping to coordinates, whereas a fixed-feature head confines adaptation to the coordinate mapping.Source: §1 · p. 1
• A lightweight regression head predicts continuous box coordinates from the last valid input-token state, returning the complete box in one multimodal forward pass.The prediction uses the input state without subsequent coordinate generation or text parsing.Source: §1 · pp. 1–2
• Jointly adapting the language backbone, visual merger, and regression head substantially improves accuracy over training the head alone.Source: §1 · p. 2
• Jag improves grounding accuracy over Base and NExT-Chat, is competitive with LocateAnything, and uses less single-request latency and GPU memory than compared specialized models.Source: §1 · p. 2

• Visual grounding includes region-matching and one-stage methods, language-conditioned detectors, and direct box-regression architectures.Source: §2 · p. 2
• NExT-Chat and Jag use MLP heads for continuous box regression, while LocateAnything predicts discrete coordinates in parallel within box-aligned blocks.Source: Table 1 · p. 2
• Multimodal grounding approaches include coordinate strings, discrete location tokens, and task-specific post-training for localization; Jag predicts a continuous single box from an existing input-token state.Source: §2 · p. 2
• Coordinate approaches range from autoregressive sequences to parallel prediction; Jag predicts four continuous values together, unlike LocateAnything’s within-block attention over discrete coordinates.Source: §2 · p. 2
• Geometric supervision includes overlap losses and rewards, while Jag directly applies L1 and GIoU losses as it adapts the backbone, visual merger, and regression head.Source: §2 · p. 3

3 Method

• Jag retains the pretrained vision encoder, visual merger, and language backbone, and trains a lightweight regression head with geometric supervision, followed by joint updates to the head and part of the backbone.Source: §3 · p. 3

3.1 Task Formulation

• The task is to localize the object referred to by an image–expression pair with one axis-aligned box, represented by normalized center coordinates, width, and height.Source: §3.1 · p. 3

3.2 Multimodal Representation

• The vision encoder and visual merger feed image and text into the language backbone, whose final hidden state at the last valid input position provides the prediction representation.This existing position follows the valid visual and textual inputs; selecting it from the mask works with left or right padding and requires no task-specific regression token.Source: §3.2 · p. 3

3.3 Bounding Box Prediction

• A regression head maps the selected representation through layer normalization and a hidden projection to a normalized box, predicting all four coordinates together.Source: §3.3 · p. 3

W1 LN(z) + a1

• The regression head predicts all four coordinates together; center–size outputs are converted to corners, with clipping applied only to final predictions.The sigmoid constrains predicted centers and dimensions to the unit range, though resulting corners can extend beyond the image.Source: W1 LN(z) + a1 · p. 3

3.5 Adaptation and Inference

• Jag’s task adaptation improves grounding accuracy over the base model, while direct output reduces inference latency relative to autoregressive localization.Source: Figure 1 · p. 4

3.4 Training Objective

• The sole training objective combines coordinate supervision with GIoU on unclipped corner boxes, retaining out-of-image extent in the training penalty.The coordinate term directly supervises normalized centers and dimensions, while GIoU couples them through box geometry.Source: §3.4 · p. 4

3.5 Adaptation and Inference

• Training first fits the new regression head with the backbone frozen, then jointly updates the head, language backbone, and visual merger while keeping the vision encoder frozen.Source: §3.5 · p. 4
• Inference runs the multimodal backbone once, then returns the box after coordinate conversion and clipping, without sequential coordinate-token generation or text parsing.Source: §3.5 · pp. 4–5

4.1 Experimental Setup

• Evaluation uses official RefCOCO, RefCOCO+, and RefCOCOg splits, with Acc@0.5 as the primary metric; Base is Qwen3.5-0.8B without task-specific fine-tuning.Jag is compared with Base for task adaptation and with released LocateAnything for grounding; Hi-Token is included in the inference-cost comparison.Source: §4.1 · p. 5

4.2 Main Results

• Jag improves Acc@0.5 over Base on every split by 9.63–17.51 points, beats NExT-Chat on every split, and is competitive with LocateAnything overall.Jag achieves 88.30% across all test expressions, compared with 88.00% for LocateAnything and 79.57% for NExT-Chat; gains also hold on RefCOCO+ splits where location words are restricted.Source: §4.2 · p. 5
• 65.97 ms mean latency and 2.52 GiB peak memory accompany a 19.60× speedup over Base; Figure 2 shows the lowest latency and less memory than the compared specialized models.Jag is 1.82× faster than NExT-Chat, 3.62× faster than LocateAnything, and 9.35× faster than Hi-Token; its peak memory is 84.22%, 73.88%, and 68.10% lower, respectively.Source: §4.2 · pp. 5–6, Figure 2 · p. 5

4.4 Effect of Training Data Budget

• Increasing the jointly varied example and update budgets improves Acc@0.5 on all three datasets; RefCOCO+ rises from 80.33% to 85.80%.Even the smallest subset exceeds Base on all three benchmarks.Source: Figure 4 · p. 6, §4.4 · pp. 7–8

4.3 Ablation Studies

• Jointly adapting the language backbone, visual merger, and head improves Acc@0.5 by more than 60 points per dataset over head-only training.Mean-pooling expression states substantially reduces accuracy; an input marker and immediate joint training each remain within one point of the default, while the backbone update matters more than these choices.Source: §4.3 · pp. 6–7, Figure 3 · p. 6
• Continuous regression beats four independent categorical heads across all three datasets and multiple seeds; full-data Jag remains within one Acc@0.5 point of fine-tuned AR on each dataset.AR is more accurate at IoU 0.75 and 0.9 on all datasets, while a within-backbone timing comparison gives Jag an 8.90× mean end-to-end speedup for single-request inference.Source: §4.3 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.5 Diagnostic Analysis: Groups and Cases

• Gains span every nonempty target-area, expression-length, and spatial-keyword group, while qualitative cases distinguish wrong-instance selection from incorrect box extent.RefCOCOg medium-area accuracy rises from 70.96% to 82.16%; expression length shows no consistent gain gradient, and both models overestimate the table region above the bowl in one case.Source: Figure 5 · p. 7, §4.5 · p. 8, Figure 6 · p. 8

5 Conclusion

• Updating the language backbone and visual merger alongside the regression head is important for accurate box prediction.Geometric supervision makes the existing multimodal representation suitable for localization.Source: §5 · p. 8
• Across evaluated benchmarks, Jag improves grounding accuracy over Base and is competitive with existing grounding models, with lower single-request latency and low GPU memory use.Direct prediction returns all four coordinates in one forward pass while retaining image–language processing.Source: §5 · p. 8
• Compared with a fine-tuned autoregressive model, Jag has numerically close accuracy at the standard overlap criterion but a trade-off in precise box placement at stricter criteria.The paper presents direct prediction as a practical choice when responsiveness matters alongside target localization.Source: §5 · p. 8

Limitations

• Jag focuses on single-object localization in static images.Extending the setting to multiple referents, video, or segmentation calls for task-specific design and evaluation.Source: Limitations · p. 9
• The evaluation uses natural-image referring-expression benchmarks, so specialized-domain applications should be assessed on representative data.Source: Limitations · p. 9
• Inference efficiency depends on input resolution, expression length, hardware, and deployment settings.Reported speedups describe the measured end-to-end workloads; other operating conditions should be measured directly.Source: Limitations · p. 9

Ethical Considerations

• Use of the publicly available datasets, pretrained models, and derived models should follow applicable licenses and attribution requirements.Source: Ethical Considerations · p. 9
• Applications involving images of people should respect privacy.Source: Ethical Considerations · p. 9
• Because localization predictions can be incorrect, downstream systems should include verification suited to their intended use, particularly for decisions involving individuals.Source: Ethical Considerations · p. 9

A.1 Data and Training Configuration

• Training uses 321,327 examples from the official RefCOCO, RefCOCO+, and RefCOCOg training partitions, covering 28,158 images.Training uses head adaptation followed by joint training; the vision encoder outside the merger remains frozen.Source: §A.1 · p. 11

A.2 Model Evaluation

• Jag, Base, LocateAnything, and NExT-Chat are evaluated with their specified model configurations and a shared grounding instruction, with protocol details differing by model.The evaluation uses official benchmark splits, and Jag and Base share the stated input and sequence limits.Source: §A.2 · pp. 11–12

A.3 Scoring and Statistical Analysis

• Scoring converts valid predictions to normalized corners in the original image, clips them to [0, 1], and counts malformed, nonfinite, or zero-area outputs as incorrect.Paired image-cluster bootstrap intervals use 2,000 draws for fixed trained models; multi-seed variation is assessed separately.Source: §A.3 · p. 12

B.1 Accuracy and Paired Outcomes

• Jag’s net increases over Base were 1,240, 1,667, and 937 correct expressions on RefCOCO, RefCOCO+, and RefCOCOg, respectively, exceeding Base’s 53, 111, and 78 invalid outputs; Jag produced none.Table 7 reports the four mutually exclusive correctness outcomes; Table 5 gives pooled Acc@0.5 comparisons with paired image-cluster 95% intervals.Source: §B.1 · pp. 12–13, Table 7 · p. 13, Table 5 · p. 13

B.2 Subgroup and Qualitative Analysis

• Groups use three expression-length bands, normalized-area bands, and a spatial-term proxy; all below-1% area groups are empty, and the grouping axes overlap.Within each axis, groups partition the dataset; the spatial proxy includes positional adjectives and relations.Source: §B.2 · p. 13
• On RefCOCO+, spatial-keyword expressions gain 20.06 points versus 15.09 without them (+4.97 points [+1.95, +8.10]); the RefCOCOg medium-minus-large gain contrast is +2.12 points [+0.05, +4.26].Long-minus-short gain intervals include zero on all three datasets.Source: §B.2 · p. 13
• Figure 6 shows one RefCOCOg example per correctness outcome, selected as the nearest distinct-image case to that outcome’s component-wise median IoU pair.Selection uses squared Euclidean distance with deterministic tie-breaking.Source: §B.2 · p. 13

C.1 Training, Readout, and Backbone Controls

• Controls compare expression-only mean pooling, an input <|box_start|> marker, immediate joint training, and frozen-backbone head-only training; at 2B, AR is numerically higher than Jag on all three datasets.Within each data budget, updates, examples, effective batch size, and input limits are matched, but backbone updates, trainable parameters, and FLOPs differ; at full data AR gets 6,796 joint updates versus Jag’s 100 head-only then 6,696 joint updates.Source: §C.1 · pp. 13–14, Table 9 · p. 14, Table 10 · p. 14

C.2 Localization Across IoU Thresholds

• Jag improves over Base at IoU 0.75 and 0.9 on all three datasets, with mAcc gains of 10.17, 14.02, and 9.60 points; AR’s mAcc is higher by 4.24, 4.39, and 6.28 points.At IoU 0.95 on RefCOCO, Jag scores 20.89% and Base scores 22.41%.Source: §C.2 · p. 14, Table 11 · p. 15
• For long RefCOCOg expressions, AR scores 89.98% and Jag 87.93%, a Jag-minus-AR difference of -2.05 points [-3.16, -0.87]; AR also scores higher in medium-area and spatial-term groups.The long-minus-short contrast in model differences is -1.70 points [-3.38, -0.08]; contrasts with large-area and untagged groups include zero.Source: §C.2 · p. 14

D.1 Model Comparison

• Figure 2 compares five models on one A100 after warmup; synchronized end-to-end timers cover image opening through output parsing, and GPU memory is sampled during prediction.Models use differing preprocessing and attention setups, including native 336×336 CLIP preprocessing and eager attention for NExT-Chat.Source: §D.1 · pp. 14–15, Table 12 · p. 15

D.2 Direct and Autoregressive Prediction

• Jag reduces mean single-request latency by 8.90× and P95 latency from 623.98 ms to 74.28 ms; at batch size eight, it retains a 3.54× throughput advantage over AR.The timing includes image–text processing and output computation.Source: §D.2 · p. 15, Table 13 · p. 15

E.1 Artifact Use and Licenses

• The pretrained weights, benchmark data, and software retain their original licenses and attribution requirements, including conditions on derived models.Model files omit the original COCO images and referring-expression annotations, which must be obtained separately.Source: §E.1 · pp. 15–16

E.2 Artifact Documentation

• The study uses existing RefCOCO, RefCOCO+, and RefCOCOg annotations for general visual scenes with one referent per expression, introducing no new benchmark or human annotation.ModelScope ms-swift supports training, EvalScope provides evaluation, and Docker specifies the execution environment.Source: §E.2 · p. 16

E.3 AI Assistants in Research and Writing

• AI assistants helped polish the manuscript; the authors reviewed it and retain responsibility for the scientific content, results, and interpretations.Source: §E.3 · p. 16