PaperlayerView exact revision on arXiv

Source-linked AI summary

WinARDet: Compressed Window Auto-Regression for Long-Horizon Vision-Language Grounding

Chenxu Wang, Hangyi Kuang, Bo-Wen Yin, Yuxuan Li, Quan-Sheng Zeng, Yang Zhang, Xu Guo, Xiang Li, Qibin Hou

arXiv:2610.19419v1cs.CV

TL;DR

Dense-scene MLLM detectors can fall into box-repetition loops during long autoregressive decoding. WinAR compresses completed answer windows while retaining exact generation within the current window. On RSDense, WinARDet improves RexOmni F1 and reduces prediction redundancy.

  • Problem

    In dense scenes, full-history autoregressive detectors can lose new-object discovery as repeated coordinates accumulate in the answer history and consume decoding budget.

    Source: §1 · pp. 1–3

  • Method

    WinAR keeps exact autoregression within the current window and represents completed windows with fixed sets of latent tokens for subsequent decoding.

    Source: §1 · pp. 2–3

  • Results

    On RSDense, WinAR raises RexOmni F1@IoU=0.5 from 32.16 to 39.36 and reduces mean prediction redundancy from 35.48% to 13.00%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    WinAR generalizes across decoding paradigms and structured geometric outputs, supporting its use for long-horizon vision-language grounding.

    Source: §4.2 · p. 8, §5 · p. 10

Abstract

from arXiv · show

MLLM-based detectors formulate vision-language grounding as autoregressive coordinate generation, yet in dense scenes, they often collapse into a box repetition loop, repeatedly emitting identical boxes until the decoding budget is exhausted. We trace this failure to historical-answer attention takeover: Under full-history autoregression, repeated coordinate tokens in the growing answer history attract disproportionate attention, crowding out the attention sinks needed to attend to new visual evidence. % To break this loop, in this paper, we propose \textbf{Compressed Window Auto-Regression} (WinAR). This new mechanism partitions a long answer sequence into consecutive generation windows and restricts full token-level autoregression to the current window only while compressing each previous window into a fixed set of latent tokens that condition subsequent decoding. This design reduces the repeated exposure to raw historical coordinates that triggers the attention takeover, yet preserves cross-window dependencies essential for coherent multi-object generation. On RSDense, which contains 200-2,000 objects per image, our model WinARDet improves RexOmni F1-score (IoU=0.5) from 32.16 to 39.36 over the baseline, while reducing the redundancy rate from 35.48% to 13.00%. WinARDet also generalizes across decoding strategies and output formats, including parallel box decoding and polygon generation, providing a practical paradigm for long-horizon vision-language grounding across scene densities.

1 INTRODUCTION

Dense-scene grounding requires detectors to keep discovering new objects over long decoding horizons, but full-history generation can collapse into repeated boxes. WinAR compresses completed answer windows while retaining exact generation within the current window, improving reported results on dense-scene benchmarks.

1 INTRODUCTION

  • Under FHAR, dense-scene decoding can deteriorate into repeated boxes as new-object discovery falls, consuming budget that could recover uncovered objects.Over the initial 400 predictions, redundancy rises from approximately 10% to near 100%, while novel-object discovery falls from roughly 16% to nearly zero.Source: §1 · pp. 1–2
  • The attention analysis links this loop to growing attention on raw answer history and a sharp reduction in attention to initial system tokens.The passages describe those initial tokens as potential attention sinks and report that repeated coordinates increasingly bias decoding toward previously generated locations.Source: §1 · p. 2, Figure 2 · p. 3
  • WinAR keeps exact token-level history only in the current window and compresses each completed window into a fixed number of latent tokens.This retains cross-window conditioning while limiting exposure to raw coordinate patterns; training supervises the current window and inference compresses each completed window before proceeding.Source: §1 · pp. 2–3
  • On RSDense, WinAR raises RexOmni F1@IoU=0.5 from 32.16 to 39.36 and reduces mean prediction redundancy from 35.48% to 13.00%.RSDense contains 200–2,000 objects per image.Source: §1 · p. 2
  • WinAR also generalizes across decoding strategies and output formats, while reducing training memory for long-sequence grounding.LocateAnything results and polygon detection results support generalization beyond serial box generation; training peak GPU memory is reduced by 34.1%.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 RELATED WORK

Conventional detectors use discriminative classification and regression, while MLLM-based detectors cast detection as generation to support complex language instructions. Existing generative approaches still retain growing raw answer histories during decoding.

2 RELATED WORK

  • Conventional regression-based detectors use classification and localization losses, while open-vocabulary variants align category names or prompts with regional visual features.The passage says these systems remain limited in complex instruction understanding and heterogeneous vision-language supervision.Source: §2 · p. 3
  • MLLM-based detectors unify vision-language tasks under a generative objective, with object categories and coordinates represented as sequence tokens.RexOmni uses geometry-aware GRPO post-training, while LocateAnything treats each box or point as an atomic unit for parallel decoding.Source: §2 · pp. 3–4
  • RexOmni generates a flat answer sequence token by token, whereas LocateAnything predicts coordinate tokens in box-aligned blocks in parallel.WinAR uses fixed-length token windows for RexOmni outputs and windows of complete blocks for LocateAnything.Source: Figure 3 · p. 4, §2 · p. 4
  • Despite these advances, the cited MLLM detectors retain the growing raw answer history and do not explicitly address long-horizon dense grounding.Accumulated coordinate tokens can dominate later predictions and induce a self-reinforcing repetition loop.Source: §2 · p. 4

3 METHOD

WinAR replaces full-history autoregression with windowed generation conditioned on compact historical context. It preserves local token-level history while carrying completed windows forward as latent memory.

3.1 PRELIMINARIES

  • MLLM detectors condition generation on a multimodal prefix and encode object categories and quantized box coordinates as structured answer sequences.RexOmni serializes detections as a flat sequence, whereas LocateAnything uses fixed-length atomic blocks to support parallel box decoding.Source: §3.1 · pp. 4–5

3.2 COMPRESSED WINDOW AUTO-REGRESSION

  • WinAR partitions answers into consecutive windows, retaining explicit token history only within the current window and conditioning later windows on compressed representations of completed windows.The multimodal prefix remains available throughout generation, while the answer context grows through latent tokens rather than raw coordinate sequences.Source: §3 · p. 4, §3.2 · p. 5
  • The Window History Compressor preserves B tokens at each window boundary and compresses only the middle into N latent tokens initialized from uniformly pooled intervals.Learnable window-local positional embeddings supply relative order for cross-attention; the resulting compressed representation includes the uncompressed boundary tokens.Source: §3.2 · pp. 5–6
  • Training processes windows sequentially, supervises only the current target window, and represents preceding windows solely through their compressed memory.At inference, each completed window is compressed and removed from explicit answer context before generation proceeds with the prefix, accumulated memory, and current-window tokens.Source: §3.2 · pp. 5–6

4 EXPERIMENT

The experiments evaluate WinAR on standard and extremely dense detection settings, across decoding paradigms and polygon outputs, and examine its configuration and efficiency. WinAR improves dense-scene grounding while suppressing duplicate predictions, with trade-offs across window and compressor settings.

4.1 TRAINING DETAILS AND EVALUATION SETUP

  • RexOmni variants use four training epochs, while the evaluation reports category-averaged Recall, Precision, and F1 at IoU 0.5.RSDense comprises evaluation images with more than 200 objects; its duplicate and over-prediction rates are image-averaged.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • On RSDense, RexOmni-WinAR raises F1 from 32.16 to 39.36 and lowers Mean Rdup from 35.48% to 13.00% versus FHAR*.The improvement is more pronounced under higher scene density.Source: Table 1 · p. 7, §4.2 · p. 7
  • RexOmni-WinAR reduces peak training memory from 57.62 GB to 42.97 GB and raises Valid Boxes/s from 3.57 to 5.40 on RSDense.Raw throughput does not consistently improve because of history-compression and KV-cache-prefilling overhead at window boundaries.Source: Table 2 · p. 7, §4.2 · p. 8
  • For polygon detection, WinAR raises F1 from 38.64 to 44.00 on SODA-A and from 47.25 to 48.93 on DOTA2 versus FHAR*.On SODA-A, Mean Rdup decreases from 3.72% to 1.93%; polygons contain eight coordinate tokens per object.Source: Table 3 · p. 8, §4.2 · p. 8
  • On RSDense, LocateAnything-WinAR under PBD raises F1 from 26.15 to 27.54 and lowers Mean Rdup from 14.89% to 2.13%.Across NTP, PBD, and Hybrid decoding, WinAR suppresses redundant predictions while maintaining comparable or improved grounding accuracy.Source: Table 1 · p. 7, §4.2 · p. 8

4.3 ABLATION STUDY

  • On RSDense, reducing window length from 300 to 100 lowers Mean Rdup from 5.38% to 0.89% but lowers F1 from 44.78 to 43.71.Removing boundary preservation at L = 200 lowers RSDense F1 from 44.56 to 32.47, despite increasing compressed tokens from 6 to 20.Source: Table 4 · p. 9, §4.3 · p. 9
  • A 6-layer Window History Compressor achieves the best F1 on both DOTA2 & SODA-A (67.97) and RSDense (39.36).Eight layers raise RSDense Recall from 45.23 to 45.75 but lower F1 from 39.36 to 39.32 and increase Mean Rdup from 13.00% to 16.87%.Source: Table 5 · p. 9, §4.3 · p. 9

5 CONCLUSIONS

WinAR addresses long-horizon box repetition by keeping exact autoregression within the current window and representing completed windows with compact latent tokens. It benefits denser scenes and generalizes across decoding paradigms and structured geometric outputs while improving efficiency and reducing training memory.

5 CONCLUSIONS

  • WinAR keeps exact autoregression within the current window and represents completed windows with compact latent tokens, limiting repeated exposure to raw coordinates while retaining cross-window dependencies.This addresses a long-horizon failure mode in which dense-scene generation collapses into a self-reinforcing box repetition loop.Source: §5 · p. 9
  • WinAR benefits denser scenes and generalizes across decoding paradigms and structured geometric outputs, while improving effective detection efficiency and reducing training memory for long sequences.The authors characterize compressed window autoregression as a practical paradigm for reliable long-horizon vision-language grounding.Source: §5 · p. 10

APPENDIX CONTENTS

The appendix provides benchmark construction and statistics, implementation configurations, evaluation protocols, and additional all-prediction comparisons.

APPENDIX CONTENTS

  • Benchmark Details covers construction, preprocessing, splits, and statistics for DOTA2 & SODA-A, RSDense, DOTA2-Poly, SODA-A-Poly, SKU110K, and COCO.Source: APPENDIX CONTENTS · p. 13
  • Additional Implementation Details gives training configurations for RexOmni and LocateAnything, while Evaluation Protocol describes capped and complementary all-prediction evaluation.The protocols analyze object discovery, prediction redundancy, and generation termination.Source: APPENDIX CONTENTS · p. 13
  • More Experiments reports all-prediction comparisons across COCO, DOTA2 & SODA-A, SKU110K, and RSDense.Source: APPENDIX CONTENTS · p. 13

A.1 BENCHMARK DETAILS

The experiments use benchmarks spanning dense remote-sensing and retail scenes, quadrilateral outputs, and ordinary visual scenarios. RSDense specifically targets highly dense scenes using images with more than 200 objects.

A.1 BENCHMARK DETAILS

  • DOTA2 and SODA-A are combined into a dense-grounding benchmark, with DOTA2 validation and SODA-A test data forming its evaluation set.DOTA2 training uses 1024 × 1024 crops with 200-pixel overlap; SODA-A uses 800 × 800 crops at a stride of 650 pixels.Source: §A.1 · p. 13
  • RSDense selects images with more than 200 objects from the DOTA2 & SODA-A evaluation set and contains 712 images averaging 412 objects each.Ship and Small Vehicle are the two representative dense categories analyzed.Source: §A.1 · p. 13
  • DOTA2-Poly and SODA-A-Poly test quadrilateral outputs, which use eight coordinate tokens per object rather than four for standard horizontal boxes.The datasets are trained and evaluated separately using oriented-box annotations.Source: §A.1 · pp. 13–14
  • SKU110K tests dense object detection on supermarket-shelf products, using its 8,769-image training split and 2,920-image test split.It assesses generalization beyond remote-sensing imagery.Source: §A.1 · p. 14
  • COCO is used to assess whether gains in dense scenes come without sacrificing competitive grounding in typical scene densities.The setup trains on COCO train and evaluates on its validation set.Source: §A.1 · p. 14

A.2 ADDITIONAL IMPLEMENTATION DETAILS

The implementation compares standard FHAR training with WinAR and exposure-matched controls for RexOmni, and applies corresponding training strategies to LocateAnything. WinAR windowing increases exposure to long-answer samples, so the comparisons control for that change.

A.2 ADDITIONAL IMPLEMENTATION DETAILS

  • RexOmni-FHAR reproduces the baseline protocol with four training epochs, 1,424 optimizer updates, and a maximum sequence length of 20,000 tokens.Samples are exposed once per epoch regardless of answer length.Source: §A.2 · p. 14
  • RexOmni-WinAR uses 300-token windows, creating approximately ⌈T/L⌉ training instances for an answer of T tokens and increasing long-answer exposure.With the training set, it performs 1,864 optimizer updates over four epochs.Source: §A.2 · pp. 14–15
  • The exposure-matched RexOmni-FHAR* baseline repeats samples to match WinAR's effective exposure and 1,864 updates while retaining full-history autoregressive training.This comparison separates the effect of historical-answer representation from gains that could arise from additional training exposure.Source: §A.2 · p. 15
  • RexOmni-FHAR*-GRPO starts from the FHAR* SFT checkpoint and receives one epoch of GRPO post-training on training images containing more than 200 objects.Training uses eight H100 GPUs, with four rollouts per prompt in groups of four candidate trajectories.Source: §A.2 · p. 15
  • LocateAnything uses an exposure-matched FHAR* comparison, while its WinAR windows are constructed over box-aligned atomic blocks rather than individual tokens.Other optimization settings, including the compressor learning rate, follow the RexOmni-WinAR strategy.Source: §A.2 · p. 15

A.3 EVALUATION PROTOCOL

The evaluation uses category-averaged Recall, Precision, and F1 at IoU 0.5, with a distinct class-agnostic protocol for RSDense. Because capped metrics can obscure generation behavior, the paper also reports uncapped all-prediction results as a complement.

A.3 EVALUATION PROTOCOL

  • Recall, Precision, and F1 are averaged across categories at IoU 0.5, retaining 100 predictions per category except for DOTA2 and SODA-A, which retain up to 1,000.RSDense instead uses all predicted boxes with class-agnostic matching within each image, averages metrics over images, and prompts only with categories present in each image.Source: §A.3 · pp. 15–16
  • Top-K evaluation can hide redundant late predictions and discard valid detections, so it gives only a partial view of generation termination and efficiency.The passage notes that excluding late predictions can limit Recall while filtering redundant predictions can make Precision appear overly optimistic.Source: §A.3 · p. 16
  • The supplementary all-prediction evaluation retains every generated prediction and uses category-wise metrics averaged across categories to characterize discovery, redundancy, and termination.It complements rather than replaces standard evaluation and is intended to assess behavior without an additional per-image cap.Source: §A.3 · p. 16

A.4 MORE EXPERIMENTS

Under all-prediction evaluation, WinAR improves F1 over the exposure-matched FHAR* baseline across datasets, with its largest gains on dense RSDense. It also reduces training memory and increases valid-prediction throughput, while qualitative comparisons report fewer repeats and more complete polygon predictions.

Additional Results on More Datasets

  • RSDense Recall rises from 38.95 to 47.69 and F1 from 45.91 to 50.37 with WinAR under all-prediction evaluation.WinAR improves F1 over exposure-matched FHAR* across evaluated datasets; reported F1 changes include 65.48 to 66.04 on COCO and gains of 0.75 and 0.48 points on DOTA2 & SODA-A and SKU110K, respectively.Source: §A.4 · pp. 16–17, Table 9 · p. 17

Training and Inference Efficiency

  • On RexOmni, Valid Boxes/s increases from 3.57 to 5.40, while LocateAnything increases from 5.14 to 5.37 under NTP and 24.63 to 25.61 under PBD.Raw prediction throughput does not consistently improve because compression and KV-cache prefilling add overhead at window boundaries.Source: Table 10 · p. 17, §A.4 · p. 17
  • Training peak memory falls from 57.62 GB to 42.97 GB on RexOmni and from 67.38 GB to 44.43 GB on LocateAnything.The window-wise formulation lets the model discard completed windows' raw token representations and activations, fitting within the 48 GB NVIDIA A40 memory budget.Source: §A.4 · p. 17

Qualitative Comparison

Related papers

Loading 2610.19419v1…