PaperlayerView exact revision on arXiv

Source-linked AI summary

Query-adaptive Recurrent Tokenization

Bavesh Balaji, Pranav Gopinath, Pouya Sadeghi, Sirisha Rambhatla

arXiv:2610.22957v2cs.CV

TL;DR

Visual evidence and token budgets should depend on the text query, but common models largely encode images independently of queries with fixed budgets. QART selects query-relevant evidence and recurrently builds a compact representation, adding tokens according to reconstruction quality. Across evaluated tasks it preserves strong downstream performance with fewer tokens, including 81.96% accuracy on Visual7W pointing with 29 tokens.

  • Problem

    Common visual representations are query-independent and use fixed token budgets even though the relevant evidence and required capacity vary with the query.

    Source: §1 · pp. 1–2

  • Method

    QART uses query-conditioned relevance selection and recurrently refines visual slots, adding tokens until reconstruction of query-relevant features is sufficient.

    Source: §1 · p. 2

  • Results

    Across retrieval, grounding, compositional matching, and VQA, QART produces compact representations with substantially fewer tokens while preserving strong downstream performance; Visual7W pointing reaches 81.96% with 29 tokens.

    Source: §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Visual representations can adapt both their content and capacity to what each query requires.

    Source: §5 · p. 9

    The code cannot be released because the project was conducted with an industrial partner, limiting reproducibility through code access.

    Source: REPRODUCIBILITY STATEMENT · p. 10

Abstract

from arXiv · show

Visual information worth extracting from an image and the token budget it warrants depend on the text query. Yet modern vision and vision-language models largely encode images independently of the query and allocate a fixed visual token budget to every input. We introduce the Query-Adaptive Recurrent Tokenizer (QART), a backbone-agnostic module that makes visual representations both query-steerable and budget-adaptive. QART steers the visual representation toward query-relevant evidence while adapting its token budget to the amount of information required. Its key principle is simple. Additional tokens are allocated until the visual features relevant to the query are represented with sufficient fidelity. Starting from frozen encoder features, QART recurrently constructs a compact representation and uses reconstruction quality over the query-relevant visual support to control token allocation. On Visual7W and VQAv2, QART achieves 81.96% and 76.37% accuracy using only about 29 visual tokens, reducing token counts by 20-25x relative to dense VLM inputs. It further reaches 92.0% P@1 on CORE and 36.3% PR-AUC on PODS, outperforming SteerViT on PODS by 8.4%. Together, these results show that visual representations need not be fixed in either content or capacity and can instead adapt to what each query requires.

1 INTRODUCTION

The visual evidence needed from an image depends on the query, but common encoders use query-independent representations and fixed token budgets. QART couples query steering with adaptive capacity, allocating tokens until query-relevant evidence is represented sufficiently.

1 INTRODUCTION

  • Visual information requirements vary by query, while common encoders produce the same image representation regardless of the query.A traffic-light color question may depend on a small region, whereas counting turning vehicles depends on information distributed across objects and spatial relationships.Source: §1 · p. 1
  • Fixed token budgets can increase downstream computation when large and discard necessary details when small.The paper distinguishes which evidence to preserve from how much capacity to allocate.Source: §1 · p. 1
  • Prior work addresses compact, steerable, or adaptive tokenization, but the paper identifies coupling query-relevant evidence selection with capacity allocation as less explored.QART allocates capacity according to how well the current representation preserves query-relevant visual evidence.Source: §1 · p. 2
  • QART steers a compact representation toward query-relevant evidence and adds tokens until reconstruction quality supports stopping.Frozen encoder features and a text query inform recurrent refinement; reconstruction quality controls whether tokens are added or computation halts.Source: §1 · p. 2
  • 81.96% accuracy with 29 tokens on Visual7W pointing retains 92.6% of dense-model accuracy and outperforms DART by 22.1%.On VQAv2, QART retains 93.8% of dense accuracy while reducing tokens by 25× and improves end-to-end throughput by 1.53×.Source: §1 · p. 2

2 RELATED WORK

Prior work separately studies compact or variable-length tokenization, language-conditioned visual representations, and token pruning or merging. QART’s distinction is to couple query-conditioned evidence selection with recurrent allocation of representation capacity.

2.1 COMPACT AND VARIABLE-LENGTH VISUAL TOKENIZATION

  • Adaptive-length methods vary capacity with image content; QART instead adapts to the image–query pair and halts when query-relevant features are sufficiently reconstructed.The same image can therefore produce different token contents and lengths under different queries.Source: §2.1 · p. 3

2.2 QUERY-CONDITIONED AND STEERABLE VISUAL REPRESENTATIONS

  • Language-steerable methods condition visual representations on queries, but QART couples this steering to recurrent capacity allocation under an explicit representation-adequacy criterion.Its token count is not externally fixed or separately predicted.Source: §2.2 · p. 3

2.3 VISUAL TOKEN PRUNING AND MERGING

  • Pruning and merging methods retain, merge, or discard existing tokens; QART constructs and recurrently expands slots from frozen patch features until query-conditioned reconstruction criteria are met.This supplies a shorter visual sequence to downstream models.Source: §2.3 · p. 3

3 METHOD

QART selects query-relevant visual features, builds a recurrent slot representation, and uses reconstruction coverage to determine whether to add slots or halt. Its trainable components use query-weighted reconstruction and relevance-prediction objectives.

3 METHOD

  • QART converts dense encoder features into variable-length query-conditioned tokens through relevance selection, recurrent slot construction, and joint optimization.The recurrent Transformer is weight-shared, begins with query-initialized slots, and adds slots when reconstruction coverage falls below a criterion.Source: §3 · p. 4

3.1 IDENTIFYING QUERY-RELEVANT EVIDENCE

  • A cross-modal relevance module predicts patch relevance from frozen visual and text features, selecting support A and corresponding visual features VA for compact representation.At inference, the predicted relevance map and reconstruction coverage jointly determine token count and recurrent rounds without relevance supervision.Source: §3.1 · p. 4, §3.4 · p. 5

3.2 RECURRENT QUERY-CONDITIONED REPRESENTATION

  • Query-pooled features initialize K0 slots, which are recurrently refined with selected visual features and text through a shared Transformer.When the slots cannot represent the selected features sufficiently, ∆K query-initialized slots are added up to Kmax.Source: §3.2 · pp. 4–5

3.3 RECONSTRUCTION-GUIDED TOKEN ALLOCATION

  • After each round, a shared decoder reconstructs the visual feature grid, and reconstruction coverage over query-relevant support governs slot growth and recurrence halting.Growth and halt thresholds are decoupled, allowing slot addition to stop while existing slots continue refining; query-dependent support yields different allocation trajectories for the same image.Source: §3.3 · p. 5

3.4 LEARNING OBJECTIVE

  • Soft relevance targets derived from segmentation or alignment annotations define per-patch importance weights for query-weighted reconstruction, while relevance prediction uses soft binary cross-entropy.The combined objective includes reconstruction and relevance losses.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments test whether QART can steer frozen visual representations, preserve downstream performance under compression, and translate compactness into inference gains. Across tasks, results show how query conditioning shapes representation content and token allocation.

4.1 EXPERIMENTAL SETUP

  • With frozen SigLIP2 encoders, QART produces at most 32 tokens from 729 patch features; one pretrained checkpoint serves representation benchmarks without task-specific finetuning.For VQA, only a lightweight visual projector is trained while QART and the language model remain frozen.Source: §4.1 · p. 5

4.2 DOES QART REDIRECT A PRETRAINED VISUAL REPRESENTATION?

  • QART reaches 92.0% P@1 vs 68.5% native SigLIP2 on CORE and improves GeneCIS R@1/R@2/R@3 by 2.7/5.0/7.1%, transferring without tuning.The GeneCIS gain increases with retrieval depth, and QART approaches Combiner despite not being designed specifically for composed retrieval.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 CAN QART PRESERVE REASONING AND GROUNDING UNDER EXTREME VISUAL COMPRESSION?

  • 81.96% Visual7W accuracy with 29 tokens beats DART by 22.1%, while 76.37% VQAv2 accuracy retains 93.8% of dense accuracy using 24.8× fewer tokens.With token growth disabled, VQAv2 accuracy rises from 46.27% after one round to 50.93% after three rounds.Source: §4.3 · p. 7, Table 1 · p. 7

4.4 DOES THE CONDITIONING QUERY ACTUALLY CONTROL THE REPRESENTATION?

  • Replacing sample-specific queries with fixed conditioning lowers Winoground Image accuracy to 15.25% and Group accuracy to 12.00%, indicating that QART responds to query composition.The intervention isolates query conditioning from the tokenizer itself.Source: Figure 3 · p. 6, §4.4 · p. 8

4.5 DOES QUERY CONDITIONING PRESERVE INFORMATION BEYOND THE PROMPT?

  • On PODS, QART reaches 36.3% PR-AUC and 61.3% NDCG, beating SteerViT by 8.4% PR-AUC with coarse supercategory prompts.The result indicates that the representation retains fine-grained instance information beyond what the prompt explicitly names.Source: Figure 3 · p. 6, §4.5 · p. 8

4.6 DOES COMPACTNESS TRANSLATE INTO REAL INFERENCE GAINS?

  • At batch size eight, QART reaches 1.53× end-to-end and 2.54× prefill speed-up, raising throughput from 21.0 to 32.2 samples/s.At batch size one, tokenization overhead exceeds the savings, yielding a 0.87× end-to-end speed ratio; the benefit grows as downstream processing dominates.Source: §4.6 · pp. 8–9, Figure 4 · p. 8

4.7 WHAT DOES QUERY CONDITIONING CONTROL?

  • Query-relevant support steers representations (79.7% P@1 without it), while query-aware growth and halting average 17.25 slots and 4.25 rounds at essentially unchanged accuracy.Making budget decisions query-agnostic activates all 32 slots and runs all six rounds; query-aware allocation uses roughly 46% fewer tokens and 29% fewer rounds.Source: §4.7 · p. 9, Table 2 · p. 9

5 CONCLUSION

QART combines query steerability with adaptive token allocation, producing compact representations that preserve strong performance across several visual tasks. Its strongest gains occur when predictions depend on localized evidence.

5 CONCLUSION

  • QART combines query-steerable visual representations with budget-adaptive token allocation.It adds tokens when query-relevant evidence is not yet represented with sufficient fidelity.Source: §5 · p. 9
  • Across retrieval, grounding, compositional matching, and visual question answering, QART uses substantially fewer tokens while preserving strong downstream performance.Source: §5 · p. 9
  • QART’s gains are strongest for predictions relying on localized visual evidence, while it remains competitive with token-reduction methods on general VQA.Source: §5 · p. 9

AI USE STATEMENT

Generative AI tools assisted with feedback, dataset and experiment work, figure modification, and editing. The authors state that they retained responsibility for the final content.

AI USE STATEMENT

  • Generative AI assisted with experiment feedback, dataset reformatting, experiment design, figure modification, and editing paper text and software code.Source: AI USE STATEMENT · p. 10
  • The authors state that AI was not used to generate synthetic datasets, develop theoretical models, formulate mathematical claims, design methodology, implement methods, or interpret qualitative data.Source: AI USE STATEMENT · p. 10
  • The authors take responsibility for the final content, including text, claims, or artifacts produced with AI assistance.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work uses public datasets and pretrained models and does not collect new human-subject data. The authors note that deployment should account for possible inherited biases and application requirements.

ETHICS STATEMENT

  • The work uses publicly available datasets and pretrained models, with no new human-subject data collection.Source: ETHICS STATEMENT · p. 10
  • QART may inherit biases and limitations from its training data and pretrained encoders.Source: ETHICS STATEMENT · p. 10
  • Downstream deployment should consider data and model provenance, biases, and the target application’s requirements.Source: ETHICS STATEMENT · p. 10

A.1 DATASETS

QART is pretrained on diverse image–text annotations with dense query-region supervision, then uses recurrent query-conditioned slots and reconstruction to adapt its visual representation. The setup keeps the backbone encoders frozen and trains only a lightweight projector for LLaVA integration.

A.1 DATASETS

  • 2,078,367 image–text annotation records from referring-expression, scene-understanding, and region-description datasets make up the pretraining mixture.The count refers to annotations, not unique images.Source: §A.1 · p. 15
  • Soft query-region heatmaps supervise relevance prediction and query-weighted reconstruction, while the recurrent tokenizer uses predicted support for growth and halting.The support is defined by patches whose relevance scores exceed τ and remains the basis for decisions during training and inference.Source: §A.1 · p. 15

A.2 SETUP

  • For LLaVA integration, two projector-training stages align QART slots to language-model embeddings while QART and the language model remain frozen.The stages use Visual Genome region descriptions, then LLaVA-Instruct-665K; only the projector is optimized.Source: §A.1 · p. 15, §A.2 · p. 17
  • QART starts with two query-conditioned slots, jointly processes slots, selected visual features, and text, then uses slot-to-patch attention to gather evidence.Only slot states recur; selected visual and text features are supplied again at each round.Source: §A.2 · p. 16
  • A shared decoder reconstructs the projected visual features over selected support, and reconstruction adequacy controls slot growth and inference halting.The maximum-capacity sequence is 2, 8, 14, 20, 26, 32 slots; inference may halt after at least two rounds and otherwise runs up to six.Source: §A.2 · pp. 16–17

A.3 EVALUATION PROTOCOLS

The evaluation protocols test QART across retrieval, matching, visual question answering, and grounding. Representation benchmarks use the same frozen tokenizer, while task-specific protocols define the conditioning queries, scoring, and comparison setups.

A.3 EVALUATION PROTOCOLS

  • CORE and GeneCIS rank normalized mean-slot representations by cosine similarity, while PODS uses coarse supercategory queries and reports object-macro-averaged classification and retrieval metrics.Transfer experiments use the SigLIP2 QART checkpoint and τ = 0.25 unless stated otherwise; each image–query pair runs independently for recurrent halting.Source: §A.3 · pp. 17–18
  • VQAv2 inserts active QART slots before the textual question in frozen LLaVA-1.5-7B and evaluates autoregressively generated answers with the standard protocol.Generation uses greedy decoding with at most 16 tokens; QART uses τ = 0.05.Source: §A.3 · p. 18
  • Visual7W pointing asks the model to choose among four candidate boxes, and all compared methods are reproduced under a common evaluation pipeline.Accuracy requires an exact match to the correct answer letter; pruning baselines share the LLaVA+SigLIP2 base setup, with learnable baselines additionally finetuned on Visual7W.Source: §A.3 · p. 18
  • Winoground scores all four image–caption pairings across its 400 examples, with Text, Image, and Group accuracy computed from each 2×2 similarity matrix.The fixed-query control conditions QART on the same generic query for every image while preserving the official scoring protocol.Source: §A.3 · p. 18

B.1 DATA AND SETUP

The DINOv2 variant uses independently pretrained visual and text encoders but keeps QART’s main architecture and training recipe. Its VQAv2 integration follows the SigLIP2 variant’s protocol.

B.1 DATA AND SETUP

  • The DINOv2 setup uses frozen DINOv2-B/14 image features and a frozen RoBERTa-large text encoder, which do not share a jointly pretrained cross-modal space.Images yield 729 patch features on a 27 × 27 grid, and the text features are projected into the 768-dimensional tokenizer space.Source: §B.1 · pp. 18–19
  • The remaining QART architecture, allocation policy, objective, data mixture, and training schedule are unchanged; the VQAv2 variant follows the SigLIP2 protocol.The model starts with two slots, allows up to 32 slots and six recurrent rounds, and uses the same LLaVA integration and projector-training protocol.Source: §B.1 · p. 19, Table 4 · p. 19

B.2 RESULTS

The DINOv2 results remain close to SigLIP2 on CORE and competitive on VQAv2 despite independently pretrained encoders. Across threshold, recurrence, and query controls, the results show that QART adjusts both its represented evidence and token capacity.

B.2 RESULTS

  • 90.33% P@1 at τ = 0.25 is achieved by DINOv2 QART with 23.34 active slots on average, compared with 92.01% P@1 for SigLIP2 QART.The paper describes the CORE gap as relatively small despite the absence of a jointly pretrained vision–language representation.Source: §B.2 · p. 19
  • 72.93% overall accuracy for DINOv2 QART compares with 76.37% for SigLIP2 QART under the matched VQAv2 training protocol.The largest difference is on number questions, and DINOv2 uses slightly more active slots on average.Source: §B.2 · p. 19

B.3 DISCUSSION

  • QART remains close to SigLIP2 on CORE and competitive on VQAv2 with independently pretrained encoders, while the paper identifies native cross-modal alignment as important for its strongest performance.The DINOv2 variant trails SigLIP2 on VQAv2; the paper attributes the remaining gap primarily to weaker cross-modal initialization.Source: §B.3 · pp. 19–20, §B.2 · p. 19

C.1 THRESHOLD SENSITIVITY AND ADAPTIVE BUDGETING

  • At GeneCIS, lowering τ from 0.20 to 0.05 raises mean slots from 25.38 to 30.04 while R@1 falls from 15.10% to 12.81%.The task-dependent optima differ: CORE performs best at τ = 0.25, GeneCIS at τ = 0.20, and Winoground favors an intermediate threshold; broader support does not guarantee better performance.Source: §C.1 · pp. 20–21

C.2 DOES RECURRENCE REFINE THE REPRESENTATION?

  • Across recurrent rounds, support shifts from distractors or coarse regions toward the queried person, backpack, or jersey number, allowing QART to revise its representation.The examples suggest iterative refinement is useful when the relevant region is visually ambiguous or surrounded by similar content.Source: §C.2 · p. 21, Figure 5 · p. 22

C.3 IS QART QUERY-STEERABLE AND BUDGET-ADAPTIVE?

  • For the same image, changing the query from “left pizza” to “right pizza” redirects support and changes the budget from K = 8 to K = 20.Across examples, active capacity spans K = 2 to K = 32 and varies with the evidence required, not simply the number of queried objects.Source: §C.3 · p. 21, Figure 6 · p. 22

C.4 LIMITATIONS

QART’s evaluation is limited to single-image settings, and its current formulation relies on task-sensitive thresholds and dense query-region supervision during pretraining.

C.4 LIMITATIONS

  • QART is evaluated only in single-image settings; video, multi-turn interaction, and temporally evolving observations remain future directions.The paper identifies these settings as extensions, not as evaluated capabilities.Source: §C.4 · p. 22
  • QART uses thresholds for relevance selection, slot growth, and halting, and preferred operating points can vary across tasks.The authors suggest learned or calibration-free allocation criteria as a way to simplify deployment across heterogeneous settings.Source: §C.4 · p. 22
  • QART uses dense query-region supervision during tokenizer pretraining; weaker or self-supervised relevance signals could reduce annotation requirements.The dense supervision is used to train the relevance predictor and reconstruction objective.Source: §C.4 · pp. 22–23

Related papers

Loading 2610.22957v2…