PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Multi-Image Understanding through Delimiter Token Scaling

Minyoung Lee, Yeji Park, Dongjun Hwang, Yejin Kim, Seong Joon Oh, Junsuk Choe

arXiv:2602.01984v2cs.CV

TL;DR

LVLMs struggle with cross-image information leakage because delimiter tokens do not fully isolate visual contexts. The paper scales delimiter-token hidden states to reduce cross-image interaction while preserving intra-image interaction, improving multi-image and text-only separation tasks without added training or inference cost.

  • Problem

    LVLM performance degrades on multi-image inputs because information can mix across images, while existing delimiter tokens do not fully prevent cross-image interaction.

  • Method

    The method scales image delimiter-token hidden states to strengthen attention absorption and image-wise tagging while preserving intra-image interaction.

  • Results

    The method consistently improves multi-image benchmarks and text-only multi-document and multi-table benchmarks without additional training or inference cost.

  • Takeaways & Limitations

    Delimiter-token scaling is easy to integrate and generalizes across multi-image and text-only settings requiring clear separation.

  • Takeaways & Limitations

    The method is currently not applicable to videos lacking explicit frame-separating tokens.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) achieve strong performance on single-image tasks, but their performance declines when multiple images are provided as input. One major reason is the cross-image information leakage, where the model struggles to distinguish information across different images. Existing LVLMs already employ delimiter tokens to mark the start and end of each image, yet our analysis reveals that these tokens fail to effectively block cross-image information leakage. To enhance their effectiveness, we propose a method that scales the hidden states of delimiter tokens. This enhances the model's ability to preserve image-specific information by reinforcing intra-image interaction and limiting undesired cross-image interactions. Consequently, the model is better able to distinguish between images and reason over them more accurately. Experiments show performance gains on multi-image benchmarks such as Mantis, MuirBench, MIRB, and QBench2. We further evaluate our method on text-only tasks that require clear distinction. The method improves performance on multi-document and multi-table understanding benchmarks, including TQABench, MultiNews, and WCEP-10. Notably, our method requires no additional training or inference cost.

1 INTRODUCTION

LVLMs perform strongly with one image but degrade on multi-image inputs because information can leak across images. The paper analyzes delimiter tokens and scales their hidden states to strengthen image separation while preserving intra-image interaction.

  • Multi-image LVLM performance declines because models struggle to distinguish individual images, causing information to mix across them.
  • Existing delimiter tokens help distinguish images but do not fully isolate visual contexts, so cross-image interaction persists.
  • Scaling delimiter-token hidden states strengthens attention absorption and intra-image interaction, reducing cross-image interaction.
  • The method improves multi-image benchmarks including Mantis, MuirBench, MIRB, and QBench2.
  • The method also improves TQABench, MultiNews, and WCEP10 without additional training or inference overhead.

2 RELATED WORK

Prior work addresses multi-image understanding through training-based or training-free methods, while this paper studies delimiter tokens as localized mechanisms for separating images without added computation or architectural changes.

  • Training-based approaches improve multi-image understanding but require costly curation of high-quality instruction data and substantial training resources.
  • AVAM reduces visual redundancy by selecting relevant regions, but relies on an external text encoder and separate preprocessing module.
  • This method enhances multi-image understanding without additional training, inference-time overhead, or architectural modifications.
  • The paper analyzes image delimiter tokens as a distinct sink-like behavior that prior work had not examined in multi-image LVLM settings.
  • Unlike globally influential text sink tokens, LVLM delimiter tokens receive substantial attention and operate as localized bias terms for corresponding images.

3 DO IMAGE DELIMITER TOKENS REALLY WORK?

Image delimiter tokens create attention boundaries that help LVLMs distinguish images, but they do not completely prevent cross-image interaction.

  • The analysis tests delimiter-token function by comparing attention maps with tokens present, removed, or replaced.
  • With delimiter tokens, triangular attention patterns delineate image boundaries; removing them eliminates these patterns, while replacement special tokens produce similar effects.
  • Approximately 10 percentage points of performance are lost when image delimiter tokens are removed or replaced.
  • Despite supporting image distinction, delimiter tokens leave some cross-image interaction in the attention map, so their isolation effect is incomplete.

4 IMAGE-WISE TAGGING VIA DELIMITER TOKENS

Delimiter tokens provide image-wise tagging by contributing localized shared biases to tokens in their corresponding images. This reinforces intra-image interaction and motivates scaling their hidden states.

  • The paper identifies two delimiter-token properties that contribute to distinguishing images.
  • Each delimiter token corresponds to its associated image, forming an image-specific correspondence.
  • For Image 3, the third delimiter contribution is about 15 times larger than the second and 30 times larger than the first.
  • The paper calls this localized shared-bias mechanism image-wise tagging.
  • Delimiter tokens reinforce intra-image interaction through a shared additive contribution to tokens within the same image.

5 METHOD

The method scales image delimiter hidden states to strengthen delimiter attention and image tagging, reducing undesired cross-image interaction while preserving intra-image interaction. It modifies hidden states rather than attention mechanisms, retaining compatibility with optimized implementations and avoiding added inference overhead.

  • 5 METHOD: Scaling delimiter hidden states strengthens attention absorption and image tagging, helping separate images while preserving intra-image interaction.The method amplifies both delimiter properties identified by the authors.
  • 5 METHOD: λ > 1 scales the hidden state of each image delimiter token, while non-delimiter token hidden states remain unchanged.D denotes the set of image delimiter token indices.
  • 5.2 EMPIRICAL EVIDENCE: Approximately 50% lower attention from Image 3 to Images 1 and 2 indicates substantially reduced cross-image interaction after scaling.The interaction is measured as average attention between token pairs from different images.
  • 5.2 EMPIRICAL EVIDENCE: Interactions within Image 3 remain largely unaffected, while Image 3 tokens receive their strongest attention from the third delimiter token.These findings support preserved intra-image interaction and enhanced image tagging.
  • 5 METHOD: The method modifies hidden states without changing attention, enabling compatibility with FlashAttention and avoiding additional inference overhead.Direct attention modification can substantially increase memory usage on multi-image inputs.
  • 5.3 DISCUSSIONS: Text-to-image interaction scores drop by only 10%, while overall text-vision interaction remains robust.The authors report minimal interference with text-vision alignment.

6 EXPERIMENTS

The experiments evaluate delimiter token scaling across multi-image, multi-document, and multi-table tasks, finding consistent performance gains across models and benchmarks without added inference cost. Qualitative and comparative analyses further indicate improved image distinction and reduced cross-image leakage.

  • 6.1 BENCHMARKS AND SETTINGS: The evaluation covers multi-image benchmarks including Mantis, MuirBench, MIRB, and QBench2, plus multi-document and multi-table benchmarks.The multi-image benchmarks test skills such as co-reference, comparison, temporal reasoning, perception, and cross-image reasoning; the additional tasks include MultiNews, WCEP10, and TQABench.
  • 6.2 RESULTS: The method consistently improves performance across Qwen2.5-VL, InternVL3, and LLaVA-OneVision on diverse multi-image benchmarks.For example, Qwen2.5-VL-3B improves from 37.31 to 42.42 on Muirbench, while InternVL3-2B improves from 52.07 to 54.38 on Mantis.
  • 6.2 RESULTS: ROUGE-1, ROUGE-2, and ROUGE-L scores consistently improve on WCEP10 and MultiNews, while TQABench also shows consistent gains.Qwen2.5-3B with the method even outperforms the Qwen2.5-7B baseline on TQABench.
  • 6.2 RESULTS: Qualitative examples show that delimiter token scaling reduces cross-image leakage and preserves correspondence between image-specific content and answers.The method correctly identifies which image contains a bicycle rider and preserves the correspondence between polar bears and camels across images.
  • 6.2 RESULTS: M-RoPE alone performs below the baseline, while combining it with image delimiter tokens improves performance but remains below the proposed method.The comparison supports the effectiveness of hidden-state scaling for distinguishing images relative to the tested temporal-embedding strategy.
  • 6.2 RESULTS: The method improves performance on larger models, few-shot and interleaved settings, and diverse input modalities without additional memory or inference-time cost.Average and peak VRAM usage and inference time remain unchanged from the baseline.

7 LIMITATION

The method has a stated scope limitation: it is not currently applicable to videos without explicit frame-separating tokens.

  • 7 LIMITATION: The method is currently not applicable to videos that lack explicit frame-separating tokens.Future work proposes incorporating mechanisms for modeling temporal transitions between frames.

8 CONCLUSION

The paper addresses cross-image information leakage by enhancing image delimiter tokens through hidden-state scaling. The method improves multi-image and text-only separation tasks without additional training or inference cost.

  • The method enhances image delimiter tokens to suppress cross-image interactions while preserving interactions within each image.It is designed to improve image distinction in multi-image inputs.
  • Performance consistently improves across multi-image benchmarks and text-only multi-document and multi-table tasks.Reported benchmarks include Mantis, MuirBench, MIRB, QBench2, TQABench, MultiNews, and WCEP10.
  • The approach introduces no additional training or inference cost and is easy to integrate.

A.1 ROLE OF IMAGE-DELIMITER TOKENS

Image delimiter tokens help separate visual inputs, but replacing them with other special tokens or removing them fails to separate images effectively. These findings support the functional importance of the original delimiter tokens.

  • Replacing the image-start delimiter with tokens such as <|im end|>, <|box start|>, or <|endoftext|> fails to separate images effectively.
  • Removing image delimiter tokens or replacing them with other special tokens causes a performance drop.
  • Special tokens such as <|vision_start|> effectively serve as image delimiters that separate visual content.

A.2 EMPIRICAL EVIDENCE OF ASSUMPTION

Attention increases most strongly toward the delimiter corresponding to the queried image. For tokens from the third image, the third delimiter receives substantially more attention than the other delimiters.

  • The third image’s delimiter receives the largest attention increase when the query token belongs to the third image.
  • The third delimiter’s increase is approximately 52 times greater than the first delimiter’s and 9 times greater than the second delimiter’s.

A.3 EXPERIMENTAL SETTINGS

The experiments evaluate delimiter-token behavior, scaling choices, implementation settings, ablations, efficiency, and qualitative attention patterns. Results support delimiter-specific scaling as a lightweight method for reducing cross-image interaction while preserving image separation.

  • Role of delimiters: Tokens in Image 3 show the largest attention increase toward their delimiter.
  • Experimental settings: Hyperparameters are tuned using validation data for multi-image tasks but directly on the test set for multi-document and multi-table tasks.
  • Ablations: The study includes ablations of first-token scaling, delimiter replacement, query/key/value scaling, layer selection, and comparisons with FOCUS.
  • Ablations: Scaling the first token improves performance slightly over the baseline but remains substantially below the proposed delimiter-token scaling method.
  • Ablations: Scaling a token that merely appears between images drops performance below the baseline, whereas scaling the actual delimiter token improves performance.
  • Implementation and efficiency: The method primarily selects consecutive early layers, with the selected layers fixed across benchmarks for the same model.
  • Attention analysis: Delimiter-token scaling further reduces attention entropy in image regions by suppressing cross-image attention, while an entropy-based extension adds memory and inference-time cost.
  • Implementation and efficiency: Our method consistently outperforms FOCUS on Mantis while FOCUS uses substantially more memory.
Loading 2602.01984v2…