Source-linked AI summary
See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
Boyuan Sun, Bowen Yin, Yuanming Li, Xihan Wei, Qibin Hou
TL;DR
Fine-grained object understanding remains challenging when MLLMs must ground user-specified objects from text alone, partly because object nouns receive diffuse visual attention. SWIM constructs NL-Refer and applies mask-guided cross-attention alignment during training, achieving state-of-the-art fine-grained benchmark performance without visual prompts at inference.
Problem
MLLMs struggle to identify and describe user-specified objects from textual prompts because object nouns often produce diffuse visual attention and weak text–visual alignment.
Method
SWIM constructs NL-Refer and aligns multi-layer cross-attention maps for object nouns with ground-truth masks during training, without architectural changes.
Results
78.3% average accuracy on VideoRefer-Q exceeds VideoRefer-7B by 6.4 percentage points, while SWIM also surpasses specialist and generalist baselines on VideoRefer-D.
Takeaways & Limitations
SWIM enables fine-grained video object understanding from natural-language references without visual prompts during inference.
Takeaways & Limitations
Experiments are bounded by current mask-annotated data availability, leaving scalability to broader and more diverse corpora untested.
Abstract
from arXiv · showhide
We present SWIM (See What I Mean), a novel training strategy that aligns vision and language representations to enable fine-grained object understanding solely from textual prompts. Unlike existing approaches that require explicit visual prompts, such as masks or points, SWIM leverages mask supervision only during training to guide cross-modal attention, allowing the model to automatically attend to the user-specified object at inference. Our cross-attention analysis of pretrained multimodal large languagemodels (MLLMs) reveals a systematic discrepancy: Attribute words produce sharp, localized activations in the visual modality, whereas object nouns yield diffuse and scattered patterns due to semantic reference bias and distributed high-level representations. To address this misalignment, we construct NL-Refer, an enriched dataset, in which each object mask is paired with a precise natural language referring expression. SWIM extracts multi-layer cross-attention maps from object nouns and enforces spatial consistency with ground-truth masks. Experimental results demonstrate that SWIM substantially improves text-visual alignment and achieves superior performance over visual-prompt-based methods on fine-grained object understanding benchmarks. The code and data are available at \href{https://github.com/HumanMLLM/SWIM}{https://github.com/HumanMLLM/SWIM}.
1. Introduction
MLLMs struggle to consistently ground user-specified objects from text alone because object nouns produce diffuse visual attention. SWIM addresses this mismatch by aligning object-noun attention with masks during training, enabling text-only inference.
- MLLMs often struggle with fine-grained understanding of user-specified objects despite strong holistic scene understanding.
- Existing video approaches use points, masks, or boxes as explicit visual prompts to guide target-object identification.
- Cross-attention analysis finds attribute words sharply localized, whereas object nouns produce diffuse and scattered visual activations.
- SWIM constructs NL-Refer and aligns multi-layer object-noun cross-attention maps with ground-truth masks during supervised fine-tuning.
- SWIM requires no visual prompt at inference and consistently improves alignment and performance over visual-prompt-based approaches on fine-grained benchmarks.
2. Related Work
MLLMs and fine-grained video methods increasingly support multimodal understanding, but object-centric systems commonly depend on extra encoders and visual prompts. SWIM instead uses training-time alignment supervision to support natural-language-only inference without architecture modifications.
- Multimodal Large Language Model: Video MLLMs extend multimodal understanding to spatiotemporal inputs and achieve strong performance in real-world applications.
- Fine-grained Object Understanding: Fine-grained object understanding remains difficult when models must identify or describe user-specified targets from textual prompts alone.
- Fine-grained Object Understanding: Object-centric methods commonly use points, boxes, or masks together with additional encoders to improve local-region comprehension.
- Fine-grained Object Understanding: These methods require extra encoders and visual prompts during inference, increasing computational cost and departing from typical user interaction.
- Fine-grained Object Understanding: SWIM uses explicit cross-modal alignment supervision to enable natural-language references without architecture modifications or inference-time visual prompts.
3. See What I Mean (SWIM)
SWIM constructs NL-Refer to pair explicit object references with masks, then regularizes cross-attention from tagged nouns toward their corresponding visual regions during training. This alignment signal supports text-only object localization at inference without modifying the base architecture or requiring visual prompts.
- NL-Refer: Dataset Construction: The dataset preserves the video, descriptive response, and pixel-level target mask while adding a linguistically explicit object reference.The refined message contains the generated reference, and the mask remains the visual annotation for the target instance.
- NL-Refer: Dataset Construction: NL-Refer replaces region placeholders with concise referring expressions and links a marked object noun to its corresponding instance mask.The dataset retains the original response while embedding explicit object identity and a deterministic noun span in the text.
- Attention Regularization: SWIM extracts cross-attention weights from tagged noun tokens to visual tokens across decoder layers, where each weight measures attention to a visual position.The noun query interacts with visual-token keys, and softmax is applied over visual-token positions.
- Attention Regularization: Attention vectors are mapped to the mask-aligned feature grid, interpolated when necessary, and averaged across selected layers to form a stable spatial map.The resulting aggregated map represents cross-modal correspondence between the tagged noun and its visual region.
- Attention Regularization: A pixel-wise binary cross-entropy loss supervises the aggregated attention map against the binary target mask.This training signal concentrates noun activation on the relevant visual region.
- Attention Regularization: Unlike approaches requiring a visual prompt mask during inference, SWIM uses the mask only for attention regularization during supervised fine-tuning.Inference therefore incurs no additional visual-prompt burden.
4. Experiments
SWIM is evaluated on fine-grained and general video-understanding benchmarks, with ablations examining attention-layer selection, fusion, and loss design. It achieves strong fine-grained performance while maintaining competitive general video understanding.
- Fine-grained benchmark results: SWIM attains 78.3% average accuracy on VideoRefer-Q, exceeding VideoRefer-7B by 6.4 percentage points and outperforming the reported generalist baselines.It gains 5.8% over Qwen2.5-VL-7B on Basic and 5.3% on Sequential cases.
- Fine-grained benchmark results: SWIM achieves an average score of 3.78 on VideoRefer-D, surpassing DAM-8B at 3.68 and GPT-4o at 3.25.Its component scores are 4.92 for SC, 3.85 for AD, 3.43 for TD, and 2.96 for HD.
- Overall results: SWIM improves referring capability across both question-answering and description tasks without architectural changes or inference-time visual prompts.The method uses explicit attention-regularization alignment supervision only during training.
- General video understanding: SWIM remains within a competitive range on MVBench, Video-MME, and ActivityNet-QA despite being primarily optimized for fine-grained object understanding.These benchmarks assess broader video-language understanding beyond fine-grained object grounding.
- Ablation studies: VideoRefer-D performance rises from 3.43 with one supervised shallow layer to 3.78 with six layers, then remains within 0.02 across larger configurations.The ablation indicates initial gains from increasing supervised layers, followed by stability beyond six layers.
- Ablation studies: Mean attention-map aggregation achieves the highest average score at 3.81, exceeding addition at 3.57 and pooling at 3.49.The reported explanation is that mean aggregation preserves consistent spatial patterns while smoothing noise across layers.
4.4. Scalability with Mask-Annotated Data Volume
SWIM benefits consistently from increasing mask-annotated training data, with alignment improving through the largest tested scale. It also outperforms Qwen2.5-VL on localization and fine-grained alignment evaluations.
- Scalability with Mask-Annotated Data Volume: Alignment scores increase monotonically as NL-Refer expands from 30K videos toward larger mask-annotated data scales.The reported trend continues through the largest tested scale.
- Scalability with Mask-Annotated Data Volume: SWIM continues gaining from additional mask supervision without reaching a plateau at the 125K data scale.The experiments are bounded by current data availability, leaving broader scaling untested.
- GamePoint-based Attention Localization: SWIM consistently outperforms Qwen2.5-VL across GamePoint@P settings, with largest gains of +6.3% at P = 1 and +5.5% at P = 5%.These results indicate stronger coverage of the target object among the highest-attention pixels.
- Fine-Grained Text-Visual Alignment Metrics: SWIM exceeds the Qwen2.5-VL baseline on AUC, NSS, Precision, and AP: 0.62 →0.67, 0.39 →0.50, 0.28 →0.39, and 0.26 →0.30.The improvements indicate more precise target coverage, fewer false activations, and stronger threshold discriminability.
- Qualitative Comparison: Qualitative examples show SWIM following natural-language references while Qwen2.5-VL can instead describe the scene’s most visually salient object.The comparison concerns examples requiring precise object reference from text.
5. Conclusions
The paper presents SWIM and NL-Refer to improve alignment between object nouns and visual regions for fine-grained video understanding. SWIM requires neither architectural changes nor visual prompts during inference.
- Conclusions: SWIM explicitly supervises cross-modal alignment between object nouns and visual regions to enhance fine-grained object understanding.NL-Refer pairs natural-language object references with mask annotations to provide this supervision.
- Conclusions: SWIM requires no architectural changes and no visual prompt inputs during inference.Mask supervision is used to train the alignment rather than supplied as an inference-time prompt.
- Conclusions: Experiments report state-of-the-art fine-grained understanding benchmark results while maintaining competitive general-benchmark performance.The paper also reports stronger fine-grained video understanding in quantitative analyses.
A. Benchmarks
The paper evaluates general video understanding across benchmarks spanning action-focused question answering, diverse long-context videos, and fine-grained temporal reasoning. Together, they cover varied domains, durations, and temporal challenges.
- ActivityNet-QA: ActivityNet-QA is a large-scale video question-answering benchmark with human-annotated pairs focused on action-related content.Its questions require understanding dynamic scenes and temporal sequences rather than only static visual cues.
- VideoMME: VideoMME covers sports, documentaries, instructional content, and entertainment across videos lasting minutes to hours.Its diversity tests long-context reasoning and adaptation across visual-text scenarios.
- MVBench: MVBench contains 20 multiple-choice tasks targeting temporal comprehension, including event ordering, cause–effect reasoning, motion tracking, and activity prediction.These tasks require integrating information across time rather than analyzing a single frame.
- Benchmark Coverage: Together, the benchmarks provide broad coverage of domains, video lengths, and fine-grained temporal reasoning challenges.ActivityNet-QA and VideoMME emphasize broad video understanding, while MVBench emphasizes varied temporal reasoning.
B.1. GamePoint@K
GamePoint@K evaluates whether relevant visual positions appear among the highest-scoring attention locations. SWIM outperforms Qwen2.5-VL across retrieval depths, including the top-ranked position.
- GamePoint@K: GamePoint@K measures the fraction of relevant elements among the top-K highest-scoring positions in an attention map.Higher scores mean relevant visual targets rank closer to the top.
- GamePoint@K: For K = 1, SWIM achieves 0.373 versus 0.330 for Qwen2.5-VL, indicating stronger top-rank placement of the correct target.The advantage persists at broader retrieval depths, including 0.375 at K = 5 (+4.7% over baseline).
- GamePoint@K: SWIM consistently outperforms Qwen2.5-VL across all evaluated K values.The consistent margins indicate reliable ranking distributions across retrieval depths.
B.2. Robustness to Synonym-based Linguistic Noise
SWIM is evaluated for robustness to variations in referring expressions by replacing selected prompt words with semantically equivalent synonyms. The resulting synonym-based noise preserves meaning while changing surface form, and performance is reported in Table 8.
- The evaluation replaces words enclosed in <ins> tags within VideoRefer-Bench-D prompts with semantically equivalent synonyms.
- This modification preserves the prompts’ overall meaning while altering their surface form.
- The altered prompts introduce lexical noise that may challenge models relying on exact token matches.
- Table 8 reports performance comparisons on VideoRefer-Bench-D, with an asterisk marking synonym-based noise.