PaperlayerView exact revision on arXiv

Source-linked AI summary

Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation

Su Ho Han, Jeongseok Hyun, Pilhyeon Lee, Minho Shim, Dongyoon Wee, Seon Joo Kim

arXiv:2510.19592v2cs.CV

TL;DR

Joint-training approaches to video reasoning segmentation incur substantial computation, while attention rollout can be noisy and dominated by irrelevant regions. DecAF fuses object-background and video-frame attention, then uses attention-guided SAM2 prompting; it outperforms training-free methods and is comparable to training-based methods overall, with a reported advantage over them on ReasonVOS.

  • Problem

    Jointly adapting MLLMs and segmentation models for video reasoning segmentation is computationally costly, and raw attention rollout can be dominated by noise and irrelevant regions.

    Source: §1 · p. 1, §3.3 · p. 5

  • Method

    DecAF combines contrastive object-background and complementary video-frame attention fusion, then uses attention-guided SAM2 prompting to produce fine-grained masks.

    Source: §1 · pp. 2–3

  • Results

    DecAF outperforms prior training-free approaches and achieves performance comparable to training-based methods across referring and reasoning VOS benchmarks; the conclusion reports it surpasses training-based methods on ReasonVOS.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Attention masks provide coarse localization that is sufficient to guide SAM prompting toward dense masks.

    Source: §1 · p. 3, §4.2 · p. 8

    Attention maps remain low resolution, with contour accuracy (F) much lower than region similarity (J), and their scores remain below conventional segmentation models.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Multimodal large language models (MLLMs) demonstrate strong video understanding by attending to visual tokens relevant to textual queries. To directly adapt this for localization in a training-free manner, we cast video reasoning segmentation as a video QA task and extract attention maps via rollout mechanism. However, raw attention maps are noisy and poorly aligned with object regions. We propose Decomposed Attention Fusion (DecAF), which refines these maps through two mechanisms: (1) contrastive object-background fusion and (2) complementary video-frame fusion. This method suppresses irrelevant activations and enhances object-focused cues, enabling direct conversion of attention maps into coarse segmentation masks. In addition, we introduce attention-guided SAM2 prompting for obtaining fine-grained masks. Unlike existing methods that jointly train MLLMs with SAM, our method operates entirely without retraining. DecAF outperforms training-free methods and achieves performance comparable to training-based methods on both referring and reasoning VOS benchmarks.

1 INTRODUCTION

MLLMs’ video reasoning suggests they may localize objects for complex text queries, but existing joint-training approaches are costly, while raw attention rollout is noisy. DecAF fuses attention maps and uses them to guide SAM2, outperforming prior training-free methods and matching training-based methods on the evaluated benchmarks.

1 INTRODUCTION

  • Jointly training MLLMs with segmentation foundation models requires substantial computation and may limit generalization.Prior approaches use efficient fine-tuning methods such as LoRA.Source: §1 · p. 1
  • Loc-Head selects attention heads using spatial entropy but assumes a single referring object, complicating extension to multi-object video and reasoning settings.Its heuristic for visual attention sinks may also limit generalization across MLLMs.Source: §1 · p. 1
  • DecAF subtracts a contrastive background attention map from object attention and combines video-level temporal cues with frame-level object detail.The fusion aims to suppress irrelevant activations while retaining temporal context and clearer object focus.Source: §1 · p. 2, Figure 1 · p. 2
  • Thresholding fused attention yields reliable but coarse masks; attention-derived point prompts guide SAM2, with a consistency score filtering unreliable predicted masks.The consistency score evaluates alignment between the predicted mask and the underlying attention map.Source: §1 · p. 3
  • DecAF consistently outperforms prior training-free approaches with and without SAM, while dense masks achieve performance comparable to training-based methods across referring and reasoning VOS benchmarks.The evaluation spans three MLLM families and five datasets.Source: §1 · p. 3

2 RELATED WORK

Prior work spans MLLM architectures that trade spatial detail for video-token efficiency, text-conditioned segmentation from simple referring expressions to reasoning-intensive queries, and training-free visual grounding. DecAF’s overview depicts rollout followed by contrastive and complementary attention fusion.

2 RELATED WORK

  • Some MLLMs compress video tokens to reduce attention cost, sacrificing fine spatial information, while LLaVA-style models preserve dense spatial features and Qwen2VL supports native-resolution video.These differences frame the MLLM architectures on which localization methods build.Source: §2 · p. 3
  • Referring VOS began with appearance-based, single-object expressions, while MeViS adds motion-centric cases with absent objects or multiple matching candidates.Source: §2 · p. 3
  • Video reasoning segmentation targets complex expressions requiring world knowledge and temporal context; existing approaches fine-tune MLLMs and often fine-tune a SAM mask decoder.The paper contrasts these methods with its training-free use of MLLMs and SAM.Source: §2 · p. 3
  • VL-SAM and TAM use attention rollout for image grounding, whereas Kang et al. select localization heads but report poor transfer from referring to reasoning-intensive datasets.VL-SAM further refines masks with SAM; VL-SAM and TAM enumerate categories during decoding.Source: §2 · p. 3
  • The DecAF overview traces attention rollout to contrastive background suppression, complementary video-frame fusion, and their combined refinement of attention maps.Source: Figure 2 · p. 4

3 METHODOLOGY

The method refines MLLM attention maps into coarse video masks, then uses attention-guided SAM2 prompting to generate dense masks. DecAF combines contrastive object-background fusion with complementary video-frame fusion.

3.1 OVERVIEW

  • The two-stage pipeline derives coarse masks from MLLM attention, then generates dense masks with SAM2 conditioned on those attention maps.Source: §3.1 · p. 4

3.2 ATTENTION ROLLOUT WITH VISION-AWARE NORMALIZATION

  • Vision-aware rollout recursively accumulates attention across layers and weights heads by visual-attention strength before aggregation to reduce noisy-head effects.Source: §3.2 · pp. 4–5

3.3 DECOMPOSED ATTENTION FUSION

  • Object-focused and target-excluding background prompts produce maps whose smoothed difference is clamped and normalized, suppressing irrelevant responses.Source: §3.3 · p. 5
  • Video-level maps contribute temporal context while frame-level maps add spatial precision; upsampled video maps are averaged with independently normalized frame maps.Source: §3.3 · pp. 5–6

3.4 SAM2 PROMPTING WITH ATTENTION MAPS

  • Thresholded attention tokens become center-point queries for sequential SAM2 mask propagation; attention-scored NMS removes redundant overlapping candidates.Source: §3.4 · p. 6
  • An attention-consistency score evaluates tracklet overlap with high-attention regions across frames; combined scores retain candidates above τtrk for propagation.Source: §3.4 · p. 7

4 EXPERIMENTS

Evaluation spans referring and reasoning VOS benchmarks, comparing DecAF with training-free and training-based methods and testing its components. DecAF performs competitively, while ablations examine fusion, rollout, prompting, and model scale.

4.1 EVALUATION SETTING

  • Evaluation covers three referring and two reasoning VOS datasets, using J, F, and their mean J &F; ReasonVOS is evaluated zero-shot.Attention-map masks use Otsu thresholding, and the default SAM prompting thresholds are τtrk = 0.8 and τpq = 0.8.Source: §4.1 · p. 7

4.2 COMPARISON WITH EXISTING METHODS USING MLLMS

  • On Ref-DAVIS with LLaVA-7B, DecAF prompting scores 55.6 versus Loc-Head’s bbox prompting at 30.3; attention-only masks remain low-resolution and below conventional segmentation models.The lower contour accuracy (F) than region similarity (J) indicates that attention masks are too coarse for precise boundaries, but provide coarse localization for SAM prompting.Source: §4.2 · p. 8
  • DecAF consistently outperforms Loc-Head across MLLMs and datasets, including MeViS and Ref-YTVOS, with especially large margins on reasoning-intensive datasets.Loc-Head is slightly higher on Ref-DAVIS with InternVL3-8B but drops substantially on ReasonVOS.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • On Ref-DAVIS, Qwen2.5VL-7B exceeds VISA and VideoLISA by 5.8 and 6.4 J &F, respectively; on MeViS, it scores 48.1 versus 44.5 and 44.4, near VRS-HQ’s 50.6.DecAF also surpasses all training-based methods on ReasonVOS, despite Veason-R1 additionally training the same MLLM with an RL-based algorithm.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 ABLATION STUDY

  • Object-background fusion raises Ref-DAVIS attention-mask results from 12.4 to 20.7 and 14.5 to 25.3; video-frame fusion raises Qwen2.5VL results from 65.9 and 67.4 separately to 75.2 combined.The passage reports similar object-background gains for SAM masks and consistent video-frame gains on ReasonVOS and with InternVL3.Source: §4.3 · p. 9, Table 4 · p. 9
  • Multi-scale fusion improves results particularly for InternVL, whose attention maps have very low resolution without tiling; vision-aware head-weighted rollout normalization also improves accuracy, and a middle LLM layer performs best.The multi-scale scheme uses higher-resolution frame inputs.Source: §4.3 · p. 9, Table 5 · p. 9
  • SAM prompting remains robust to threshold choices despite dataset-dependent optima; omitting the attention consistency score causes a significant accuracy drop.The same τpq = 0.8 and τtrk = 0.8 are used across datasets and models; thresholding with δ favors mean thresholding because it covers the object while included background has low attention and low sac.Source: Table 6 · p. 9, Table 7 · p. 9, §4.3 · p. 10
  • Larger MLLMs generally perform better, with InternVL3 improving from 53.5 to 63.3 on Ref-D and 54.1 to 65.6 on ReasV; qualitative results show target-aligned attention and fine-grained SAM2 masks.Qwen2.5VL-7B achieves the best results across datasets.Source: Table 8 · p. 9, §4.3 · p. 10, Figure 4 · p. 10

5 CONCLUSION

The conclusion frames video reasoning segmentation as video QA and introduces DecAF to turn MLLM attention into robust localization, with attention-guided SAM2 prompting for dense masks. The appendices describe manuscript polishing and prompt templates for object and background attention.

5 CONCLUSION

  • DecAF casts video reasoning segmentation as video QA, produces robust attention maps for coarse masks, and guides SAM2 prompting to obtain fine-grained masks without training.The conclusion also reports that the method surpasses training-based methods on ReasonVOS and suggests extending DecAF beyond vision.Source: §5 · p. 10

A USE OF LARGE LANGUAGE MODELS

B PROMPT TEMPLATES

  • The object-focused prompt asks the MLLM to identify the expression’s main object, prioritizing the primary subject or agent and requesting a single-word answer.Source: B PROMPT TEMPLATES · p. 14
  • The background-focused prompt asks for a brief description of the video background while excluding the target object category.Source: B PROMPT TEMPLATES · p. 14
  • Category selection combines video- and frame-level predictions with an explicit query, and the same prompt templates are used across MLLMs and datasets without specific modifications.For vague expressions, the template uses a candidate class list as support and cautions against defaulting to the most frequent class.Source: B PROMPT TEMPLATES · p. 14

C MORE DETAILS ABOUT PREVIOUS METHODS

The comparison section highlights instability in TAM’s token-dependent localization and Loc-Head’s sensitivity to sampling, heuristics, and mask-prompt construction. It contrasts these issues with DecAF’s reported consistency across models and datasets.

TAM

  • TAM’s attention maps vary with predicted tokens and tokenization, while its best-token evaluation selects the highest-IoU token and can overstate practical robustness.The examples include misalignment for “bike” and the first token of “bicycle,” with relatively better alignment for “icycle.”Source: C MORE DETAILS ABOUT PREVIOUS METHODS · p. 15

Loc-Head

  • Loc-Head head discovery was sensitive to its sampling dataset: video-sampled heads reduced Attn-mask (J ) from 24.2 to 19.2 on Ref-DAVIS and from 18.6 to 4.2 on ReVOS.The reproduction therefore used heads discovered from RefCOCO across datasets.Source: C MORE DETAILS ABOUT PREVIOUS METHODS · p. 15
  • The bottom-row attention-sink heuristic failed on Qwen2VL, where excluding the right-most column instead improved Attn-mask J to 23.8; tiling choices also affected InternVL3.Source: C MORE DETAILS ABOUT PREVIOUS METHODS · p. 16
  • Loc-Head’s coarse attention-derived bounding boxes often yielded inaccurate SAM masks, with the RefCOCO validation score dropping from 74.2 to 34.4 in the reproduction.The authors applied their SAM prompting process consistently across video datasets for comparison.Source: C MORE DETAILS ABOUT PREVIOUS METHODS · p. 16
  • DecAF is reported to work reliably across MLLMs and datasets, whereas Loc-Head depends heavily on dataset-specific sampling and model-specific heuristics.Source: C MORE DETAILS ABOUT PREVIOUS METHODS · p. 16
  • For LLaVA-OV-7B, the reproduction disabled tiling during head discovery and found that excluding the left-most column in addition to the bottom row was needed for reasonable results.Source: C MORE DETAILS ABOUT PREVIOUS METHODS · p. 16

D QUALITATIVE RESULTS

Qualitative examples show DecAF attention and resulting masks aligning with instruction-referred objects across varied scenarios, while the failure cases expose limits in consistency scoring and attention coverage. The section also points to a comparison of Qwen2.5-VL grounding settings on Ref-DAVIS and ReasonVOS.

D QUALITATIVE RESULTS

  • DecAF’s attention maps and resulting attention and SAM masks are reported to align with targets across single- and multi-object, small-object, temporal-reasoning, and world-knowledge cases.The qualitative discussion reports robust localization even for multiple objects and small targets, and accurate masks without additional training for temporal reasoning or world-knowledge cases.Source: D QUALITATIVE RESULTS · p. 16, Figure 6 · p. 17, Figure 7 · p. 17
  • Attention-consistency scoring can falsely penalize correctly generated tracklets when large objects are only partly attended or small objects appear briefly, while thin elongated targets may yield poor masks.Source: D QUALITATIVE RESULTS · p. 16
  • Table 9 compares Qwen2.5-VL-7B grounding and SAM2 baselines on Ref-DAVIS and ReasonVOS across four video-grounding setups.The caption names All frames, First frame, Ref & key frames, and 16 frames as the setups.Source: Table 9 · p. 17

E NATIVE GROUNDING OF QWEN2.5VL

Native Qwen2.5-VL grounding is reported below DecAF across the evaluated settings, including a comparison using the same 16-frame sampling and SAM2 process. The section describes frame-wise, first-frame, reference-frame, and sampled-frame prompting schemes.

E NATIVE GROUNDING OF QWEN2.5VL

  • The evaluation covers Ref-DAVIS and ReasonVOS, using Qwen2.5-VL-7B keypoints as SAM2 prompts or propagation inputs.Source: E NATIVE GROUNDING OF QWEN2.5VL · p. 17
  • The All frames setup grounds and segments each frame independently, while First frame grounds only the initial frame and propagates its mask through the video.Source: E NATIVE GROUNDING OF QWEN2.5VL · pp. 17, 20
  • Ref & key frames supplies 16 reference frames for temporal context before grounding a key frame, whereas the 16 frames setup extracts keypoints on 16 sampled frames for the SAM2 pipeline.The passage notes that Qwen2.5-VL does not support video-level spatial grounding.Source: E NATIVE GROUNDING OF QWEN2.5VL · p. 20
  • With uniform 16-frame sampling and the same SAM2 prompting and propagation process, Qwen2.5-VL grounding performs better than Loc-Head but falls significantly short of DecAF.Across settings, native grounding shows reasonable performance but remains below the proposed method.Source: E NATIVE GROUNDING OF QWEN2.5VL · p. 20

F PROMPT ROBUSTNESS ANALYSIS

The prompt robustness analysis varies object-focused, background-focused, and object-category prompts. Results are broadly similar across formulations, and the paper reports using one fixed prompt set without prompt tuning.

F PROMPT ROBUSTNESS ANALYSIS

  • The analysis tests alternative formulations of the object-focused, background-focused, and object-category prompts for their effects on performance.Alternative versions were generated using Chat-GPT.Source: F PROMPT ROBUSTNESS ANALYSIS · p. 20
  • Three object-focused prompt variants produce similar performance, with a slight Ref-DAVIS drop and slight ReVOS improvement for V1.The variants are a single-sentence prompt, a rephrased prompt, and a shorter prompt.Source: F PROMPT ROBUSTNESS ANALYSIS · p. 20
  • Background-prompt variants perform similarly overall, though omitting the object category can slightly lower performance when the target is not the scene’s primary object.Without the category, the model may mistake the target for part of the background.Source: F PROMPT ROBUSTNESS ANALYSIS · p. 21
  • Three object-category prompt variants yield similar performance, indicating robust category selection across these formulations.The variants use a single sentence, a shorter prompt, or the expression alone without the object-focused prompt’s class.Source: F PROMPT ROBUSTNESS ANALYSIS · p. 22
  • Across prompt variations, DecAF remains robust, and all main-paper and appendix experiments use the same fixed prompts without tuning for MLLMs or benchmarks.The authors report consistent results across diverse formulations.Source: F PROMPT ROBUSTNESS ANALYSIS · p. 22

G ADDITIONAL RESULTS OF FIGURE 1

  • Figure 10 shows fused attention, thresholded attention masks, SAM2 query points, and dense masks; the authors report that target-region query points yield accurate dense masks.Frame-wise thresholding can also turn weak attention responses into foreground regions.Source: G ADDITIONAL RESULTS OF FIGURE 1 · p. 22

H ANALYSIS OF SIMILAR MULTIPLE OBJECTS SCENARIO

On similar multiple-object cases, DecAF’s performance declines less than VRS-HQ’s on the ReasonVOS subset. Qualitative comparisons also report correct-target localization where VRS-HQ often selects an incorrect object.

H ANALYSIS OF SIMILAR MULTIPLE OBJECTS SCENARIO

I PART-LEVEL SEGMENTATION ANALYSIS

Although the method is evaluated on object-level segmentation, qualitative examples show attention maps aligning with part-level expressions. SAM2 dense masks do not consistently preserve that fine-grained localization.

I PART-LEVEL SEGMENTATION ANALYSIS

J ANALYSIS OF NON-EXPLICIT OBJECT EXPRESSION CASE.

Some expressions do not name or visually describe the target, requiring reasoning about scene context. Framing segmentation as video QA lets DecAF use reasoning-driven attention to localize the inferred object and produce dense masks.

J ANALYSIS OF NON-EXPLICIT OBJECT EXPRESSION CASE.

  • In non-explicit cases, expressions give no direct visual attributes or class cues, so identifying the target requires reasoning over scene context.One example asks which object is likely awaited during a construction-site work stoppage.Source: J ANALYSIS OF NON-EXPLICIT OBJECT… · p. 28
  • Casting the task as video QA lets reasoning-driven attention highlight the correct target despite no explicit object description, supporting accurate dense masks for the inferred target.The attention maps are generated when the MLLM answers a question formatted with the expression.Source: J ANALYSIS OF NON-EXPLICIT OBJECT… · p. 28

Related papers

Loading 2510.19592v2…