Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration

Introduction

• MLLMs remain prone to hallucinations and over-reliance on language priors, often without adequately using task-relevant visual evidence; this is especially acute in fine-grained reasoning dependent on few informative visual cues (Leng et al. 2024; Bai et al. 2024).Recent analyses associate these failures with misallocated text-to-image attention, while existing methods manipulate attention at inference or align attention maps during training (Seil et al. 2025; Woo et al. 2025; Leng et al. 2024; Chen et al. 2025).Source: Introduction · p. 1

Methodology; Problem Formulation

• sPMC guides attention toward segmentation-derived relevant regions, while Adaptive Head Selection leaves non-selected heads unconstrained to preserve their complementary functions.Its region-level objective increases aggregate in-mask probability mass rather than prescribing a distribution across individual visual tokens, and selective guidance is reported as more effective than all-head guidance.Source: Introduction · pp. 1–2, Methodology · p. 2, Problem Formulation · p. 2

Introduction

• Across six multimodal benchmark suites, sPMC reports an average zero-shot gain of 3% and gains of up to 11.3% over corresponding base models.The framework is evaluated across multiple MLLMs.Source: Introduction · p. 2

Multimodal Large Language Models; Enhancing Multimodal Reasoning

• MLLMs connect vision encoders to language models through learned adapters or represent visual and textual inputs in shared autoregressive sequences, with attention providing an interface to visual tokens.Related work includes multimodal chain-of-thought methods such as ICoT and MVoT, attention interventions and contrastive decoding, and Lavender’s patch-wise attention regression; these approaches may inherit evidence-selection errors or add inference overhead. (Bai et al. 2023; Dai et al. 2023; Liu et al. 2024; Team 2024; Bai et al. 2025; Zhang et al. 2024; Gao et al. 2025; Wu et al. 2024; Zheng et al. 2023; Chen et al. 2024b; Mitra et al. 2024; Li et al. 2025a; Seil et al. 2025; Woo et al. 2025; Leng et al. 2024; Chen et al. 2025)Source: Introduction · p. 1, Multimodal Large Language Models · p. 2, Enhancing Multimodal Reasoning · p. 2

Semantically Enriched Grounding Masks

• Grounding masks use concrete, fine-grained noun phrases from image-associated captions or VQA reference answers to prompt segmentation, then merge candidate regions and discard masks covering over 60% of the image.The masks are weak spatial priors rather than pixel-accurate ground truth; favoring coverage makes the objective more tolerant of moderate over-inclusion than missing relevant regions. SAM 3 supplies the text-conditioned segmentation prior (Carion et al. 2026).Source: Semantically Enriched Grounding Masks · pp. 2–3

Adaptive Attention Aggregation; Attention Aggregation.

• Selected head-layer text-to-image attention logits are aggregated into a spatial distribution and reshaped to the visual-token grid.This aggregation provides the spatial representation used to guide attention toward the target mask.Source: Methodology · p. 2, Adaptive Attention Aggregation · p. 3, Attention Aggregation · p. 3

Adaptive Head Selection.

• Adaptive Head Selection targets head-layer pairs with strong visual engagement and spatial alignment, based on calibration-example statistics averaged over the calibration set.The engagement and alignment thresholds are determined from calibration statistics using the elbow method (Sahoo, Soltani, and Wong 1988).Source: Adaptive Head Selection · pp. 3–4

Selective Probability Mass Concentration

• The sPMC loss increases total attention probability mass within the target region without requiring point-wise agreement for every masked token.The model can concentrate on any informative subset within the mask, making the objective tolerant of moderately over-inclusive masks and false-positive regions.Source: Selective Probability Mass Concentration · p. 4

Grounding Mask Dataset Generation.

• The method generates 332,480 image-question-mask tuples from Flickr30k and RLAIF-V 83k using semantically enriched caption prompts and SAM3 segmentation masks.Qwen3-Instruct-4B extracts the prompts used as textual queries for grounding.Source: Grounding Mask Dataset Generation · p. 4

Multimodal benchmarks

• Evaluation covers six benchmark suites reported as seven sets: general vision-language understanding and hallucination, using Qwen3-VL-2B, Qwen3-VL-8B, and Llama3.2-11B-Vision.The benchmarks include MMVP, VisOnly, V*, MME, POPE, and HallusionBench.Source: Multimodal benchmarks · p. 4

Implementation details

• Fine-tuning used LoRA on one NVIDIA RTX Pro 6000 GPU, with effective batch size 32, AdamW at 1e−4, and cosine scheduling with 100-step warm-up and one mid-training reset.Gradient accumulation was performed over 4 or 8 steps depending on base-model size.Source: Implementation details · pp. 4–5

Main Results

• sPMC yields 2.1%-3.7% average gains across baseline models, and Qwen3-VL-8B scores 59.5 on HallusionBench and 69.3 on MMVP, surpassing GPT-4V and Gemini 1.5 Pro despite being 7× smaller.On POPE and HallusionBench, sPMC also gives average relative gains of 0.7% and 4.5%, respectively, across base models.Source: Table 1 · p. 5, Main Results · p. 5
• Only 3.03%-15.4% of heads are selected, with about 3% in Qwen3-VL-8B; sPMC largely preserves overall text-to-image attention while redirecting it toward relevant cues and lowering alignment loss.The head-selection results support restricting alignment to grounding-responsive heads instead of supervising all attention heads.Source: Main Results · p. 5, Table 2 · p. 6, Figure 3 · p. 6

Training objective variants

• AutoR. and Cross-Entropy reduce average zero-shot performance by 3.03% and 9.42%, respectively, versus pretrained No Guidance, while sPMC improves it.The larger Cross-Entropy drop is attributed to its overly restrictive exact patch-wise target, whereas sPMC uses flexible region-level probability-mass guidance.Source: Table 3 · p. 6, Training objective variants · p. 6

Ablation study

• Adaptive-Late tops all five metrics; All Heads trails by 6.24% on average, Random-Late by 3.9 points, Adaptive-Early by approximately 10 points, and Adaptive-Full improves 4.9 points over Adaptive-Early.Adaptive-Late applies adaptive head selection to layers 17–28; the compared variants include all-head, random late-layer, adaptive early-layer, and adaptive all-layer selection.Source: Ablation study · p. 6, Figure 4 · p. 7

Qualitative analysis

• Figure 5 examples span diagram interpretation, OCR, commonsense reasoning, and fine-grained visual discrimination, but sPMC can miss ambiguous low-salience evidence such as a blurred pet bed.The reported failure suggests attention concentration may bias the model against weak or uncertain visual signals.Source: Qualitative analysis · pp. 6–7, Figure 5 · p. 7

• The work frames attention misallocation as suboptimal probability mass distribution over visual regions and proposes sPMC to regularize cross-modal attention.sPMC concentrates attention on semantically relevant regions and selectively tunes grounding-relevant heads while preserving the model’s inherent reasoning capabilities.Source: Conclusion and Future Directions · p. 7
• sPMC consistently enhances zero-shot performance over base models across multiple benchmarks.The conclusion reports this result qualitatively without specifying benchmark scores.Source: Conclusion and Future Directions · p. 7
• Future work could incorporate synthetic data and compositional scenes to provide richer supervision and improve robustness under complex reasoning conditions.Source: Conclusion and Future Directions · p. 7

• Only a small subset of attention heads consistently concentrates attention on semantically relevant visual regions, localizing visual evidence aggregation to specialized components.Source: Preliminary Experiment · p. 12
• Across model architectures and scales, a limited number of heads dominate grounding-related attention while others serve complementary roles such as contextual reasoning or diffuse exploration.Source: Preliminary Experiment · p. 12
• Figures 6 and 7 illustrate representative grounding heads and visual evidence aggregation across model backbones, motivating targeted rather than uniform head regulation.Source: Preliminary Experiment · p. 12

Mask Generation

• The pipeline transforms raw image-text pairs into grounded training data through a two-stage extraction-and-segmentation framework.It pairs extracted semantic queries with generated masks for downstream probability mass concentration tuning.Source: Mask Generation · p. 12
• Qwen3-4B-Instruct (Yang et al. 2025) extracts literal, query-ready noun phrases, preserving modifiers while excluding abstract concepts and generalized categories.The phrases identify individual physical objects or regions, rather than collections.Source: Mask Generation · p. 12
• SAM3 (Carion et al. 2026) performs zero-shot instance segmentation by pairing each extracted phrase with its image to predict a binary mask.Each predicted mask has the same resolution as its image.Source: Mask Generation · p. 12
• A detection threshold of τdet = 0.15 filters low-confidence proposals, while τmask = 0.5 binarizes logits into pixel-level assignments.The configuration aims to retain challenging or small objects while defining foreground pixels sharply.Source: Mask Generation · p. 12
• The semantic-density filter discards masks with foreground ratio ρ > 0.6 to remove hallucinated segments or masks that conflate an object with the background.The ratio is defined over binary mask values M(x,y) ∈ {0, 1}.Source: Mask Generation · p. 12
• The curated dataset maps semantic queries to masks that Figure 8 examples show are well-aligned, spatially precise, and minimally noisy.The authors describe these samples as the foundation for visual reasoning training.Source: Mask Generation · p. 12

Quality Evaluation of Segmentation Masks

• A 200-record audit used human and blinded multimodal-AI judgments of translucent-red mask overlays against target phrases, withholding source questions and answers.It assessed useful semantic localization rather than pixel-level accuracy; imperfect boundaries, partial target coverage, moderate over-coverage, and merged nearby regions were not automatically penalized.Source: Quality Evaluation of Segmentation Masks · p. 15
• The usable-mask quality was sufficient for sPMC because masks provide weak spatial guidance rather than pixel-level ground truth, allowing limited noisy or ambiguous supervision.Excluding Unjudgeable cases, the usable-mask rate was 195/198 = 0.985 (98.5%; Wilson 95% CI [95.6%, 99.5%]).Source: Quality Evaluation of Segmentation Masks · p. 15
• Residual failures involved wrong-target or over-global masks and ambiguous phrases; AI scoring was slightly stricter than human scoring, but mask quality remained sufficient for sPMC training.Examples included masks dominated by irrelevant sky, targeting a man or foreground instead of “women,” highlighting an illustration instead of a guitar, or focusing on other bicycles.Source: Quality Evaluation of Segmentation Masks · p. 15
• At least 195/200 examples (97.5%) were Accurate or Usable; excluding Unjudgeable cases, the usable-mask rate was 195/198 = 0.985 (98.5%; Wilson 95% CI [95.6%, 99.5%]).Table 4 summarizes the segmentation mask quality audit statistics.Source: Table 4 · p. 15, Quality Evaluation of Segmentation Masks · p. 15

Theoretical justification

• High visual engagement selects heads that allocate substantial attention to visual tokens, excluding heads whose attention collapses onto text and contributes little to visual evidence aggregation.The criterion lower-bounds the marginal probability mass assigned to visual tokens, ensuring non-trivial mutual information between the head and visual inputs.Source: Theoretical justification · p. 15
• Spatial consistency selects heads whose attention aligns sufficiently with the supervision-mask region, corresponding to the negative log-likelihood of attention mass within that region.Source: Theoretical justification · p. 15
• Together, the criteria define a feasible set of heads with non-trivial visual engagement and attention to semantically relevant regions; in practice, softplus provides a stable, smooth upper bound while preserving monotonicity.Source: Theoretical justification · p. 18

Adaptive thresholds

• Rather than use a fixed heuristic threshold, adaptive selection finds the point of diminishing returns by locating maximum curvature in the sorted head-score curve relative to its endpoint chord.The threshold is defined by the optimal head index k* for sorted scores and the chord connecting the curve's endpoints.Source: Adaptive thresholds · p. 18
• The curvature-based threshold selects statistically significant score outliers, yielding a sparse set of grounding-active heads.Source: Adaptive thresholds · p. 18

More Implementation Details

• LoRA adapters are trained for one epoch while backbone parameters remain frozen, with placements varying by backbone and projection type.All adapters use rank 16, alpha 16, dropout 0.1, and no bias; training uses bfloat16, AdamW, learning rate 10^-4, weight decay 0.1, 100 warmup steps, a two-cycle cosine schedule with hard restarts, effective batch size 32, and seed 42. Qwen3-VL-2B uses query/value projections in the first 14 of 28 language layers; Qwen3-VL-8B uses query/key projections in the final 15 of 36; Llama3.2-11B-Vision uses query/key projections in cross-attention modules.Source: More Implementation Details · p. 18

Benchmarks

• General vision-language evaluation covers MMVP (Tong et al. 2024), VisOnly (Kamoi et al. 2025), V* (Wu and Xie 2023), and MME, spanning visual grounding, geometry, and broad multimodal capabilities.The benchmarks probe CLIP-blind visual features, geometric perception, high-resolution search for small details, and perception and cognition subtasks including OCR and reasoning.Source: Benchmarks · p. 18
• Hallucination evaluation uses POPE (Li et al. 2023) for object-presence questions and HallusionBench (Guan et al. 2024) for multimodal hallucination scenarios.POPE contains 3,000 questions across sampling strategies; HallusionBench includes 346 images and 1,129 questions.Source: Benchmarks · p. 18

Baseline models

• The final backbones are Qwen3-VL-2B-Instruct, Qwen3-VL-8B-Instruct, and Llama3.2-11B-Vision, spanning two model scales and self- versus cross-attention designs.Other open- and closed-source MLLMs serve as reference points, but are not directly comparable because they use larger scales (≥20B parameters) and complex post-training strategies.Source: Baseline models · pp. 18–19

Computation Time

• sPMC adds no inference-time overhead versus prior attention-modification approaches, with training completed within a few hours on a single GPU.The design is described as compatible with existing architectures and deployment settings.Source: Computation Time · p. 19

Adaptive Head Selection Results

• Table 6 summarizes full head selection across base models, defining selection rate as selected heads divided by total heads.The table also notes models using self-attention only for cross-modal attention and reports total attention layers and selected head indexes.Source: Adaptive Head Selection Results · p. 19, Table 6 · p. 20