Source-linked AI summary
Reinforced Attention Learning
Bangzheng Li, Jianmo Ni, Chen Qu, Ian Miao, Liu Yang, Xingyu Fu, Muhao Chen, Derek Zhiyuan Cheng
TL;DR
Multimodal post-training with verbose rationales provides limited perception gains and can degrade core capabilities because token-level optimization does not directly shape visual information allocation. The paper proposes RAL, which optimizes attention policies and extends them to on-policy attention distillation. Across diverse image and long-video benchmarks, RAL consistently outperforms base and GRPO baselines, while attention transfer complements output imitation.
Problem
Verbose textual reasoning offers limited gains and may degrade multimodal perception, while token-level objectives optimize outputs rather than internal information allocation.
Method
RAL uses attention distributions as the policy and optimizes their reward-weighted divergence, while on-policy attention distillation transfers token and attention distributions.
Results
RAL consistently outperforms base Qwen-2.5-VL-7B and GRPO across diverse image and long-video benchmarks, with more stable gains than token-level RL.
Takeaways & Limitations
Attention distributions are a first-class optimization target for multimodal alignment, and transferring where to focus provides a complementary signal beyond output imitation.
Takeaways & Limitations
Token-level optimization can cause diversity collapse, surface-form overfitting, and weaker generalization across reasoning patterns through linguistic reward hacking.
Abstract
from arXiv · showhide
Post-training with Reinforcement Learning (RL) has substantially improved reasoning in Large Language Models (LLMs) via test-time scaling. However, extending this paradigm to Multimodal LLMs (MLLMs) through verbose rationales yields limited gains for perception and can even degrade performance. We propose Reinforced Attention Learning (RAL), a policy-gradient framework that directly optimizes internal attention distributions rather than output token sequences. By shifting optimization from what to generate to where to attend, RAL promotes effective information allocation and improved grounding in complex multimodal inputs. Experiments across diverse image and video benchmarks show consistent gains over GRPO and other baselines. We further introduce On-Policy Attention Distillation, demonstrating that transferring latent attention behaviors yields stronger cross-modal alignment than standard knowledge distillation. Our results position attention policies as a principled and general alternative for multimodal post-training.
1. Introduction
The paper argues that verbose textual reasoning offers limited or harmful benefits for multimodal perception because token-level objectives do not directly optimize visual information allocation. It introduces RAL, which treats attention distributions as the policy, and extends this idea to on-policy attention distillation.
- Motivation: Verbose textual reasoning provides only marginal gains and may degrade core perception tasks such as image and video question answering.
- Motivation: Standard RLHF optimizes output tokens rather than the internal attention process needed to identify task-relevant multimodal information.
- Reinforced Attention Learning: RAL rewards high-utility attention patterns by reducing divergence from reference policies and penalizes low-reward patterns by increasing that divergence.
- On-Policy Attention Distillation: On-Policy Attention Distillation transfers knowledge through attention distributions alongside token distributions, improving students’ fine-grained perceptual and grounding behaviors.
- Reinforced Attention Learning: RAL directly optimizes attention-distribution alignment, reinforcing visual grounding instead of relying on textual outputs as indirect supervision.
- Empirical Validation: Extensive experiments report consistent RAL improvements over GRPO across visual question answering benchmarks requiring fine-grained visual understanding.
2. Related Works
Prior work applies post-training and distillation to multimodal alignment, but persistent visual hallucination, modality bias, and deployment-distribution challenges remain. The paper positions attention-level supervision as a way to shape cross-modal information selection within reinforcement and on-policy distillation frameworks.
- Multimodal Alignment: Multimodal post-training must address visual hallucination and robust cross-modal grounding beyond text-only alignment.
- Multimodal Alignment: Existing methods adapt RLHF, RLVR, and DPO to improve visual grounding and reduce hallucinations.
- Multimodal Alignment: Modality bias can produce reliance on linguistic priors or superficial visual cues, motivating rewards that discourage shortcuts and spurious correlations.
- Attention-Based Alignment: The paper targets cross-modal reasoning by directly shaping attention weights to control evidence selection across visual and textual modalities.
- Knowledge Distillation: Knowledge distillation commonly matches softened outputs, while related approaches also supervise intermediate representations, attention maps, and hidden states.
- On-Policy Distillation: On-policy distillation supervises student-generated trajectories, mitigating exposure bias and aligning the student’s generation distribution with deployment behavior.
- Attention Distillation: The paper studies attention distillation within on-policy distillation to regularize internal information allocation while maintaining policy alignment.
3. Reinforced Attention Learning
RAL shifts reinforcement learning from output-token distributions to internal attention distributions, treating information allocation as a policy for multimodal reasoning. Its combined training and distillation objectives align attention with reward or teacher behavior while retaining token-level learning.
- 3. Reinforced Attention Learning: RAL treats aggregate attention over an input prompt as an information-gathering policy, shifting optimization from external output distributions to internal attention distributions.The approach targets how the model allocates computational focus over contextual information rather than strictly constraining output tokens.
- 3.1. Aggregated causal Attention Distribution Policy: For each generated token, the causal attention policy distributes weight over preceding prompt and response positions, capturing attention to visual input, instructions, and emerging reasoning.The policy uses final-layer attention averaged across heads and includes the model’s own generated rationale as context.
- 3.2. Advantage-Weighted Attention Divergence: The attention objective uses advantage-weighted divergence between current and old attention policies, pulling toward successful patterns and away from suboptimal ones.A positive advantage minimizes divergence from the old strategy, whereas a negative advantage pushes the current policy away from it.
- 3.2. Advantage-Weighted Attention Divergence: The per-token objective preserves supervision for late response tokens, addressing vanishing gradients that can arise when attention is averaged across long sequences.The method applies the internal attention objective at token-level granularity.
- 3.3. Combined Optimization Objective: The final objective combines token-level policy gradients with an attention regularizer, using λ_attn to balance output maximization and attention-level exploration.This dual-objective design is intended to retain linguistic flexibility while maintaining structured, reward-aligned reasoning.
- 3.5. On-Policy Attention Distillation: On-policy attention distillation minimizes divergence between teacher and student attention distributions over trajectories sampled from the student, without an advantage term.The framework combines attention alignment with output-logit distillation and can produce teacher-like attention patterns on student-generated states.
4. Experiments
Experiments evaluate RAL and attention distillation across image and video benchmarks, finding consistent improvements over GRPO and standard distillation. Gains persist across dense visual contexts and even without explicit thinking processes.
- Main Results: RAL consistently surpasses GRPO and the base Qwen-2.5-VL-7B model across image and video VQA benchmarks.The evaluation covers spatial, temporal, and reasoning-intensive capabilities.
- Image VQA: RAL outperforms GRPO across all eight image benchmarks, including gains of +5.8 on V∗, +94.1 on MME, +2.8 on ChartQA, and +3.8 on VizWiz.The gains are associated with stronger visual grounding and compositional reasoning, especially on perception-intensive and document-based tasks.
- Image VQA: Attention distillation outperforms standard distillation on 7 of 8 image benchmarks, with gains of +3.6 on V∗ and +1.8 on MuirBench.V∗ tests fine-grained object attributes and spatial relationships.
- Video VQA: RAL outperforms GRPO on 6 of 7 long-video datasets, with improvements of +2.2 on LongVideoBench, +3.4 on NExTQA, and +1.5 on MVBench.These benchmarks require temporal understanding and multi-hop reasoning; GRPO retains a marginal lead on VideoMMMU.
- Video VQA: Under on-policy distillation, attention-enhanced alignment improves every video benchmark except ties on LongVideoBench and MVBench.The largest reported gains are +4.4 on NExTQA and +2.6 on VideoMME.
- Scaling with Visual Density: RAL’s advantage widens with visual density, increasing on V∗ from +1.6 at 512 tokens to +6.3 at 2048 tokens.Across LongVideoBench, RAL consistently outperforms GRPO at 32, 64, and 128 maximum frames.
- Ablation Studies: RAL-zero surpasses the base model on 5 of 7 long-video benchmarks and full GRPO on 5 benchmarks without an explicit thinking process.It achieves state-of-the-art performance on NExTQA, VideoMME, and LVBench.
- Ablation Studies: RAL-zero improves over the base model in 4 of 8 image benchmarks and achieves the highest scores among RL-based methods on MMMU-Pro and VizWiz.These results support attention optimization as a route to cross-modal reasoning without verbalized logic.
5. Conclusion
RAL shifts MLLM post-training from text-token optimization to internal attention distributions, directly reinforcing visual grounding and perceptual focus. Across diverse benchmarks, it consistently outperforms established baselines and extends to attention-centered on-policy distillation.
- RAL shifts optimization from text token distributions to internal attention distributions, treating attention as a policy for directly reinforcing visual grounding.This targets the underlying cross-modal reasoning process rather than relying only on output-token optimization.
- RAL consistently outperforms the base Qwen-2.5-VL-7B and GRPO baselines across diverse image and long-video benchmarks.The gains are described as more stable and uniform than those from token-level RL, which can occasionally degrade base-model performance.
- RAL extends naturally to on-policy distillation, where transferring attention about “where to focus” surpasses simple output imitation.The attention signal is described as complementary and semantically rich for cross-modal alignment.
- Attention distributions are established as a first-class optimization target for multimodal alignment and a process-aware alternative to standard RLHF.The paper identifies fine-grained credit assignment and other internal structures, including MoE routing and cross-modal fusion, as future directions.