PaperlayerView exact revision on arXiv

Source-linked AI summary

DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning

Hangui Lin, Yan Shu, Zhengyang Liang, Chi Liu, Xiangrui Liu, Minghao Qin, Teng Long, Zheng Liu, Nicu Sebe

arXiv:2606.08035v1cs.CV

TL;DR

Existing RLVR methods overlook the dynamic cross-modal coordination needed during visual reasoning. DyCo-RL adds token-level role-alignment signals to RLVR, consistently improving four algorithms across seven visual and mathematical reasoning benchmarks.

  • Problem

    Existing visual RLVR methods overlook the dynamic alternation between extracting visual evidence and anchoring reasoning to textual context during CoT generation.

  • Method

    DyCo-RL assigns token roles using Fisher–Rao attention shifts and reweights policy advantages according to role-attention alignment.

  • Results

    DyCo-RL consistently improves four representative RLVR algorithms across seven benchmarks spanning visual-centric and mathematical reasoning.

  • Takeaways & Limitations

    Explicit token-level cross-modal coordination is an effective path toward faithful multimodal reasoning.

  • Takeaways & Limitations

    DyCo-RL incurs additional rollout-time computation and memory overhead, and its generalization to substantially larger MLLMs remains unverified.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a leading paradigm for enhancing visual reasoning in Multimodal Large Language Models (MLLMs). However, existing RLVR methods optimize primarily for the reasoning outcome, fundamentally overlooking the fine-grained cross-modal coordination required during the generation process. Through token-level analyses and controlled interventions, we reveal that during Chain-of-Thought (CoT) reasoning, MLLMs frequently fail to dynamically alternate between extracting visual evidence and synthesizing textual context-a coordination breakdown that is causally linked to reasoning failures. Motivated by these findings, we propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization. Specifically, DyCo-RL uses the Fisher-Rao geodesic distance to measure within-modality attention shifts, assigning tokens to either visually-oriented or text-oriented functional roles. It then evaluates the alignment between a token's actual attention allocation and its assigned role, leveraging this score for alignment-guided advantage reweighting during policy optimization. Extensive experiments demonstrate that the algorithm-agnostic DyCo-RL, when applied to Qwen2.5-VL-3B/7B, consistently improves four representative RLVR algorithms across seven benchmarks spanning visual-centric and mathematical reasoning.

1 Introduction

DyCo-RL addresses cross-modal coordination failures in MLLM visual reasoning, where tokens fail to dynamically alternate between visual evidence extraction and textual-context synthesis during CoT. It integrates token-level attention-shift roles and modality-specific alignment into RLVR optimization, improving multiple algorithms across visual and mathematical reasoning benchmarks.

  • Problem: During CoT, models must alternate between extracting visual evidence and anchoring logic to preceding textual context, but coordination breakdowns produce hallucinated visual features and incoherent reasoning.The illustrated failure includes incorrectly asserting ∠ADE = 80° and baselessly claiming ∠CED = ∠ADE.
  • Problem: Existing RLVR methods overlook dynamic coordination by applying uniform token optimization or focusing exclusively on visual perception.These strategies neglect the interplay between modalities required for robust multi-step reasoning.
  • Evidence: Token-level attention analysis finds that erroneous samples under-attend to the image with visually-oriented tokens and insufficiently anchor text-oriented tokens to preceding textual context.The analysis examines how each token distributes attention between visual patches and textual context.
  • Method: DyCo-RL assigns token roles using Fisher–Rao geodesic distances between consecutive within-modality attention distributions and reweights advantages using modality-specific alignment signals.The method embeds dynamic, token-level cross-modal coordination directly into RLVR optimization.
  • Results: DyCo-RL consistently improves GRPO, DAPO, SAPO, and GSPO on seven benchmarks with Qwen2.5-VL-3B/7B, spanning visual-centric and mathematical reasoning.Ablations attribute the gains to improved dynamic coordination rather than naive modality bias.

2 Related Work

Prior work largely improves visual reasoning through trajectory-level RLVR optimization, while other studies use attention for visual-perception enhancement, inference-time intervention, and interpretability. These approaches motivate finer-grained analysis of cross-modal interactions during generation.

  • RLVR for Visual Reasoning: GRPO, DAPO, GSPO, and SAPO optimize policy gradients primarily at the trajectory level with uniform credit assignment across tokens.These algorithms are presented as foundational RLVR methods for enhancing visual reasoning in MLLMs.
  • RLVR for Visual Reasoning: PAPO introduces an implicit perception loss based on KL divergence between outputs under original and masked inputs.The method explicitly targets visual perception within RLVR-based visual reasoning.
  • Cross-Modal Attention Analysis in MLLMs: Attention-based research studies cross-modal interactions through inference-time interventions and interpretability analysis.Attention refocusing and visual-aware decoding dynamically reweight attention distributions during generation to mitigate hallucinations and improve visual faithfulness.

3 Preliminaries

The preliminaries define modality-specific attention in autoregressive MLLMs and outline GRPO’s group-normalized, clipped policy optimization. They also identify GRPO’s uniform scalar advantage as a limitation because it ignores token-level visual–textual coordination.

  • Modality-Specific Attention: Autoregressive MLLMs partition context tokens into visual and textual sets, with the decoder attending to both at each generation step.Textual tokens include the input prompt and previously generated tokens.
  • Modality-Specific Attention: A generated token’s modality-specific attention score measures the total attention density allocated to visual or textual tokens during generation.The score is defined for modality m ∈ {vis, txt}.
  • GRPO Optimization: GRPO generates a group of candidate responses, assigns outcome-based rewards, and computes group-normalized advantages from within-group reward statistics.The reference policy generates G candidates, while rewards depend on whether final answers match ground truth.
  • GRPO Optimization: GRPO optimizes a clipped surrogate objective using importance sampling ratios between the current and reference policies.The ratio ρi,t compares token probabilities at generation step t.
  • GRPO Limitation: GRPO broadcasts one scalar advantage uniformly across response tokens, giving identical learning signals to tokens that extract visual evidence or synthesize textual logic.This formulation ignores the fine-grained, dynamic modality coordination required for robust visual reasoning.

4 Method

Section 4 shows that cross-modal attention misalignment is associated with and causally contributes to token-level reasoning errors. DyCo-RL addresses this by assigning visual or textual roles using Fisher–Rao attention shifts and reweighting token-level advantages according to role alignment.

  • Token Role Assignment: DyCo-RL assigns each token a visual, textual, or neutral role by comparing consecutive within-modality attention distributions with the Fisher–Rao geodesic distance.The Fisher–Rao distance is symmetric and bounded, providing a stable measure for noisy attention dynamics; a stability margin suppresses ambiguous transitions.
  • Coordination Breakdown: Correct visually-oriented tokens attend more to images, while correct text-oriented tokens attend more to preceding context than their erroneous counterparts.This establishes an association between modality-specific attention allocation and token-level correctness.
  • Coordination Breakdown: Moderate role-specific attention enhancement recovers performance on previously erroneous samples, whereas excessive enhancement degrades performance by disrupting attention balance.Visually-oriented tokens show higher sensitivity to correction.
  • Alignment-Guided Optimization: DyCo-RL dynamically reweights the standard sequence-level advantage for each token before integrating the result into the policy-gradient objective.The reweighting uses ˜At = (1 + α · wt) · ˆA, with |o| preserving the overall advantage magnitude after softmax normalization.
  • Alignment-Guided Optimization: The method scores whether each token’s actual attention matches its assigned role, rewarding visual and textual tokens for attending to images and textual context while leaving neutral tokens unbiased.This alignment score provides fine-grained reinforcement-learning guidance.

5 Experiments

Experiments apply DyCo-RL to two Qwen2.5-VL model scales, four RLVR algorithms, and seven mathematical or visual-centric benchmarks. Results show broad gains, algorithm-agnostic transfer, scalability, and benefits from Fisher–Rao role assignment and coordinated rather than single-modality reweighting.

  • Experimental Setup: DyCo-RL is integrated into Qwen2.5-VL-3B/7B and evaluated with GRPO, DAPO, GSPO, and SAPO on seven mathematical and visual-centric benchmarks.Training uses ThinkLite-hard-11K, while evaluation reports top-1 accuracy at decoding temperature 0.1.
  • Main Results: DyCo-RL consistently improves baseline performance across the vast majority of benchmarks, functioning as an effective plug-and-play module for existing RLVR pipelines.The evaluation covers four representative RLVR algorithms and two model scales.
  • Algorithm-Agnostic Generality: +5.7 is the largest reported individual benchmark gain across GRPO, DAPO, SAPO, and GSPO, with average performance improving across all baselines.The gains transfer across clip-based, sequence-level, and soft-gated objectives without algorithm-specific hyperparameter tuning.
  • Scalability and Cross-Domain Performance: +13.1 is the reported individual surge on MMBench for the 7B model, while DyCo-RL continues elevating average performance despite narrower headroom.Improvements also span mathematical and visual-centric reasoning, strengthening visual evidence grounding while preserving or enhancing chain-of-thought coherence.
  • Ablation Studies: Fisher–Rao role assignment outperforms the evaluated alternatives, whereas Random provides marginal unstable gains and Reverse causes the most severe degradation.Fisher–Rao supplies a symmetric, geometrically principled measure that addresses Entropy’s static snapshot and KL’s asymmetric divergence limitations.
  • Ablation Studies: Single-modality reweighting improves modality-biased tasks but severely degrades the other reasoning type, whereas DyCo-RL’s role-conditional coordination is more robust.Universal visual weighting helps visually intensive tasks but harms textual reasoning; universal textual weighting shows the opposite pattern.

6 Discussion and Analysis

DyCo-RL improves cross-modal reasoning by strengthening alignment between token roles and modality attention, while reshaping temporal dynamics toward interleaved visual re-grounding and textual reasoning.

  • Analysis Setup: The analysis samples 200 generated MathVerse and MathVision instances from vanilla GRPO and GRPO+DyCo-RL, independently annotating tokens as visually- or text-oriented.The annotations follow the diagnostic protocol in Section 4.1 and account for distinct reasoning trajectories.
  • Strengthening Role-Aligned Attention: DyCo-RL strengthens role-aligned attention by increasing image attention and suppressing text attention for visually-oriented tokens, while amplifying text attention for text-oriented tokens.These changes are illustrated in Fig. 4(a).
  • Reshaping Temporal Dynamics: DyCo-RL replaces the baseline’s phase-locked dynamics with interleaved reasoning: visually-oriented tokens persist at 0.4–0.6, while text-oriented tokens activate earlier and distribute more evenly.The sustained visual presence supports continuous visual re-grounding, as shown in Fig. 4(b-c).

7 Conclusion

The conclusion identifies cross-modal coordination breakdowns as a primary bottleneck in visual RLVR and presents DyCo-RL as an algorithm-agnostic solution. DyCo-RL assigns functional token roles using Fisher–Rao attention distance and dynamically reweights advantages according to role–attention alignment, improving four representative baselines.

  • Conclusion: Cross-modal coordination breakdowns are identified as a primary bottleneck in visual RLVR.Visually- and text-oriented tokens systematically under-attend to their designated modalities.
  • Conclusion: DyCo-RL is proposed as an algorithm-agnostic plugin for addressing this bottleneck.It assigns functional token roles via Fisher–Rao attention distance and dynamically reweights advantages based on role–attention alignment.
  • Conclusion: DyCo-RL consistently improves four representative baselines across the evaluated benchmarks.

Limitations … C Broader Impact

DyCo-RL incurs additional rollout-time training cost and has only been evaluated on Qwen2.5-VL-3B and 7B models. Its broader-impact rationale is to make cross-modal coordination an explicit, algorithm-agnostic optimization target for trustworthy multimodal reasoning.

  • Limitations: DyCo-RL adds rollout-time training time and GPU memory overhead by computing per-token attention statistics and Fisher–Rao distances.The method introduces no additional trainable parameters, but further engineering optimization is needed, especially for longer rollouts.
  • Limitations: All experiments are conducted on Qwen2.5-VL-3B and 7B models because of computational resource constraints.The supplied passage continues beyond this limitation, but does not provide additional visible details here.
  • A Overview of Appendix: The appendix includes sections on LLM usage, broader impact, experimental setup, implementation details, diagnostic annotation, ablations, computational overhead, training dynamics, generalization, and visualizations.These topics are listed as appendix sections B, C, D, E, F, G, H, I, J, and L.
  • B LLM Usage Statement: An LLM assisted only with proofreading and clarity rephrasing, not core ideas, data analysis, or the main content.This statement limits the reported LLM use to editorial assistance.
  • C Broader Impact: DyCo-RL’s attention-based role assignment and alignment-guided advantage reweighting extend beyond visual mathematical reasoning to long-chain multimodal tasks.The paper identifies dynamic alternation between visual grounding and textual deduction as a pervasive bottleneck in such tasks.
  • C Broader Impact: Relevant application domains include visual document and chart understanding, long-video reasoning, remote sensing reasoning, and medical image analysis.These domains require continuous coordination between perceptual grounding and higher-level reasoning, whose breakdown can produce hallucinated details, temporal inconsistencies, or evidence-drifting derivations.
  • C Broader Impact: By making cross-modal coordination an explicit token-level signal, DyCo-RL offers an algorithm-agnostic framework for more faithful and hallucination-resilient multimodal reasoners.The Fisher–Rao assignment uses intrinsic attention geometry, and the reweighting strategy integrates with existing reward design.
  • C Broader Impact: The paper envisions role-aware token-level optimization for trustworthy multimodal systems in healthcare, autonomous systems, and scientific discovery.In these high-stakes domains, step-by-step reasoning fidelity is presented as equally important as final-answer correctness.

D Details of Experimental Setup · E Implementation Details · F Diagnostic Annotation Protocol

The study evaluates DyCo-RL against four policy-optimization baselines across seven benchmarks spanning mathematical and visual-centric reasoning, using a standardized VLM-R1 training setup. A token-level diagnostic protocol labels visual- and text-oriented reasoning behavior and achieves substantial inter-annotator agreement.

  • D Details of Experimental Setup: DyCo-RL is compared with four reinforcement learning baselines representing recent policy-optimization methods for reasoning-oriented language models.The baselines include GRPO, DAPO, SAPO, and a fourth method described in the experimental setup.
  • D Details of Experimental Setup: Evaluation covers seven benchmarks targeting faithful visual grounding and consistent textual anchoring across mathematical and visual-centric reasoning.The benchmark suite includes WeMath, MathVision, MathVerse, LogicVista, HallusionBench, MME, and MMBench.
  • E Implementation Details: The experiments use the VLM-R1 framework, train for one epoch on ThinkLite-VL-Hard-11k, and sample four responses per prompt at temperature 0.9.ThinkLite-VL-Hard-11k contains 11,031 complex reasoning instances.
  • E Implementation Details: The vision tower remains unfrozen, while AdamW training uses a peak learning rate of 1 × 10^-6, linear decay, gradient checkpointing, and a global gradient-norm cap of 1.0.These settings are applied across the evaluated baseline algorithms.
  • E Implementation Details: Training uses bfloat16, DeepSpeed ZeRO-3 without CPU offloading, per-device batch size 4, two-step gradient accumulation, and a global batch size of 64 on 8 × NVIDIA A100 GPUs.Average training requires approximately 48 GPU-hours for the 3B model and 72 GPU-hours for the 7B model per algorithm.
  • F Diagnostic Annotation Protocol: The diagnostic set contains 200 incorrect Qwen2.5-VL-3B GRPO rollouts uniformly sampled from MathVerse and MathVision.The sampled problems cover diagram understanding, geometric reasoning, symbolic derivation, and chart interpretation.
  • F Diagnostic Annotation Protocol: Annotators assign each token one of four functional categories based on its local reasoning role: correct or incorrect visually-oriented, and correct or incorrect text-oriented.Labels are assigned at the semantic-span level and projected to individual tokens, with connective tokens following the role of the subsequent span.
  • F Diagnostic Annotation Protocol: Double-blind annotation by two graduate-level mathematically trained annotators achieves Cohen’s κ = 0.85, with approximately 3% of tokens disputed mainly near perception–reasoning transitions.Annotators review the image, question, full response, and reference answer while labeling trajectory consistency and visual faithfulness rather than only the final answer.

G Additional Ablation Studies

The ablations identify moderate modality-aware advantage reweighting as most effective, while excessive reweighting harms coordination. Advantage reweighting is more consistent than trajectory-level reward shaping, and increasing rollouts improves performance with diminishing returns.

  • Ablation on Reweight Strength (α): α = 0.2 provides the strongest or near-strongest results across most datasets, outperforming the unweighted baseline (α = 0).The reweight strength α controls the intensity of modality-aware advantage adjustment during RL training.
  • Ablation on Reweight Strength (α): α = 0.1 provides insufficient signal, whereas α ≥ 0.4 precipitously degrades performance by overly constraining cross-modal coordination.The results support balanced modulation over aggressive reweighting across benchmarks.
  • Advantage Reweighting vs. Reward Shaping: Advantage reweighting yields more consistent and overall superior gains than trajectory-level reward shaping, which only occasionally surpasses vanilla GRPO.Reward shaping augments the binary accuracy reward with an aggregated alignment score.
  • Advantage Reweighting vs. Reward Shaping: Reward shaping destabilizes policy updates by altering reward scale and variance before group normalization, whereas advantage reweighting adds fine-grained guidance afterward.The resulting noise amplification affects group-relative advantage estimates under GRPO.
  • Scaling the Number of Training Rollouts: Performance scales monotonically with R ∈ {4, 8, 16}, but the gains exhibit sharply diminishing returns as rollout count increases.Larger R lowers advantage-estimate variance but linearly increases training costs.

H Analysis of Computational Overhead … L More Visualization Results

DyCo-RL adds moderate training overhead while preserving vanilla inference efficiency, and its role-aware optimization yields faster, consistently stronger learning dynamics and broad zero-shot generalization. Additional prompt specifications and visualizations document the evaluation format and illustrate dynamic cross-modal coordination.

  • H Analysis of Computational Overhead: 27% lower training throughput results from token-level Fisher–Rao distance computation and role assignment at each generation step.These operations require extracting and aligning cross-modal attention weights throughout decoding.
  • H Analysis of Computational Overhead: Inference speed and memory footprint remain identical to the vanilla backbone because advantage reweighting is detached after optimization.The added cost is confined to training.
  • I Analysis of Training Dynamics: DyCo-RL converges faster early and maintains a strictly higher average accuracy reward than GRPO throughout training.Training curves use a rolling mean with window size 10 for smoothing.
  • J Extended Generalization Analysis: DyCo-RL consistently outperforms standard GRPO across zero-shot benchmarks spanning commonsense visual understanding, real-world image reasoning, chart interpretation, and multimodal question answering.The evaluation tests whether alignment learned on visual-mathematical data transfers across different visual formats, reasoning styles, and task structures.
  • K Prompt Template: The adopted GRPO-style prompt places intermediate reasoning inside <think> tags and final responses inside <answer> tags.The format presents a system instruction, a user question, and explicit reasoning and answer delimiters.
  • L More Visualization Results: DyCo-RL exhibits role-aligned attention and dynamic cross-modal alternation across diverse visual reasoning cases.These visualizations further corroborate the main-paper findings.
Loading 2606.08035v1…