PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence

Yiran Zhang, Hanbo Huang, Jielan Li, Zhen Zhao, Shiyu Liang, Hongxia Hao

arXiv:2610.21671v1cs.CL

TL;DR

Self-distilled multimodal RLVR can miss whether token support comes from the image or a language shortcut, and applies fixed weighting even when the teacher is uninformative. PACE contrasts correct and incorrect hints under full and null vision, then adapts credit redistribution to trajectory confidence. It achieves the best average accuracy across three main backbones and five multimodal reasoning benchmarks, with both components contributing to gains.

  • Problem

    Self-distilled RLVR cannot distinguish image-based token support from language shortcuts and applies fixed weighting across trajectories, including ones where the teacher is uninformative.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    PACE compares correct and incorrect hints under full and null vision to form token contrasts, then uses batch-normalized trajectory confidence to control their credit redistribution.

    Source: §1 · pp. 1–2, §4.3 · p. 5

  • Results

    PACE achieves the best average accuracy on all three main backbones across five multimodal reasoning benchmarks, and both components contribute to gains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Hint contrasts provide more stable token credit than single-view signals, while batch-normalized trajectory confidence keeps the mixing rate stable during training.

    Source: §6 · p. 8

    Whether gains transfer to other tasks remains untested, and scoring four teacher views adds training cost, although inference is unaffected.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but group-relative policy optimization broadcasts a single sequence-level advantage to every token of a response, although errors usually localize to a few decisive steps. Recent self-distilled variants refine credit with a teacher--student likelihood gap. However, this signal cannot tell whether it relies on the image or on a language shortcut, and is applied uniformly even when the teacher is uninformative. To address this, we propose Pairwise token contrast and Adaptive trajectory ConfidencE (PACE). Pairwise token contrast scores each response under four teacher views, crossing a correct or incorrect answer hint with the image present or removed, and compares the two hints for every token, which cancels shifts shared by both hints and reveals whether the token's support relies on the image. Adaptive trajectory confidence measures how well the teacher predicts each response given the correct hint and the image, normalizes this score within the batch, and uses it to set how far the token contrast redistributes credit within that response. PACE achieves the best average accuracy on three backbones and ranks first on all evaluated multimodal reasoning benchmarks with Qwen3-VL-8B-Instruct. Experiments show that both components matter: replacing pairwise token contrast with a single-view signal underperforms our method, while adaptive trajectory confidence adds further improvement.

1 Introduction

Prior token-level self-distillation leaves multimodal credit unable to distinguish image-based support from language shortcuts and weights every trajectory alike. PACE combines paired teacher contrasts with confidence-adaptive credit redistribution, and achieves the best average accuracy across its evaluated backbones and benchmarks.

1 Introduction

  • Single-view, fixed-weight token supervision cannot distinguish image-based support from language shortcuts and weights informative and uninformative trajectories equally.The paper identifies these challenges as aggravating drift in the teacher–student gap.Source: §1 · p. 1
  • PACE compares correct and incorrect answer hints under full and null vision, then uses trajectory confidence to control how strongly token contrasts redistribute credit.The paired contrasts cancel shifts shared across hints, while adaptive confidence applies stronger contrast weighting to trajectories the teacher predicts confidently.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • PACE achieves the best average accuracy on all three main backbones across five multimodal reasoning benchmarks.The method’s pairwise token contrast and adaptive trajectory confidence both contribute to the gains.Source: §1 · p. 2

2 Related Works

Prior work applies RLVR and self-distillation to multimodal reasoning, but existing approaches differ from PACE in how they form token-level signals. PACE contrasts correctness-conditioned views across both full and null vision, while related visual-grounding and reflection methods take different approaches.

RLVR for language and multimodal model reasoning

  • GRPO and multimodal variants use group-relative rewards but broadcast one sequence-level advantage across all response tokens; PACE instead assigns that credit at token level.These approaches retain rule-based verification and group advantages.Source: §2 · p. 2

On-policy self-distillation and token-level credit assignment

  • VCSD, GCPO, PEPO, and CEPO use distinct token-credit signals, but none combines PACE’s answer-conditioned contrast with a full/null vision factorial design.CEPO contrasts correct and incorrect answer teachers while holding the image fixed; PACE uses the contrast for sign-preserving GRPO credit rather than standalone distillation.Source: §2 · pp. 2–3

Visual grounding and reflective multimodal RL

  • Visual-grounding methods prioritize visual dependence or retention, while reflection-centric systems add rethink or re-attention stages; PACE scores already-sampled responses without changing inference.Visual dependence alone is unsigned with respect to correctness, and reflection changes the sampled reasoning process.Source: §2 · p. 3

3 Preliminaries

The preliminaries describe group-relative policy optimization for rule-verified responses and motivate redistributing sequence-level credit across tokens. They identify shortcomings in single-view self-distilled gains and requirements for more reliable token-level weighting.

Multimodal RLVR

  • GRPO normalizes rule-based rewards across sampled responses and assigns the resulting sequence advantage to every token, despite errors often localizing to decisive tokens.The verifier and group advantage remain unchanged in PACE; only the allocation of credit across tokens changes.Source: §3 · p. 3

Limitations of self-distilled RLVR

  • A single-context teacher–student gain cannot distinguish image support from language-only patterns and applies the same weighting even when the teacher barely separates contexts.The passage also attributes gain drift to changing teacher log-probability offsets during training.Source: §3 · p. 3

Problem formulation

  • Redistribution weights remain positive and unit-mean within a response, preserving the verifier advantage’s sign and mean; uniform weights recover GRPO.The mixing form combines normalized token weights with uniform weighting.Source: §3 · p. 3
  • Token credit should compare answer hints with and without the image so shared offsets cancel, while trajectory-level mixing should reflect teacher confidence relative to the batch.The proposed factors instantiate these requirements as pairwise token contrast and adaptive trajectory confidence.Source: §3 · p. 3

4 Methodology

PACE scores sampled responses under counterfactual teacher views and uses pairwise token contrasts and adaptive trajectory confidence to redistribute GRPO credit. The weighting preserves trajectory-level direction and mean at the coefficient level.

4.1 Counterfactual Teacher Views

  • The teacher scores each sampled response across correct or incorrect answer hints and full or null vision, forming four counterfactual views.Incorrect hints come from valid sibling responses; when none exists, the method falls back to single-view RLSD or uniform weights.Source: §4.1 · pp. 3–4, Figure 2 · p. 4

4.2 Pairwise Token Contrast

  • Pairwise token contrast compares correctness under full and null vision, exposing how much of the correctness signal remains without the image.The contrast combines the two correctness contrasts; shared shifts cancel because each pair differs only in its answer hint. Tokens whose correctness preference holds without the image receive a moderate additional estimate rather than being penalized.Source: §4.2 · pp. 4–5

4.3 Adaptive Trajectory Confidence

  • Adaptive trajectory confidence measures the teacher’s separation of the full-information view from degraded views, normalizes that score within the micro-batch, and maps it to a trajectory-specific mixing rate.The rate ranges from 0.15 to 0.65; trajectories without a valid sibling pair use single-view RLSD weights at the minimum rate, while those without any hint use uniform GRPO weights.Source: §4.3 · p. 5

4.4 Contrast-Weighted Credit Redistribution

  • Contrast-weighted credit uses direction-aware token weights, normalizes them within each response, and mixes them with uniform weights before replacing the GRPO advantage.At the coefficient level, the weights preserve the advantage’s sign and response-level mean; these guarantees do not extend to the clipped nonlinear PPO update.Source: §4.4 · pp. 5–6

5 Experiments

PACE is evaluated against multimodal RLVR baselines on five reasoning benchmarks, across three backbones, with ablations of its token-credit and adaptive-mixing components. It ranks first across the Qwen3-VL-8B benchmarks and achieves the best average accuracy on all three backbones.

5.1 Experimental Setup

  • The shared training setup uses MMFineReason-123K, 200 optimization steps, and eight rollouts per prompt; it compares PACE with GRPO, OPSD, RLSD, RLCSD, PAPO, and VPPO.Training uses three backbones, and evaluation covers five benchmarks: GSM8K-V, LogicVista, MMMU, MathVision, and MathVista.Source: §5.1 · p. 6

5.2 Main Results

  • PACE exceeds RLSD by 3.09 points and RLCSD by 3.96 points on average; on the other two backbones, it improves over GRPO by 3.64, 1.48, and 1.12 points.The largest Qwen3-VL-8B gaps over RLSD and RLCSD are on LogicVista (+5.59 and +6.26) and MathVision (+4.11 and +5.06).Source: §5.2 · pp. 6–7
  • PACE ranks first on all five Qwen3-VL-8B benchmarks (58.47% average) and has the best averages on Qwen2.5-VL-7B (45.17%) and InternVL3.5-4B (48.00%).On InternVL3.5-4B, RLSD narrowly leads PACE on GSM8K-V, 15.92 versus 15.54.Source: §5.2 · pp. 6–7, Table 1 · p. 7
  • PACE’s gains over Base shrink from +6.38 points on Qwen3-VL-8B-Instruct to +4.04 on Qwen2.5-VL-7B-Instruct and +2.05 on InternVL3.5-4B.The paper attributes the smaller gains to noisier token contrasts when smaller self-teachers solve fewer hard training problems, limiting the added value of token-level credit assignment.Source: §5.2 · p. 7

5.3 Ablation Study

  • Replacing hint contrasts with the vision gap lowers average accuracy to 51.05%, below Base (52.09%); fixed-rate hint-contrast variants reach 57.06–57.68%.The 0.62-point spread across hint-contrast variants is small beside the six-point drop from the vision gap; hint contrasts also stay stable while the vision gap drifts during training.Source: §5.3 · pp. 7–8, Table 2 · p. 8
  • With adaptive mixing, average accuracy peaks at contrast coefficients (α, β) = (1.5, 0.5); removing the null-vision contrast or weighting it heavily degrades accuracy.The next-best tested coefficients are (1.25, 0.25).Source: §5.3 · p. 8
  • Adaptive trajectory confidence improves over fixed λ = 0.5 by 1.24 points, with 1.17 points attributed to per-trajectory adaptation rather than the average rate.No fixed rate exceeds 57.63%, and adaptive mixing outperforms every fixed rate on GSM8K-V, MMMU, and MathVision.Source: §5.3 · p. 8, Figure 3 · p. 8
  • Within the tested range, accuracy depends more on the mixing-rate range than on the confidence measure; at [0.25, 0.65], the three measures perform similarly (57.60–57.64%).The tested measures include mean-based and dispersion-based confidence scores.Source: §5.3 · p. 8

6 Conclusion

PACE assigns credit at both token and trajectory levels: token contrast selects which tokens receive more credit, while trajectory confidence controls the redistribution strength. The paper reports the best average accuracy across evaluated benchmarks for models from 4B to 8B.

6 Conclusion

  • PACE uses pairwise token contrast to decide which response tokens receive more credit and adaptive trajectory confidence to set how far credit is redistributed.Hint contrasts provide more stable token credit than single-view signals, while batch-normalized confidence keeps the mixing rate stable during training.Source: §6 · p. 8
  • PACE achieves the best average accuracy on evaluated benchmarks across model families and model sizes from 4B to 8B.Source: §6 · p. 9

Limitations

PACE’s limitations include incomplete visual-grounding guarantees, restricted evaluation scope, and added training cost. The accompanying related-work and implementation passages clarify its credit-assignment design, training procedure, and confidence normalization.

Limitations

  • PACE’s null-vision intervention cannot rule out correlated language shortcuts, and its confidence score does not guarantee visual grounding; transfer beyond verifiable math and logic remains untested.Gains are smaller on smaller backbones, and PACE does not achieve the best average on Qwen3-VL-2B-Instruct.Source: Limitations · p. 9
  • Four teacher-view scores add training cost, although inference is unaffected.Source: Limitations · p. 9

A Related Works

  • Unlike reflection-based methods, PACE scores sampled trajectories without adding an inference-time reflection module, and its four-view contrast tests correctness-aligned visual support.It combines full- and null-vision answer contrasts as a detached, sign-preserving, mean-conserving credit signal rather than a distillation target.Source: A Related Works · pp. 12–13

B.1 Training Procedure

  • PACE samples responses, builds positive and incorrect-sibling answer hints, scores each response under four teacher views, then redistributes group advantage across tokens for a PPO-clipped update.Without a valid negative-hint sibling it falls back to RLSD weights; without any hint it uses uniform GRPO weights, and teacher scoring adds no inference-time cost.Source: §B.1 · p. 13

B.2 Training Configurations

  • The shared training stack uses full-parameter student fine-tuning and a BF16 teacher refreshed every 20 steps; the 2B configuration instead refreshes every 10 steps and narrows mixing to [0.05, 0.30].OPSD is trained separately with rank-64 LoRA adapters and the ms-swift GKD trainer.Source: §B.2 · pp. 13–14

B.3 Teacher Prompt

  • Teacher prompts use a privileged answer candidate; negative candidates come from incorrect sibling rollouts, while visual-null views zero image embeddings but retain image-token positions.Source: §B.3 · p. 14

B.4 Shared-Offset Cancellation in Pairwise Token Contrast

  • Pairwise contrasts remove offsets shared by the paired hints, retaining relative correctness support rather than an absolute confidence level.Batch means of the paired contrasts stayed within 0.02 of zero and shifted by at most 0.011, but this does not rule out token- or view-specific shifts; cancellation is not general robustness to policy drift.Source: §B.4 · p. 15

B.5 Adaptive Trajectory Confidence: Implementation Details

  • Adaptive trajectory confidence uses robust, batch-relative normalization because teacher confidence shifts during training and its distribution is heavy-tailed.The confidence measures teacher separation of full-information from degraded views, not image necessity; a high mixing rate therefore does not guarantee visual grounding.Source: §B.5 · p. 16

B.6 Sign- and Mean-Preserving Properties: Scope and Limitations

The token coefficients preserve each response’s mean advantage and sign before policy optimization. Nonlinear clipping means these coefficient-level properties do not fully determine the resulting token updates.

B.6 Sign- and Mean-Preserving Properties: Scope and Limitations

  • Coefficient-level token advantages preserve each response’s mean advantage and the sign of its trajectory advantage.The mean property follows because the trajectory advantage is a scalar multiplier; the positive mixing formula preserves its sign.Source: §B.6 · p. 16
  • Normalization makes the mean of the normalized token weights equal to one before they are mixed with the trajectory advantage.This follows from dividing each weight by its trajectory-level mean.Source: §B.6 · p. 16
  • After policy optimization, clipping makes token-level gradient flow depend on the active objective term and does not guarantee exact mean or relative weighting.The magnitude of token updates also depends on the clipping regime and advantage magnitudes.Source: §B.6 · pp. 16–17

B.7 Training Dynamics

Across the shared 200-step budget, training reward rises similarly for PACE, RLSD, and GRPO, while their response-length trajectories differ. PACE ends with the shortest responses.

B.7 Training Dynamics

  • Training reward rises from about 0.20 to 0.39, with the methods’ curves close and converging again by step 200.PACE is slightly ahead around steps 120–170, peaking near 0.41.Source: §B.7 · p. 17
  • At step 200, PACE has the shortest responses at about 620 tokens, while GRPO has re-inflated to about 800 tokens.All methods sharply compress responses during the first 50 steps; PACE stays around 600–700 tokens thereafter.Source: §B.7 · p. 17

B.8 Teacher Signals During Training

The teacher-signal analysis finds that hint contrasts remain stable despite drift in the vision gap, and batch-relative confidence keeps mixing rates stable. The supplied results also report a training-time cost and a capacity-related limitation on Qwen3-VL-2B-Instruct.

B.8 Teacher Signals During Training

  • The vision gap doubles from 0.20 to 0.42, while hint contrasts stay within 0.02 of zero and move by at most 0.011.The shared drift cancels when the two hints are subtracted; a 0.4 drift would produce weight factors e^0.4 ≈ 1.49, exceeding the clip bound.Source: §B.8 · pp. 17–18

B.9 Training-Time Cost Analysis

  • After the first 50 steps, a PACE step takes about 350–380 s, versus 250–280 s for GRPO and 300–350 s for RLSD and RLCSD.Generation takes about 50 s per step, close to GRPO; the additional work is from teacher scoring and is confined to training.Source: §B.9 · p. 18, Figure 6 · p. 18

C.1 Main Results with Standard Deviation

  • Table 5 reports per-benchmark accuracy with standard deviation for four models, including Qwen3-VL-2B-Instruct.The reported models are Qwen3-VL-8B-Instruct, Qwen2.5-VL-7B-Instruct, InternVL3.5-4B, and Qwen3-VL-2B-Instruct.Source: §C.1 · p. 18

C.2 Results on Qwen3-VL-2B-Instruct

  • PACE is second on average on Qwen3-VL-2B-Instruct at 38.23% (+4.19 over Base), behind RLSD at 38.78%.PACE leads on GSM8K-V and LogicVista but trails RLSD on MMMU, MathVision, and MathVista.Source: §C.2 · p. 18
  • On the 2B model, the authors attribute weaker teacher signals to its limited capacity for the difficult filtered training data.Low-confidence or inconsistent predictions under correct-answer contexts can make both pairwise contrasts and adaptive confidence unreliable.Source: §C.2 · p. 18

C.3 Full Ablation Results

The ablation table lists each configuration once and describes how the full PACE setup differs from alternative contrast and confidence settings. The supplied table caption identifies accuracy as its measure.

C.3 Full Ablation Results

  • Table 6 lists each ablation configuration once; full PACE uses full-plus-null vision contrast, Smean confidence, and λi ∈ [0.15, 0.65].Other rows change only the setting listed for that row.Source: §C.3 · p. 18
  • The contrast definitions are full vision c = ∆, full-minus-null vision c = 2∆−¯∆, and full-plus-null vision c = 1.5∆ + 0.5¯∆.The vision-gap variant is c = ℓ+,V − ℓ+,∅.Source: §C.3 · p. 18
  • The supplied table caption identifies accuracy (%) with mean@8 and standard deviation as the reported measure.It describes five multimodal reasoning benchmarks.Source: Table 5 · p. 19

D Case Study

Figures 7–9 compare PACE and RLSD token weights on three sampled responses, showing where each method places trajectory credit. The examples illustrate PACE emphasizing decisive readouts and answer-bearing tokens, but are qualitative and do not establish that its credit is correct on every response.

Credit on visual readouts

  • PACE emphasizes chart readouts, the comparison token “both”, and “All 7”, while RLSD emphasizes question-restating tokens and leaves the readouts nearly unweighted.The example asks how many items exceed 3 units in at least one store; the response concludes that all seven qualify.Source: D Case Study · p. 19, Figure 7 · p. 21

Credit on computation

  • For the share calculation, PACE weights copied counts, key total and quotient digits, and answer-decision tokens; RLSD favors connective and planning tokens and weights counts nearly uniformly.The response sums four visitor counts to 437,380, divides to obtain 18.86%, and selects option (B).Source: D Case Study · p. 19, Figure 8 · p. 21

Credit on an incorrect response

  • On an incorrect answer, PACE concentrates the negative-advantage penalty on extracted values and their context, while RLSD spreads it over generic planning tokens.The response reads 58.09% and 44.00% and answers 51.05 instead of 51.04; the passage locates the error in extracted values or rounding and says the arithmetic is consistent with the readouts.Source: D Case Study · pp. 19–20, Figure 9 · p. 22

Case study scope

  • The three selected examples are qualitative illustrations of credit placement, not evidence that PACE assigns credit correctly on every response.Both methods preserve the trajectory advantage’s sign and mean, differing only in where credit is placed within the response.Source: D Case Study · pp. 19–20

Related papers

Loading 2610.21671v1…