PaperlayerView exact revision on arXiv

Source-linked AI summary

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

Yuhang Han, Dingning Liu, Yujie Chen, Yaojie Zhang, Jingwei Song, Zequn Yang, Xuanming Shang, Li Ding, Zhijiang Guo, Linfeng Zhang

arXiv:2610.32324v1cs.AI

TL;DR

Response-level rewards in multimodal RLVR cover visual observations and subsequent reasoning together, while visual affinity alone does not determine learning credit. OVC calibrates context-relative visual and predictive evidence against advantage-group outcomes, then allocates normalized token weights; it improves six-benchmark mean accuracy over GSPO across three models.

  • Problem

    Response-level rewards evaluate complete solutions despite their distinct visual and reasoning token roles, and visual affinity alone is insufficient to determine token credit.

    Source: §1 · pp. 1–2

  • Method

    OVC measures visual affinity relative to prompt context, calibrates it with predictive features using advantage-side contrasts and regularized variation, then assigns positive normalized token weights.

    Source: §1 · p. 2

  • Results

    OVC improves six-task average over GSPO by +2.04, +2.39, and +1.86 points on 4B, 8B, and 9B, respectively.

    Source: §4.2.1 · p. 5

  • Takeaways & Limitations

    Across three models and six benchmarks, OVC improves average accuracy over GSPO while preserving advantage signs and sequence-level clipping.

    Source: §1 · p. 2, §5 · p. 9

    Reference-based calibration assumes both positive- and negative-advantage sides are present in the actor mini-batch.

    Source: §3 · p. 3

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose \textbf{Outcome-Conditioned Visual Credit (OVC)}, a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.

1. Introduction

Response-level rewards guide multimodal reasoning despite diverse token roles, making finer-grained credit assignment a challenge. OVC measures visual association relative to context and calibrates its contribution using outcome differences before allocating token weights.

1. Introduction

  • RLVR supplies verifiable outcome supervision without dense intermediate-step annotations, but a single response reward spans visual observations, symbolic transformations, and intermediate inferences.This motivates using visual signals for finer-grained token credit assignment.Source: §1 · p. 1
  • Absolute image affinity can be ambiguous when predictive states also align with prompt representations, and rewarding visual association can amplify negative updates when later reasoning fails.The relevant question is how to interpret measured visual association under outcome feedback.Source: §1 · pp. 1–2
  • Figure 1 observations motivate measuring visual relevance relative to context and calibrating it against outcome differences, rather than relying on image affinity alone.The highest image-affinity group does not have the highest observed mean accuracy, while pairs matched on image affinity show different outcome statistics across context-affinity levels.Source: §1 · p. 2, Figure 1 · p. 2
  • OVC’s Relative Visual Encoding contrasts predictive-state affinity to image and prompt-background representations under shared diagonal scaling, without directly prescribing update magnitude.This yields a context-relative visual coordinate.Source: §1 · p. 2
  • Outcome-conditioned Evidence Calibration combines the relative visual coordinate with predictive features, using positive- versus negative-advantage contrasts and regularized within-side variation.With statistics fixed, the visual contribution depends on outcome contrast and variance scale, and is zero when visual reference means coincide.Source: §1 · p. 2
  • Normalized Credit Allocation produces positive, globally normalized token coefficients that preserve the sequence-advantage sign while modulating individual token-gradient contributions.Through GSPO-token, OVC retains sequence-level importance ratios and clipping, without an auxiliary reward model, added process or region supervision, or an inference-time module.Source: §1 · p. 2
  • The paper contributes an analytical distinction between response-level reweighting and within-response allocation to support controlled evaluation of token credit assignment.Its contributions also include a context-relative visual measure and outcome-conditioned calibration without a fixed positive update preference for visual affinity.Source: §1 · p. 2

2. Related Work

Prior work advances multimodal optimization, supervision, and visual-guided credit assignment. This work instead treats visual association as evidence to calibrate against outcome feedback, rather than as a direct update priority.

Policy Optimization and Perceptual Supervision for Multimodal Reasoning

  • RLVR and multimodal methods advance optimization, training procedures, and perceptual supervision, while this work studies fine-grained credit from visual representations under outcome feedback.The cited examples include GRPO [29], GSPO [55], MM-Eureka [25], VL-Rethinker, PAPO [39], and VisualPRM.Source: §2 · p. 2

Visually Guided Fine-Grained Credit Assignment

  • VPPO [11], PGPO [47], PRPO [15], VGPO [40], and KAWHI [8] connect visual signals to policy updates through filtering, advantage redistribution, compensation, or region-guided credit.These approaches use predictive-distribution changes, prediction instability, attention-related scores, or structured visual evidence.Source: §2 · pp. 2–3
  • Visual sensitivity, representational affinity, and region alignment alone do not determine a token’s contribution to outcome improvement; this work calibrates visual association using advantage-side contrasts and within-side variance.It separates measuring visual relevance from allocating learning credit.Source: §2 · p. 3

3. Method

The method treats visual relevance as a measurement calibrated against outcome feedback, then uses the calibrated evidence to allocate policy updates across tokens.

3. Method

  • For each multimodal input, the behavior policy samples response groups and assigns group-relative advantages; calibration statistics aggregate across the actor mini-batch and are detached during the optimizer step.Advantage signs indicate relative reward, not token correctness, and both advantage sides must be represented for reference-based calibration.Source: §3 · p. 3

3.1. A Calibration Principle for Visual Credit

  • Evidence compares a token’s squared distances to positive- and negative-advantage reference means under a shared diagonal metric, with each feature contributing by its mean contrast relative to regularized within-side variation.Positive evidence indicates greater closeness to the positive-advantage reference; the calibration is a fixed-statistic characterization, not an actor-optimized auxiliary objective.Source: §3.1 · p. 3

3.2. Estimating Relative Visual Evidence

  • Relative Visual Encoding measures image affinity relative to prompt context using shared scaling, while predictive features retain sampled-token information through uncertainty scaling.The context prototype comes from the original multimodal input, and the predictive feature is gradient-inspired rather than an exact parameter gradient.Source: §3.2 · p. 4

3.3. Credit Allocation and Policy Optimization

  • Normalized Credit Allocation maps signed evidence to positive coefficients and normalizes their token mean globally across the actor mini-batch.Applied through GSPO-token, detached coefficients retain sequence-level clipping while adjusting individual token-gradient contributions; OVC reuses sampled responses and adds no learnable head or inference-time module.Source: §3.3 · p. 5

4. Experiment

Across three model scales, OVC improves six-benchmark averages over GSPO; ablations, chart-centric tests, alternative optimizers, transfer evaluations, visualizations, and efficiency analyses examine its scope and behavior.

4.1. Experimental Setup

  • Main comparisons use matched instruction-tuned initializations and six benchmarks, with Qwen3-VL trained on ViRL39K and Qwen3.5-9B on Math77K.The first five benchmarks use matched LMMs-Eval decoding and scoring; MMK12-val follows a fixed validation protocol.Source: §4.1 · p. 5

4.2.1. Main Results

  • OVC raises the six-benchmark mean over GSPO by +2.04, +2.39, and +1.86 points on 4B, 8B, and 9B, respectively.It exceeds Base by +4.80, +4.75, and +3.32 points, respectively.Source: §4.2.1 · p. 5, Table 1 · p. 6

4.2.2. Ablation Studies

  • Ablations show that context-relative encoding, within-side variance calibration, and token-specific allocation each contribute to full OVC's four-benchmark mean of 62.72.The mean exceeds uniform-credit GSPO (60.42) by 2.30 points; removing the coordinate, variance calibration, or token-specific correspondence reduces performance.Source: §4.2.2 · pp. 6–7

4.2.3. Chart-centric Reasoning

  • On chart-centric reasoning, OVC raises the six-benchmark average from 54.13 to 55.36 (+1.23 points) over GSPO and performs best on 5/6 benchmarks.The evaluation uses matched initialization, training data, and rollout budgets.Source: Table 3 · p. 7, §4.2.3 · p. 7

4.2.4. Extension to Other Policy Optimizers

  • With DAPO and GRPO, OVC increases six-task averages by 1.73 and 0.76 points, respectively.Improvements occur on 5/6 benchmarks for DAPO and all six for GRPO.Source: Table 4 · p. 7, §4.2.4 · p. 7

4.2.5. Zero-shot Generalization

  • In zero-shot transfer, OVC improves four-task VQA means over GSPO by 0.14, 0.72, and 0.31 points on 4B, 8B, and 9B, respectively.On five scientific and spatial benchmarks, it beats GSPO on all five and raises the average from 62.95 to 63.91.Source: Figure 3 · p. 7, §4.2.5 · pp. 7–8, Table 5 · p. 8

4.3. Visualizations

  • A qualitative example shows OVC assigning relatively high credit to the decisive angle conversion, while DelTA assigns relatively low credit to its erroneous angle reasoning.OVC obtains 11.8 km after converting the bearing to 65°; DelTA uses 25° and obtains 25.3 km.Source: Figure 4 · p. 8, §4.3 · p. 8

4.4. Efficiency

  • During Qwen3.5-9B Math77K training, OVC generally achieves higher reward and MMK12-val performance; credit-estimation passes add 189 s, or 5.16% of 3663 s total step time.OVC adds no learnable parameters or inference-time computation.Source: Figure 5 · p. 8, §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

OVC treats visual association as outcome-conditioned evidence for fine-grained policy optimization, rather than as direct update priority. Across three models and six benchmarks, it improves average accuracy over GSPO without additional process supervision or inference-time modules.

5. Conclusion

  • OVC combines context-relative visual affinity with predictive features and calibrates their contributions using advantage-side contrasts and regularized within-side variation.Positive, normalized token weights modulate token-gradient contributions while preserving advantage signs and sequence-level clipping.Source: §5 · p. 9
  • Across three models and six benchmarks, OVC improves average accuracy over GSPO without additional process supervision or inference-time modules.Source: §5 · p. 9
  • The analysis distinguishes response-level reweighting from within-response allocation and characterizes how calibrated coefficients affect individual token gradients.Source: §5 · p. 9

A.1. Benchmarks and evaluation settings

The appendix specifies benchmark protocols and validation practices, then details ablation controls, generation and scoring procedures, and zero-shot transfer evaluations across general-purpose, scientific, and spatial reasoning tasks.

A.1. Benchmarks and evaluation settings

  • MMK12’s official 2,000-example test split serves as fixed validation, excluded from optimization, checkpoint selection, and early stopping; the reported score is from the final scheduled event.The main comparisons also use fixed benchmark splits and evaluation procedures within each setting.Source: §A.1 · p. 12

A.2. Ablation configurations

  • Ablations vary visual representations, calibration metrics, and allocation rules while retaining the remaining OVC procedure and computing statistics from each run’s actor and rollouts.Allocation controls modify the standard normalized weights; permutation and affinity ranking preserve the within-response weight multiset and total weight.Source: §A.2 · pp. 12–13

A.3. Generation protocol

  • Main evaluations sample responses at temperature 0.7, nucleus threshold 0.8, and top-k 20, with a 32,768-token context window and up to 16,384 generated tokens.The native thinking mode is disabled, though prompts may request a written reasoning trace.Source: §A.3 · p. 13

A.4. Scoring protocol

  • MathVerse and MathVision use DeepSeek V4 binary correctness judgments, MathVista uses judged answer extraction followed by deterministic evaluation, and MMMU and We-Math use native deterministic evaluators.Judging instructions and parsing rules are fixed across compared checkpoints.Source: §A.4 · pp. 13–14

A.5. Zero-shot generalization

  • Zero-shot transfer compares Base, GSPO, and OVC checkpoints without target-benchmark adaptation, holding task-specific prompting, decoding, and scoring fixed within each comparison.The transfer evaluation spans chart understanding, visual question answering, scientific reasoning, and spatial reasoning.Source: §A.5 · p. 14, §A.5.1 · p. 14, §A.5.2 · p. 14

A.5.1. General-purpose VQA

  • General-purpose transfer evaluates ChartQA, CharXiv, A-OKVQA, and SimpleVQA across three Qwen model sizes, reporting absolute scores and changes relative to Base.The four-task average is unweighted, with CharXiv counted once after averaging its descriptive and reasoning scores.Source: §A.5.1 · p. 14, Table 7 · p. 14

A.5.2. Scientific and spatial reasoning

  • Scientific and spatial transfer evaluates Qwen3.5-9B Base, GSPO, and OVC checkpoints on ScienceQA, RxnBench, SUPERChem, EmbSpatial, and ERQA; the reported average is an unweighted mean of five scores.EmbSpatial and ERQA assess embodied-relevant visual reasoning through question answering rather than closed-loop robot control.Source: §A.5.2 · pp. 14–15

A.6. Chart-centric evaluation

The section specifies chart-centric evaluation and formalizes how response-scaled visual and predictive features enter outcome-conditioned calibration. It also bounds the estimator’s scope: the predictive feature is gradient-inspired, and the calibration references require both advantage sides.

A.6. Chart-centric evaluation

  • Evaluation covers six chart benchmarks, with CharXiv averaging reasoning and descriptive scores and Avg. taking the unweighted mean across benchmarks and evaluation seeds.Generation uses one answer per checkpoint with temperature 1.0 and top-p 1.0; benchmark scoring rules are followed, with model-based judging using deepseek-v4-flash [5].Source: §A.6 · p. 15

B.1. Response-wise scaling and geometric interpretation

  • The response-wise scale applies the same diagonal transform to the predictive state and both prototypes; the resulting relative-affinity bound is zero for identical normalized prototypes.The scaling uses floored response-wise population standard deviations, and the predictive state is not subtracted in the cosine.Source: §B.1 · p. 15

B.2. Predictive feature and gradient interpretation

  • The uncertainty-scaled intermediate-layer feature is gradient-inspired rather than an exact parameter-gradient identity, because the selected state and final probabilities come from different actor stages.The selected intermediate layer is fixed.Source: §B.2 · p. 16

C. Properties of Outcome-conditioned Calibration

  • The two-reference estimator requires positive reference mass on both advantage sides; under its stated conditions, the diagonal calibration matrix remains positive definite.The conditions include 0 ≤ η ≤ 1, r_min ≥ 0, and ϵ_D > 0.Source: C. Properties of Outcome-conditioned… · p. 16

C.1. Optimal coefficients and normalized contrast

The section characterizes the calibration direction through a regularized contrast of positive- and negative-side reference means. It also distinguishes this fixed-statistic characterization from claims about downstream reward or generalization outcomes.

C.1. Optimal coefficients and normalized contrast

  • The regularized contrast criterion is uniquely maximized by b* = D^-1Δμ, while nonzero scalar multiples of that direction maximize its normalized form when Δμ is nonzero.The result is stated for a positive-definite D and Δμ = μ+ − μ−.Source: §C.1 · p. 16
  • The denominator is a regularized diagonal variation surrogate, not generally the full empirical variance, because cross-coordinate covariances are not modeled.The result requires neither a Gaussian assumption nor a fitted probability model.Source: §C.1 · p. 16

C.2. Equivalence to a distance contrast

  • Under the fixed-statistic comparison, calibration rescales nonzero outcome mean differences by the inverse diagonal scale, and equal-reference coordinates contribute zero directly.With D = I, the score is an ordinary squared-distance contrast.Source: §C.2 · p. 16

C.3. Direct contribution of the visual coordinate

  • The visual coordinate contributes to the fixed-statistic diagonal criterion, but this contribution does not measure image-unique information, and refitting after RVE removal can change hidden references.The contribution does not remove redundancy with hidden coordinates.Source: §C.3 · p. 17

D. Soft Reference Refinement

  • Soft assignment and reference updating are separate variational subproblems; the estimator performs one refinement rather than iterating to convergence.The variational characterization does not identify the full procedure as EM or coordinate ascent on a common objective.Source: D. Soft Reference Refinement · p. 17

E.1. Normalization, bounds, and signs

The section explains how globally normalized positive coefficients preserve advantage signs while changing token contributions within sequence-level clipping. It separates response-level weighting from within-response allocation and limits what the detached surrogate implies.

E.1. Normalization, bounds, and signs

  • With raw coefficients in [0.8, 1.2], global token-mean normalization gives final bounds [2/3, 3/2], but does not preserve response means, either side’s advantage mass, or policy-gradient norm.Zero-advantage tokens affect the normalization factor but contribute no direct policy gradient.Source: §E.1 · p. 17

E.2. Token-specific gradients under sequence clipping

  • Sequence clipping selects active response-level surrogate terms, while each retained token-gradient term receives its own coefficient; setting all coefficients to one recovers the corresponding GSPO surrogate gradient.All valid tokens in a response share the numerical ratio and advantage sign.Source: §E.2 · p. 18

E.3. Separating response weighting from token allocation

  • The update separates response-level weighting from a within-response term that depends on the correspondence between token weights and gradients; replacing weights by their response mean removes that allocation term.The detached surrogate does not establish an unbiased gradient of expected reward, monotonic improvement, or independent per-token probability changes from advantage-sign preservation.Source: §E.3 · p. 18

F.1. Complete algorithm and implementation details

The implementation computes detached credit statistics and normalized token weights from current rollouts, then applies them in a GSPO-token update. The described setup uses fixed actor parameters during the statistics passes and specifies training recipes and checkpoint scope.

F.1. Complete algorithm and implementation details

  • Reference moments, margin scales, and normalization aggregate across actor-mini-batch micro-batches and data-parallel ranks, while response-wise RVE scales are computed before those aggregations.The method uses scalar sequence rewards, and advantage signs indicate relative sequence rewards within rollout groups rather than token correctness.Source: §F.1 · pp. 18–19
  • The actor remains fixed through the credit-statistics passes, which recompute response-wise features and aggregate reference moments, calibration scales, refined references, and normalized weights before one optimizer update.The algorithm requires nonempty valid tokens and positive reference mass on both advantage sides.Source: §F.1 · p. 19
  • For one refinement, the implementation uses four no-gradient statistics forwards and one training forward per micro-batch; its O(NH) calibration arithmetic excludes model forwards, activation storage, and communication and is not a runtime or speedup claim.Standard inference does not require the credit estimator.Source: §F.1 · p. 19

F.2. Training recipes and checkpoints

  • The chart-centric experiment shares a Chart SFT initialization at optimizer step 782 and uses the checkpoint after 72 rollout steps; compared methods share training data and rollout configuration unless otherwise noted.The training-recipe table summarizes main comparisons, optimizer extensions, and the chart-centric experiment.Source: §F.2 · p. 19, Table 9 · p. 19

F.2.1. Chart-centric training

Chart-centric training initializes both RL methods from a Qwen3-VL-4B-Instruct checkpoint fine-tuned on ChartVerse examples. The shared RL set contains 18,280 questions selected using verifier-scored responses.

F.2.1. Chart-centric training

  • The Base checkpoint fine-tunes Qwen3-VL-4B-Instruct for one epoch on 100,000 ChartVerse-SFT-600K examples and initializes both RL methods.The 512 validation examples and their images are held out from SFT and RL training.Source: §F.2.1 · pp. 19–20
  • Table 10 is captioned as listing defaults for the supplied Qwen3.5-9B recipe.Source: Table 10 · p. 20
  • The shared RL dataset contains 18,280 ChartVerse-RL-40K training questions, retained after filtering for questions with one to seven correct sampled responses and deduplication.CompassVerifier-7B judges eight responses per question sampled from the SFT model at temperature 1.Source: §F.2.1 · p. 20
Loading 2610.32324v1…