Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Michael Jerge; Joseph Pelczar; Justin Downes

Source-linked AI summary

Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Michael Jerge, Joseph Pelczar, Justin Downes

arXiv:2608.21595v1Aug 21, 2026cs.CV

TL;DR

Existing rollout diversification for VLM RLVR uses decoding stochasticity or pixel perturbations, leaving open whether perturbing latent representations is more appropriate. NC-GRPO injects calibrated Gaussian noise at the last hidden layer during prompt encoding and uses mixed-group contrasts for training. On Qwen2.5-VL-7B trained on Geometry3K, it improves OOD reasoning across five held-out benchmarks while also improving in-domain accuracy and hallucination robustness, with independent stochastic diversity identified as the active ingredient.

Problem
Existing temperature and image-space diversification do not directly perturb the model’s internal problem representation, motivating latent-space rollout diversification.
Method
NC-GRPO injects scale-calibrated Gaussian noise into the last hidden layer during prompt encoding for half of each rollout group, leaving the objective, reward, decoding, and evaluation protocol unchanged.
Results
NC-GRPO significantly improves OOD mathematical reasoning over vanilla GRPO across five held-out benchmarks, while improving in-domain accuracy and hallucination robustness.
Takeaways & Limitations
Independent stochastic diversity, rather than noise budget or direction, is the active ingredient, and σ0=0.2 improves reasoning without measurable general-capability cost.
Takeaways & Limitations
The perturbation displaces only the departure state; sustained propagation into the KV cache is untested, and hallucination and general-capability evaluations are single-seed.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar $p \le 0.001$) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.

1. Introduction

NC-GRPO asks whether rollout diversification should perturb the model’s latent representation rather than decoding or pixels. It branches noisy rollouts from displaced hidden states while preserving the GRPO objective and reports broad reasoning, robustness, and mechanism gains.

Approach: NC-GRPO injects Gaussian noise into hidden representations during prompt encoding for half of each rollout group, then reinforces branches that remain successful.The group-normalized advantage turns differing responses to the displacement into policy-gradient signal.
Design advantages: Latent perturbation is modality-agnostic, computationally cheap, and precisely controllable in representation space.The paper notes applicability to text-only models, although validation here is limited to VLMs.
Results: Significant OOD reasoning gains over vanilla GRPO occurred on five multimodal mathematics benchmarks, alongside improved in-domain accuracy and hallucination robustness.The contribution summary reports pooled McNemar p=0.001–p<10^-4 across noise scales and significant POPE improvement at p=0.007.
Mechanism: Mechanism probes identify independent stochastic diversity, rather than noise budget or direction, as the active ingredient.Matched antithetic and steered interventions are used to separate these possibilities.
Scale: A noise-scale study exposes a trade-off between reasoning specialization and general capability, including σ0=0.2 with reasoning gains and no measurable general-capability cost.This scale is presented as a regime rather than a universally optimal setting.

2. Background and Related Work

The paper situates NC-GRPO within GRPO-based RLVR and prior rollout-diversification methods. Its controlled comparison changes the perturbation site while retaining the established training scaffold.

GRPO and RLVR: GRPO generates multiple rollouts per prompt, normalizes their rewards within each group, and favors above-average outcomes.The method is applied to multimodal mathematical reasoning in VLMs.
Prior diversification: Prior VLM work primarily diversifies rollouts through temperature sampling, while NoisyRollout adds Gaussian image distortion to half of each group.NoisyRollout anneals the distortion and leaves the objective unchanged.
Scope: The rollout-generation change is complementary to improvements in the optimization objective and reward model.
Controlled comparison: NC-GRPO inherits NoisyRollout’s two-pass structure, 50/50 split, schedule, and training protocol while changing only the perturbation site from pixels to hidden states.This makes the image-versus-latent comparison a controlled pipeline experiment.
Positioning: Related stochastic methods use forward-pass or model diversity for uncertainty, whereas NC-GRPO uses representation-level diversity within one model for training.The paper distinguishes it from regularization, exploration, flat-minima optimization, and embedding-noise methods.

3. Method: NC-GRPO

NC-GRPO creates calibrated branch-point perturbations during prefill and uses mixed-group reward contrasts to train robustness to representational displacement. Ablations distinguish independent stochastic spread from antithetic cancellation or fixed steering.

3.1. Noise-induced branching: NC-GRPO retains NoisyRollout’s annealing schedule, using stronger exploration early and reducing noise toward zero as training converges.
3.1. Noise-induced branching: Noise is calibrated so E∥˜h_t−h_t∥^2=σ_k^2∥h_t∥^2, with relative magnitude concentrating around σ_k∥h_t∥ as hidden width grows.The calibration fixes Euclidean perturbation size across widths, not behavioral equivalence across models.
3.1. Noise-induced branching: The hidden-state hook perturbs the final prefill representation, shifts the first-token distribution, and leaves subsequent decoding noise-free.The KV cache remains unperturbed, so the intervention displaces only the branch point rather than sustaining noise during generation.
3.2. Mixed-group advantage decomposition: Mixed-group normalization adds a contrast term that reinforces the clean or noisy half according to its expected reward relative to the other half.The contrast vanishes when expected reward is invariant to the perturbation.
3.3. Agreement view: This contrast trains policies whose reward is insensitive to displacement at the branch point, which the paper connects to a restricted form of representational agreement and flatness.The agreement view distinguishes stable states, whose noisy branches remain successful, from fragile states that diverge under perturbation.
3.4. Perturbation geometry: Pixel noise is encoder-filtered and sustained through attended keys and values, whereas latent noise is isotropic, exactly calibrated, and applied only at the branch point.This geometric difference predicts stronger perception effects for pixel noise and stronger hallucination or in-domain effects for latent noise.
3.5. Mechanism probes: Antithetic pairing cancels the first-order reward fluctuation while retaining the even curvature component, testing whether independent reward spread carries the training signal.If independent stochastic diversity is active, cancellation should hurt rather than help.
3.5. Mechanism probes: SPSA steering converts rank-one random perturbations into a learned direction, testing whether a beneficial fixed direction outperforms undirected noise.The probe distinguishes directional structure from the value of independent randomness.

4. Experiments

Across matched evaluations, NC-GRPO improves held-out reasoning and hallucination robustness without improving training reward, with independent latent noise emerging as the active ingredient and noise scale trading specialization against general capability.

Seed variance: Over two seeds, NC-GRPO raises OOD average by 1.0 and 0.7 points relative to vanilla, while pooled per-question testing gives p=2.7 × 10−4.The in-domain effect is interpreted as a paired per-seed delta because Geo3K varies more across seeds and ranges overlap.
Training dynamics: Training reward matches vanilla step-for-step, yet NC-GRPO leads on four of five OOD benchmarks, in-domain accuracy, and POPE by step 60.The reported training-reward mean difference is −0.003 ± 0.005, with ± denoting standard deviation across steps.
Mechanism analysis: Noise direction does not recover the benefit: SPSA steering follows a stable learned direction but yields no OOD improvement over antithetic noise and degrades in-domain accuracy.The probes therefore support independent stochastic diversity rather than direction or magnitude as the active ingredient.
Image-space comparison: Image noise achieves a larger OOD average of 56.2 versus 53.5 but does not improve in-domain accuracy, while latent noise is stronger on several benchmarks and hallucination robustness.Image noise’s OOD margin is carried almost entirely by WeMath and MathVerse; latent noise beats it on POPE with p=0.007.
Specialization dial: Noise scale acts as a specialization dial: σ0=0.5 gives the strongest math-OOD point but a significant pooled general-capability regression, while σ0=0.2 is indistinguishable from vanilla on that battery.The σ0=0.4 OOD dip is driven entirely by WeMath, whose response is especially sensitive to rollout-distribution changes.
Scales and generations: NC-GRPO improves pooled OOD by +1.3pp on Qwen3-VL-8B-Thinking but produces a null pooled OOD result on Qwen2.5-VL-3B, 45.0 versus 45.0 with p=0.81.The 3B result is single-seed and uses σ0=0.2 transferred from 7B, so it supports only a narrow capacity-bound reading.

5. Limitations

The paper’s evidence is constrained by limited seeds and model coverage, incomplete mechanism controls, and a strict scoring protocol.

Most configurations use single-run results, so reported spreads are indicators rather than seed distributions.Only three load-bearing configurations have two seeds; paired tests quantify model differences, not seed variance.
Uncorrected p-values span approximately 15 tests, although headline results survive Bonferroni correction.Values from 0.006–0.05 are treated as suggestive, and the antithetic result is confirmatory only when pooled across seeds.
The in-domain claim relies on paired per-seed deltas, while hallucination and general-capability batteries are single-seed.Geo3K seed spread exceeds the reported effect.
The intervention shifts only the departure state and first-token distribution; sustained or propagating perturbations remain untested.The paper also lacks a matched first-token temperature control, and the antithetic baseline-cancellation interpretation remains open.
Rule-based scoring may deflate free-form benchmark performance, limiting direct comparison with results using other scoring protocols.

A. Proofs

The proofs formalize antithetic cancellation, calibrated perturbation moments, and the induced geometry of pixel noise. Together, they establish which stochastic and geometric terms the method retains or removes.

Perturbation calibration: The perturbation h̃ = h + aϵ is calibrated through E∥ϵ∥2 = d, giving E∥h̃−h∥2 = σ2∥h∥2.
Advantage decomposition: The contrast term has zero group mean and vanishes exactly when expected reward is invariant to perturbation.
Noise-smoothed reward: The noise-smoothed reward expansion uses the Hessian trace, with mean curvature represented by the mean eigenvalue of ∇2φ.
Noise-smoothed reward: The Taylor remainder is bounded by O(Cσ3∥h∥3), which becomes O(σ3) uniformly in dimension after norm scaling.
Lemma 1: Antithetic cancellation: Antithetic pairing preserves each rollout’s marginal distribution while replacing noisy-half means with pair means.
Lemma 1: Antithetic cancellation: The pair mean removes the odd reward-response component, including the first-order term, while retaining the even curvature component.The removed first-order contribution is proportional to ∇φ(h)⊤ϵ; the retained curvature contribution is O(σ2).
Lemma 1: Antithetic cancellation: For small σ, antithetic pairing removes the dominant noisy-half mean variance because odd-component variance is O(σ2), whereas even-component variance is O(σ4).
Induced geometry of pixel noise: Pixel noise induces Δh = Jδ + O(∥δ∥2), producing a first-order Gaussian perturbation with covariance s2JJ⊤.

B. Workflow, Training-Step Algorithm, and Additional Plots

NC-GRPO creates clean and latent-noisy rollout branches, normalizes their rewards jointly, and trains with a noise-free policy-gradient pass. The workflow compares both noise scales and tracks training, validation, and OOD behavior across methods.

Workflow: NC-GRPO pre-fills each prompt cleanly and with scale-calibrated Gaussian noise, then decodes noisy branches from a clean cache without decode-time noise.The perturbation shifts the final prompt-position logits before generation.
Additional Plots: At σ0 ∈ {0.2, 0.5}, Figure 3 compares NC-GRPO with vanilla GRPO using accuracy gains in percentage points across five held-out benchmarks.The shaded column reports the OOD average, while asterisks mark exact McNemar p<0.05 on paired outcomes.
Training-Step Algorithm: The training step samples n clean rollouts and n noisy rollouts, perturbing last-layer prefill states independently for the noisy half.The noisy branches use perturbed first-token logits, while the remainder is decoded from the clean key-value cache.
Training-Step Algorithm: Rewards are normalized over the joint 2n-rollout group before the parameter update, allowing clean–noisy disagreement to contribute to the learning signal.The normalized advantage is (ri−r̄)/(std(r)+δ), and the update uses a noise-free forward pass.
Additional Plots: Figure 4 plots Geometry3K training reward and validation accuracy for vanilla GRPO, NoisyRollout, and NC-GRPO, whose in-domain curves are nearly indistinguishable.The methods separate more clearly OOD than in-domain.