PaperlayerView exact revision on arXiv

Source-linked AI summary

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Longyue Wang, Chenyang Lyu

arXiv:2601.11061v2cs.LGcs.CL

TL;DR

Spurious RLVR can improve LLM benchmark performance despite rewards unrelated to genuine reasoning, raising the question of how incorrect optimization unlocks memorized answers. Using mechanistic analyses and causal interventions, the paper identifies an Anchor-Adapter circuit whose middle-layer anchor triggers retrieval and whose later adapters accommodate it, while steering experiments show that contamination-driven performance can be amplified or suppressed.

  • Problem

    Spurious or incorrect rewards can produce benchmark gains, but how they trigger Qwen2.5 to replace general reasoning with memorization remains unclear.

  • Method

    The paper combines Path Patching, Logit Lens, JSD, NDE trajectory analysis, ablations, and MLP-key scaling to localize and causally test the shortcut circuit.

  • Results

    The study identifies L18-L20 as a Functional Anchor that injects a memorization trigger and L21+ as Structural Adapters that reorganize representations around it.

  • Takeaways & Limitations

    The findings provide a mechanistic roadmap for detecting contamination and using causal steering to suppress or amplify shortcut reliance in RLVR-tuned models.

  • Takeaways & Limitations

    The same mechanistic techniques could theoretically amplify reliance on specific datasets, requiring responsible use to support transparency and safety.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.

1. Introduction

The paper asks how spurious RLVR can unlock memorized answers despite incorrect rewards, and identifies a mechanistic shortcut that sacrifices prompt coherence for answer retrieval. It localizes this shortcut to an Anchor-Adapter circuit and validates its causal role through ablations and neuron interventions.

  • Spurious RLVR is defined as verifiable-reward training decoupled from genuine reasoning quality, including random, format-only, or intentionally incorrect rewards.
  • The Perplexity Paradox is a divergence in which answer-token perplexity falls while prompt perplexity rises, indicating shortcut formation with degraded prompt-side coherence.
  • Path Patching, Logit Lens, JSD, and NDE analysis identify L18-L20 as a Functional Anchor that injects a memorization trigger and L21+ as Structural Adapters that reorganize representations.
  • Scaling identified MLP keys bidirectionally amplifies or suppresses contamination effects, providing causal steering of shortcut reliance.
  • The paper causally localizes Qwen 2.5’s spurious gains to Functional Anchor and Structural Adapter components rather than newly stored knowledge.
  • Ablations eliminate performance gains on contaminated benchmarks while preserving general reasoning capabilities, supporting dataset-dependent shortcut circuits.

2. Related Work

Prior work shows that spurious RLVR can improve benchmark accuracy under random or incorrect rewards and links such gains to contamination and shortcut exploitation. Interpretability studies distinguish memorization from reasoning through neuron activations and steerable residual-stream directions, while this paper supplies a mechanistic account of the reconfiguration.

  • Qwen2.5 Math can improve substantially under random or incorrect reward signals, challenging the assumption that RLVR gains require ground-truth feedback.
  • Existing behavioral work argues that spurious RLVR gains primarily reflect data contamination and that RLVR often favors shortcut exploitation over robust reasoning exploration.
  • This study extends those observations by explaining how spurious signals physically reconfigure the model’s internal architecture.
  • Interpretability research detects memorization through specific neuron activations and identifies a steerable residual-stream direction mediating reasoning-memory interplay.

3. Preliminaries

The paper models Transformer components as causal memory and trajectory-analysis tools for locating shortcut activation. Path Patching identifies responsible computational edges, Logit Lens and JSD track intermediate commitments, and NDEs locate trajectory bifurcation.

  • The residual stream is modeled to locate memory activation through mechanistic interpretability.
  • MLPs are treated as key-value memories, while Path Patching swaps Leakage and Stable activations to identify edges responsible for memory-retrieval shortcuts.
  • Logit Lens projects intermediate hidden states into token distributions, and JSD measures their alignment with the final output.
  • High JSD indicates that substantial computation remains before the model commits to an answer.
  • NDEs approximate residual updates as a continuous dynamical system and use separation force to locate where memorized and reasoning trajectories bifurcate.

4. Empirical Study

Spurious RLVR produced contamination-linked gains on MATH-500 and MinervaMath, while leakage-free LiveMathBench showed no comparable improvement. Mechanistic analyses localized a Qwen-specific memorization circuit whose anchor emerges around L18–L20 and whose later layers adapt and amplify the shortcut signal.

  • Experimental setting: MATH-500 and MinervaMath showed the strongest performance shifts under spurious RLVR, whereas LiveMathBench served as a leakage-free control with only marginal improvement.LiveMathBench’s uncertain improvement, high perplexity, and partial-prompt behavior were consistent with format alignment or elicitation rather than answer memorization.
  • Perplexity analysis: On contaminated MATH-500, Qwen’s answer-only perplexity decreased while full-text perplexity increased, unlike leakage-free LiveMathBench and the control models.The divergence indicates lower uncertainty on specific answers alongside degraded language modeling on prompts.
  • Functional localization: Path Patching found higher accuracy recovery for Qwen MLPs than attention heads, with sustained recovery through L18–L20 before a sharp decline at L21.LLaMA showed neither comparable recovery nor layer-specific trends.
  • Structural analysis: Counterfactual JSD scores for Qwen Wup and Wgate peaked at L21–L22 and then declined, while Wdown remained elevated, identifying later structural adaptation.The pattern is consistent with Wup storing candidate features, Wgate selecting them, and Wdown maintaining the transformed output projection.
  • Dynamic analysis: Logit Lens showed L19 priming, L21–L22 transforming the representation, and L23 injecting the correct answer token “4” in a successful trajectory.When the anchor signal was weaker, the stream drifted toward the incorrect token “3” despite later MLP attempts to inject “4”.
  • Dynamic analysis: Leakage and stable trajectories bifurcated in middle layers: Separation Force peaked at L18–L20, later velocity differences increased, and linear-probe AUC-ROC peaked at L20.These results place the strongest directional steering at the anchor layers and subsequent signal amplification in later layers.

5. Ablation Study

Ablations show that the L18–20 Anchor is the integrity-critical trigger for contaminated-data retrieval, while L21–22 Adapters collaborate with it and clean-sample reasoning remains robust to manipulation.

  • Necessity Tests: Anchor Reset reduced contaminated accuracy from 98% to 86% on MATH-500 and from 88% to 72% on MinervaMath.Adapter Reset caused smaller drops of Δ=−6% and −10%, respectively.
  • Necessity Tests: Anchor Reset removed most RLVR-only gain but preserved pre-existing contamination, showing the shortcut is anchor-triggered yet partially distributed.On MATH500, the base model reached 78%, while randomizing the Anchor reduced accuracy to 34%.
  • Sufficiency Tests: Keep Only experiments showed neither group alone recovered RLVR performance, whereas Keep Both reached 46% on MinervaMath.Keep Only Adapter reached 32% and Keep Only Anchor 42%, supporting a collaborative Anchor–Adapter circuit.
  • Contaminated versus Clean Data: LiveMathBench accuracy remained stable at 70% across ablation conditions, indicating Anchor layers encode contaminated knowledge rather than general reasoning.Stable samples likewise remained nearly identical to the RLVR baseline across reset experiments.
  • Cross-Scale Validation: The same qualitative Anchor-centered pattern appeared in the Qwen2.5-Math-1.5B leakage subset, but the authors frame this as within-family evidence rather than universality.The smaller model showed a weaker leakage signal overall.

6. Mechanistic Intervention

The intervention identifies task-relevant MLP neurons and scales their key activations to control memorization retrieval. Steering is strongest at the Functional Anchor, targets contaminated circuits selectively, and can be gated for auditing.

  • Task-Relevant Neuron Identification: The method selects top-k neurons with k=10 using key activation magnitude and semantic overlap between projected neuron values and answer tokens.The relevance score uses λ to control semantic alignment.
  • Causal Steering: Multiplicative scaling of selected MLP key activations applies factor α uniformly across generation steps.The intervention directly manipulates neuron keys rather than retraining the model.
  • Dataset-Level Steering: Layer 18 showed maximal sensitivity: suppression changed accuracy by −3.8%, while amplification changed it by +4.4%.Layer 25 degraded uniformly under manipulation, consistent with a Structural Adapter role.
  • Dataset-Level Steering: Leakage-free datasets showed no systematic steering pattern, indicating the intervention targets contamination-dependent circuits rather than general reasoning pathways.This comparison is summarized in Figure 10 and its caption.
  • Trajectory Analysis: Under α=3.0, amplification advanced trigger-token top-1 emergence to Layer 22 and increased answer-token probability, whereas α=0.2 delayed triggering and reduced it.A failed baseline retrieval could also be converted into abrupt late-layer recovery by amplification.
  • Selective Auditing: The probe-gated audit activated on 61.7% of the MATH500 wrong→right subset and reduced accuracy from 100.0% to 85.16%.It fired on 0/30 AIME-2025 and 0/100 LiveMathBench examples, leaving those conditional accuracies unchanged.

7. Conclusion

The paper characterizes spurious RLVR memorization as a Perplexity Paradox supported by an Anchor–Adapter circuit and demonstrates bidirectional causal steering of contamination-driven performance.

  • Conclusion: The Perplexity Paradox is rising prompt perplexity alongside falling answer perplexity, indicating sacrificed language coherence during shortcut learning.The conclusion presents this divergence as a macroscopic fingerprint of spurious-RLVR memorization.
  • Conclusion: Path Patching, Logit Lens, JSD analysis, and NDEs localize L18–20 as the Functional Anchor and L21+ as Structural Adapters.The Anchor injects the memorization trigger, while later layers reorganize to accommodate it.
  • Conclusion: Ablations selectively degrade contaminated benchmarks while leaving clean benchmarks unaffected, supporting dataset-specific shortcuts rather than general reasoning improvements.This conclusion follows the paper’s manipulation-based comparison of contaminated and clean evaluations.
  • Conclusion: Scaling specific MLP keys enables bidirectional causal steering by amplifying or suppressing contamination-driven performance.The intervention provides active control over the identified shortcut circuit.

Impact Statement

The paper frames spurious RLVR as a risk to trustworthy evaluation because models may exploit contamination rather than develop robust reasoning. It combines mechanistic localization, causal intervention, and interpretability tools to identify and address these shortcuts.

  • The work links spurious RLVR to reward optimization through memorization rather than generalization, motivating stronger evaluation and reward design.The paper highlights data contamination and recommends evaluation benchmarks resistant to leakage.
  • Scaling selected MLP keys provides a route to causally steer contamination-driven behavior after training.The paper presents this as a possible approach for reducing performance based on contaminated data and out-of-distribution failure risk.
  • The same steering tools could theoretically amplify reliance on specific datasets, creating an ethical requirement for responsible use.The paper advocates using mechanistic interpretability to improve transparency and safety rather than mask architectural flaws.
  • Path Patching, Logit Lens, JSD, and Neural Differential Equations are used to analyze internal computation and locate shortcut-related transformations.These methods respectively attribute causal paths, track intermediate token emergence, quantify distributional divergence, and model hidden-state trajectories.

C. Partial Prompt Evaluation Details

Partial Prompt Evaluation tests whether models can complete exact answer strings from question text alone, exposing differences in contamination-related retrieval across models. Qwen models show retrieval-linked gains, whereas LLaMA and OLMo remain consistently weak.

  • Evaluation setup: Question-only prompting without Chain-of-Thought tests whether models possess pre-existing knowledge of ground-truth answer strings.The evaluation measures exact-string completion from the question text alone.
  • Cross-model comparison: Qwen3-8B shows high baseline accuracy and substantial post-RLVR gains across six benchmarks, unlike LLaMA-3.1-8B and OLMo-2-1124-7B.The cross-model comparison is presented before and after spurious RLVR with incorrect rewards.
  • Qwen2.5-Math-7B: Qwen2.5-Math-7B shows a dramatic increase in contaminated-dataset “Wrong→Right” cases for both ROUGE-L and exact completion accuracy after RLVR.The result is reported on MATH-500 and MinervaMath and is interpreted as retrieval of memorized strings.
  • Qwen3-8B: Qwen3-8B retrieves memorized answers as prompt ratios increase, but its pre- versus post-RLVR separation is less pronounced than Qwen2.5-Math-7B’s.The weaker separation is especially evident in the “Wrong→Right” group.
  • Control models: LLaMA-3.1-8B and OLMo-2-1124-7B remain at low ROUGE-L and completion accuracy without significant prompt-ratio trends, indicating no detected pre-existing contamination.OLMo has no samples categorized as “Wrong→Right” or “Stable” on LiveMathBench.

D. Complete Perplexity Analysis

Perplexity heatmaps distinguish contaminated from clean datasets and reveal model-specific divergence after spurious RLVR. Qwen shows answer-perplexity drops, while OLMo lacks the paradox and the associated mechanistic signatures.

  • Perplexity heatmaps: Answer perplexity drops on contaminated datasets such as MATH-500, AIME, and MinervaMath but remains high on clean LiveMathBench.The heatmap presents this contaminated-versus-clean dataset dichotomy across six benchmarks and four checkpoints.
  • Cross-model comparison: Only Qwen exhibits the characteristic answer-perplexity drop on contaminated datasets in the compact three-model comparison.The figure also annotates base accuracy and improvement percentages.
  • OLMo control: OLMo-2-1124-7B shows uniformly increasing full-text and answer-only perplexity with minimal accuracy changes, indicating no Perplexity Paradox.The analysis attributes this absence to failure to activate memorization shortcuts.
  • Path patching: OLMo lacks Qwen2.5-Math-7B’s middle-layer path-patching signal, while Qwen3-8B shows weaker contamination effects and no characteristic middle-layer peak.The recovery patterns for OLMo and Qwen3-8B resemble the control-model pattern rather than Qwen2.5-Math-7B’s.
  • MLP JSD: OLMo’s MLP-component JSD rises monotonically, whereas Qwen3-8B shows a peak-and-decline pattern at Layer 33 with sustained Wdown divergence.The Qwen3-8B pattern is presented as evidence of structural-adapter generalizability within the Qwen family.

G. Additional Results on Logit Lens

Additional Logit Lens analyses contrast late answer-token emergence in control models with a multi-stage memorization circuit in Qwen models. The Qwen pattern recurs across samples and across the Qwen architecture family.

  • Control models: LLaMA and OLMo show negligible answer-token probability through most layers, with emergence only in the final few layers.Neither control model exhibits the peak-valley-rise pattern or aggressive middle-layer MLP signal injection.
  • Qwen2.5-Math-7B: Additional Qwen2.5-Math cases show a precursor at L18–L20, a structural-adaptation valley, and aggressive correct-token injection at Layer 23.The sequence is reported across different samples.
  • Qwen3-8B: Qwen3-8B reproduces the peak-valley-rise activation pattern in successful cases, while failed cases show weakened precursor activation.The result supports the Functional Anchor mechanism as a systematic Qwen-family property on contaminated data.

H. Additional Results on Ablation Study

Ablation results show that contaminated samples depend primarily on the Functional Anchor, while structurally different LiveMathBench subsets exhibit reversed sensitivity to layer manipulation.

  • Study design: The ablation study compares overall accuracy trends and three-dataset heatmaps across different layer-reset conditions.Leakage samples are highly sensitive to Anchor/Adapter resetting, while Stable samples remain robust in the reported comprehensive results.
  • MinervaMath: Resetting Anchor layers L18–20 drops accuracy from 88% to 72%, compared with 88% to 78% after resetting Adapter layers.The larger Anchor-related decline identifies these layers as the primary causal driver for retrieving contaminated knowledge.
  • MinervaMath: Keeping only the Adapter layers yields 32% accuracy, below the Base Model’s 50%, showing dependence on the Anchor signal.Structural Adapters L21–22 cannot sustain the memorization circuit without the upstream trigger.
  • LiveMathBench: LiveMathBench reverses the contaminated-data pattern: its leakage subset is more stable under layer manipulation, whereas its stable subset is more sensitive.The passage attributes this inversion to the absence of actual contamination.

I. Additional Results on Mechanistic Intervention

Mechanistic intervention experiments identify layer-specific steering effects and illustrate how memorized answer pathways appear in case-level analyses. The included examples also show that generated solutions can produce conflicting answers for the same problems.

  • Mechanistic intervention: Amplifying factors at layer 18 improves accuracy, while zeroing factors at layers 22 and 23 degrades performance across the 28-layer intervention.These effects support distinct functional roles for early anchor features and later circuit components.
  • Logit-lens cases: Successful Qwen2.5-Math retrieval shows Anchor priming at L19, structural adaptation at L21–22, and correct-answer injection at L23.In a failure case, a weaker Anchor signal allows drift toward token “3” despite continued MLP attempts to inject “4”.
  • Cross-model signatures: Additional Qwen2.5-Math leakage cases reproduce strong priming at L18–L20, a transitional valley, and answer injection beginning at Layer 23.Qwen3-8B successful retrieval likewise shows a single peak-valley-rise trajectory, unlike oscillatory failed cases.
  • Steering effects: Amplification can unlock a dormant memorization pathway, producing a high-confidence answer plateau and abrupt answer injection at L26–L27.Partially memorized samples show gradual probability modulation, whereas failed-retrieval samples show binary pathway activation.
  • Case studies: The case studies contain conflicting outputs for identical or related mathematical prompts, including values 3 and 4 for the same inequality problem.The supplied examples include step-by-step solution fragments, code confirmations, and incorrect or inconsistent results.
Loading 2601.11061v2…