PaperlayerView exact revision on arXiv

Source-linked AI summary

CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards

Zhuofan Chen, Ziqian Jiao, Yikai Cui, Zhixin Cai, Jun Bai, Wenge Rong

arXiv:2609.07183v1cs.CLcs.LG

TL;DR

Existing RLVR selection criteria treat a problem’s training value as intrinsic, despite the possibility that its relationship to reasoning capacity differs across models. CRS scores problem engagement of 46 reasoning-sensitive heads identified by contrastive ablation in one frozen-base-model forward pass, without reward labels or rollouts. CRS-bottom selection improves over random by +2.0 pp on GSM8K, +1.6 pp on OlympiadBench, and +2.9 pp on Minerva Math; CRS-top’s gain is smaller.

  • Problem

    Existing RLVR selection criteria treat a problem’s training value as intrinsic, despite the possibility that its relationship to reasoning capacity differs across models.

    Source: §1 · p. 1

  • Method

    CRS scores problem engagement of 46 reasoning-sensitive heads identified by contrastive ablation in one frozen-base-model forward pass, without reward labels or rollouts.

    Source: §1 · p. 2

  • Results

    CRS-bottom selection improves over random by +2.0 pp on GSM8K, +1.6 pp on OlympiadBench, and +2.9 pp on Minerva Math; CRS-top’s gain is smaller.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Within the tested Qwen2.5-Math settings, selection value depends on pool diversity, model capacity, and evaluation regime rather than a static ranking of problem quality.

    Source: §6 · pp. 8–9

    Experiments cover mathematical data from Qwen2.5-Math at 7B and 1.5B; generalization to other architectures and verifiable-reward domains is untested, and CRS is a predictive rather than verified circuit signal.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria--difficulty filtering, hand-curation, reward-trajectory scoring--assess data value as an intrinsic property of problems, independent of the model that will learn from them. We introduce Circuit Reasoning Score (CRS), a selection signal derived from 46 reasoning-sensitive attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts. CRS runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces better training data: on Qwen2.5-Math-7B, the lowest-engagement decile improves over random selection on three medium-difficulty benchmarks (GSM8K +2.0 pp, OlympiadBench +1.6 pp, Minerva +2.9 pp), while the highest-engagement decile gains less and is indistinguishable from the middle decile. The advantage has boundary conditions: on a domain-curated pool no selection method separates from the others; at 1.5B scale the useful direction differs; and the lowest-reward training condition produces the strongest downstream generalization. Within the Qwen2.5-Math settings tested, RLVR data selection appears regime-dependent rather than reducible to a static ranking of problem quality.

1 Introduction

The introduction proposes CRS as a model-internal RLVR data-selection signal, challenging the assumption that higher reasoning-circuit engagement identifies better training problems. Its results make the useful selection direction conditional on model scale, pool diversity, and evaluation regime.

1 Introduction

  • CRS scores each problem from 46 reasoning-sensitive heads identified by contrastive ablation, using one frozen-base-model forward pass without reward labels or rollouts.The score is intended to reflect the relationship between a problem and the model’s reasoning state, unlike criteria that treat training value as problem-intrinsic.Source: §1 · pp. 1–2
  • Existing selection methods use difficulty, hand-curation, reward trajectories, or single-example criteria, treating training value as a property of the problem alone.CRS instead makes selection model-internal; within the tested settings, its usefulness varies with pool diversity, model capacity, and evaluation difficulty.Source: §1 · pp. 1–2
  • +2.0 pp on GSM8K, +1.6 pp on OlympiadBench, and +2.9 pp on Minerva Math: CRS-bottom beats random, while CRS-top is not separable from CRS-middle.The reported pattern runs against the intuitive hypothesis that stronger circuit engagement yields better training data.Source: §1 · p. 1
  • The advantage is conditional: no selection method separates on a domain-curated Olympiads pool, the useful direction differs at 1.5B, and the lowest-reward condition generalizes best.Across 58 GRPO runs on two model scales, the findings challenge the claim that higher circuit engagement consistently identifies better RLVR data.Source: §1 · pp. 1–2

2 Related Work

Related work covers verifier-based RLVR, data-selection approaches, and circuit analysis, then positions this paper as asking when a model-internal ranking helps, fails, or reverses. That framing makes data selection a regime-dependent matching problem rather than a static ranking.

2 Related Work

  • GRPO uses within-group reward variance for chain-of-thought training, and prompts with identical completion rewards provide no gradient signal, making prompt selection consequential.Prior work also reports RLVR output-distribution narrowing and gains under spurious rewards (Shao et al., 2025).Source: §2 · p. 2
  • Prior RLVR selection uses reward-trajectory alignment, difficulty filtering, or dataset scaling, while model-aware selection studies cited here use supervised losses rather than RLVR.The paper distinguishes CRS from curricula that adjust sampling by reward variance or environment difficulty, and reports CRS is not a difficulty proxy (R2 < 2%).Source: §2 · p. 2
  • Circuit studies identify task-specific heads and interpretable reasoning circuits; Wang et al. (2025a) applied circuit signals to SFT selection using attention-pattern variance.This paper targets RLVR, uses contrastive ablation to isolate reasoning-specific heads, and tests both engagement directions instead of assuming higher is better.Source: §2 · p. 2
  • The paper asks when model-internal rankings help, fail, or reverse, finding the answer depends on pool diversity, model capacity, and evaluation regime.This reframes RLVR selection from a static ranking problem to regime-dependent matching.Source: §2 · p. 2

3 Method

The method constructs CRS as a model-internal, per-problem signal from reasoning-sensitive head activations in a single forward pass, then residualizes it against input length. It also defines selection subsets and controls for evaluating the signal.

3.1 Reasoning-Head Identification

  • Contrastive ablation compares reasoning and trivial probes to identify reasoning-specific heads; the final set contains 46 stable heads spanning 21 layers.The authors call these heads “reasoning-sensitive” and explicitly do not claim a verified causal pathway.Source: §3.1 · p. 3

3.2 CRS Computation

  • CRS averages each selected head’s maximum output activation across tokens, capturing peak engagement rather than diluting it across boilerplate.A single forward pass records the head outputs for each problem.Source: §3.2 · p. 3

3.3 Length Residualization

  • Residualizing raw CRS against log input length reduces its length correlation to r = 0.013; top-decile overlap with the longest prompts falls to 0.057.Raw CRS correlates with input length at r = 0.568.Source: §3.3 · p. 3

3.4 Selection and Controls

  • The evaluation ranks DAPO-Math-17K problems into top, middle, and bottom CRS subsets and includes random, random-head, cross-pool, and cross-scale controls.The CRS subsets contain 1,718 problems each; the Olympiads cross-pool subsets contain 1,389 each.Source: §3.4 · p. 3

4 Experiments

Across the experiments, CRS-bottom improves over random selection on three medium-difficulty benchmarks at 7B, but the useful selection direction depends on the pool, model scale, and evaluation regime. Training reward does not predict downstream generalization in these settings.

4.1 Setup

  • The 7B results use shared GRPO settings, with top, middle, and bottom CRS deciles compared against random selection across five primary benchmarks.The CRS pipeline residualizes activation scores for length before forming deciles.Source: §4.1 · pp. 3–4, Figure 1 · p. 4

4.2 CRS Is Not a Monotonic Quality Score

  • CRS is not a monotonic quality score: CRS-bottom has the highest Tier 1 accuracy on three medium-difficulty benchmarks, while no condition dominates all benchmarks.CRS-top’s margin over Random is small on every benchmark.Source: §4.2 · p. 4, Figure 2 · p. 5

4.3 CRS-Bottom Improves Medium-Difficulty Reasoning

  • CRS-bottom beats Random by +2.0 pp on GSM8K, +2.9 pp on Minerva, and +1.6 pp on Olympiad-Bench.The advantage is positive on 13 of 15 seed-by-benchmark cells and yields a +2.15 pp composite.Source: §4.3 · pp. 4–5, Figure 3 · p. 5, Table 4 · p. 5

4.4 Ruling Out Alternative Explanations

  • Baseline comparisons indicate that CRS-bottom’s advantage is not reducible to reward-trajectory selection, arbitrary head signals, data volume, CRS magnitude, or difficulty filtering.Full-data training trails the 10% subsets on MATH-500 under the fixed training budget.Source: §4.4 · p. 6

4.5 Selection Effects Require Pool Diversity

  • On diverse DAPO-17K, CRS stratification produces up to 2.9 pp differences; on the domain-curated Olympiads pool, it produces none.The Olympiads subsets differ in problem identity but converge in outcome.Source: §4.5 · p. 6

4.6 The Useful Direction Differs at 1.5B

  • At 1.5B, CRS-top leads on MATH-500 (65.4), GSM8K (77.5), and OlympiadBench (31.4), reversing the useful direction seen at 7B.The authors interpret the reversal through GRPO reward variance and leave intermediate-scale verification for future work.Source: Table 7 · p. 7, §4.6 · p. 7, Figure 4 · p. 7

4.7 Training Reward Does Not Predict Generalization

  • CRS-bottom has the lowest reward (0.265) yet highest GSM8K (85.81) and Minerva (22.50); reward–accuracy correlation is −0.18 across runs and +0.03 across condition means.The authors advise against optimizing RLVR data selection for training reward because the lowest-reward condition generalizes best.Source: §4.7 · pp. 7–8, Figure 5 · p. 7

5 Discussion

CRS is not simply a difficulty ranking: its relationship to rollout success is weak, and CRS-bottom problems are nearly disjoint from the easiest problems. Selection effects vary with pool diversity, evaluation regime, and model capacity, so no single direction dominates.

5.1 CRS Is Not a Difficulty Proxy

  • Residualized CRS explains 1.57% of success-rate variance (ρ = 0.125), while CRS-bottom and SR-bottom overlap at 0.055, essentially random.The analysis uses rollout-estimated success rates for 16,478 problems; CRS-bottom’s advantage therefore cannot be attributed to selecting the easiest problems.Source: §5.1 · p. 8

5.2 Three Axes of Selection Sensitivity

  • Pool diversity, model capacity, and evaluation difficulty determine whether selection helps, hurts, or is irrelevant.CRS-bottom is most effective with a diverse pool, a capable model, and medium-difficulty targets; outside that regime, the signal vanishes or reverses.Source: §5.2 · p. 8
  • On diverse DAPO-17K, CRS-bottom outperforms Random by up to 2.9%; the same procedure has no detectable effect on the domain-narrow Olympiads pool.The discussion identifies pool diversity as a prerequisite for a selection effect.Source: §5.2 · p. 8
  • CRS-bottom helps on medium-difficulty benchmarks but provides no benefit on AIME, so the useful selection direction depends on the evaluation regime.The practical guidance is to match selection direction to the target.Source: §5.2 · p. 8
  • At 7B, low-CRS problems are more valuable; at 1.5B, they fall below the productive reward-variance threshold and high-CRS problems become more valuable.The discussion frames low-CRS problems as unfamiliar territory for the 7B model.Source: §5.2 · p. 8

6 Conclusion

Within the Qwen2.5-Math settings tested, the value of RLVR training problems is not an intrinsic property of the problems alone. CRS exposes regime-dependent selection effects, while its weak relationship with rollout success distinguishes it from a simple difficulty ranking.

6 Conclusion

  • Within tested Qwen2.5-Math settings, CRS exposes selection effects that vary with pool diversity, model scale, and training reward rather than providing a universal selector.The lowest-reward condition produced the strongest generalization; low-engagement data favored 7B, whereas high-engagement data favored 1.5B.Source: §6 · p. 8
  • CRS and rollout-estimated success rate rank problems in nearly independent orders, with ρ=0.125 and R2=1.57%.The median success rate shifts only from ∼0.15 to ∼0.30 across CRS deciles.Source: Figure 6 · p. 9
  • For a diverse pool and a 7B-scale model, CRS-bottom improved medium-difficulty reasoning in the reported runs; on curated pools or smaller models, the signal was undetectable or reversed.The passage cautions that these observations apply to comparable settings.Source: §6 · p. 9

Limitations

The evidence is limited to mathematical tasks and two Qwen2.5-Math model sizes, and the authors characterize CRS as a predictive signal rather than a verified reasoning circuit.

Limitations

  • All experiments use Qwen2.5-Math models at 7B and 1.5B on mathematical data; generalization to other architectures and verifiable-reward domains is untested.The scope limitation leaves those settings beyond the tested evidence.Source: Limitations · p. 9
  • The 7B analysis uses n=5 seeds and Tier 4 / 1.5B use n=3; each run sees 58.2% of its prompt subset, contributing to cross-seed variance.The authors report effect sizes rather than significance labels.Source: Limitations · p. 9
  • CRS is correlational, and zero-ablating its heads also harms the untrained base model; the authors present it as a selection-level predictive signal, not a verified reasoning circuit.The ablation result indicates a pathway already present in the base model rather than one built by GRPO.Source: Limitations · p. 9

Ethics Statement

The work uses publicly available mathematical benchmarks and open-weight models, with no human subjects, private data, or dual-use concerns.

Ethics Statement

  • The work uses publicly available mathematical benchmarks and open-weight models, with no human subjects, private data, or dual-use concerns.Source: Ethics Statement · p. 9

A.1 7B Main Grid: All 7 Benchmarks

Supplementary analyses characterize benchmark coverage, seed-level comparisons, item-level consistency, selector overlap, and the CRS ranking distribution. They also report Tier 2 and 1.5B comparisons and document the per-seed results.

A.1 7B Main Grid: All 7 Benchmarks

  • The seven-benchmark grid includes AMC 2023 and AIME 2025, which are excluded from the primary analysis because their small sample sizes yield high per-problem variance.Source: §A.1 · p. 12

A.2 Full Seed-Level Difference Table (Tier 1)

  • The six Tier 1 pairwise comparisons across seven benchmarks underlie the forest plot and summary.Source: §A.2 · p. 12

A.3 Tier 2 and 1.5B Seed-Level Differences

  • Tier 2 reports a −1.6 pp LIMR–Random difference on AIME 2024; at 1.5B, Bot–Top is −3.0 pp on MATH-500 and xmod–Random is +1.0 pp on GSM8K.Source: §A.3 · p. 12

B Per-Seed Results

  • Per-seed accuracy plots show cross-seed spread across five primary benchmarks, with tables providing the underlying accuracies and seed ordering.Source: B Per-Seed Results · p. 12

C Supplementary Analyses

  • The supplementary analyses cover item-level tests, subset overlap, CRS distribution, post-training ablation, hierarchical modeling, CRS-ranking ends, and difficulty-filtering construction.Source: C Supplementary Analyses · p. 12

C.1 McNemar Item-Level Tests

  • GSM8K shows item-level consistency: CRS-bottom solved 38 more problems than Random, while MATH-500 and OlympiadBench were not significant in a single seed pair.The GSM8K McNemar test gives p = 3.3 × 10−4.Source: §C.1 · p. 12

C.2 Subset Overlap Analysis

  • CRS, success-rate selection, and LIMR rank problems in nearly independent orders, with Jaccard similarities at or near chance.Source: §C.2 · p. 12

C.3 CRS Distribution

  • The 17,186-problem length-residualized CRS distribution is approximately Gaussian with mild right skew, and its bottom, middle, and top deciles occupy distinct regions.Source: §C.3 · p. 12

C.4 Post-Training Head Ablation

Post-training ablation removes the CRS-bottom advantage, while supplementary analyses quantify the condition effect and compare the two ends of the CRS ranking. Additional analyses describe difficulty-filtering construction and the identified heads’ distribution across layers.

C.4 Post-Training Head Ablation

  • Ablating the 46 identified heads removes CRS-bottom’s advantage on every benchmark; on GSM8K, base accuracy falls from 81.20% to 0.23%, though coarse ablation also disables mathematical generation broadly.Parse-failure rates rise to 73–88%, so the result establishes the heads’ importance to mathematical generation, not that they form a reasoning-specific circuit.Source: §C.4 · pp. 12, 15

C.5 Hierarchical Model of the Condition Effect

  • The condition effect is +0.256 log-odds, with 91.3% of its posterior above zero; the seed-level slope SD is 0.372, and removing any seed preserves the advantage’s sign.Source: §C.5 · p. 15

C.6 The Two Ends of the CRS Ranking

  • CRS-bottom’s advantage is positive in 13 of 15 seed-by-benchmark cells, while CRS-top’s weaker effect is not separable from CRS-middle at this seed count.The pattern is described as an asymmetric U.Source: §C.6 · p. 15

C.7 Difficulty-Filtering Subset Construction

  • Difficulty-filtering bands each contain 1,647 problems; constructing them required 65,912 generations, and SR=0 problems were excluded because their group reward provides no GRPO gradient.Source: §C.7 · p. 15

D Circuit Identification Details

  • The excluded L0H3 head shows extreme loss increases on both probe types, while selected heads are distributed across layers, concentrated near layer 0 and mid-to-late layers.Layers 26–27 contribute few selected heads.Source: D Circuit Identification Details · p. 15

E Sensitivity Analyses

Sensitivity analyses test the stability of the head set and CRS ranking to design choices and report training-reward trajectories. Implementation details compare CRS scoring costs with training and difficulty-filtering costs.

E Sensitivity Analyses

  • The head set and CRS ranking remain stable for contrastive weights w ∈ [0.25, 1.0], but switching off the contrastive term lowers head-set Jaccard similarity to 0.48.Across w ∈ [0.25, 1.0], Jaccard is at least 0.69 and Spearman correlation is at least 0.94.Source: E Sensitivity Analyses · pp. 15–16
  • Bottom deciles under n, √n, and log n residualization overlap at Jaccard 0.740–0.891, while log n gives the best fit and zero residual correlation with log-length.Source: E Sensitivity Analyses · p. 16, Table 21 · p. 16
  • Bootstrap intervals for Corr(∆r, ∆t) are wide because a few high-magnitude heads dominate the covariance, and omitting them pulls estimates toward zero.Source: E Sensitivity Analyses · p. 16, Table 22 · p. 16

G Implementation Details

  • CRS identification and scoring account for less than 0.1% of ∼18,100 GPU-hours; scoring 17,186 problems takes ∼10 minutes, compared with 65,912 generations for difficulty scores.Source: G Implementation Details · p. 17

F Training Curves

  • LIMR reaches ∼0.78 training reward while CRS-bottom plateaus at ∼0.27, yet CRS-bottom achieves stronger downstream performance on three medium-difficulty benchmarks.Source: F Training Curves · p. 16, Figure 11 · p. 18
Loading 2609.07183v1…