PaperlayerView exact revision on arXiv

Source-linked AI summary

Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning

Md Tanvirul Alam, Nidhi Rastogi

arXiv:2510.27044v2cs.LG

TL;DR

It remains unclear whether RLVR fosters genuine mathematical reasoning, particularly when benchmarks make reasoning correctness difficult to verify. The paper tests RLVR on uniquely solvable Activity Scheduling and LIS tasks, finding that answer-level gains can arise without stronger underlying reasoning.

  • Problem

    Many benchmarks make reasoning correctness difficult to verify, leaving unclear whether improvements reflect mathematical competence or superficial pattern matching.

    Source: §1 · p. 1

  • Method

    The study tests RLVR on Activity Scheduling and LIS using instances with unique optimal sequences and evaluates answer accuracy and sequence fidelity.

    Source: §1 · p. 1

  • Results

    RLVR improves answer-level generalization on both tasks, but reasoning gains appear only in Activity Scheduling; LIS gains stem from superficial heuristics or formatting strategies.

    Source: §3.1 · p. 4

  • Takeaways & Limitations

    These findings highlight the need for benchmarks and diagnostics that distinguish genuine reasoning from shortcut exploitation.

    Source: §1 · p. 2, §4 · p. 6

    The controlled study uses two tasks and one base model, so its conclusions may not generalize across tasks, model families, or scales.

    Source: §4 · p. 6

Abstract

from arXiv · show

Mathematical reasoning is a central challenge for large language models (LLMs), requiring not only correct answers but also faithful reasoning processes. Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising approach for enhancing such capabilities; however, its ability to foster genuine reasoning remains unclear. We investigate RLVR on two combinatorial problems with fully verifiable solutions: \emph{Activity Scheduling} and the \emph{Longest Increasing Subsequence}, using carefully curated datasets with unique optima. Across multiple reward designs, we find that RLVR improves evaluation metrics but often by reinforcing superficial heuristics rather than acquiring new reasoning strategies. These findings highlight the limits of RLVR generalization, emphasizing the importance of benchmarks that disentangle genuine mathematical reasoning from shortcut exploitation and provide faithful measures of progress. Code available at https://github.com/xashru/rlvr-seq-generalization.

1 Introduction

Although RLVR has produced gains on reasoning tasks, it remains unclear whether it builds genuine mathematical reasoning or stabilizes existing abilities. This study uses uniquely solvable Activity Scheduling and LIS tasks to examine whether gains reflect reasoning or shortcuts, and what benchmarks can measure.

  • Prior work finds RLVR can improve accuracy while reducing exploration, reweighting existing solutions, and stabilizing rather than inducing new reasoning strategies.Standard Pass@K can over-credit lucky completions, while stricter CoT-Pass@K is harder to scale.Source: §1 · p. 1
  • Benchmarks with difficult-to-verify reasoning leave unclear whether apparent gains reflect mathematical competence or superficial pattern matching.The study focuses on Activity Scheduling and LIS, where solutions can be verified and each instance has a unique optimum.Source: §1 · p. 1
  • The study uses verifiable Activity Scheduling and LIS instances with unique optimal sequences to measure answer accuracy, sequence fidelity, and structural behaviors such as sorting.Activity Scheduling has a unique greedy optimum; LIS can be solved with dynamic programming.Source: §1 · p. 1
  • The findings aim to distinguish gains from heuristic shortcuts versus genuine reasoning and inform mathematical benchmark design.Source: §1 · p. 2

2 Experimental Setup

The experiments use unique-optimum Activity Scheduling and LIS datasets, compare rewards for answers, formats, exact or partial sequences, and sorting, then evaluate answer and sequence accuracy using Pass@k and self-consistency.

2.1 Tasks

  • Each task has a unique canonical optimal sequence: Activity Scheduling uses earliest-finish-time greedy selection, while LIS uses dynamic programming for uniqueness and patience sorting for reconstruction.The dataset has 2,000 instances per task, half with hints; train and test use disjoint length ranges, leaving 462 Activity and 428 LIS test cases.Source: §2.1 · p. 2

2.2 Reward Functions

  • Rewards score answer correctness, add a small LIS formatting bonus, give exact or prefix-based sequence credit, and for Activity optionally reward matching the extracted sorted order.The formatting bonus mixes with answer correctness at λ = 0.1; Prefix-IDs gives partial credit for the longest common prefix and applies a small penalty for incorrect predicted length.Source: §2.2 · pp. 2–3

2.3 Training & Evaluation

  • Qwen2.5-7B-Instruct is fine-tuned with GRPO, and evaluation measures answer and ID-sequence accuracy under Pass@k and self-consistency across 256 samples per instance.Training uses 20 epochs, 120 updates, and a learning rate of 10^-6; metrics are averaged over the test set, whose unique ground truths make scoring unambiguous.Source: §2.3 · p. 3

3 Results & Analysis

RLVR improves answer-level performance on both tasks, but sequence-level gains differ: they appear on Activity Scheduling, while LIS improvements remain weak under exact sequence evaluation. Sequence-aware rewards improve sequence fidelity, yet sorting rewards and heuristic analyses expose limits on what these gains establish about reasoning.

3.1 Training with Exact Answer Reward

  • On Activity Scheduling, RLVR reaches Pass@256ids ≈0.64 versus 0.14 for Base, while on LIS both RLVR policies remain weak in exact-sequence Pass@k and SC.On Activity, RLVR also has SCids ≈0.34 versus 0.004 for Base; on LIS, answer-level gains do not translate into stronger exact-sequence evaluation.Source: §3.1 · p. 4

3.2 Training with Sequence Rewards

  • At k=256 on Activity, rids,exa/rids,pre raise SCids from 0.34 to 0.72/0.71, while on LIS they raise SCids from ≈0.08 to 0.42/0.40.On LIS, rids,pre has higher Accans while rids,exa has higher Accids; both also improve answer- and sequence-level consistency over Base and rans.Source: §3.2 · pp. 4–5

3.3 Sorting Performance on Activity Task

  • Sequence rewards raise exact sorting from 0.17%/0.43% to 2.01%/1.85% and mean LCS from 0.248/0.290 to 0.517/0.444, but exact sorting remains about 2%.Despite high sequence correctness on Activity, the sorting preface does not reliably drive the final schedule.Source: §3.3 · p. 5

3.4 Training with Sorting-Match Reward

  • Sorting-match reward alone collapses Accans and Accids to nearly 0%, and equal-weight combination with rans and rids restores them without improving test sorting.A 10-update rsort prefix can recover under rans, but 20 or 30 updates hinder recovery; after 30, Accans does not improve even on training data.Source: §3.4 · pp. 5–6, Figure 5 · p. 6

3.5 Heuristics Analysis for LIS

  • A Random Forest predicts RLVR-trained LIS outputs well from human-interpretable features, unlike Base, suggesting systematic task-aligned heuristics can boost answer performance without stronger reasoning.Features span input scale, order and run structure, simple LIS heuristics, and patience-sorting tails; train/test splits are by problem instance.Source: §3.5 · p. 6

4 Limitations & Future Work

The study’s controlled task and model setup limits how broadly its conclusions can be generalized, while future work could extend the scope and probe RLVR-induced behavior mechanistically.

4 Limitations & Future Work

  • The study covers two verifiable tasks with Qwen2.5-7B-Instruct, so its conclusions may not generalize across tasks, model families, or scales.The authors provide additional Llama-3.1-8B results in Appendix F as a partial response to this scope boundary.Source: §4 · p. 6
  • Future work could broaden the models and problem domains studied and use mechanistic interpretability to probe RLVR-induced behaviors.The authors say such analysis may clarify whether improvements reflect task learning or superficial strategies that exploit evaluation metrics.Source: §4 · p. 6

A.1 Activity Scheduling

Activity Scheduling instances use deterministic interval ordering and are retained only when dynamic programming confirms a unique optimum matching the greedy schedule.

A.1 Activity Scheduling

  • Instances use unweighted half-open intervals, where touching endpoints are compatible, and sorting ties are broken by finish time, start time, then ID.Times are sampled on an integer-minute grid; experiments use 5–16 activities, start bounds up to 9×60, and durations of 10–120 minutes.Source: §A.1 · p. 9
  • Dynamic programming counts optimal schedules alongside their maximum cardinality, identifying a unique optimum when the final count is 1.The recurrence compares including activity i after its predecessor with excluding it; tied branches add their optimal-schedule counts.Source: §A.1 · p. 9
  • Each candidate is accepted only if the earliest-finish-time greedy schedule matches the unique optimal set, which is reported in canonical finish-time order.The released target contains the ordered activity IDs and the optimal schedule size.Source: §A.1 · p. 9
  • Preprocessing and predecessor lookup take O(m log m), while dynamic programming, backtracking, and greedy verification each take O(m).Source: §A.1 · p. 9

A.2 Longest Increasing Subsequence (Unique Optimum)

The appendix constructs unique LIS targets and describes how free-form responses are converted into candidate ID lists and evaluated against canonical order.

A.2 Longest Increasing Subsequence (Unique Optimum)

  • LIS instances are sampled on an integer grid and retained only when the index-sequence optimum is unique and has length at least 2.Experiments typically sample lengths of 5–16 and values from 1 to 1000; duplicate values are allowed, but the subsequence must be strictly increasing.Source: §A.2 · pp. 9–10
  • An O(n2) dynamic program counts LIS endings by index to test uniqueness, then patience sorting reconstructs the unique index sequence.The released answer is that sequence in increasing row order and its length; reconstruction takes O(n log n) after the uniqueness check.Source: §A.2 · pp. 10–11

B Extracting sorted ID lists from free-form responses

  • For Activity Scheduling replies, the target order is IDs sorted by finish time, with smaller IDs breaking ties, and the method extracts candidate lists from free-form text.Candidate sources include brace-delimited ID blocks, “ID k” tokens, and comma-separated integer runs; sorted-block candidates are reserved for exact-sorting checks.Source: B Extracting sorted ID lists from… · pp. 11–13
  • A response counts as exactly sorted only when a complete normalized permutation of all IDs matches the target order.Invalid IDs are filtered out, repeated IDs are dropped after their first occurrence, and empty candidates are discarded.Source: B Extracting sorted ID lists from… · p. 13
  • Candidate extraction and subsequent scoring are linear in the response length and candidate-list lengths, respectively.Source: B Extracting sorted ID lists from… · p. 13

C Contiguous LCS (longest common substring) metric

The metric measures the longest contiguous block shared by a candidate list and the ground-truth order, then reports its normalized score and location.

C Contiguous LCS (longest common substring) metric

  • The contiguous LCS is the longest block in a candidate sequence that appears contiguously in the ground-truth order.Because IDs in the ground-truth order are unique, this is the standard longest common substring.Source: C Contiguous LCS (longest common… · p. 13
  • The method selects the best match among extracted candidates and records whether its matching block touches the candidate’s start, end, both boundaries, or neither.These anchors distinguish leading or trailing matches from a mid-sequence fragment.Source: C Contiguous LCS (longest common… · p. 13
  • It reports mean LCS-fraction over instances with a nonempty match and separately reports coverage, the percentage with any nonempty match.Exact-sorting accuracy and LCS are shown in Fig. 8.Source: C Contiguous LCS (longest common… · p. 13

D LIS Task Response Length and Entropy

On LIS, answer-only reward sharply reduces response length and entropy, while the format reward preserves both more closely to the base model. Despite these different generation styles, the trained policies converge to similar accuracy and may rely on comparable heuristics.

D LIS Task Response Length and Entropy

E Random-Forest Regression for LIS: Features, Protocol, and Training

The analysis regresses the model’s numeric LIS answer on interpretable features computed only from the input sequence. It pools stochastic runs, holds out problem instances, and uses feature importance to select a compact predictor.

E Random-Forest Regression for LIS: Features, Protocol, and Training

  • The regression target is the model’s emitted numeric answer, predicted from input-only features without using ground-truth labels as features.The analysis regresses y = pred_lis_len on features computed from the input values.Source: E Random-Forest Regression for LIS:… · p. 14
  • All stochastic runs for an instance are pooled, and a group hold-out split keeps each instance’s replicates together to prevent leakage.The split holds out approximately 25% of instances using sample_idx as the group key.Source: E Random-Forest Regression for LIS:… · p. 14
  • Input features capture sequence scale and dispersion, adjacent and pairwise order, and contiguous-run and local-extrema structure.The feature set also includes record highs and lows and counts of monotone runs.Source: E Random-Forest Regression for LIS:… · p. 14
  • Additional features encode greedy and beam-limited LIS heuristics, limited backtracking, and patience-sorting tails, alongside a random-permutation reference baseline.Patience-tail descriptors include their mean, standard deviation, interquartile range, and slope.Source: E Random-Forest Regression for LIS:… · pp. 14–15
  • The analysis uses a Random-Forest regressor with 800 trees and drops rows with missing feature values after replacing infinities with NaN.The reported settings include min_samples_leaf=2, max_features=sqrt, and random_state=42.Source: E Random-Forest Regression for LIS:… · p. 15
  • Held-out performance is reported with R2 and MAE, and the smallest feature-importance-ranked K within the specified tolerances is selected.The tolerances are ∆R2 = 0.01 and ∆MAE = 0.02 relative to the full model.Source: E Random-Forest Regression for LIS:… · p. 15

F Llama Model Performance

With Llama-3.1-8B, overall accuracy is higher on both tasks, while the relative trends between base and RLVR variants mirror those for Qwen. Across Activity and LIS, hints yield no significant performance differences.

F Llama Model Performance

  • Llama-3.1-8B attains higher overall accuracy on Activity and LIS, while base-versus-RLVR trends closely mirror those observed with Qwen.The comparison uses SC@256.Source: F Llama Model Performance · p. 15

G Evaluation of Hinted vs. Unhinted Prompts

Related papers

Loading 2510.27044v2…