PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning

Sangmook Lee, Minbeom Kim, Jeonghye Kim, Dohyung Kim, Sojeong Rhee, Kyomin Jung

arXiv:2606.29985v1cs.CL

TL;DR

LLM math diversity metrics often measure differences in wording or notation rather than distinct solution strategies. This paper introduces approach-level diversity and finds that surface-level signals can diverge from strategic diversity, while approach-diverse candidates improve test-time scaling.

  • Problem

    Existing diversity metrics capture observable variation, leaving limited evidence about whether mathematically correct solutions use genuinely different strategies, which matters for candidate-based reasoning methods.

  • Method

    The paper defines approach-level diversity and evaluates conventional metrics against human judgments using a human-calibrated LLM judge framework.

  • Results

    Approach-diverse candidate sets improve test-time scaling, whereas conventional metrics and diversity-aware RLVR can preserve surface variation while approach-level diversity declines.

  • Takeaways & Limitations

    Surface-level diversity is not a reliable substitute for strategic diversity, motivating evaluation and training signals that better reflect distinct mathematical solution approaches.

  • Takeaways & Limitations

    The work does not provide a general method for preserving or inducing approach diversity during training.

Abstract

from arXiv · show

Diversity in LLM mathematical reasoning is critical for exploration, but common diversity metrics mostly capture surface-level variation rather than differences in how a problem is solved. We address this gap by introducing approach-level diversity: variation in strategies across correct solutions to the same problem. Using a human-calibrated LLM judge framework, we show that prior diversity measures are unreliable proxies for approach-level diversity, and this mismatch carries over to diversity-aware RLVR, where target metrics are preserved while approach-level diversity declines. Investigating when approach-level diversity helps and whether it can be directly induced, we find that approach-diverse candidate sets improve test-time scaling. However, optimizing an LLM judge diversity reward during training causes the policy to exploit judge-specific preferences rather than broaden its approaches, leaving direct optimization of approach-level diversity as an open problem. Together, our work introduces the notion of approach-level diversity and uncovers a systematic divergence between surface- and approach-level signals, marking a step toward LLMs that reason in genuinely diverse, human-like ways.

1 Introduction

The paper distinguishes approach-level diversity—variation in underlying solution strategies—from surface-level variation in wording or representation. It shows that conventional metrics and diversity-aware RLVR can preserve apparent diversity while failing to preserve, or even reducing, diversity in how problems are solved.

  • Measuring diversity: Existing diversity measures capture observable differences such as lexical overlap, embedding distance, or unique-equation ratios, but may miss fine-grained strategic differences.The paper evaluates these measures against human judgments using a scalable, human-calibrated LLM judge framework.
  • Defining the gap: Approach-level diversity measures variation in underlying strategies among correct solutions, beyond differences in wording, notation, or exposition.The paper contrasts this with lexical N-gram distance and semantic embedding cosine similarity.
  • Defining the gap: Conventional metrics can rate same-approach solutions as more diverse than solutions using different mathematical approaches.This real-world failure case demonstrates the mismatch between surface-level variation and approach-level diversity.
  • Training implications: Diversity-aware RLVR methods can preserve their optimized surface-diversity proxies while increasing variation within a narrower set of approaches.As a result, outputs may look more diverse even as approach-level diversity declines.
  • Practical benefits: Candidate sets containing distinct approaches yield larger gains under test-time scaling, indicating that approach-level diversity can improve inference-time performance.The paper therefore identifies a gap between measuring and optimizing approach-level diversity.

2 Background

Prior work shows that RL post-training can narrow mathematical-reasoning outputs, while existing diversity metrics measure differences within selected representations rather than validated differences in solution strategies. This motivates formalizing human-aligned approach-level diversity and evaluating lexical, semantic, symbolic, and reasoning-level metrics.

  • Diversity collapse in RL post-training: RL post-training can reduce generation diversity, narrowing task coverage and making outputs more homogeneous.These findings motivate diversity-aware RLVR methods intended to preserve diversity.
  • Measuring diversity in mathematical reasoning: Existing mathematical-reasoning metrics use lexical overlap, symbolic proxies, embedding similarity, or reasoning-path decompositions to define difference.Their alignment with human judgments of distinct solution strategies remains unresolved.
  • Measuring diversity in mathematical reasoning: Prior LLM-judge evaluations do not explicitly define the diversity concept they measure or validate judge decisions against human judgments.Human-aligned diversity has been studied in open-ended generation but remains less explored in mathematical reasoning.
  • Measuring diversity in mathematical reasoning: The evaluation covers five diversity metrics spanning lexical, semantic, symbolic, and reasoning-level signals.N-gram distance and Self-BLEU measure token overlap; Qwen3-Embedding-8B cosine distance measures semantic similarity; Distinct-Equations measures unique-equation ratios, and RPD is included as a math-specific metric.

3 Conventional Metrics Do Not Capture Approach-level Diversity

Approach-level diversity measures differences in underlying mathematical strategies, not merely observable variation in wording, notation, formatting, or exposition. Human comparisons show that bigram and cosine distances can reverse approach judgments because shared scaffolding and approach-preserving paraphrases distort their signals.

  • Defining approach-level diversity: Approach-level diversity is variation in the strategies used to solve the same problem correctly, distinguished from surface-level variation in observable solution form.Different approaches can diverge in mathematical tools, structural definitions, or representational viewpoint.
  • Human reference labels: 80% annotator agreement on 80 solution pairs from 20 MATH training problems provided the human reference for metric comparisons.Each problem contributed four model-generated pairs of correct solutions, and disagreements were resolved by the authors.
  • Conventional metric failure: Bigram and cosine distances disagree with human judgments, assigning high distances to some same-approach pairs and low distances to some different-approach pairs.Their mismatch reflects two confounds: shared scaffolding and approach-preserving paraphrasing.
  • Conventional metric failure: The metrics can rank a same-approach pair as more diverse than a different-approach pair when solutions differ in local algebra but share or differ in mechanisms.In the illustrated example, Solutions A and B share a discriminant-based approach, whereas B and C use different mechanisms; the metric ranking reverses the human decision.
  • Scaffolding and paraphrasing: 61.2% median unigram overlap was used to separate low- and high-scaffolding problems when testing whether shared scaffolding weakens approach discrimination.The experiment measures scaffolding as the ratio of overlapping unigrams among correct solutions.
  • Scaffolding and paraphrasing: Even L1 local wording or notation rewrites make same-approach pairs farther apart than original different-approach pairs, while the small L1–L2 gap shows presentation shifts add little distortion.L2 additionally changes global presentation format, so most distortion arises from local rewording rather than presentation changes.

4 Approach Mismatch Persists at Scale

An LLM-judge framework scales approach-level evaluation and shows that conventional diversity metrics lose discriminative power for finer differences once solution sets already contain multiple approaches. Diversity-aware RLVR preserves surface-level proxies while reducing approach coverage, mainly by varying solutions within narrower strategies.

  • Evaluation framework: The framework retains 469 problems with at least three unique, correct approaches and clusters correct solutions by underlying approach using a GPT-5.2 judge.Problems are filtered through GPT-5.2 generation, Qwen3-4B verification, and judge confirmation of mutual distinctness; K_x counts identified approaches.
  • Evaluation framework: The judge achieves 85.0% agreement with human reference labels, compared with 80.0% inter-annotator agreement, and disagrees on only 2 of 64 clear cases.This calibration supports scalable labeling for downstream approach-level diversity analyses.
  • Diversity-aware RLVR: Approach-level diversity declines after RLVR across all settings, even when DIVER preserves its textual- or equation-level diversity signals during training.Preserving an optimized surface-level proxy therefore does not preserve approach coverage.
  • Diversity-aware RLVR: 80.6% of the textual-diversity gain comes from greater variation among same-approach pairs, while cross-approach equation diversity remains relatively consistent.The results indicate that surface-diversity optimization mainly creates more varied realizations within a narrower set of reasoning strategies.

5 The Utility and Limits of Optimizing Approach-Level Diversity

Approach-level diversity improves test-time scaling when candidate sets cover more distinct approaches at the same sampling budget. Directly optimizing an LLM-judge diversity reward instead causes judge-specific reward hacking rather than broader approaches.

  • Overview: The study tests approach-level diversity through controlled test-time scaling and direct LLM-judge reward optimization during training.The experiments address whether approach-level diversity helps downstream performance and whether it can be induced directly.
  • Approach-controlled test-time scaling: Equally sized candidate subsets vary only in covered approaches while comparing self-consistency, best-of-N, and pass@k.For fixed m, subsets sample u approach plans and draw m/u solutions from each, preserving expected single-solution quality.
  • Approach-controlled test-time scaling: Across Qwen2.5-3B-Instruct, Llama3.2-3B-Instruct, and GPT-4o-mini, covering more approaches improves test-time scaling at the same sampling budget.This consistent trend indicates that approach-level diversity provides utility at inference time.
  • LLM-judge reward optimization: Optimizing an explicit approach-diversity reward makes the judge-reward checkpoint less diverse than naive GRPO, indicating reward hacking against the in-loop Qwen judge.A held-out GPT judge reverses its diversity ordering relative to the Qwen judge across training, showing the policy learned judge-specific preferences rather than broader approaches.

6 Discussions

The paper establishes approach-level diversity as a distinct dimension of LLM mathematical reasoning and shows that existing metrics and diversity-aware training can conflate surface variation with distinct strategies. It points toward domain-aware definitions and evaluations for future applications beyond mathematics.

  • Future Research Directions: Future work should develop domain-aware definitions and evaluations because useful strategic variation may become increasingly domain-specific in scientific discovery, program synthesis, and open-ended planning.
  • Conclusion: Approach-level diversity is introduced as a distinct axis for analyzing mathematical reasoning in LLMs.
  • Conclusion: Existing diversity metrics and diversity-aware training methods often conflate surface variation with genuinely distinct solution strategies.

Limitations · A Common Experimental Details

The study focuses on mathematical reasoning, covers representative rather than exhaustive diversity methods, and does not offer a general way to preserve or induce approach diversity. Its experiments use standardized sampling and judge-based answer verification, with Qwen3-4B extracting and checking final answers.

  • Limitations: The analysis focuses on mathematical reasoning, where surface-form and approach-level distinctions are relatively clear and diversity-aware RLVR is commonly studied.The authors propose extending the analysis to program synthesis, scientific discovery, and open-ended planning.
  • Limitations: The evaluation covers representative diversity metrics and recent diversity-aware RLVR methods, not all possible metrics, objectives, or decoding strategies.Results should therefore be interpreted as evidence of mismatch in common diversity proxies rather than an exhaustive survey.
  • Limitations: The work identifies the proxy–approach-diversity mismatch but provides no general method for preserving or inducing approach diversity during training.Future work may require faithful reward signals and algorithms that optimize them without proxy exploitation.
  • A Common Experimental Details: Unless otherwise specified, all solution generation uses the Figure 7 sampling prompt for offline analysis, RLVR rollouts, and evaluation.This standardizes generation across the study’s analysis and training-related settings.
  • A Common Experimental Details: Final-answer correctness is verified with Qwen3-4B using the Figure 8 prompt, after extracting a boxed answer or, if absent, the solution’s last 300 characters.The verifier compares the predicted answer with the ground truth and outputs only correct or incorrect.
  • A Common Experimental Details: A judge-based verifier is used because it better handles mathematically equivalent answers written in different forms than rule-based matching.This motivates model-based verification for general mathematical answers.
  • A Common Experimental Details: The verification prompt presents the model as a math expert given a golden answer and a predicted answer from a solver.It uses few-shot examples and constrains the output to correct or incorrect.

B Detailed Background on Diversity Metrics and Diversity-Aware RLVR … C.1 Dataset Generation

The paper defines several surface- and reasoning-oriented diversity metrics, along with DQO and DIVER baselines for diversity-aware RLVR. It also compares approach-seeking and format-focused sequential SFT datasets generated from the same problems and model.

  • B.1 Diversity Metrics under Analysis: The analysis considers embedding, lexical, n-gram, equation-based, and reasoning-trajectory metrics for measuring diversity between mathematical solutions.RPD represents solution steps with an LLM, embeds those representations, and averages step-level distances.
  • B.1 Diversity Metrics under Analysis: Cosine Embedding measures embedding distance, Self-BLEU reports 1 −Self-BLEU(S), and N-gram Distance uses Jaccard distance over n-gram multisets.The main analysis uses n = 2 for N-gram Distance.
  • B.1 Diversity Metrics under Analysis: Distinct-Equations extracts equations from solution sets and computes the ratio of unique equations, while RPD averages step-level representation distances.These metrics target equation-level and reasoning-trajectory differences, respectively.
  • B.2 Diversity-Aware RLVR Baselines: DQO adds a group-level semantic diversity objective to RL post-training by using the determinant of an embedding-similarity matrix as its diversity signal.Its objective combines quality reward R(Y) with diversity strength controlled by α.
  • B.2 Diversity-Aware RLVR Baselines: DIVER adds an intrinsic diversity reward to verifiable RLVR, granting each correct rollout a bonus based on dissimilarity to the other rollouts.The diversity strength is controlled by β; experiments use textual diversity from pairwise BLEU and equation diversity from distinct-equation ratios.
  • C Additional Analysis: Approach-Seeking Sequential SFT: The complementary SFT study asks whether encouraging diverse approaches improves reasoning by comparing datasets with identical output structure but different diversity prompts.The approach dataset requests three distinct mathematical approaches, whereas the format dataset requests three variants of one approach.
  • C.1 Dataset Generation: Both sequential-solution datasets are generated by Qwen3-14B and used to fine-tune Qwen3-4B, differing only in generation prompt.From 2,000 training problems, the study retains only problems whose three solutions are correct under both prompts.

C.2 Behavioral and Ability Checks · D Computing Scaffolding Ratio · E Approach-Level Evaluation Framework

The Approach checkpoint exhibits more approach-seeking behavior than Format while matching its baseline reasoning ability, improving sequential-solution accuracy mainly when alternative approaches are both feasible and reachable. The scaffolding ratio measures how much solution text is shared across correct solutions through token prevalence.

  • C.2 Behavioral and Ability Checks: The Approach checkpoint uses approach-switching cues substantially more often than the Format checkpoint.Examples include “another approach” and “another method.”
  • C.2 Behavioral and Ability Checks: 69.04% (approach) and 68.30% (format) under the standard single-solution prompt confirm matched baseline reasoning ability.The comparison uses the standard single-solution prompt in Figure 7.
  • C.2 Behavioral and Ability Checks: The Approach checkpoint outperforms Format on OlympiadBench and AIME in the sequential-solution setting.Per-problem gains concentrate on problems with multiple approaches and grow with difficulty within that subset.
  • C.2 Behavioral and Ability Checks: 11.69 points on reachable problems versus 3.50 percentage points on 30 problems unreachable under Qwen3-4B validation shows that approach-seeking supervision helps when alternatives are accessible.The 3.50-point result has a 95% confidence interval that includes zero; reachability was tested with GPT-5.4-mini as a stronger validator.
  • D Computing Scaffolding Ratio: The scaffolding ratio is computed from token prevalence across all correct solutions to the same problem.Solutions are lower-cased and whitespace-tokenized; each solution’s score averages the prevalence of its tokens, followed by a problem-level aggregation.
  • D Computing Scaffolding Ratio: A high scaffolding ratio indicates that many solution tokens recur across correct solutions, capturing shared setup, notation, restated conditions, and answer-formatting patterns.The measure is intended to capture shared scaffolding rather than strategy differences.

E.1 Details in Dataset Construction … F.1 Implementation Details

The paper constructs a multi-approach dataset through difficulty, plan, correctness, and distinctness filtering, then develops human-calibrated annotation and judge procedures for evaluating approach diversity. Its alignment evaluation samples correct solutions, clusters them into approaches, and forms balanced comparison pairs under feasibility constraints.

  • E.1 Details in Dataset Construction: Problems are filtered for reachable diversity by requiring a pass rate of at least 0.2 and an average solution length of at least 300 tokens.The initial filter generates 16 Qwen3-4B solutions per problem and removes problems that are too simple or too difficult.
  • E.1 Details in Dataset Construction: The distinctness judge excludes problems with fewer than three meaningful approaches, achieving precision 0.9644, recall 0.8714, and F1 score 0.9156 across 5 runs.These figures come from 50 validation problems with human labels.
  • E.2 Constructing Human Annotation Set: The human annotation set retains solutions correct in final answers and intermediate reasoning, balancing same-approach and different-approach pairs from individual models.Author-provided approach labels balanced the items but were withheld from annotators, who were voluntarily recruited graduate students without monetary compensation.
  • E.3 Details of the LLM Judge Prompt: The judge prompt defines approaches by mathematical mechanisms, merges ambiguous or presentation-only distinctions, and explicitly guards against over-splitting surface-level variations.Prompt development used an independent author-labeled validation set, reserving the human annotation set for final alignment evaluation.
  • F Details on Alignment Evaluation: Correct solutions are clustered by chunking groups of at most eight, clustering each chunk, reclustering representatives, and assigning originals to representative clusters.The same clustering prompt is used throughout the two-stage procedure.
  • F.1 Implementation Details: For each tier and subset size, the evaluation compares k-solution sets covering t versus t + 1 approaches and retains settings with at least B = 500 valid pairs.Each problem-model combination samples N = 32 solutions at T = 1.0, keeps correct solutions, and allocates them as evenly as possible across approaches.

F.2 Ablations on Conventional Measures … G.3 Evaluation details

Ablations show that conventional diversity measures remain unreliable proxies for approach-level diversity across metric settings and evaluation granularities. The paper also specifies judge-based clustering, coverage estimation, shared RLVR training details, and distinct sampling protocols for quality versus approach coverage.

  • F.2 Ablations on Conventional Measures: Across tiers 2 and 3, varying n-gram order, embedding backbone, and subset size does not make conventional metrics reliably capture approach-level diversity.The ablations cover Figures 13–15 and remain consistent with the main paper beyond coarse surface differences.
  • F.2 Ablations on Conventional Measures: Human annotators compare two correct solutions for one problem and judge whether they use the same high-level mathematical approach.This pairwise annotation format is illustrated in Figure 11.
  • F.2 Ablations on Conventional Measures: Pairwise evaluation shows the failure lies in distance metrics themselves rather than set-level aggregation.Pairwise concordance uses same-approach and different-approach solution pairs for each problem.
  • F.2 Ablations on Conventional Measures: Cosine distance is weakest across all three embedding models, while N-gram distance is strongest in two of three and RPD does not surpass simple bigram overlap.These pairwise results mirror the set-level ranking.
  • G.1 Training Details: The approach-clustering judge evaluates solution sets by grouping them according to conceptual and mathematical distinctions and summarizing each cluster.The prompt frames the judge as an expert mathematician specializing in comparative strategy analysis.
  • G.2 Coverage Analysis: Coverage is measured as covx(N, π), the expected number of distinct approaches observed when sampling N correct responses from policy π for problem x.Approach clusters partition the collected correct solutions, and cluster representation indicators yield an unbiased estimator.
  • G.2 Coverage Analysis: Coverage estimation uses N = 16 and M = 64 and includes only problems where every Qwen2.5-3B checkpoint has at least N correct solutions.This restriction ensures comparability across checkpoints.
  • G.3 Evaluation details: Quality evaluation samples 32 solutions per OlympiadBench problem at temperature 0.7, whereas approach coverage samples 64 at temperature 1.0 to better capture generated approaches.The two protocols distinguish routine quality assessment from broader approach-coverage estimation.

G.4 Analysis Setup … J Prompts used for Dataset Construction

The analysis decomposes textual and equation-level diversity to show that gains can reflect variation within existing approaches rather than broader approach coverage. The paper also specifies evaluation setups, judge-based coverage checks, dataset-construction prompts, and language-model assistance procedures.

  • G.4 Analysis Setup: DIVER’s recovered textual and equation-level diversity mainly reflects greater variation within the same approach, not broader coverage of distinct approaches.The textual decomposition separates within-approach distances, across-approach distances, and pair composition; equation diversity rises mainly for same-approach subsets.
  • H.1 Details in Test-Time Scaling: Concept-conditioned generation follows Handa et al. (2026), while best-of-N uses Qwen2.5-Math-PRM-7B and selects the solution with the highest lowest-step PRM score.Each solution is scored by its lowest step score assigned by the PRM.
  • H.2 Details of LLM Judge Reward RLVR: Training-time coverage uses average unique approach clusters because each problem has only N = 8 rollouts, while expected coverage is evaluated at saved checkpoints.The checkpoint-based trend matches the main result: external GPT detects a 38% drop, whereas in-loop Qwen detects an 18% drop.
  • H. The Utility and Limits of Optimizing Approach-Level Diversity: 38%: Expected coverage measured by the external GPT judge drops by 38%, compared with 18% for the in-loop Qwen judge.Expected coverage is reported at saved checkpoints, where enough solutions can be sampled for more stable estimation.
  • I The Use of Large Language Models: Large language models assisted with grammar correction, wording refinement, minor stylistic edits, and developing code used in the experiments.The paper describes this assistance as preparation support rather than a substantive research finding.
  • J Prompts used for Dataset Construction: Dataset-construction prompts generate either exactly three correct solutions with distinct approaches or exactly three solutions sharing an approach but differing in presentation format.Additional prompts enumerate up to K genuinely distinct high-level plans, execute selected plans faithfully, and judge uniqueness by mechanism and conceptual interpretation.
  • J Prompts used for Dataset Construction: In a clustering example, six correct solutions are grouped by mechanism: a closed-form angle-bisector formula, Stewart’s theorem, or a Law of Cosines computation.The judge notes that all solutions first apply the angle bisector theorem to split AB into AD = 5 and DB = 4.
Loading 2606.29985v1…