Source-linked AI summary
Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi
TL;DR
RL reasoning post-training can concentrate on dominant patterns, limiting rollout-level diversity and pass@k gains. The paper introduces a rollout-level objective that clusters same-problem solutions by high-level strategy and rewards correct rare strategies more strongly. Across mathematics, physics, and medicine, it reports improved pass@k and AUC@K while maintaining exploration and pass@1.
Problem
RL training can prematurely concentrate on dominant reasoning patterns, limiting rollout-level diversity and pass@k gains.
Method
Uniqueness-Aware RL uses an LLM judge to cluster same-problem rollouts by high-level strategy and reweights advantages inversely with cluster size.
Results
Across mathematics, physics, and medicine, the method consistently improves pass@k and AUC@K across sampling budgets while maintaining strong pass@1 performance.
Takeaways & Limitations
The findings support treating reasoning diversity as a set-level property and explicitly modeling strategy uniqueness during RL.
Takeaways & Limitations
The LLM judge adds computational overhead, may miscluster ambiguous strategies, and measures rarity only within one problem’s rollout set.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.
1 Introduction
RL post-training can collapse onto a few dominant reasoning patterns, making token-level variation an inadequate proxy for strategy-level exploration. Uniqueness-Aware RL instead rewards correct, rare strategies within rollout sets and improves coverage across reasoning domains and sampling budgets.
- Exploration collapse concentrates policies on a few high-probability reasoning patterns, limiting solution-space coverage.
- Token entropy and embedding distance can increase wording variation without producing distinct high-level solution strategies.
- Pass@k depends on maintaining multiple conceptually distinct strategies, whereas RL can improve pass@1 while reducing rollout-level diversity.
- Uniqueness-Aware RL clusters same-problem rollouts by high-level plans, amplifying correct rare strategies and downweighting redundant correct ones.
- Evaluations span mathematics, physics, and medicine, with exploration maintained as sampling budgets increase up to k=256.
2 Related Work
Prior exploration-aware methods often use token entropy, embedding distance, semantic uncertainty, or pass@k-oriented objectives as indirect diversity signals. Uniqueness-Aware RL instead clusters full rollouts by high-level strategy and reweights advantages at the rollout-set level.
- Existing methods encourage diversity through token entropy, embedding distance, semantic uncertainty, or pass@k-oriented objectives.
- These approaches may increase variation or novelty without necessarily broadening solution-strategy coverage.
- Uniqueness-Aware RL clusters full reasoning traces into high-level strategies for each problem and reweights group-based advantages inversely with cluster size.
3 Methodology
The method extends GRPO by measuring strategy rarity within each problem’s rollout group and incorporating it into policy advantages. An inference-only LLM judge clusters high-level approaches while ignoring superficial differences, so rare correct strategies receive stronger updates without changing the GRPO objective form.
- Overview: The method uses GRPO group-normalized advantages computed from verifier-assigned rollout rewards.
- Training Objective: The training objective retains the GRPO form and modifies only its advantage term, with standard regularization and stochastic optimization.
- Uniqueness Calculation: An inference-only LLM judge partitions same-problem reasoning traces into clusters sharing a high-level solution idea.
- Uniqueness Calculation: The judge ignores superficial differences such as variable naming, algebraic rearrangement, and verbosity when grouping strategies.
- Uniqueness Calculation: Small strategy clusters represent rare approaches, whereas large clusters represent common approaches repeatedly produced by the policy.
- Combining Quality and Creativity: Uniqueness weights downweight common strategies and preserve larger effective weights for rare strategies while remaining bounded.
- Combining Quality and Creativity: Incorrect rollouts remain penalized because uniqueness reweighting amplifies positive quality signals rather than overriding correctness.
- Training Objective: The objective allocates probability mass across multiple high-level strategies, aligning updates with pass@k and creative problem-solving behavior.
4 Experiments
Experiments evaluate uniqueness-aware RL across reasoning domains, model families, sampling budgets, and strategy-coverage diagnostics. The method generally improves pass@k and AUC@K while sustaining exploration and recovering rarer solution strategies.
- Experimental Setup: Training uses mathematics, physics, and medicine datasets, with evaluation on AIME, HLE, OlympiadBench, and MedCaseReasoning.The models include Qwen-2.5-7B, OLMo-3-7B, and Qwen-3-8B, compared against instruction and RL baselines.
- Pass@k Performance: OURS matches or exceeds instruction and SimpleRL baselines across domains, with larger pass@k gains at medium-to-large budgets around k ≳32.The improvement is clearest on AIME, HLE, and OlympiadBench-Physics; medicine shows consistent gains without low-k degradation.
- AUC@K Comparison: AUC@K is highest for OURS across all four domains and budgets K=64/128/256 on Qwen2.5-7B.Against SimpleRL, the gains are largest on harder AIME/HLE settings; at K=64 they are +0.044 on AIME and +0.026 on HLE, while gains shrink as curves saturate at K=256.
- Additional Model Families: OURS also achieves the best AUC@K on HLE and Physics for OLMo-3-7B and Qwen-3-8B, surpassing DAPO and Forking Token.For Qwen-3-8B at K=64, OURS improves over DAPO from 0.201→0.217 on HLE and 0.361→0.365 on Physics.
- Entropy Dynamics: Uniqueness-aware training maintains higher and more stable policy entropy than SimpleRL, preserving a broader exploration horizon during optimization.The analysis connects this behavior with continued search for long-tail strategies and improved diversity coverage.
- Human Solution Coverage: The cover@32 analysis shows higher human-method coverage on the four most complex problems, including 100% versus 40% on one geometry problem.On aime24_i_p10, OURS recovers all five canonical ideas, including Symmedian Similarity and Pure Trigonometry; on aime2025_ii_p3, it reaches 75% cover@32.
5 Conclusion
The paper presents Uniqueness-Aware Reinforcement Learning as a strategy-level approach to mitigating exploration collapse and promoting diverse, high-quality reasoning paths. It also describes an LLM-based clustering pipeline and notes computational and modeling limitations.
- 5 Conclusion: Uniqueness-Aware Reinforcement Learning reweights policy updates to favor correct but rare reasoning paths within each problem.This aligns reinforcement learning with discovering diverse, high-quality solutions rather than optimizing a single dominant mode.
- 5 Conclusion: The approach relies on an LLM-based judge, adding computational overhead and risking imperfect clustering for ambiguous or overlapping reasoning structures.Its high-level strategy definition is task-dependent, and rarity is measured only within one problem’s rollout set.
- 5 Conclusion: The three-stage pipeline converts judge-generated strategy descriptions into structured category mappings and then into an integer label list of length K.The stages comprise natural-language clustering, dictionary extraction, and ordered label conversion.
- 5 Conclusion: The strategy-clustering pipeline asks an LLM judge to group solutions by high-level methods while disregarding sub-steps and implementation details.For physics, the prompts distinguish overarching physical principles or modeling frameworks and assign every solution to one group.
B.1 Training Examples
This training-example section contains short quantitative and algebraic problems, including equations, inequalities, rates, and mixture calculations. The examples are presented as individual question fragments and complete prompts.
- B.1 Training Examples: The examples include solving a cubic-root equation and expressing the smaller real solution as p − √q.The task asks for p + q.
- B.1 Training Examples: Other examples ask for integer counts satisfying a quadratic inequality and a travel-time difference between two distances at constant speed.The stated car speed is 50 miles per hour, with trip lengths of 450 and 475 miles.
- B.1 Training Examples: A mixture problem asks how much pure water must be added to 30 ounces of a 30% acid solution to obtain a 20% acid solution.The prompt includes the initial solution quantity and concentration and the target concentration.
B.1.2 Physics (TextbookReasoning-Physics subset) 1. Question
The physics training subset spans fluid flow, heat transfer, vibration-driven motion, aerodynamics, and a vector-calculus identity. Its questions combine numerical parameters with derivations and proofs.
- B.1.2 Physics: The subset includes a brine core-sample problem that provides time-dependent brine heights and asks for permeability.The table lists heights from 100.0 cm at 0 seconds to 13.5 cm at 4000 seconds.
- B.1.2 Physics: The subset also includes an aerodynamic speed calculation for a thrust-limited car-plane at 8200 feet and a proof involving a constant-length vector.The vector proof uses differentiation of its dot product with itself.
- B.1.2 Physics: A furnace problem asks for the heat-transfer rate to brass plates heated from 75°F toward 1000°F in a 1300°F oven.The plates are processed at 300 plates per minute and have specified dimensions and density.
- B.1.2 Physics: A vibrotransporting-tray problem asks for the minimum vibration amplitude at 50 Hz and the minimum frequency for a 0.01 mm amplitude.The tray angle is 10° and the friction coefficient is 0.2.
B.1.3 Medical (MedCaseReasoning train subset) 1. Question
The medical training subset contains diagnostic cases across musculoskeletal, pulmonary, neurologic, dermatologic, and maxillofacial presentations, alongside additional mathematics examples. The cases pair clinical findings with imaging or pathology information.
- B.1.3 Medical: One case describes fever, painful leg swelling, abnormal muscle imaging, and an inguinal lymph node, with a highlighted diagnosis of diffuse large B-cell lymphoma.Imaging showed diffuse muscle involvement and no thrombosis.
- B.1.3 Medical: Another case presents an asymptomatic right lower-lobe mass with calcifications and imaging findings supporting a presumptive diagnosis of unicentric Castleman’s disease.The lesion was evaluated with radiography, CT, MRI, endobronchial ultrasound, and bronchoscopy.
- B.1.3 Medical: A maxillary swelling case includes mixed radiopaque–radiolucent imaging features and a highlighted diagnosis of chondroblastic osteosarcoma.The lesion showed a cotton-wool pattern and peripheral sunray appearance.
- B.1.3 Medical: Additional examples ask about base representations, integer solutions, geometric area, and flavor assignments under ordering constraints.These prompts include divisibility in base b, a quadratic Diophantine equation, a heptagon area, and counting assignments.
B.2.3 Physics Test Set: OlympiadBench (Text-only, English, Competition) 1. Question
The physics test set contains several quantitative mechanics and thermal-energy problems, including string tension, propulsion, viscosity, conveyor motion, and neutrino heating.
- Physics problems: A conveyor-belt problem asks for the maximum density of a replacement hopper material under constant-velocity and no-slip conditions.The supplied setup includes 20.3 m/s belt speed, 15 MW maximum power, 2 m wheel diameter, and 18 m^3 hopper delivery.
- Physics problems: A neutrino-heating problem asks for the integer exponent N in the time 1 × 10^N seconds required to heat Earth’s inner core by 1°C.The model uses neutrino energy and flux at Earth, plus the core’s radius, density, and specific heat.
- Physics problems: The set includes a violin-string scaling problem asking for the frequency ratio after all lengths increase by √2.The string density remains unchanged, and the original and enlarged strings are tuned to their highest possible frequencies.
- Physics problems: A propeller problem compares initial forces when the same battery-powered propeller pushes air versus water.The stated densities are 997 kg/m^3 for water and 1.23 kg/m^3 for air, with unchanged propeller efficiency assumed.
B.2.4 Medical Test Set: MedCaseReasoning 1. Case
The medical test set presents varied clinical cases spanning dermatology, oral pathology, nephrology, gastrointestinal imaging, and diagnosis evaluation procedures.
- Clinical cases: The set also includes a bullous fixed drug eruption diagnosed after dicloxacillin exposure for presumed cellulitis.The patient developed a new itchy truncal rash after antibiotics, while laboratory studies remained within normal limits.
- Clinical cases: An odontogenic lesion showed epithelial cords, strands, and islands in a myxoid stroma without enamel or dentin formation.The lesion had peripheral columnar cells, central stellate-reticulum-like cells, variable cellularity, partial encapsulation, and no atypia or mitoses.
- Clinical cases: A patient with progressive skin thickening developed abrupt severe hypertension, worsening renal function, and new proteinuria after corticosteroid exposure and intravitreal VEGF blockade.Scleroderma renal crisis was suspected, while complement-mediated thrombotic microangiopathy related to VEGF inhibition could not be ruled out.
- Clinical cases: A child with episodic epigastric pain had imaging findings consistent with a nonvascularized, displaced gallbladder containing sludge and a neck abnormality.Ultrasound and CT documented wall irregularity, hemorrhage-like hyperdensity, pericholecystic changes, and fundus deviation.
- Clinical cases: A bullous eruption after Shingrix vaccination occurred in a patient with Crohn’s disease receiving infliximab, with mucosal sparing and negative viral PCR testing.The eruption involved erythema, swelling, and bullae across the chest, axilla, and upper arm.
- Evaluation procedure: Medical diagnosis rewards are binary after extracting a diagnosis and using an LLM judge with a strict equivalence rubric.The evaluator maps y to 1 and n to 0, producing rmed ∈ {0, 1}.
C.1 Geometry: AIME 2024 I Problem 10 (aime24_i_p10)
The geometry problem asks for m + n after expressing AP = m/n in a triangle whose tangent intersection determines the A-symmedian. Multiple solution routes use power, similarity, tangent, or cyclic-geometry relations.
- Problem: The problem gives AB = 5, BC = 9, and AC = 10, with AP = m/n for relatively prime integers m and n.The requested output is m + n.
- Result: The reported final result is AP = 13/22, giving m + n = 113.The answer passage states the reduced numerator-and-denominator sum explicitly.
- Symmedian characterization: Tangents at B and C identify AD as the A-symmedian of triangle ABC.The supplied characterization connects the tangent construction with equal-angle relations involving chords AB and AC.
- Power and tangent route: One route combines tangent lengths, circumradius relations, algebraic simplification, and Power of a Point at D.It uses DB = R cos A and the secant relation involving D, A, and P.
- Similarity route: A symmedian-similarity route introduces the midpoint M of BC and computes AM using Apollonius’ median-length formula.The method relates the symmedian intersection with the circumcircle to the median geometry of triangle ABC.
- Ptolemy route: A Ptolemy route applies the symmedian chord ratio in cyclic quadrilateral ABPC, followed by Ptolemy’s theorem.The stated relation is AP · BC = AB · PC + AC · PB.
C.2 Combinatorics: AIME 2025 II Problem 3 (aime2025_ii_p3)
The combinatorics problem counts red-blue edge colorings of a 2 × 2 grid in which every unit square has exactly two red and two blue sides. Several equivalent enumerations yield 82.
- Problem: The configuration has 12 unit segments, each colored red or blue, with every unit square containing two red and two blue sides.The stated answer is 82.
- Interior-edge enumeration: For k = 2, opposite red interior edges contribute 32 colorings while adjacent red interior edges contribute 16.There are two opposite patterns with 16 completions each and four adjacent patterns with four completions each.
- Interior-edge enumeration: Casework conditions on the number k of red interior edges, with k = 0 and k = 4 each contributing one coloring.The all-blue and all-red interior assignments force the boundary edges uniquely.
- Interior-edge enumeration: For k = 1 and k = 3, the enumeration contributes 16 colorings in each case.The k = 3 count follows by swapping red and blue from the k = 1 configurations.
- Alternative viewpoints: A trail viewpoint interprets red edges as local degree-2 path pieces that connect across shared interior edges.Each square’s red sides form either a straight segment from opposite sides or a turn from adjacent sides.