Source-linked AI summary
Reinforcement Learning with Rubric Anchors
Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, Xijun Gu, Peiyi Tu, Jiaxin Liu, Wenyu Chen, Yuzhuo Fu, Zhiting Fan, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, Jianguo Li, Junbo Zhao
TL;DR
RLVR’s dependence on objectively verifiable answers limits its reach into open-ended tasks, where reliable and scalable rewards are difficult to construct. The paper extends RLVR with structured rubric-based rewards and evaluates Rubicon-preview, reporting gains on open-ended benchmarks, stylistic control, and preserved general abilities.
Problem
RLVR is limited to tasks with objectively verifiable solutions, creating a need for reliable, scalable reward signals for open-ended outputs without explicit ground truth.
Method
The paper extends RLVR with structured, interpretable rubrics that provide multidimensional reward signals and trains Rubicon-preview through a multi-stage RL framework.
Results
+5.2% absolute improvement: Rubicon-preview 30B-A3B outperforms 671B DeepSeek-V3 by +2.4% percentage points on subjective, humanities-centric tasks using only 5K data samples.
Takeaways & Limitations
Rubrics serve as controllable anchors for more human-like, emotionally expressive, less formulaic responses while preserving general competence and adding reasoning gains.
Takeaways & Limitations
The authors note that current benchmarks inadequately evaluate rubric-based performance and that the optimal hierarchical rubric structure remains insufficiently studied.
Abstract
from arXiv · showhide
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing Large Language Models (LLMs), exemplified by the success of OpenAI's o-series. In RLVR, rewards are derived from verifiable signals-such as passing unit tests in code generation or matching correct answers in mathematical reasoning. While effective, this requirement largely confines RLVR to domains with automatically checkable outcomes. To overcome this, we extend the RLVR paradigm to open-ended tasks by integrating rubric-based rewards, where carefully designed rubrics serve as structured, model-interpretable criteria for automatic scoring of subjective outputs. We construct, to our knowledge, the largest rubric reward system to date, with over 10,000 rubrics from humans, LLMs, or a hybrid human-LLM collaboration. Implementing rubric-based RL is challenging; we tackle these issues with a clear framework and present an open-sourced Qwen-30B-A3B model with notable gains: 1) With only 5K+ samples, our system improves by +5.2% on open-ended benchmarks (especially humanities), outperforming a 671B DeepSeek-V3 model by +2.4%, while preserving general and reasoning abilities. 2) Our method provides fine-grained stylistic control, using rubrics as anchors to mitigate the "AI-like" tone and produce more human-like, expressive responses. We share key lessons in rubric construction, data selection, and training, and discuss limitations and future releases.
1 Introduction
The paper extends RLVR to open-ended tasks by using rubric-based rewards, then presents Rubicon-preview with gains on subjective benchmarks, stylistic control, and preserved general abilities.
- Motivation: RLVR is restricted by its reliance on objectively verifiable question–answer pairs, limiting its applicability to a narrow subset of tasks.Open-ended tasks require reward signals that remain reliable and scalable without explicit ground truth.
- Approach: Rubric-based rewards use structured, interpretable criteria to assess multidimensional response quality beyond binary correctness.The paper studies rubric diversity, granularity, quantity, training routines, and data curation as determinants of effectiveness.
- Approach: The system constructs a rubric reward bank containing over 10,000 rubrics generated by human experts, LLMs, or hybrid collaboration.Rubrics span dataset-, task-, and data-point-level scopes and are ablated before integration into training.
- Results: +5.2% absolute improvement: Rubicon-preview 30B-A3B outperforms 671B DeepSeek-V3 by +2.4% percentage points on subjective, humanities-centric tasks using only 5K data samples.The result is reported as the paper’s principal token-efficient performance outcome.
- Results: Rubric-based RL provides fine-grained style control, producing more human-like, emotionally expressive, and less formulaic responses.Rubrics act as controllable anchors that reduce AI-like or didactic tone.
- Results: Rubicon-preview preserves general competence while improving reasoning benchmarks, including AIME 2024 (+4.1%) and AIME 2025 (+0.8%).The rubrics were not tailored specifically to STEM-oriented tasks such as mathematics or coding.
- Outlook: The authors identify token efficiency as an open scaling-law question: 5K samples combined with many rubrics may represent a new form of post-training scaling.They frame this as a question for future investigation rather than an established scaling law.
2 Rubric System
The rubric system converts open-ended evaluation into structured, interpretable feedback by combining model-verifiable criteria, multidimensional scores, and nonlinear reward aggregation.
- 2.1 Rubrics Design & Tasks Curation: The rubric-first workflow constructs model-verifiable rubrics before curating or synthesizing matching data for supervision, reward shaping, and evaluation.Using the same criteria across acquisition, training, and assessment promotes consistency.
- 2.1 Rubrics Design & Tasks Curation: Rubric quality depends on quantity, diversity, granularity, training routine, and meticulous data curation rather than a single construction strategy.Rubrics may be dataset-specific, task-level, or associated with individual data points.
- 2.1 Rubrics Design & Tasks Curation: A rubric R is a set of K critic dimensions, each containing a criterion description, ordered score tiers, and a relative-importance weight.This structure supports both general-purpose open-ended evaluation and fine-grained programmatically verifiable criteria.
- 2.1 Rubrics Design & Tasks Curation: The multidimensional rubric formalization translates evaluation criteria into granular, interpretable reward signals for policy optimization.The framework unifies diverse evaluation protocols under a common representation.
- 2.2 Rubric-Based Reward Framework: Given rubric R, the reward function maps response y to a vector of scores across the rubric’s specified criteria.Each component rk(y|x) is the score for the k-th dimension, providing interpretable feedback across criteria.
- 2.2 Rubric-Based Reward Framework: A weighted sum provides a scalar baseline reward, but the framework supports more sophisticated aggregation for nonlinear interdependencies between dimensions.The design moves beyond linear combination when simple addition fails to capture criterion relationships.
- 2.2 Rubric-Based Reward Framework: Veto mechanisms can nullify other rewards after critical failures, while saturation-aware aggregation encourages balanced improvement across dimensions.These mechanisms address hard constraints and diminishing marginal returns.
- 2.2 Rubric-Based Reward Framework: Pairwise interaction modeling captures synergistic or antagonistic relationships between criteria that a simple sum would ignore.This allows aggregation to represent dependencies among rubric dimensions explicitly.
3 Implementation of Rubicon Framework
Rubicon uses multi-stage reinforcement learning with filtered data and task-specific sequencing to develop instruction-following, creative, and socially grounded capabilities. An adaptive defense rubric addresses reward hacking and improves training stability.
- 3 Implementation of Rubicon Framework: A multi-stage RL protocol progressively develops instruction-following, creative, and social reasoning capabilities from a diverse 900K+ instance corpus.The corpus combines community Q&A, examinations, and conversational data with strategic sampling for broad topical coverage.
- 3 Implementation of Rubicon Framework: Offline filtering retains candidate instruction–rubric pairs within a calibrated central score quantile, excluding low-quality and weak-learning-signal instances.Base-model responses are scored by critic models before filtering and stage-specific adjustment.
- 3 Implementation of Rubicon Framework: Stage-wise RL separates constraint-handling training from later open-ended, socially grounded, and creative training to mitigate conflicting objectives.The first phase uses programmatic checks and static rubrics; the second uses reference-based and instance-specific criteria.
- 3 Implementation of Rubicon Framework: Reward hacking emerges when early training focuses on a small number of capabilities, allowing rubric exploitation without genuine improvement.The authors identify recurrent high-level failure patterns through offline analysis of anomalously high-reward rollouts.
- 3 Implementation of Rubicon Framework: A defense rubric synthesized from observed failure modes is added to later RL stages as a supervisory constraint, improving stability and preventing reward-hacking collapse.The defense rubric mitigates catastrophic reward spikes and enables longer, more productive training epochs.
4 Experimental Results
Rubicon is evaluated on open-ended, humanities-centric, general, reasoning, and stylistic tasks. It improves open-ended performance and human-like expressiveness while preserving general abilities, though mixed task rubrics create trade-offs.
- 4 Experimental Results: Rubicon is evaluated quantitatively on open-ended and human-centric benchmarks, qualitatively through output showcases, and against general-ability benchmarks.The benchmark suite includes writing, emotional intelligence, instruction-following, and broader capability evaluations.
- 4 Experimental Results: The benchmark set covers Creative Writing V3, Writingbench, Judgemark V2, EQ-Bench3, IFEval, Collie, and IFScale.These tasks emphasize open-ended outputs and humanities-centric capabilities that lack directly verifiable rewards.
- 4 Experimental Results: The broader evaluation includes MMLU, HellaSwag, StoryCloze, social and conversational benchmarks, plus AIME24, AIME25, Math500, GPQA-Diamond, and LiveCodeBench v5.These benchmarks test potential regressions in general and reasoning capabilities.
- 4 Experimental Results: 5.2% average improvement across the open-ended benchmark suite lets Rubicon surpass the 671B DeepSeek-V3 model by 2.4%.Table 1 compares Rubicon-preview with its Qwen3-30B-A3B base model and DeepSeek-V3.
- 4 Experimental Results: Rubicon achieves significant gains on writing and emotional-intelligence benchmarks, while showing a minor drop on IFEval but strong results on the other instruction-following benchmarks.The reported gains are concentrated in writing and emotional intelligence rather than uniformly across instruction-following tasks.
- 4 Experimental Results: Rubric-based RL produces plainer, more informative, human-like, and emotionally expressive responses with less AI-like or didactic tone.The style evaluation prioritizes authenticity and avoidance of artificial, preachy, or formulaic language over literary polish or technical correctness.
- 4 Experimental Results: Rubicon does not degrade general benchmarks and improves AIME24 by +4.17% and AIME25 by +0.83%.The results indicate preserved general competence alongside additional gains on selected reasoning benchmarks.
- 4 Experimental Results: The seesaw effect reflects trade-offs between rubric types: instruction-following rubrics improve compliance but reduce creativity, while creativity and empathy rubrics harm strict adherence.The creativity-focused model drops on Collie (-6.0%) and IFEval (-5.9%), while the instruction-following model declines on EQ-Bench3 (-2.2%).
5 Outlook
The outlook identifies benchmark limitations, unresolved rubric-system design questions, and the open challenge of combining Rubicon with RLVR. It also frames the seesaw effect as a key issue for combined training.
- Benchmarks: Current benchmarks inadequately evaluate rubric-based approaches and do not consistently reflect large-scale human feedback on open-ended abilities.The paper notes a scarcity of benchmarks capturing open-ended, anthropomorphic capabilities.
- Rubric system: The optimal hierarchical structure of rubrics for maximizing performance gains and token efficiency remains unresolved.Rubric quantity, diversity, granularity, quality, and data curation all influence success, but their best organization requires systematic study.
- Figure 2: Figure 2 compares baseline Qwen3-30B-A3B with models trained exclusively on creativity or instruction-following tasks across task categories.The horizontal axis shows corresponding-task performance, while the vertical axis lists task categories.
- Scaling RL training: Rubicon targets non-verifiable answers, complementing RLVR’s focus on tasks with verifiable rewards.The paper identifies combining these paradigms as an important future direction.
- Scaling RL training: Combined RL training raises an open question about how the seesaw effect would emerge and be managed.The paper does not present a definitive solution for this combined setting.
A Rubric Demonstration
The rubric demonstrations illustrate how Rubicon converts diverse evaluation goals into structured scoring procedures, spanning reward-hacking defenses, nuanced style assessment, and programmatic hard-constraint checks.
- A.1 Rubrics for Defensing Reward Hacking: Rubicon’s reward-hacking defense detects opening praise and self-evaluation in responses using explicit feature definitions and decision rules.The procedure separates main response content from additional explanatory content and checks the first sentence for opening praise.
- A.1 Rubrics for Defensing Reward Hacking: The defense rubric requires exact JSON fields, boolean values, empty strings when absent, and no additional explanation.Matching text is copied into the corresponding output field, while unmatched features remain false and blank.
- A.2 Rubrics for Creativity & Empathy: The creativity-and-empathy rubric evaluates responses beyond functional correctness, emphasizing meaningful intellectual and emotional connection.Its criteria include relational efficacy, intellectual contribution, and compositional excellence.
- A.2 Rubrics for Creativity & Empathy: Its compositional criteria assess logical coherence, linguistic sophistication, rhetorical artistry, and the integration of substance with style.The rubric also penalizes responses that lack bespoke adaptation to the user’s context.
- A.2 Rubrics for Creativity & Empathy: The style rubric’s output is a JSON object containing a criterion identifier, rationale, and integer score.The examples specify a consolidated scoring scale for the evaluation criteria.
- A.3 Rubrics for Hard Constraints: The hard-constraint dataset pairs natural-language writing instructions with Python scoring functions that validate specified constraints.Generated rubrics are filtered for syntactic validity and appropriate constraint complexity.
- A.3 Rubrics for Hard Constraints: Example hard constraints check passage count and whether the word “education” appears at least once.The scoring functions apply programmatic checks to the generated text.
B Output Cases of Our Rubicon-preview
The output cases present Rubicon-preview as a model for emotionally textured creative writing, using concrete objects and sensory details to develop themes of departure, memory, and uncertainty.
- Cross-case patterns: Across the examples, Rubicon-preview uses concrete objects and restrained emotional cues to connect physical settings with personal histories.Examples include the clock, garden, piano, scarf, photograph, notebook, and key.
- Departure narrative: The story closes with the character carrying selected objects while treating uncertain beginnings as distinct from failed endings.The ending emphasizes movement into an uncharted world without looking back.
- Departure narrative: Rubicon-preview’s suitcase narrative turns selected belongings into fragments of identity while presenting departure as a form of freedom.The packed objects include a photograph, notebook, locket, clothing, soil, passport, and map.
- Departure narrative: The same narrative contrasts carried memories with a house, furniture, key, pet, routines, and unresolved grief left behind.The passage frames departure as severing ties to a former life.
- Lighthouse narrative: The lighthouse story establishes mystery through a storm, an unfamiliar bottle, an unknown script, and the warning that the light is not what it seems.The message also connects the mystery to Elias’s vanished wife and the lighthouse’s flickering light.
- Lighthouse narrative: Its prose builds atmosphere through sensory descriptions of sea spray, wind, iron rails, salt, and the lighthouse’s movement during the storm.The scene’s tension increases as the light flickers independently of the storm.
C Ablation Study: Style Control
The ablation study evaluates style control through rubrics emphasizing plain, authentic, emotionally restrained writing rather than polished or technically correct prose. The examples contrast Rubicon-preview’s narrative with a more formulaic response.
- Style definition: The Plain Narrative style is characterized by simple, restrained language and quiet resilience.The rubric evaluates whether the model adopts this specific narrative style.
- Evaluation principle: Style evaluation prioritizes authenticity and whether a response feels right over literary polish or technical correctness.Artificiality, including “AI-speak” and “preachy-speak,” is treated as a major failure.
- Core Evaluative Criteria: Relational efficacy is assessed through calm acceptance, grounded realism, and understated emotion in the narrative voice.These indicators emphasize concrete physical details and restrained expression of complex feelings.
- Core Evaluative Criteria: Intellectual contribution measures whether a narrative contains substantive insights, novel reframing, or transformative potential.The criterion focuses on the cognitive impact of embedded ideas.
- Compositional Excellence: Compositional excellence evaluates colloquial simplicity, textured sensory details, and natural rhythm without overly polished prose.The rubric also rejects clichés, generic life statements, and formulaic responses.
- Evaluation format: The evaluation output records each criterion’s identifier, detailed rationale, and score on a 1–5 scale.Results are required in a JSON object.
- Output examples: One comparison contrasts an autobiographical-sounding Chinese response with a direct acknowledgment that an AI lacks real parental experience.The examples differ in whether they present invented personal experience or explicitly disclose the model’s limitations.
- Output examples: The Chinese example combines emotional family memories with advice about material care and overprotection’s possible effect on independence.The passage presents both narrative detail and generalized parenting commentary.