LexReward: A Taxonomy-Driven Reward Framework for Legal Language Models

• Legal responses can reach correct conclusions while omitting decisive facts, citing inappropriate statutes, or using incomplete reasoning, which final-answer accuracy alone cannot capture.Source: §1 · p. 1
• LexReward defines Style, Element, and Chain dimensions and refines fine-grained criteria using expert knowledge and model-response error analysis.Style covers lexical and syntactic properties; Element addresses legal subjects, facts, statutes, and decisions; Chain assesses the order, completeness, correctness, and non-redundancy of reasoning.Source: §1 · p. 2, Figure 2 · p. 3
• Rubric-based rewards use rule-based or LLM-as-a-judge evaluators and reliably distinguish response quality along their corresponding dimensions.Source: §1 · pp. 2–3
• Rubric-derived preference data improve legal response generation through DPO, and LexRM supports response selection and dimension-specific reinforcement-learning optimization.The introduction also reports LexRM-guided selection outperforming random selection and GRPO gains across dimensions over rule-based outcome rewards.Source: §1 · pp. 2–3
• LexReward’s contributions include a Chinese legal reward-model family and integration of learned rewards into a GRPO pipeline to improve policy generation quality.Source: §1 · p. 3

2.1 LEGAL LANGUAGE MODELS AND DOMAIN-SPECIFIC EVALUATION

• Task-level legal benchmarks evaluate knowledge, rule understanding, and reasoning, but do not fully capture response-level expression, legal-element coverage, statutory grounding, and reasoning coherence.Source: §2.1 · pp. 3–4

2.2 RUBRIC-BASED REWARD MODELING

• Reward modeling progressed from holistic human preferences to finer-grained supervision over errors, segments, and multiple objectives (Wang et al., 2024).Source: §2.2 · p. 4
• Rubric-based evaluation makes criteria and quality levels explicit, while prior legal process-reward work targets criminal-law knowledge-graph reasoning as a specific task (Chen et al., 2026).LexReward builds on structured supervision by defining legal reward dimensions and criteria, then operationalizing them into rubrics and cross-task preference data for reward-model training and reinforcement learning.Source: §2.2 · p. 4

• Experts combine top-down legal requirements with bottom-up analysis of model responses against reference answers, then integrate and refine the identified requirements into a unified taxonomy.The analysis spans responses from multiple models across legal AI benchmarks.Source: §3 · p. 4
• Style assesses lexical and syntactic quality, Element assesses inclusion of legally relevant subjects, facts, statutes, and decisions, and Chain assesses reasoning links to a conclusion.Source: §3 · p. 4
• Element measures coverage of relevant information, whereas Chain evaluates whether inferential steps are ordered, complete, legally valid, and non-redundant.Source: §3 · p. 4
• The three dimensions organize expression, substantive content, and reasoning into explicit rubric criteria for fine-grained assessment and reward modeling.Source: §3 · p. 4

4.1 RUBRIC-BASED REWARD OPERATIONALIZATION

• Evaluation strategy depends on context: Style uses rules, Element uses an LLM judge, and Chain uses rules for predefined reasoning structures or a judge when the path must be inferred.Context-independent criteria use deterministic scores calibrated against high-quality legal texts; context-dependent criteria use criterion-specific rubrics to assess the input–response relationship.Source: §4.1 · p. 5
• Style scores similarity to authentic Chinese judicial writing across five lexical and syntactic attributes using normalized feature discrepancies.The attributes are word specificity, subjective word control, sentence cohesion, sentence structure, and collocation correctness; the reward is the negated mean normalized discrepancy.Source: §4.1 · p. 5
• Element averages 0–1 judge scores for subjects, facts, statutes, and decisions, assessing task fit, accuracy, coverage, specificity, and unsupported content without a reference answer.Source: §4.1 · p. 6
• Chain evaluates reasoning order, completeness, correctness, and non-redundancy, using prescribed steps when available and an LLM judge otherwise.Source: §4.1 · pp. 5–6

4.2 REWARD MODEL TRAINING FROM RUBRIC-GUIDED PREFERENCES

• Because rubric scores can tie across responses, the authors build within-query preference pairs across high, medium, and lower score ranges to train a separate reward model per dimension.The preference dataset contains query, preferred-response, and less-preferred-response tuples; Bradley–Terry training encourages higher scalar scores for preferred responses.Source: §4.2 · p. 6
• Dimension-specific updates have low pairwise cosine similarity, motivating task-arithmetic composition into one reward model that scores all dimensions without running three backbones at inference.Across backbone weight matrices, task-vector cosine similarities average at most 0.03 in absolute value and never exceed 0.14.Source: §4.2 · p. 6

5 EXPERIMENTS AND RESULTS

• The evaluation covers rubric-based and reward-model selection among five-model candidates, followed by DPO and reinforcement learning from an SFT checkpoint.The selection setting also includes MoE-based test-time scaling.Source: §5 · p. 7

5.1.1 DATASETS

• Style uses CLASE, Element draws on criminal questions from JEC-QA and civil judgments from LexChain, and Chain uses LexChain; each has a specified training and evaluation set.The datasets include 4,000 Style training instances and a 1,000-instance official test set; Element uses 2,244 preference pairs from 2,736 instances, and Chain has 9,550 training instances and a 1,000-instance official test set.Source: §5.1.1 · p. 7

5.2.2 DPO AND RL EVALUATION

• GRPO (RM) achieves the highest score on every dimension, with Element’s average rising 6.04 points over SFT.The downstream policy table covers all three LexReward dimensions.Source: Table 2 · p. 8, §5.2.2 · p. 9

5.1.2 MODELS

• Qwen3-8B is the backbone for trained models, while five candidate-generation models and four reward-model or legal-language-model baselines are used in evaluation.DeepSeek-V4-Flash supplies the LLM-based components of the rubric evaluators.Source: §5.1.2 · p. 8

5.2.1 RUBRIC AND RM EVALUATION

• Rubric rewards outperform five-model averaging and random selection on all three datasets; LexRM-Style and LexRM-Chain lead their datasets, while LexRM-Element ranks second.LexRM uses substantially less training data than the Skywork reward models; the merged model matches the Element expert on Legal∆ but falls below the experts on CLASE and LexChain.Source: §5.2.1 · p. 8

5.2.2 DPO AND RL EVALUATION

• DPO improves over the vanilla model across all three dimensions; GRPO (rule) yields smaller Style gains than GRPO (RM) and leaves Element and Chain averages largely unchanged.GRPO (rule)’s gains are confined to the final answer, while dimensions grounding that answer, such as Legal Basis, stagnate or decline.Source: §5.2.2 · pp. 8–9

5.2.3 DIMENSION ABLATION ANALYSIS

• Aggregating criteria gives the best overall score in all three dimensions, while no single criterion consistently performs best across tasks.Within Element, adding statutes restores statute prediction after a subjects-only rubric misses cited provisions; Chain’s aggregate gains concentrate in later reasoning stages.Source: §5.2.3 · p. 9

6 CONCLUSION

• LexReward characterizes legal response quality through the complementary dimensions Style, Element, and Chain.The dimensions cover stylistic expression, substantive legal content, and reasoning.Source: §6 · p. 10
• Fine-grained rubrics distinguish responses of different quality and produce preference data that improve legal response generation through DPO.Source: §6 · p. 10
• LexRM supports multimodel response selection and reinforcement-learning rewards for separately optimizing policies in Style, Element, and Chain.Source: §6 · p. 10

AI USE STATEMENT

• Generative AI assisted with language refinement, partial code implementation, preference-data generation, and interpretation of results.Source: AI USE STATEMENT · p. 10
• The authors state that AI tools were not used to develop theoretical models or conceptual frameworks or to formulate mathematical claims or essential proof components.Source: AI USE STATEMENT · p. 10
• The authors reviewed and validated AI-assisted outputs and take responsibility for the final content and artifacts.They report checking refined text, testing generated code and data, and independently verifying interpretations.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Section 3 describes the legal quality taxonomy, and Section 4 specifies rubric operationalization, preference-data construction, and reward-model training objectives.Source: REPRODUCIBILITY STATEMENT · p. 10
• Section 5 presents the experimental datasets, models, and evaluation protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
• Appendix A covers rubric-reward implementation; Appendix B documents data statistics, preference-data construction, and training, inference, and evaluation configurations.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 STYLE

• Style references authentic Chinese judgments for lexical statistics, estimates normalization scales separately, and averages its five normalized rewards equally.The reference statistics use 1,000 CJO judgments; normalization scales use the LY fields of 986 CJO documents.Source: §A.1 · pp. 12, 14, Table 4 · p. 13
• Word specificity uses legal-term counts to form a response distribution, assigns absent terms 10^-10, and measures divergence against reference frequencies.Legal terms are identified with a domain-specific lexicon and forward maximum matching.Source: §A.1 · pp. 12–13
• Subjective-word control compares sentiment-word distributions, while sentence cohesion measures deviation from the reference conjunction rate.The former uses the Chinese HowNet sentiment lexicon; the latter normalizes absolute deviation from the reference value.Source: §A.1 · p. 13
• Pronoun frequency is omitted because preliminary experiments found it provided little discrimination between authentic and generated responses.Source: §A.1 · p. 13
• Sentence structure compares response sentence-length statistics with reference statistics and normalizes their Euclidean distance.Sentence lengths are measured in Chinese characters; the normalization scale is 26.0567.Source: §A.1 · p. 13
• Collocation correctness measures response coverage of retained 3- to 6-grams; bigrams are excluded because they include many generic or non-legal combinations.Reference n-grams must occur in at least ten documents.Source: §A.1 · pp. 13–14

A.2 ELEMENT

• The Element evaluator receives task context, a candidate response, and element-specific rubric criteria, then returns a structured assessment.The displayed prompt is an English translation; experiments used the Chinese original.Source: §A.2 · p. 14

A.3 CHAIN

• Chain scoring averages criterion-level rewards for order, completeness, correctness, and non-redundancy, using rule-based checks when available and an LLM judge otherwise.When predefined reasoning steps are unavailable, the LLM-based evaluation prompt is used; the displayed prompt is translated from Chinese.Source: §A.3 · p. 14

B.1 DATASETS

• For Style, 4,000 training queries supply preference data, with 2,000 reserved as the CLASE-Mix reference corpus and the remaining 2,000 split equally between SFT and GRPO.Evaluation uses the official CLASE test set of 1,000 response pairs; no model is trained on authentic judicial texts.Source: §B.1 · p. 14
• Element preference data use 2,736 JEC-QA and LexChain instances to form 2,244 pairs, and evaluation uses 3,000 Legal∆ in-domain test instances.Figure 3 presents the rubric-guided prompt for Element evaluation as an English translation of the Chinese experimental prompt.Source: §B.1 · p. 14, Figure 3 · p. 15
• For Chain, the 9,550 LexChain training instances are allocated to SFT, GRPO, and preference training, with evaluation on the official 1,000-instance test set.Figure 4 describes reasoning-step extraction for Chain evaluation when predefined steps are unavailable.Source: §B.1 · p. 14, Figure 4 · p. 16
• Across dimensions, preference pairs come from differently scored responses among five model candidates; queries with identical scores for all candidates are discarded.Source: §B.1 · p. 14

B.2 HYPER-PARAMETERS

• LLM rubric evaluation uses DeepSeek-V4-Flash with an 8,192-token input limit; candidate responses are sampled at temperature 0.8 and top-p 0.95.All inputs fit within the limit without truncation.Source: §B.2 · p. 15
• Reward models use the Bradley–Terry objective, while DPO uses the same two-epoch, learning-rate, and batch-size configuration.Both use a learning rate of 1 × 10^-7 and batch size 32; SFT, DPO, and GRPO use LoRA.Source: §B.2 · p. 16
• For each dimension, SFT precedes GRPO; GRPO (RM) uses its dimension-specific LexRM, whereas GRPO (rule) uses the baseline reward.The two GRPO variants start from the same dimension-specific SFT checkpoint; DPO is initialized directly from Qwen3-8B.Source: §B.2 · p. 16

C LIMITATIONS

• Because the taxonomy draws on Chinese legal materials, evaluation is limited to Chinese-language datasets and applicability elsewhere remains untested.Source: C LIMITATIONS · p. 17
• The study primarily evaluates dimension-specific rewards rather than a unified multi-dimension reward.Source: C LIMITATIONS · p. 17
• Combining dimensions into a unified rubric reward or reward model remains open, including choices about aggregation weights, preference-data composition, and training strategy.Source: C LIMITATIONS · p. 17