PaperlayerView exact revision on arXiv

Source-linked AI summary

JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Prediction

Zhaolu Kang, Yantao Liu, Tailong Luo, Leqi Zheng, Lei Wei, Chenghua Zhu, Junhao Gong, Jiachen Qian, Eric Hanchen Jiang, Jiaxin Liu, Yuan Wang, Hao Zhang, Zixia Wang, Rong Fu, Zheng Lin, Richeng Xuan, Zhichao Hu

arXiv:2608.29616v1cs.CL

TL;DR

Criminal judgment prediction must connect facts, statutes, charges, and sentences coherently, whereas existing methods often optimize final labels and assess reasoning indirectly. JPO combines teacher-supervised four-step rationales with reinforcement learning over legal accuracy, reasoning completeness, and cross-step consistency, and experiments show consistent gains in judgment prediction and reasoning quality across three Chinese benchmarks and multiple open-source backbones. Its automatic rewards remain approximations of professional legal judgment and its process metrics are partly coupled to JPO’s output structure.

  • Problem

    Criminal judgment prediction requires coherent links among facts, statutory articles, charges, and sentencing, but existing methods and evaluations often emphasize final labels or indirect LLM-generated rubrics.

  • Method

    JPO uses teacher-generated rationales for standardized fact extraction, statutory analysis, charge determination, and sentence prediction, followed by reinforcement learning with composite rewards and legally salient token reweighting.

  • Results

    JPO consistently improves judgment prediction and reasoning-oriented metrics over supervised fine-tuning and standard reinforcement-learning baselines across multiple open-source backbones and three Chinese legal benchmarks.

  • Takeaways & Limitations

    The results support optimizing intermediate dependency structures connecting facts, statutes, charges, and sentencing outcomes alongside final judicial answers.

  • Takeaways & Limitations

    JPO relies on approximate automatic rewards, and its 4-Step Completeness and Full-Chain Consistency metrics are coupled to its output structure and therefore treated as auxiliary evidence.

Abstract

from arXiv · show

Criminal judgment prediction requires models to infer statutory articles, charges, and sentencing outcomes from case facts. Unlike standard classification tasks, it involves a structured reasoning process in which statutes should be matched with facts, charges should be justified by statutes, and sentencing outcomes should remain consistent with charges. Existing approaches optimize final labels, and while some have attempted to evaluate reasoning quality, their evaluations are indirect, often relying on LLM-generated rubrics that reflect model-internal preferences rather than the inherent logical structure of legal adjudication. We propose Juris Policy Optimization (JPO), a post-training framework for structured legal reasoning in Chinese criminal judgment prediction. JPO first uses teacher-generated rationales to supervise a standardized four-step reasoning process, and then applies reinforcement learning with a composite reward over legal prediction quality, reasoning structure completeness, and cross-step consistency. JPO further introduces token-level advantage reweighting and adaptive clipping for legally salient reasoning segments. Experiments on multiple open-source language models and three Chinese legal benchmarks show that JPO consistently improves both judgment prediction and reasoning quality over supervised fine-tuning and reinforcement learning baselines.

1 Introduction

Criminal judgment prediction requires coherent reasoning from facts through statutes and charges to sentencing, but existing methods and evaluations often focus on final labels or indirect rubrics. JPO addresses this gap with structured four-step supervision and reinforcement learning that improves prediction and reasoning quality.

  • Criminal judgments require statutes, charges, and sentences to remain coherent with the legally relevant case facts.
  • Existing approaches primarily optimize final labels, while reasoning evaluations often rely on LLM-generated rubrics rather than legal adjudication’s structured dependencies.
  • JPO uses teacher-generated rationales to supervise four steps: fact extraction, statutory analysis, charge determination, and sentence prediction.
  • JPO’s composite reinforcement-learning reward jointly targets legal prediction accuracy, reasoning completeness, and cross-step consistency.
  • JPO’s computable rewards correlate substantially with expert judgments while remaining approximations rather than a complete account of legally grounded reasoning.The reported Spearman ρ range is 0.64–0.72 across expert-rated reasoning steps.
  • Experiments across multiple open-source backbones and three Chinese legal benchmarks show consistent improvements in judgment prediction and reasoning quality over strong baselines.The reported gains are especially clear on compact models.

2 Related Work

Legal judgment prediction has traditionally emphasized final-label benchmarks, while newer work seeks process-oriented reasoning and post-training alignment. JPO is situated among these approaches through structured supervision, rule-based rewards, and multi-stage legal reasoning.

  • Legal judgment prediction infers applicable statutes, charges, and sentencing results from case facts, with Chinese benchmarks establishing standard prediction settings.
  • Most existing approaches supervise final labels without explicitly optimizing a standardized multi-stage reasoning process.
  • Structured SFT uses teacher-generated reasoning traces, while JPO adds multi-dimensional rewards and token-level advantages during post-training.
  • Unconstrained rationale generation does not by itself ensure legally grounded reasoning or coherence across reasoning steps.
  • Related compliance methods apply structured traces, checklists, or rule-based rewards to policy, privacy, and safety reasoning rather than criminal adjudication.

3 Method

JPO is a two-stage framework that combines structured four-step supervision with reinforcement learning rewards for legal correctness, reasoning completeness, and cross-step consistency.

  • Stage I: Structured Supervised Fine-Tuning: JPO trains models to generate fact extraction, statutory analysis, charge determination, and sentence prediction alongside final judgments.A teacher-generated reasoning trace provides the four-stage structure during supervised fine-tuning.
  • Stage II: Multi-Dimensional Rewards: The reinforcement learning stage jointly rewards legal prediction quality, structural completeness, and consistency across adjacent reasoning steps.The reward covers fact-to-article, article-to-charge, and charge-to-sentence transitions.
  • Legal Prediction Reward: Legal prediction rewards assess response format, statutory articles, charges, and sentencing outcomes, using graded relative-error supervision for sentences.The sentence reward preserves graded supervision when the predicted sentence is only partially accurate.
  • Structural Completeness Reward: Structural completeness rewards require all four reasoning stages and discourage empty or trivial segments through segment-length terms.The indicator term checks segment presence, while the length term penalizes insufficient generation.
  • Cross-Step Consistency Reward: Consistency rewards connect facts to articles, articles to charges, and charges to plausible sentences using computable proxy scores.The article–charge association uses a rule-based mapping, while charge-conditioned sentence plausibility is estimated from training-data distributions.
  • Token-Level Advantage Reweighting: Token-level advantage reweighting assigns unequal learning signals to tokens according to importance signals and their reasoning segments.The reasoning trace is segmented into fact extraction, statutory analysis, charge determination, and sentence prediction.
  • Adaptive Policy Clipping: Token-aware clipping replaces uniform thresholds so legally important tokens can receive larger updates while less informative tokens remain more conservatively optimized.The base clipping coefficient sets the reference threshold, and adaptation strength controls token-level adjustment.

4 Experiments

JPO is evaluated across five open-source backbones and three Chinese criminal judgment benchmarks using both prediction and reasoning-oriented metrics. Across these settings, JPO improves final judgment quality, reasoning coherence, and local reasoning transitions over supervised and reinforcement-learning baselines.

  • Experimental setup: Experiments cover JPO-Dataset, CAIL2018, and LawBench across five open-source backbones, with additional baseline comparisons on representative 3B models.The evaluated backbones include Qwen2.5-3B/7B-Instruct, Qwen3-4B-Instruct, Llama-3.2-3B-Instruct, and Llama-3-8B-Instruct; the 3B studies add RL-only, Vanilla PPO, Legal∆, and Issue Tree Rubrics.
  • Experimental setup: Evaluation reports article F1, charge F1, sentence score, 4-Step Completeness, and Full-Chain Consistency.The final two metrics measure inclusion of all four reasoning stages and coherence across fact extraction, statutory analysis, charge determination, and sentence prediction.
  • Main results: JPO consistently improves prediction and reasoning metrics over pre-trained and structured SFT baselines across all backbones and datasets.Improvements include article, charge, and sentence prediction as well as 4-Step Completeness and Full-Chain Consistency.
  • Main results: Structured SFT substantially improves over pre-trained models, while JPO adds clear gains over SFT in legal grounding and cross-step consistency.The results support the paper’s two-stage design: structured initialization followed by policy optimization.
  • Detailed analysis: The gains are especially consistent on compact and midsized models, sentence prediction, and Full-Chain Consistency.On two representative 3B backbones, JPO also improves fact-to-article, article-to-charge, and charge-to-sentence consistency, with the largest gains in the final transition.
  • Detailed analysis: Ablations show that reward components, token-level advantage reweighting, and adaptive clipping each contribute to performance.Removing structural reward most affects completeness, while removing consistency reward most degrades sentence score and Full-Chain Consistency.

5 Conclusion

JPO is a two-stage post-training framework that aligns Chinese criminal judgment models with both final outcomes and the reasoning dependencies connecting facts, statutes, charges, and sentences. Across three benchmarks and multiple open-source backbones, it consistently improves judgment prediction and reasoning metrics over strong baselines.

  • Conclusion: JPO combines structured four-step supervision with reinforcement learning to optimize prediction quality, reasoning completeness, and cross-step consistency.Token-level advantage reweighting and adaptive policy clipping further target legally salient segments in long responses.
  • Conclusion: JPO’s results support optimizing intermediate legal dependency structure alongside final answers in criminal judgment prediction.The targeted dependencies connect facts, statutory articles, charges, and sentencing outcomes.

Limitations

JPO’s evaluation is bounded by proxy-based reasoning rewards, a narrow criminal-law setting, and unresolved weaknesses in multi-party and counterfactual reasoning. These limitations leave broader legal validation and deployment outside the paper’s demonstrated scope.

  • Domain scope: The study focuses on Chinese criminal judgment prediction and excludes procedural disputes, civil claims, and complex multi-party cases.This setting provides a controlled testbed but does not cover all forms of judicial decision-making.
  • Multi-party generalization: On held-out multi-charge and multi-defendant subsets without retraining, both SFT and JPO degrade substantially, especially for multi-defendant cases.JPO retains a margin over SFT, but the absolute drop indicates unresolved headroom and the need for per-defendant reasoning chains and charge-level consistency terms.
  • Reward and evaluation scope: JPO’s automatic rewards and process metrics approximate rather than fully capture expert legal judgment.The authors treat format-independent outcome metrics as primary because 4-Step Completeness and Full-Chain Consistency are coupled to JPO’s output structure.
  • Supervision and reasoning reliability: Teacher-generated rationales may contain imperfections and post-hoc rationalization because the teacher observes the gold judgment.Counterfactual testing finds greater responsiveness to legally decisive facts than SFT or the pretrained model, but some cases still fail to update as required.
  • Deployment boundary: Real-world legal deployment would require human oversight, domain-specific validation, and institutional safeguards.The paper frames JPO as a research framework rather than an autonomous legal decision system.

Ethics Statement

The ethics statement addresses privacy, bias, and the limits of computational legal reasoning. Public judicial documents are processed for research, but JPO is not presented as a substitute for expert judgment or autonomous legal practice.

  • Data risks: The publicly available judicial data may still contain sensitive information and historical biases.The study uses public criminal judgment documents, but public availability does not eliminate these ethical risks.
  • Interpretive limits: JPO’s structured rationales and reasoning rewards are weak supervision and computational proxies, not authoritative legal explanations.They are not substitutes for expert legal judgment.
  • Responsible use: JPO should not be used as an autonomous decision system in real legal practice.Practical use would require careful human oversight and broader validation.

B JPO-Dataset Construction

JPO-Dataset is a recent, normalized benchmark built from public Chinese criminal judgments and organized around facts, statutory articles, charges, and sentences. Its construction emphasizes consistent targets, leakage control, and a structured four-step reasoning interface.

  • Dataset motivation: JPO-Dataset uses public Chinese criminal judgments from 2024–2026 to reflect newer case distributions and legal expressions than earlier benchmarks.The dataset is intended as a cleaner, more recent testbed rather than one maximizing label coverage.
  • Field construction: Each case is normalized into fact descriptions, statutory articles, charges, and sentencing outcomes.Article references, charge expressions, and sentencing expressions are mapped to unified targets for consistent prediction.
  • Filtering and scope: The dataset currently restricts cases to single defendants with sufficiently complete facts and explicit judicial outcomes.It excludes missing-key-field, duplicate, severely truncated, multi-defendant, highly atypical, and complex procedural cases.
  • Target normalization: Sentencing outcomes are converted into normalized scalar values for training and evaluation.This resolves surface-form variation such as different expressions of imprisonment length and equivalent textual formulations.
  • Reasoning supervision: The SFT split teaches the four-step reasoning template, while the RL split supports composite-reward post-training.Teacher rationales contain fact extraction, statutory analysis, charge determination, and sentence prediction aligned with the final answer block.
  • Preprocessing controls: Input processing caps all datasets at 2,048 tokens, with truncation affecting fewer than 0.1% of JPO-Dataset cases, 4.2% of CAIL2018, and 1.3% of LawBench.The same tail-truncation policy is applied across training and evaluation.
  • Reproducibility: JPO-Dataset and related artifacts are released to support reproducibility.Released materials include dataset splits, training and evaluation code, teacher prompts, consistency-scoring artifacts, and additional evaluation protocols.
  • Output and evaluation interface: The unified response schema combines a four-step reasoning trace with normalized article, charge, and sentence predictions.Parsing maps outputs into the dataset label space, while schema failures receive lower format rewards and may yield zero component scores.

F Implementation Details of Consistency Scoring

JPO implements three local consistency scores that approximate dependencies from facts to articles, articles to charges, and charges to sentences. These signals support scalable policy optimization but are not complete models of legal doctrine or expert reasoning.

  • Consistency components: JPO defines fact-to-article, article-to-charge, and charge-to-sentence consistency scores.They are denoted SFA, SAC, and SCS and correspond to successive links in the legal reasoning chain.
  • Fact-to-article scoring: SFA uses a Naive Bayes article predictor trained on fact-side lexical features to estimate whether predicted articles are supported by facts.It is a lightweight consistency signal, not a standalone article predictor.
  • Article-to-charge scoring: SAC uses a rule-based article–charge association matrix derived from statutory references and charge annotations.The matrix is intended as a compatibility table rather than a complete codification of legal doctrine.
  • Charge-to-sentence scoring: SCS estimates a charge-conditioned sentence distribution to measure whether a sentence lies in a plausible region for the predicted charge.It is a statistical plausibility signal rather than a normative sentencing model.
  • Interpretive boundary: All three consistency scores are computable proxies that support policy optimization and reasoning-oriented evaluation.They are not intended to fully represent expert legal analysis or replace professional legal judgment.
  • Experimental control: Post-training comparisons use the same structured output schema and normalized target space, with reinforcement learning initialized from corresponding SFT checkpoints.This design focuses comparisons on post-training objectives rather than interface or initialization differences.
  • Baseline interpretation: Proprietary systems are reported only as zero-shot references, not directly comparable post-training baselines.They are not post-trained on JPO-Dataset.

H Definitions of Reasoning-Oriented Metrics

The paper evaluates reasoning beyond final judgment labels using structural completeness and cross-stage coherence metrics, while treating them as process signals rather than substitutes for expert legal evaluation.

  • Process Metrics: 4-Step Completeness measures whether each response explicitly contains non-empty fact extraction, statutory analysis, charge determination, and sentence prediction stages.The score is the evaluation-set average completeness rate.
  • Process Metrics: Full-Chain Consistency averages fact-to-article, article-to-charge, and charge-to-sentence consistency scores across evaluation samples.The three local scores are denoted SFA, SAC, and SCS.
  • Metric Interpretation: Article F1, charge F1, and sentence score evaluate final outcomes, whereas the process metrics assess reasoning structure and coherence across stages.Full-Chain Consistency uses three local consistency proxies corresponding to adjacent reasoning transitions.
  • Metric Interpretation: Because both process metrics are coupled to JPO’s output structure, the paper treats format-independent outcome metrics as primary evidence and process metrics as auxiliary.The authors address this concern with robustness checks, expert-rating correlations, and format-agnostic judging.

I Training Dynamics

JPO’s training dynamics separate format acquisition from later legal refinement: supervised fine-tuning improves validity and coarse predictions, while reinforcement learning produces larger gains in sentence prediction and consistency.

  • Training Dynamics: During SFT, output validity and coarse legal prediction improve first as structured supervision teaches the desired reasoning template.The SFT stage mainly stabilizes the reasoning format and exposes intermediate legal steps.
  • Training Dynamics: During RL, the largest gains shift to sentence prediction and Full-Chain Consistency, extending beyond template acquisition toward legal coherence and decision calibration.The paper identifies sentence prediction and Full-Chain Consistency as the most visible RL-stage improvements.
  • Consistency Ablations: Removing charge-to-sentence consistency causes the largest sentence-prediction drop, while removing fact-to-article consistency most directly harms statutory grounding.The ablation indicates that different consistency transitions affect different parts of the judgment chain.
  • Sensitivity Analysis: The best reward composition keeps legal prediction dominant while using structure and consistency as auxiliary guidance; overemphasizing one component weakens overall performance.This supports balancing final-answer supervision with process-level reasoning supervision.
  • Sensitivity Analysis: Moderate token-level advantage scaling performs best, whereas no scaling or excessive scaling produces weaker results.The result supports importance-aware but conservative token-level optimization.
  • Evaluation Caveats: Sentencing comparisons across papers are not directly absolute because methods use different metrics, including top-1 accuracy, Macro-F1, and continuous sentence scores.The paper also distinguishes per-label and per-case Macro-F1 conventions.

K.3 Pilot Human Evaluation of Reasoning Quality

The pilot evaluation finds that JPO improves human-rated quality across all four reasoning stages, with the largest gain in sentence prediction, while broader evidence frames process metrics as useful but imperfect proxies.

  • Pilot Design: A blinded pilot evaluated four binary reasoning-step ratings on 400 randomly sampled test cases, comparing Qwen2.5-7B-Instruct JPO with its SFT counterpart.The evaluated stages were Fact, Article, Charge, and Sentence.
  • Pilot Results: JPO outperforms the SFT baseline across all four reasoning dimensions in the pilot evaluation.The reported dimensions are Fact, Article, Charge, and Sentence.
  • Pilot Results: +0.22 is the largest pilot improvement, occurring on Sentence prediction and aligning with SFT’s under-calibrated sentencing rationales.JPO rationales were also rated as more coherent, better grounded in statutes, and more logically connected from facts to sentencing.
  • Remaining Errors: JPO’s remaining failures center on secondary article omission, fine-grained sentence calibration, and confusion between neighboring charges with similar facts.The paper characterizes fine-grained legal distinction, rather than gross formatting failure, as the main remaining challenge.
  • Proxy Validation: Automatic consistency proxies correlate with expert ratings from ρ = 0.64 to ρ = 0.72, but their sub-unity correlations confirm they remain approximations.Charge-to-sentence consistency has the lowest reported correlation because case-specific mitigating and aggravating circumstances are difficult to represent statistically.
  • Structured Reasoning: The four-step scheme is strongest overall against free-form CoT, especially on sentence prediction and Full-Chain Consistency, but its benefits are expected to depend on tasks with this dependency structure.The authors do not claim universal optimality.

O.6 Within-Subjects Expert Evaluation

The larger within-subjects study and supporting analyses independently corroborate JPO’s reasoning gains, while ablations attribute improvement to legal-specific reward shaping and token optimization rather than generic components alone.

  • Expert Evaluation: Three legal experts rated pretrained, SFT, and JPO four-step traces on 800 cases under blinded, randomized within-subjects evaluation.The study uses a shared protocol independent of the reward.
  • Expert Evaluation: Fleiss’ κ = 0.73 indicates substantial inter-annotator agreement, and JPO’s consistent superiority corroborates genuine reasoning-quality differences.The largest gap is on sentence prediction, matching the automatic-metric pattern.
  • Component Analysis: The legal composite reward adds 0.035 sentence score and 0.057 Full-Chain, while legal-logic-aware token optimization adds a further 0.035 and 0.044.Generic entropy-only token weighting adds only 0.012 sentence score on its own.
  • Teacher Robustness: Across five teachers, final JPO models span 0.930–0.941 article F1, 0.919–0.930 charge F1, and 0.532–0.558 sentence score.The results indicate robustness to teacher choice, while stronger teachers mainly raise the SFT starting point.
Loading 2608.29616v1…