JusticeAxis: Benchmarking Legal Judgment between Rigid Rule Application and Ungrounded Discretion

1. INTRODUCTION

• Existing legal judgment systems face a challenge because the same charge can lead to markedly different dispositions and sentences depending on culpability, harm, and local practice.The introduction identifies quantitative comparison of systems' positions between rigid rules and ungrounded discretion as an open question.Source: §1 · p. 1
• The introduction reports that open-weight models tend toward unsupported grounds, while commercial models tend toward the statute's default outcome.It says these failures divide by scale and diverge from what a court would decide.Source: §1 · p. 1
• JusticeAxis and JusticeAgent address how models can apply law to established facts while weighing circumstances, and how that balance can be assessed.The introduction presents this as the central research question.Source: §1 · p. 1
• JusticeAxis supplies three lawyer-written judgments per case: the court's recorded decision and references for rigid rule application and ungrounded discretion.The benchmark contains 256 real-world criminal cases from 18 countries.Source: §1 · p. 1
• The proposed JusticeAgent assigns agents to offence elements to establish facts on a shared graph, then has a judge agent apply law under verified experience.The introduction describes it as a plug-in harness that can lift a frozen open-weight backbone to commercial level.Source: §1 · p. 1

Agents for legal judgment

• Legal judgment systems have progressed from prompting one model over retrieved statutes and precedents to courtroom debate, element-based agents, and evolving skill libraries.JusticeAgent builds on both element-based and skill-library approaches.Source: §2 · p. 2

Benchmarks for legal reasoning

• Legal benchmarks expanded from charge and article classification to task suites, exam-style reasoning, cross-jurisdictional text, and citation grounding.Released sets are described as almost entirely textual, while courtroom-speech corpora serve conversation analysis and outcome prediction rather than judgment.Source: §2 · p. 2
• Existing benchmarks score a produced label or rubric but lack a written per-case judgment for each failure mode, leaving the balance unmeasurable.The passage identifies this as the gap JusticeAxis addresses.Source: §2 · p. 2

Reference-anchored legal judgment

• A system receives available audio, keyframes, text evidence, case background, and candidate statutes and precedents, then generates a charge, disposition, sentence, and reasoning citing its grounds.Missing modalities are allowed.Source: §3 · p. 2
• Each case has references for the court's recorded decision, rigid application of the matched statute, and unsupported legal grounds drawn from the narrative.A judgment is placed nearest to one of the three references under an outcome distance; signed misses indicate which failure direction it takes.Source: §3 · p. 2

The JusticeAxis benchmark

• JusticeAxis comprises 256 concluded criminal cases spanning 18 countries and 18 charge types, with recordings or footage, written descriptions, case background, and legal materials.Recorded outcomes are withheld from solver input, and reconstructed background is marked with its source and inference limits.Source: §3 · p. 2
• Law professors and practising lawyers annotate evidence as atomic fact units and write three reference judgments with reasoning, while a second annotator reviews every case.Answers are sealed during inference.Source: §3 · p. 2

Metrics

• The benchmark scores charges by exact-match and family-level accuracy and outcomes by disposition and exact-sentence accuracy, including conditional scores when the coarser decision is correct.Sentence distance uses the log-difference between sentence lengths, while argumentation is constrained rather than scored, so fluency earns no credit.Source: §3 · p. 2

4. JUSTICEAGENT

• JusticeAgent wraps a frozen backbone in a harness that separates fact establishment from weighing circumstances.The paper motivates the separation by noting that reading the record once establishes no facts and weighs no circumstances.Source: §4 · p. 2

4.1. Element Graph and Fact-Finding

• The Element Graph is a directed acyclic graph whose nodes are offence elements and whose edges encode legal dependencies.Node attributes record the element, gathered evidence, a finding with its evidence chain, and an open, found, or unestablished status; the graph is closed when no node remains open.Source: §4.1 · p. 2
• The harness maintains the graph, skill library, transcription, keyframe, retrieval, and alignment tools, and verifiers.These are the callable resources held by the environment.Source: §4.1 · p. 2
• The orchestrator makes one atomic graph edit per turn and dispatches an element node to an agent, which returns a finding with its evidence chain.Each dispatched agent issues at most K tool calls.Source: §4.1 · p. 2

4.2. Adjudication under Experience

• The judge applies the law to a closed graph and legal materials, using contextual skills to weigh circumstances within a syllogistic, evidence-grounded reasoning form.The context comprises jurisdiction, offence family, and seriousness band; infeasible judgments are rewritten.Source: §4.2 · p. 3

4.3. Evidence-Driven Skill Evolution

• Recurring evidence-to-finding routes become fact-finding skills, while recurring circumstance–outcome associations become contextual skills stored with their context.Each skill invocation is independently checked by a verifier that does not see the judgment score, so a lucky outcome earns no credit.Source: §4.3 · p. 3
• A hierarchical Beta prior lets sparse contexts borrow evidence across a skill’s contexts, and credible-bound decisions retain, defer, refine, or prune skills.Skills may also be split when their reliability differs across contexts, or generated when none meets the reliability threshold.Source: §4.3 · p. 3
• A candidate skill library is accepted only when its paired mean gain on held-out development cases is non-negative for Acc, Disp, and Sent.Scores gate the library as a whole, not individual skills.Source: §4.3 · p. 3

JusticeAgent performance

• JusticeAgent raises Qwen3.8-27B Acc from 47.7 to 78.9 and J∗share from 54.7% to 85.2%, while Pol. shifts from −0.34 to +0.26.The shift removes the unsupported-grounds failure rather than trading it for the rigid one; it leads peer frameworks on Acc, Avg, and J∗share, though margins are inside Wilson intervals at 128 cases.Source: §5 · p. 4

Where systems sit on the axis

• The three open-weight backbones have negative Pol. and unsupported-ground misses, while commercial models and peer frameworks have positive Pol. and statutory-default misses.Placement and charge accuracy rank-correlate at 0.98 across fourteen systems; Acc/Fam exceeds 82% throughout, while Sent/Disp is under half for open-weight backbones and 78–88% for commercial models.Source: §5 · p. 4

Ablation

• Removing audio in favor of its written description costs 24.0 points of Avg, more than twice any other component; removing the graph or contextual skills shifts misses toward different failure poles.Without the graph, misses drift to unsupported grounds; without contextual skills, they drift to the statutory default, and a frozen library loses most of the skills’ gains.Source: §5 · p. 4, Table 3 · p. 4

Across backbones

• On three further backbones, Gemma 4 31B gains 39.1 Acc with JusticeAgent, and Claude Sonnet 5 reaches 93.8 Acc, above every stand-alone Table 2 model.Sentence performance moves most on all three backbones.Source: §5 · p. 4, Figure 3 · p. 4

• Adjudication is formalized as a judgment anchored between rigid rule application and ungrounded discretion, scored against three lawyer-written references.Source: §6 · p. 4
• The reported failures divide by scale, and JusticeAgent closes part of the gap with an Element Graph and verified experience.Source: §6 · p. 4
• The benchmark is retrospective and criminal only, and the harness is framed as decision support rather than a court.Source: §6 · p. 4

• The study was retrospective and used human-subject data made available in open access by courts and public agencies.Source: §7 · p. 5
• The source materials included recordings, footage, and court records released publicly by courts and agencies.Source: §7 · p. 5
• The authors report that ethical approval was not required and that participants are aliased in all annotations.Source: §7 · p. 5