Source-linked AI summary
PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
Yuzhen Shi, Huanghai Liu, Yiran Hu, Gaojie Song, Xinran Xu, Yubo Ma, Tianyi Tang, Li Zhang, Qingjing Chen, Di Feng, Wenbo Lv, Weiheng Wu, Kexin Yang, Sen Yang, Wei Wang, Rongyao Shi, Yuanyang Qiu, Yuemeng Qi, Jingwen Zhang, Xiaoyu Sui, Yifan Chen, Yi Zhang, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Weixing Shen, Bing Zhao, Charles L. A. Clarke, Hu Wei
TL;DR
Existing legal benchmarks often simplify practice and rely on coarse evaluation, leaving ambiguity and fine-grained legal reasoning insufficiently assessed. PLawBench models realistic legal workflows with expert-designed rubrics, and its evaluation finds that current models struggle with legal knowledge and reasoning. The benchmark is grounded in Chinese legal practice, limiting generalizability across jurisdictions.
Problem
Existing legal benchmarks inadequately capture ambiguous real-world workflows, complex legal reasoning, and fine-grained evaluation needs.
Method
PLawBench evaluates consultation, case analysis, and document generation using realistic scenarios, explicit reasoning steps, and task-specific rubrics.
Results
Current models achieve suboptimal performance on PLawBench and struggle with legal knowledge and reasoning.
Takeaways & Limitations
PLawBench provides guidance for developing and evaluating domain-specific legal LLMs in realistic practice settings.
Takeaways & Limitations
PLawBench is grounded in the Chinese legal system and may have limited generalizability to other jurisdictions.
Abstract
from arXiv · showhide
As large language models (LLMs) are increasingly applied to legal domain-specific tasks, evaluating their ability to perform legal work in real-world settings has become essential. However, existing legal benchmarks rely on simplified and highly standardized tasks, failing to capture the ambiguity, complexity, and reasoning demands of real legal practice. Moreover, prior evaluations often adopt coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. To address these limitations, we introduce PLawBench, a Practical Law Benchmark designed to evaluate LLMs in realistic legal practice scenarios. Grounded in real-world legal workflows, PLawBench models the core processes of legal practitioners through three task categories: public legal consultation, practical case analysis, and legal document generation. These tasks assess a model's ability to identify legal issues and key facts, perform structured legal reasoning, and generate legally coherent documents. PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics, resulting in approximately 12,500 rubric items for fine-grained assessment. Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs. Experimental results show that none achieves strong performance on PLawBench, revealing substantial limitations in the fine-grained legal reasoning capabilities of current LLMs and highlighting important directions for future evaluation and development of legal LLMs. Data is available at: https://github.com/skylenage/PLawbench.
1 Introduction
PLawBench addresses the gap between standardized legal benchmarks and the ambiguity, complexity, and fine-grained reasoning demands of professional legal practice. It evaluates realistic workflows through authentic task settings, explicit reasoning steps, and task-specific rubrics.
- Motivation: Real legal scenarios require models to retrieve knowledge, clarify relevant facts, reason through disputes, and verify conclusions.These demands motivate evaluating whether LLMs can handle complex legal tasks like human practitioners.
- Motivation: Existing legal evaluations use coarse criteria that overlook the diversity and complexity of legal practice.Examples include final-conclusion correctness, statute citation, and syllogistic structure.
- Benchmark Design: PLawBench simulates legal practice through public consultation, practical case analysis, and legal document generation.The tasks are adapted from authentic legal workflows and include vague queries, emotional narratives, and omitted facts.
- Benchmark Design: The benchmark explicitly evaluates issue identification, fact clarification, legal analysis, and conclusion validation rather than only final answers.This design targets multi-stage legal reasoning instead of shallow pattern matching.
- Evaluation: Experts construct personalized rubrics by first defining task-level reasoning frameworks and then tailoring criteria to individual cases.This two-stage process is intended to make evaluation principled and context-sensitive.
2 Related Work
Prior legal benchmarks broadened evaluation across knowledge, reasoning, and task types, but often remain disconnected from the ambiguity and integrated reasoning of real legal work.
- Existing Benchmarks: Earlier benchmarks evaluated legal information extraction, legal reasoning, holdings, and judgment prediction through structured datasets.Examples include JEC-QA, CaseHOLD, and other classification-oriented resources.
- Existing Benchmarks: More recent benchmarks expanded coverage across cognitive levels and diverse legal reasoning tasks.LawBench, LegalBench, LexEval, and LAiW provide broader or layered evaluations at different scales.
- Remaining Gaps: Many existing benchmarks use exam-style or isolated tasks that diverge from ambiguous practitioner scenarios and holistic reasoning chains.They may separate statutory retrieval from judgment prediction even though practical legal work integrates these activities.
3 PLAWBENCH
PLawBench organizes realistic legal workflows into complementary tasks and evaluates them with risk-aware, fine-grained rubrics. Its scenarios require models to transform ambiguous narratives into legally reasoned analyses and formal documents.
- Framework: PLawBench structures legal practice into consultation, case analysis, and document generation tasks.These target issue and fact identification, dispute analysis, and structured presentation of legal claims.
- Framework: Its rubric framework evaluates task-specific legal risks rather than relying only on abstract accuracy.The design draws on risk classification, auditing, and legal risk-management principles.
- Framework: Six rubric dimensions cover issue and fact identification, legal reasoning, knowledge application, procedural awareness, outcome construction, and professional compliance.These dimensions are mapped to specific task settings and their legal risk points.
- Public Legal Consultation: Public consultation tasks test whether models can clarify omitted facts and distinguish users’ descriptions from legally accurate classifications.The task emphasizes iterative questioning when narratives are ambiguous, emotional, or incomplete.
- Practical Case Analysis: Practical case analysis emphasizes coherent integration of legal reasoning, rule application, and claim construction.Its rubrics require identifying legally determinative facts and preserving logical and doctrinal coherence.
- Legal Document Generation: Document-generation tasks require filtering information, analyzing legal relationships, formulating litigation strategy, and drafting documents from disorganized client narratives.The scenarios add multiple parties, ambiguous responsibility boundaries, layered obligations, and legacy or newly enacted laws.
- Data Procurement: The dataset combines expert-edited judgments, consultation records, and complex cases curated by leading Chinese law firms.These open-ended cases are designed to exceed the complexity of standard examination questions.
4 Experiment
The experiments evaluate LLMs across three legal tasks using rubric-based scoring and an LLM judge selected for concordance with human experts. Results show task-dependent strengths, distinct error patterns, scaling effects, and divergent failure modes despite similar aggregate scores.
- Experiments cover Public Legal Consultation, Practical Case Analysis, and Legal Document Generation using 24 API-accessed LLMs.
- Evaluation Metrics: Gemini-3.0-Pro-Preview was selected as the final judge after achieving the highest concordance with human experts.The pilot compared candidate judges against scores from three human legal experts.
- Evaluation Metrics: The benchmark uses fine-grained rubrics, including four dimensions for case analysis and separate plaintiff and defendant documents for document generation.Scoring Rate aggregates item-level scores normalized by each question’s maximum possible score.
- Overall Performance: Model rankings fluctuate across tasks: GPT-5.2 and Claude-4.5 lead procedural tasks, whereas Gemini and Qwen3-Max perform better on deep logical deduction.This pattern suggests a trade-off between safe generation and reasoning depth.
- Overall Performance: Models show distinct errors, including evidence-complexity problems for Gemini and citations of repealed or outdated statutes in GPT outputs.The latter pattern is attributed to likely temporal misalignment in pre-training data.
- Overall Performance: Performance generally improves with model scale and iteration, with Qwen3-Max surpassing smaller same-series models and GPT-5 substantially outperforming GPT-4o.
- Error Analysis: Error analysis reveals substantial divergences in failure patterns even when frontier models achieve comparable aggregate scores.
1) The “Fluency-Logic” Gap in Consultation.
The error analysis exposes a gap between fluent legal outputs and reliable legal reasoning. Models struggle with factual causality, statutory syllogisms, procedural judgment, and sequential constraints across consultation, case analysis, and document generation.
- Consultation: In consultation, models often fail to connect reported events to legal consequences and omit corroboration of key facts before advising.
- Case Analysis: In practical case analysis, Statute and Reasoning scores remain below average because models make logical jumps or omit core statutory provisions.Their conclusions may appear plausible while remaining legally unsound.
- Document Generation: In document generation, Plaintiff drafting scores are lower than Defendant drafting scores because models often follow instructions without detecting standing or jurisdictional defects.This reflects subject-suitability errors and insufficient legal judgment.
- Sequential Constraints: Sequential tasks produce a universal performance drop, with top models losing over 6–8% in Reasoning under ordered multi-step constraints.GPT-4o’s 0.2% drop likely reflects a floor effect from its low 42.00 baseline.
5 Conclusion
PLAWBENCH evaluates LLM agents in realistic legal workflows through three tasks spanning 13 scenarios, 850 questions, and approximately 12,500 fine-grained rubric items. Experiments find that current models still struggle with legal knowledge and reasoning.
- PLAWBENCH covers three core legal tasks, 13 scenarios, 850 questions, and approximately 12,500 fine-grained rubric items.
- The benchmark integrates legal reasoning into evaluation to align assessment more closely with real-world practice.
- Experiments reveal suboptimal performance and continuing difficulties with legal knowledge and reasoning among current models.
- The authors position PLAWBENCH as guidance for domain-specific LLM evaluation, with future work exploring model training and alignment.
6 Ethical Considerations
PLAWBENCH applies privacy, anonymization, content-screening, and expert-review measures to meet legal and ethical standards in benchmark construction and release.
- All benchmark data underwent privacy screening and anonymization, with personal and sensitive information removed.
- The benchmark excludes discriminatory, explicit, violent, and offensive content.
- Legal experts conducted an ethical review intended to minimize risks associated with the benchmark.
7 Limitations
PLAWBENCH has limited jurisdictional and system scope, evaluates single-model agents, and focuses on benchmarking rather than improving models.
- PLAWBENCH is grounded in the Chinese legal system, limiting direct generalizability across jurisdictions.The authors identify jurisdiction-specific realism as a trade-off against broader applicability.
- The experiments evaluate single-model LLM agents rather than complex multi-agent systems.The authors note that multi-agent frameworks may perform well but involve greater operational complexity and token consumption.
- As LLM capabilities improve, performance on PLAWBENCH is expected to increase, motivating continuous benchmark updates.The authors plan an online platform for ongoing evaluation of state-of-the-art models.
- The work evaluates existing models rather than directly improving them through training or alignment.Future research will use PLAWBENCH for post-training and alignment.
A Survey on Legal Benchmarks
Legal benchmarks have evolved from classification and retrieval tasks toward broader reasoning and process evaluation, but persistent gaps remain in realistic, fine-grained assessment. PLAWBENCH distinguishes itself by centering practical legal scenarios and bespoke rubrics.
- The Era of Early Benchmarking and Classification Tasks (2014-2020): 2014–2020 legal benchmarks primarily emphasized legal judgment prediction and basic information extraction within text-classification frameworks.COLIEE and CAIL2018 exemplify this early focus on retrieval, reasoning, and factual-feature identification.
- The Transition to Specialization and Retrieval Benchmarks (2021-2022): 2021–2022 benchmarks shifted toward specialized capabilities such as case retrieval, legal summarization, and structured information extraction.Examples include LeCaRD, CaseHOLD, IN-Abs, IN-Ext, UK-Abs, and related extraction datasets.
- The Period of LLM Reasoning and Comprehensive Evaluation (2023-2024): 2023–2024 evaluations increasingly stratified legal abilities across cognitive dimensions and expanded coverage to ethics, privacy, and evidence-based responses.LegalBench, LawBench, LAiW, LexEval, EQUALS, LLeQA, and CLSum represent this broader evaluation stage.
- The Frontier of Logical Process and Reasoning Chain Evaluation (2025-Now): Recent benchmarks examine reasoning-process quality, including transparency, auditability, due diligence, issue trees, and reasoning-trace integrity.PRBench, LEGIT, and MSLR exemplify this process-oriented direction.
- Outcome-based metrics can overlook whether models reach correct legal answers through clear and valid step-by-step reasoning.The literature identifies a persistent gap in maintaining consistent reasoning across complex legal problems.
- Existing benchmarks often diverge from authentic practice by using textbooks or standardized exams, isolating interdependent tasks, and emphasizing classification or short-form QA.PLAWBENCH instead focuses entirely on practical legal scenarios and broader professional capabilities.
- The benchmark comparison organizes prior systems by data sources, statistical scale, and functional features.Table 9 provides the comparative framework used to position PLAWBENCH among existing legal benchmarks.
- PLAWBENCH develops three specialized sub-datasets, each paired with a bespoke rubric and scoring framework.The separate designs reflect the different objectives and evaluative focuses of the three tasks.
B.3 Dataset Characteristics
The dataset spans three legal-practice task categories and 13 scenarios, with practical case analysis dominating its composition. Its prompts and rubric allocation vary substantially by task type.
- Practical Case Analysis contains 750 items, compared with 60 for Public Legal Consultation and 40 for Legal Document Generation.The dataset also covers 15 specific subtasks across civil, commercial, procedural, and drafting contexts.
- Public Legal Consultation uses prompts averaging 1,171 characters and approximately 550 rubric items overall.Its evaluation averages 21 rubric items per task in the task-level description.
- Practical Case Analysis uses prompts averaging 692 characters and approximately 11,000 rubric items.The rubric system averages 14.1 items per task and targets deep logical reasoning.
- Legal Document Generation uses the longest prompts, averaging 2,574 characters, with approximately 900 rubric items.The task averages 23 rubric items per task to assess formal legal drafting precisely.
- Figure 6 presents the dataset composition hierarchically across primary task categories and finer-grained legal domains.The figure is intended to show how the benchmark’s cases are distributed across its task structure.
C Details of Rubrics
PLAWBENCH uses task-specific, expert-designed rubrics to assess factual reconstruction, substantive legal reasoning, and professional document generation. The rubric designs emphasize case-sensitive content over surface form.
- C Details of Rubrics: The benchmark’s rubric system evaluates factual reconstruction, legal-issue extraction, practical reasoning, and standardized document generation.These competencies are assessed across heterogeneous legal tasks using multiple scoring layers.
- C.1 Rubrics of Public Legal Consultation Task: Public Legal Consultation tests whether models detect contradictions, uncover omitted facts, and formulate targeted follow-up questions.Cases adapt real judgments and embed misleading subplots, concealed evidence, and factual pitfalls.
- C.1 Rubrics of Public Legal Consultation Task: Consultation rubrics require each inquiry to identify a unique missing fact, be necessary for closing an information gap, and affect legal determination.The framework also rewards clarification under uncertainty and penalizes generalized questioning or fraud-inducing guidance.
- C.1 Rubrics of Public Legal Consultation Task: The consultation subset contains 60 cases and approximately 550 rubric points, averaging 8.7 rubric items per case.The associated rubric table defines the scoring framework for this task.
- C.2 Rubrics of Practical Case Analysis Task: Practical Case Analysis references four answer components: legal conclusion, fact interpretation, reasoning process, and statute location.Its approximately 11,000 rubric items distinguish sequential from non-sequential reasoning and score validated rubric aggregates.
- C.3 Rubrics of Legal Document Generation Task: Document-generation rubrics assign less than 10% weight to formal presentation and prioritize substantive content and filtering of irrelevant information.They also impose uniform cumulative penalties for inaccurate provisions, wrong document types, and unmet document standards.
- C.3 Rubrics of Legal Document Generation Task: Plaintiff-document criteria cover jurisdiction, claims, relief calculations, factual structuring, and evidence organization, while defendant criteria cover procedural and damages defenses.These criteria are represented in the task’s rubric tables and include strategies such as challenging standing, timelines, causation, and claimed amounts.
- E.1.1 Prompt Design for Task 1: Public Legal Consultation: An adapted case tests whether analysts look beyond formal records and identify missing substantive labor elements amid selective disclosures and factual distortions.The scenario foregrounds organizational subordination, employer wage payment, and a bona fide job position as legally relevant elements.
F.1 Judge Model Validation Results
The study validates candidate LLM judge models against human expert scoring across legal consultation, case analysis, and document drafting. Gemini 3 Pro Preview is selected because it performs best across two of the three categories, despite not leading document drafting alignment.
- Validation Procedure: At least three human experts evaluated each response, enabling comparison between individual judgments and the human group average.The validation set included sampled case-analysis responses, ten complaint and defense drafting cases each, and 20 consultation items.
- Human Agreement: Pearson correlations between individual experts and the human average ranged from 0.8391 to 0.8841, with MAE values from 5.1937 to 7.9817.These statistics quantify agreement among human evaluators before comparing candidate judge models.
- Judge Model Selection: Gemini 3 Pro Preview was selected as the final judge model after comprehensive cross-task evaluation of three candidate judge models.The candidates were Gemini 3 Pro Preview, GPT-5.1-1113-Global, and Qwen3-Max.
- Judge Model Selection: Gemini 3 Pro Preview did not achieve the highest alignment in document drafting but performed best in the other two task categories.The selection therefore reflects stronger overall cross-task alignment rather than dominance in every category.
- Scoring Procedure: Scores are reported as Earned Points / Total Points, with raw judge-model scores extracted and summed programmatically to reduce calculation errors.Regular expressions process scores from JSON or list outputs generated under the scoring prompts.
G.1 Error Analysis of Public Legal Consultation
Error analysis shows that current models struggle with factual causality, evidence handling, legal characterization, reasoning order, statutory application, and procedural judgment. These weaknesses persist across consultation, case analysis, and document generation, especially when legal reasoning requires linked premises or complex evidence.
- Public Legal Consultation: Consultation models often fail to connect reported events to legal consequences and to identify the corroborating facts needed before giving advice.They also show weak sensitivity to the specific evidence types required for different legal claims.
- Public Legal Consultation: Complex evidence cases expose fragmented reasoning, including questions that lack sufficient legal basis and divergent follow-up inquiries that depart from key facts.GPT-5.2 is described as deviating from key facts, while Gemini-3.0 struggles with probative requirements.
- Practical Case Analysis: Case-analysis errors include omitted legally material facts, broken syllogistic sequences, missing statutory bases, and fundamental errors in legal characterization.The analysis identifies these problems across case facts, reasoning process, statutory basis, and conclusions.
- Practical Case Analysis: Current models have not reached professional-level practical case analysis, despite producing structured text.Reported shortcomings include omitting key facts, misordering reasoning, breaking logical chains, and mixing old and new laws or general and special provisions.
- Practical Case Analysis: Skipping contract-validity analysis caused later reasoning about liability and remedies to lose its legal basis in the mining-machine case.The model moved directly to breach, applied rescission rather than void-contract restitution, and failed to analyze mutual fault regarding interest.
- Legal Document Generation: Document-generation errors include inaccurate legal application and weak jurisdiction review, with hallucinations and procedural-logic problems appearing across models.These issues concern the timeliness, relevance, and citation of laws as well as sensitivity to jurisdictional rules.
- Model-Specific Patterns: DeepSeek, Kimi, and Qwen3-Max more often hallucinate or follow legally unsound client instructions, whereas GPT-5.2, Claude-4.5, and Gemini-3.0 perform better on basic legal logic and fact summarization.The comparison concerns model-specific differences in instruction following, stability, hallucination, and procedural judgment.