Source-linked AI summary
OBJECTION! Lawyer Agents Mitigate Guilty Bias in Legal Judgment Prediction
Jaehoon Jeong, Jay-Yoon Lee
TL;DR
The paper argues that guilty bias in Legal NLP must be addressed at inference time to neutralize it effectively. OBJECTION integrates an adversarial Lawyer Agent into three-step legal reasoning, reducing False Guilty Rate while maintaining strong Macro-F1 across settings.
Problem
The paper argues that guilty bias should be handled at inference time to neutralize it effectively.
Method
OBJECTION integrates a Lawyer Agent into the offense, unlawfulness, and culpability reasoning steps, actively hypothesizing exculpatory contexts and refining decisions through adversarial intervention.
Results
OBJECTION reduces FGR from 53.21% for the combined structured baseline to 10.71% when the Lawyer Agent is incorporated.
Takeaways & Limitations
The system implements the Presumption of Innocence at the architecture level while attaining the highest or tied-highest Macro-F1 across backbone–dataset settings.
Takeaways & Limitations
The pipeline is designed for Civil Law, while adaptation to Common Law systems requires further study.
Abstract
from arXiv · showhide
Legal Judgment Prediction (LJP) models are typically trained on documents that describe facts from a prosecutorial perspective. Existing datasets further exhibit severe label imbalance toward guilty outcomes. Consequently, these models suffer from "Guilty Bias", blindly accepting the prosecution's narrative as objective truth. Previous studies employing three-step reasoning structures or training on synthetically generated innocence data improve overall accuracy, but they still fail to mitigate bias at inference time. In this paper, we introduce OBJECTION, an inference-time pipeline that integrates an Adversarial Lawyer Agent into each 3-step reasoning of offense, unlawfulness, and culpability. Unlike generic critics, our agent actively challenges the model's presumptions of guilt by injecting legal defense arguments at each reasoning stage. To thoroughly evaluate this, we present a new "Natural Innocent" dataset including 3.4k real-world cases, overcoming the limitations of synthetic innocence benchmarks. Test results show that OBJECTION drastically reduces the False Guilty Rate (FGR) from 82.93% (SOTA baseline) to 16.69%, proving its capability to perform substantive legal reasoning. This work denotes a key progress toward aligning Legal AI with the presumption of innocence.
1 Introduction
LJP systems inherit guilt-biased prosecutorial narratives, while prior structured-reasoning and synthetic-data approaches do not fully address this bias at inference time. OBJECTION introduces a training-free adversarial Lawyer Agent and a Natural Innocent dataset to challenge guilt assumptions during reasoning.
- LJP training data commonly presents prosecutorial narratives rather than neutral event summaries, exposing models to guilt-biased descriptions.
- Prior structured-reasoning methods and synthetic innocent cases largely inherit or address guilty bias only during training, leaving inference-time vulnerability.
- OBJECTION adds a training-free Lawyer Agent that actively excavates reasonable doubt and challenges overconfident guilt assumptions during inference.
- OBJECTION quantifies guilty bias using False Guilty Rate and presents adversarial inference as a way to neutralize bias in existing frameworks.
2 Related Works
Related work includes structured extraction, synthetic innocence, critique-and-revise, and multi-agent approaches, but these methods can retain or amplify guilt-biased reasoning. OBJECTION instead constrains an adversarial Lawyer Agent to construct grounded defenses and challenge the presumption of guilt.
- Standard LJP methods structure prosecutorial narratives through extraction or legal elements but overlook their inherent bias and inherit the prosecutorial perspective.
- Synthetic innocent data and trichotomous training address guilty bias primarily during training, leaving models vulnerable when inference lacks explicit acquittal or adversarial signals.
- Iterative critique-and-revise systems may amplify existing model biases, with refinement reinforcing an initial presumption of guilt.
- OBJECTION assigns its Lawyer Agent the sole objective of constructing the strongest grounded defense rather than seeking consensus or general knowledge refinement.
- The role-constrained adversarial process prevents bias amplification, and critic ablations attribute the effect to role asymmetry rather than merely adding a second agent.
3 Methodology
OBJECTION combines structured factual extraction, isolated three-stage criminal-law reasoning, and an adversarial Lawyer Agent to counter guilty bias during inference. The pipeline makes the Lawyer Agent challenge each stage’s judgment, supports early acquittal, and produces granular legal grounds for verdicts.
- 3 Methodology: OBJECTION orchestrates schema-based information extraction, trichotomous reasoning with issue isolation, and adversarial argumentation.The modules are individually vulnerable to guilt-biased narratives, so the Lawyer Agent connects them into a unified pipeline.
- 3.2 Trichotomous Reasoning: The reasoning chain evaluates offense, unlawfulness, and culpability sequentially, with each stage logically isolated from factors belonging to later stages.For example, unlawfulness is assessed assuming the offense is established, independently of culpability factors.
- 3.1 Input and Schema-based IE: The SOAM schema abstracts criminal facts into Subject, Object, Actus Reus, and Mens Rea components for zero-shot extraction.The extraction is used in the Offense stage to provide a structured factual anchor without granular annotation.
- 3.3 Adversarial Lawyer Agent: The Lawyer Agent actively searches for reasonable doubt and exculpatory contexts, unlike a standard critic that primarily checks logical consistency.Its review can consider contexts such as self-defense or intent negation, shifting attention from surface-valid prosecution logic to possible defenses.
- 3.3 Adversarial Lawyer Agent: At each stage, the Judge produces an initial judgment, the Lawyer generates a defense argument, and the Judge re-evaluates the case using that argument.This dialectical interaction conditions the final stage verdict on the defense narrative.
- 3.4 Verdict Generation: Early exit terminates inference with acquittal when a higher-level stage is negated, while granular acquittal labels identify the legal basis for the outcome.These mechanisms target computational efficiency, logical consistency, error containment, and explainability.
4 Experiments
Experiments evaluate OBJECTION across synthetic and real-world innocence datasets, backbones, ablations, commercial APIs, and controllability settings. Results show that adversarial legal reasoning reduces guilty bias while preserving reasoning quality, generalization, and controllable accuracy–fairness trade-offs.
- Experimental Setup: OBJECTION is evaluated on three datasets, including Natural Innocent, which contains 3,412 real criminal judgments from South Korean courts.The other datasets use synthetic innocent samples from CAIL and ELAM.
- Limitations of Naive Prompting: 53.21% FGR for the naive Schema + Trichotomy pipeline falls to 10.71% after adding the Lawyer Agent.Structured reasoning alone produces logically organized but biased predictions; adversarial intervention improves the balance between accuracy and fairness.
- Main Comparative Results: OBJECTION consistently reduces guilty bias across backbones and achieves the highest overall Macro-F1, while Debate-Feedback varies substantially by architecture.LJPIV reaches 47.60% FGR on out-of-domain ELAM, whereas Debate-Feedback ranges from insufficient bias mitigation to uncontrolled over-acquittal.
- Structural Integrity and Reasoning: OBJECTION maintains 3-step element-detection F1 scores of 0.82 on CAIL and 0.81 on ELAM, compared with Baseline D’s 0.37 on ELAM.The results associate the Lawyer Agent with more stable legal procedure and logical coherence across domains.
- Synthetic vs. Natural Innocence: On Natural Innocent, OBJECTION achieves 16.69% FGR versus LJPIV’s 82.93% and Debate-Feedback’s 42.95% with Qwen.The results indicate that the Lawyer Agent identifies reasonable doubt without relying on explicit synthetic innocence cues.
- Ablation Studies: Removing the Lawyer Agent raises FGR from 23.20% to 38.80%, while removing structural anchors causes FNRs of 62.40% and 68.00%.Replacing the Lawyer Agent with a generic critic yields 56.40% FGR; the full design therefore balances adversarial challenge with factual and logical constraints.
5 Conclusion
OBJECTION addresses prosecutorial narrative bias in Legal NLP through an inference-time adversarial pipeline, reducing false guilty predictions while maintaining balanced performance across settings.
- OBJECTION combines a Lawyer Agent with three-step reasoning to actively challenge models’ presumption of guilt.
- OBJECTION reduced the False Guilty Rate on the Natural Innocent dataset, where fine-tuned SOTA models failed.
- OBJECTION attained the highest or tied-highest Macro-F1 in every backbone–dataset setting while balancing guilty and innocent predictions.
- The framework implements the Presumption of Innocence at the system-architecture level and prioritizes preventing false convictions over raw accuracy.
- The paper presents OBJECTION as a step toward normative alignment in Legal AI and adherence to criminal-justice principles.
6 Limitations
The paper identifies boundaries involving legal-system scope, schema granularity, inference cost, and the binary decision setting.
- OBJECTION is designed for Civil Law systems; adaptation to Common Law systems requires further study.
- The generic SOAM schema supports scalability, but jurisdiction-specific schemas could better represent crime-specific constitutive elements and improve performance.
- The adversarial agent increases inference cost relative to single-pass baselines, although early exit reduces expected token consumption.
- The current framework addresses only binary guilty/not-guilty decisions, leaving charge and sentencing prediction for follow-up work.
7 Ethical Considerations
The paper frames OBJECTION as decision support rather than autonomous adjudication and describes privacy protections, human evaluation safeguards, and fact-grounded prompting practices.
- OBJECTION is intended to assist legal professionals as a Decision Support System, not to replace authorized human judgment or issue autonomous verdicts.
- The Natural Innocent judgments were collected from public South Korean records and de-identified to protect individuals’ privacy.
- Human expert evaluation involved active law-enforcement investigators in a minimal-risk professional task assessment without sensitive participant data collection.
- The pipeline’s prompts require extraction and judgments to remain grounded in case facts, with unsupported defense points ignored during final aggregation.
- The three reasoning stages separately address offense elements, justification grounds, and responsibility capacity.
- Defense prompts introduce stage-specific challenges involving constituent elements, self-defense or necessity, and responsibility capacity.
B About “Natural Innocent” Dataset
The Natural Innocent dataset uses anonymized South Korean criminal judgments and organizes acquittal cases by offense, unlawfulness, and culpability issues. Its realistic narratives require inference of innocence from contextual details rather than explicit cues.
- Dataset provenance: The Natural Innocent dataset is built from anonymized criminal judgments released by South Korean government sources and courts.
- Label construction: Class 1 contains only final acquittals, while Class 0 contains final guilty verdicts; the dataset is restricted to first-instance judgments with detailed facts.
- Stage construction: Acquittal cases were filtered by keywords and categorized into offense, unlawfulness, and culpability stages, with assignments verified by a police investigator.
- Dataset statistics: The dataset documentation covers provenance, scale, verdict distributions, offense categories, and case-description token lengths.
- Synthetic versus natural innocence: Natural Innocent cases contain realistic narratives where innocence must be inferred from implicit contextual details, unlike synthetic cases with explicit exculpatory cues.
C Detailed Case Study (Qualitative Analysis)
The case study shows how OBJECTION restructures case facts and injects stage-specific defense arguments, changing an initial unjustified judgment to a justified one. The example also illustrates the SOAM schema’s separation of physical conduct from inferred intent.
- SOAM Schema Extraction: The SOAM extraction maps the case into subject, object, actus reus, and mens rea elements.It represents the defendant’s restraint conduct separately from contextual evidence of intent.
- SOAM Schema Extraction: Contextual actions such as repeated police calls reveal an intent to restrain the attacker rather than maliciously kill.The pipeline separates inferred mens rea from the fatal physical outcome.
- Initial Judgment: The base model initially judged the conduct Not Justified because the danger appeared to end after knife removal and prolonged neck compression seemed excessive.This was the model’s initial conclusion at the Unlawfulness stage.
- Lawyer Agent Intervention: The Lawyer Agent reorganized existing evidence around persistent imminence, necessary restraint during delayed police response, and the victim’s initiating armed attack.The intervention added no new facts but made defense-relevant considerations explicit.
- Revised Judgment: The revised judgment concluded that the conduct was Justified after recognizing continued imminence, lack of reasonable alternatives, and last-resort necessity.These considerations supported justification through self-defense.
E Inference Efficiency Analysis
OBJECTION reduces inference cost through early exit, avoiding the fixed interaction burden of Debate. The resulting latency reductions are approximately 69.1% for Llama and 57.9% for Qwen.
- Early Exit: OBJECTION uses early exit to reduce expected interactions according to the verdict distribution, unlike Debate’s fixed eight calls.The efficiency evaluation measures latency and token consumption per CAIL case.
- Early Exit: Llama-3.1 averages 5.57 API calls and 5.77 seconds latency because 62.68% of cases dismiss charges at the Offense stage.The early exits follow the model’s verdict distribution.
- Efficiency Comparison: Latency decreases by approximately 69.1% for Llama and 57.9% for Qwen relative to the Debate baseline.The paper characterizes this as an efficiency–performance trade-off advantage over unstructured debate.
F Ablation on LJPIV-CAIL dataset
The ablation analysis evaluates OBJECTION across backbones, datasets, and baselines using exact paired tests on innocent cases. Its FGR reductions are significant in most comparisons and generalize beyond explicitly prompted doctrines.
- Evaluation Protocol: Exact McNemar tests evaluate False Guilty Rate directly on paired innocent-case verdicts under greedy decoding.The protocol applies the same per-case predictions used in the reported comparisons.
- Ablation Results: OBJECTION significantly reduces FGR in 20 of 27 comparisons.The effect is significant in 9 of 9 comparisons against Self-Refine and 8 of 9 against Baseline D.
- Ablation Results: Against Debate-Feedback, results are backbone-dependent, with 3 significant wins, 3 losses, and 3 ties.The comparison pattern differs from the near-uniform gains against the single-pass baselines.
- Generalization: Performance generalizes to acquittal grounds absent from the prompt, with large and consistent FGR reductions across all three stages.The effect includes the Offense stage, where evidentiary insufficiency dominates rather than named doctrines.
H.2 Failure Mode Analysis
The failure analysis identifies false acquittals as concentrated at the Offense stage and links many to inconsistencies between reasoning and emitted labels. It also reports charge-specific error patterns that motivate human oversight.
- False Acquittals: The Natural Innocent analysis examines the distribution of 404 false acquittals.The breakdown is reported for Qwen-2.5-7B.
- False Acquittals: False acquittals concentrate at the Offense stage, where roughly 68% show reasoning–verdict inconsistency.The reasoning satisfies the elements while the emitted label is NO.
- Charge-Specific Patterns: Assault errors typically over-credit self-defense narratives in mutual altercations, while fraud errors involve uncorroborated negations of intent.These charge-specific patterns bear on public safety and motivate human-in-the-loop deployment.