Source-linked AI summary
JEC-QA: A Legal-Domain Question Answering Dataset
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun
TL;DR
Legal QA lacks qualified datasets, while legal questions demand complex reasoning over retrieved materials. The paper introduces JEC-QA, a large examination-derived benchmark with legal knowledge resources and reasoning labels, and evaluates representative OpenQA methods. Existing methods reach about 28.63% accuracy, compared with 64% for unskilled humans and 81% for skilled humans.
Problem
Legal QA lacks qualified datasets, and legal questions require sophisticated reasoning and text comprehension.
Method
The paper introduces JEC-QA, a large legal QA dataset with examination questions, a legal knowledge database, and labels for question types and reasoning abilities.
Results
28.63% accuracy is achieved by the best-performing model on all questions, versus 64% for unskilled humans and 81% for skilled humans.
Takeaways & Limitations
JEC-QA provides a challenging benchmark for improving reasoning in reading-comprehension and legal question-answering models.
Abstract
from arXiv · showhide
We present JEC-QA, the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China. The examination is a comprehensive evaluation of professional skills for legal practitioners. College students are required to pass the examination to be certified as a lawyer or a judge. The dataset is challenging for existing question answering methods, because both retrieving relevant materials and answering questions require the ability of logic reasoning. Due to the high demand of multiple reasoning abilities to answer legal questions, the state-of-the-art models can only achieve about 28% accuracy on JEC-QA, while skilled humans and unskilled humans can reach 81% and 64% accuracy respectively, which indicates a huge gap between humans and machines on this task. We will release JEC-QA and our baselines to help improve the reasoning ability of machine comprehension models. You can access the dataset from http://jecqa.thunlp.org/.
Introduction
JEC-QA addresses the limited availability of qualified legal QA datasets and the complexity of legal questions by introducing a large, challenging benchmark with legal knowledge and reasoning annotations. Experiments show a substantial gap between existing methods and human performance.
- Legal question answering remains difficult because qualified datasets are scarce and legal cases and questions are complex and rigorous.
- JEC-QA collects legal questions from China’s National Judicial Examination, a difficult certification exam for prospective lawyers and judges.Only around 10% of examination participants pass each year.
- JEC-QA contains 26,365 four-option multiple-choice questions, 50 times more than the previous largest legal QA dataset.
- The dataset includes an examination-aligned legal knowledge database and human-labeled question types and required reasoning abilities.
- Answering JEC-QA requires retrieval, legal concept understanding, multi-paragraph reading, multi-hop reasoning, and numerical analysis.The example in Figure 1 combines direct and extra evidence while comparing crime severity.
- The best evaluated methods achieve about 25% on KD-questions and 29% on CA-questions, compared with 81% for skilled humans and 64% for unskilled humans.
Related Work
Prior legal QA and OpenQA systems commonly separate document retrieval from answer extraction or selection, but existing approaches remain limited in reasoning and applicability. JEC-QA is positioned as a benchmark for evaluating legal QA reasoning ability.
- OpenQA systems generally retrieve external materials and then extract or select an answer.
- Many retrieval methods emphasize lexical similarity, while reranking approaches attempt to filter noisy passages and improve semantic relevance.
- Despite progress on some tasks, existing OpenQA approaches still lack sufficient reasoning ability.
- Legal QA has been studied with both rule-based and neural systems, but applicable systems remain distant because of poor performance, reasoning ability, and interpretability.
- The authors collect JEC-QA from the National Judicial Examination as a benchmark for reasoning ability in legal-domain question answering.
Dataset Construction and Analysis
JEC-QA combines a large multiple-choice question collection with a structured legal database and labels for question types and reasoning requirements. Its analysis identifies five reasoning types, with multi-hop reasoning especially common among case-analysis questions.
- Dataset Construction: After deduplication, JEC-QA contains 26,365 questions collected from national judicial examinations and online practice exercises.
- Dataset Construction: Questions contain four candidate options and are categorized as single-answer or multi-answer, and as Knowledge-Driven or Case-Analysis questions.Knowledge-Driven questions concern legal concepts, whereas Case-Analysis questions concern actual scenarios.
- Dataset Construction: The database covers examination knowledge from a structured electronic counseling book and 3,382 Chinese legal provisions.The book contains 15 topics and 215 chapters and was manually converted rather than OCR-processed.
- Reasoning Types: JEC-QA defines five reasoning types: word matching, concept understanding, numerical analysis, multi-paragraph reading, and multi-hop reasoning.
- Reasoning Types: More than 66% of Case-Analysis questions require multi-hop reasoning, which creates substantial challenges for existing reading comprehension models.
- Reasoning Types: All five reasoning types are essential for answering JEC-QA and achieving promising performance.
Experiments
The experiments evaluate retrieval and unified OpenQA answering on JEC-QA, showing that retrieval and complex reasoning remain major bottlenecks for existing models.
- Retrieve Strategy: The pipeline retrieves relevant paragraphs before applying question answering models, following the OpenQA setting.Retrieved materials are then used to answer questions under a unified framework.
- Retrieve Strategy: Topic classification narrows retrieval to the top-2 relevant topics, while ElasticSearch retrieves K paragraphs per topic and from legal provisions.The experiments set K = 6, yielding 3K paragraphs for each option.
- Retrieve Strategy: Around 46% of questions can be answered correctly from retrieved materials, with word-matching questions achieving a 62% hit rate.KD-questions retrieve better than CA-questions, while more sophisticated reasoning types achieve substantially lower hit rates.
- Experiment Settings: The unified framework represents each question with four options and 18 reading paragraphs per option, adapting diverse baselines to shared inputs and outputs.Single-answer questions produce option scores, while all-question answering uses scores for possible option combinations.
- Experimental Results: 28.63% is the best model accuracy on all questions, compared with 64% for unskilled humans, despite both using the same retrieved materials.The authors attribute this gap mainly to insufficient model reasoning ability.
- Comparative Analysis: Even with sufficient evidence, existing methods answer only about 32% of questions correctly and perform especially poorly on multi-paragraph and multi-hop CA-questions.The results indicate difficulty understanding reading materials and handling real-case multi-hop reasoning.
- Case Study: A case study shows that models can find one-hop evidence but fail to combine additional facts that eliminate conflicting or unsupported options.The correct answer requires using the fact that Bob is under age 16 to filter options A and B.
Conclusion
JEC-QA is presented as the largest and a challenging legal question-answering dataset, requiring multiple reasoning abilities in both document retrieval and question answering. The authors hope it will advance reasoning in reading comprehension and QA models, as well as legal question answering.
- JEC-QA is presented as a new, challenging, and largest dataset for legal question answering.
- Both document retrieval and question answering on JEC-QA require multiple types of reasoning ability.
- The authors hope JEC-QA will help improve reasoning in reading comprehension and QA models and advance legal question answering.
- Future work will improve question-answering reasoning and integrate legal knowledge, which the authors identify as necessary for answering JEC-QA questions.