PaperlayerView exact revision on arXiv

Source-linked AI summary

InvestigationWorlds: An Agentic Environment for Legal Investigation

Albert Yu Sun, Andrew Benard, Sil Hamilton, Anna Teresita A. Marcelo, Yong Jae Kim, Carl-Leander Henneking, Rundong Hu, Yuhong Wang, David Mimno, Bishan Yang, Igor Labutov

arXiv:2610.04129v1cs.AIcs.CLcs.CYcs.LG

TL;DR

Legal investigation requires weighing evidence that can support competing factual accounts, a capability existing legal benchmarks do not measure. InvestigationWorlds builds a benchmark from real federal summary-judgment records, augmented with attorney-validated synthetic documents. Adding documents supporting alternatives increased agents’ commitments to those alternatives, while often reducing case-solved rates.

  • Problem

    Existing legal evaluations do not measure whether agents can distinguish incompatible conclusions that are each coherent with the same evidence.

    Source: §1 · p. 1

  • Method

    InvestigationWorlds uses real federal summary-judgment records as judicially grounded cases and augments them with attorney-validated synthetic documents supporting alternative factual hypotheses.

    Source: §1 · pp. 1–2

  • Results

    Adding 20 pro-alternative documents increased alternative-hypothesis belief from 3/506 (0.6%) to 115/506 (22.7%), while adding pro-alternative documents reduced case-solved rates for five of six models.

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    Agents can retrieve relevant evidence yet commit to incorrect hypotheses, underscoring the value of evaluating whether they distinguish court-adopted accounts from alternatives.

    Source: §1 · p. 2

    The environments cover U.S. federal civil cases reaching fully briefed summary judgment, and synthetic documents are attorney-validated rather than court-tested.

    Source: §6 · p. 10

Abstract

from arXiv · show

We introduce InvestigationWorlds, an agentic environment for legal investigation. We build on an underused artifact of U.S. civil litigation: the summary judgment motion. This motion relies upon a record composed of real evidence exhibits, and results in a court-adopted hypothesis that is treated as ground truth for the purposes of deciding the motion. Each environment is built from a real U.S. Federal Court case retrieved from Public Access to Court Electronic Records (PACER) and augmented by an attorney-validated generation pipeline that synthesizes role-tagged documents around the original record. The resulting corpus admits multiple coherent factual readings, only one of which matches the court-adopted hypothesis. Evaluating on 100 cases, we find agents often commit to incorrect hypotheses despite retrieving relevant evidence, struggling to distinguish the court-adopted hypothesis from alternative hypotheses.

1 Introduction

Legal investigation requires constructing a coherent account from evidence that may support competing interpretations, but existing legal benchmarks do not test that capability. INVESTIGATIONWORLDS uses real federal summary-judgment records and attorney-validated synthetic documents to test whether agents identify the court-adopted account among plausible alternatives.

1 Introduction

  • Existing legal evaluations do not test whether agents can distinguish multiple coherent, incompatible conclusions supported by the same evidence.They generally assume a single correct answer and/or a well-posed query.Source: §1 · p. 1
  • Summary-judgment opinions provide a complete evidentiary corpus paired with judicially validated ground truth: the court’s exhibit-traceable factual findings.Both sides submit material facts and supporting exhibits, and the judge issues a written opinion adopting specific findings.Source: §1 · pp. 1–2
  • InvestigationWorlds builds on real PACER federal cases and augments their merits records with attorney-validated, role-tagged synthetic documents that support alternative factual readings.The court-adopted disposition supplies the ground truth for evaluating agents.Source: §1 · p. 2

1.1 The Investigation Task

  • An investigation environment comprises documents, recurring entities, and a hypothesis DAG rooted in the dispositive legal question, with leaves tied to document-supported ground facts.The DAG includes court-adopted and court-rejected hypotheses plus plausible alternatives that are refutable on a complete record.Source: §1.1 · p. 2
  • Agents must construct their account before seeing the hypothesis menu, then avoid committing to synthetic alternatives that distract from the court-adopted account.This makes the task an investigation rather than simple question answering.Source: §1.1 · pp. 2–3

2 Constructing the Environment

The environment construction pipeline selects federal summary-judgment records, extracts court-grounded facts and hypotheses, and augments merits exhibits with role-tagged synthetic documents. Attorneys review the pipeline outputs, and an independent attorney recovered the court-adopted outcomes from merits evidence in a 10-case check.

2.1 Case Selection from PACER

  • The cohort contains 100 fully decided summary-judgment cases, balanced across five federal case types, from 28 states, D.C., and Puerto Rico, spanning 41 districts and all 12 regional circuits.The archived PACER PDFs total 42,283 pages.Source: §2.1 · p. 3

2.2 Ground Truth Extraction

  • The pipeline extracts court findings, legal questions, and supporting opinion passages while parsing party fact statements and counter-statements for agreed and contested facts.It also disambiguates recurring entities and annotates merits exhibits with document type, date, and mentioned entities.Source: §2.2 · pp. 3–4

2.3 Augmenting the Corpus

  • A language model generates two plausible, refutable alternative hypotheses per case and plans coherent synthetic-document series with assigned roles, target facts or hypotheses, and formats.Sub-agents render the documents, and a judge model reviews them; failed documents can be rejected and regenerated up to three times.Source: §2.3 · p. 4

2.4 Attorney Validation

  • In the 55-case subset, pass rates exceeded 94% for party hypotheses, alternative hypotheses, and chaff, with 81%+ agreement on the 10-case agreement subset.Three practicing attorneys contributed over 30 years of combined experience and approximately 100 hours of review.Source: §2.4 · p. 5
  • An attorney matched the court-adopted outcome in all 10 randomly selected cases after reviewing 459 pages of merits evidence over 12 hours, with the opinion and adopted label withheld.The sample spanned the five case categories and had not been audited during original validation.Source: Figure 3 · p. 4, §2.4 · p. 5

3 Experimental Setup

Agents investigate a corpus before receiving a shuffled hypothesis list, then commit to verdicts without further tools. The evaluation uses six frontier agents, a case-level success metric, and fixed tool budgets.

3.1 Tasks and Protocol

  • The two-turn protocol requires agents to investigate documents and form an account before seeing shuffled hypotheses, then judge each without tools as believe, do not believe, or do not know.Case-solved scores 1 only when the agent believes the court-adopted hypothesis and rejects every alternative; “do not know” counts as not believing an alternative.Source: §3.1 · pp. 5–6

3.2 Models, Harnesses, and Resources

  • Six frontier agents are evaluated through Claude Agent SDK, OpenAI Agents SDK, or a vLLM-backed harness, with the same four filesystem operations and a fixed per-task turn-1 budget.Tool requests are denied after the budget is exhausted.Source: §3.2 · p. 6

4 Results

Across the 100-case cohort, Haiku 4.5 solved the most cases, but its advantage over a stronger baseline was not statistically significant. Ablations show that adding evidence for alternatives can reduce solved cases and increase belief in incorrect alternatives, while retrieval strategy and evaluator rankings vary across models and generation pipelines.

4.1 Investigation

  • 33.3% of cases were solved by Haiku 4.5, the highest rate, but its 8.3-point gain over the 25.0% baseline was not statistically significant.Sonnet 4.6 solved 29.9% of cases; its 4.9-point gain over the same baseline was also not statistically significant.Source: Figure 4 · p. 6, §4.1 · pp. 6–7

4.2 Counterfactual Ablations over Document Roles

  • Adding pro-alternative documents reduced case-solved rates for five of six models by 4–22 percentage points; additional noise and chaff produced no consistent further decline.The paired 95% bootstrap confidence intervals excluded zero for Haiku 4.5 (−22 points) and GPT-5.4-mini (−8 points).Source: Figure 5 · p. 7, §4.2 · p. 7

4.1 Investigation

  • Five models achieved 41.9%–52.6% accuracy on their worse-performing class, while GPT-5.4-mini reached 20.0%.The always-reject policy scored 100% on alternatives but 0% on court-adopted hypotheses.Source: §4.1 · p. 7
  • Haiku committed on 80.7% of verdicts versus Sonnet’s 73.3%, while Sonnet was more accurate among committed verdicts, 74.1% versus 70.8%.On 87 cases, Haiku solved 29/87 and Sonnet 25/87; both models sometimes believed an incorrect alternative despite believing the court-adopted hypothesis.Source: §4.1 · p. 7

4.2 Counterfactual Ablations over Document Roles

  • 22.7% of paired verdicts believed a reference-incorrect alternative after pro-alternative evidence was added, up from 0.6%.Of 114 verdicts switching to believe, 55 had been do not believe and 59 had been do not know.Source: Figure 6 · p. 8, §4.2 · p. 8

4.3 Investigative Strategies

  • Models opened 0.7–10% of the corpus on average while recovering signal at higher rates, and strategy differences did not predict accuracy.Sonnet 4.6 had 7.4% coverage and 24.7% signal recall; GPT-5.4-mini had 0.7% coverage and 3.1% signal recall.Source: §4.3 · pp. 8–9

4.4 Robustness to Generator Choice

  • Evaluator rankings remained consistent up to ties across the two generation pipelines (Spearman ρ = 0.94), with no strict-order reversals.DeepSeek v4’s case-solved rate changed from 10.0% on Anthropic-generated corpora to 26.7% on OpenAI-generated corpora.Source: §4.4 · p. 9

5 Related Work

Existing benchmarks cover legal question answering, retrieval, and reasoning, but do not directly evaluate reconstructing narratives from heterogeneous evidence with multiple coherent, incompatible readings. InvestigationWorlds differs by grounding competing interpretations in real federal cases and judicially validated findings.

Agentic environments and reasoning benchmarks

  • Prior agentic environments assess tool use, browsing, or coding against objective answers or test suites, rather than adversarially constructed, coherent factual interpretations.GAIA and BrowseComp reward locating a single answer, while coding environments use passing tests.Source: §5 · p. 9

Multi-document reasoning

  • Unlike benchmarks built on synthetic corpora or single-answer retrieval, InvestigationWorlds uses real federal cases and judicially validated ground truth to assess competing factual readings.Its distractors are typed by evidentiary role against a hypothesis DAG rather than by retrieval similarity.Source: §5 · p. 9

Legal AI benchmarks

  • InvestigationWorlds evaluates legal investigation, a capability existing legal benchmarks have not directly measured.Prior work covers legal QA, retrieval, general legal reasoning, hallucination, and review tasks.Source: §5 · p. 10

6 Discussion and future work

The environments cover U.S. federal civil cases that reached a fully briefed summary judgment motion, a focused slice of legal practice. Synthetic documents are attorney-validated for plausibility and role consistency, but are not court-tested.

Scope of jurisdiction and case type

  • The benchmark is scoped to U.S. federal civil cases that reached a fully briefed summary judgment motion.The motion supplies structured ground truth for rigorous evaluation.Source: §6 · p. 10

Synthetic documents are attorney-validated, not court-tested

  • Synthetic documents are judged plausible and role-consistent by experienced litigators, not tested in court.Forensic-style realism, including metadata patterns and threading conventions, could be added incrementally.Source: §6 · p. 10

A Implementation Details

The implementation uses provider-specific agent SDKs, tool limits, and model-serving setups, while logging per-call telemetry to support reproducible cost accounting. Rankings across the two tested generation pipelines are highly consistent.

Agent harnesses

  • Claude runs use the Claude Agent SDK with Read, Bash, Glob, and Grep, while OpenAI runs use its Agents SDK with file-reading, globbing, grep, and bash tools.Claude uses a 10 MB maximum buffer; OpenAI uses automatic truncation.Source: A Implementation Details · p. 12

Resource budgets

  • Investigation runs allow 25 turns and 30 tool calls for both providers; tools are disabled in Turn 2.Factual-QA limits differ by provider, and an expanded 50-case ablation had a two-hour limit.Source: A Implementation Details · p. 12

Compute environment

  • Frontier models run through Anthropic and OpenAI APIs, while open-weight models run on local vLLM deployments with NVIDIA GPUs.The system requires no specialized storage beyond the local case corpus.Source: A Implementation Details · p. 12

Generator comparison

  • ρ = 0.94: evaluator rankings across Anthropic- and OpenAI-generated environments remain consistent up to ties on 30 held-out cases.The comparison covers six evaluators, including DeepSeek v4 and Qwen 3.6 35B.Source: Figure 8 · p. 13

Concurrency and scheduling

  • The runner caps Anthropic and OpenAI at three concurrent requests each within a twelve-worker pool, and limits construction to three concurrent case pipelines.Evaluation calls use exponential backoff on rate limits.Source: A Implementation Details · p. 13

Per-call telemetry and total compute

  • $3,300: approximate total API spend, with per-case investigation costs ranging from $2.06 for Sonnet 4.6 to $0.07 for GPT 5.4 mini.Per-call logs record token counts, duration, and dollar cost; open-weight runs incur local GPU time.Source: A Implementation Details · p. 13

B Annotator Results

Attorney validation found high pass rates and pairwise agreement across stages critical to the investigation task, with lower pooled agreement across all stages. The table’s agreement figures come from a 10-case subset independently rated by three attorneys.

Attorney validation

  • Critical stages other than document-metadata extraction exceed 94% pass and 81% pairwise agreement; pooled agreement across all stages is 78.2%.Coverage and judgment rates aggregate deep-review audits across 55 cases; pairwise agreement uses a 10-case subset rated independently by three attorneys.Source: B Annotator Results · p. 13, Table 2 · p. 13

C Generator Robustness

OpenAI and Anthropic models retain their investigation-accuracy ranking when evaluated with generators from the other family.

C Generator Robustness

  • The investigation-accuracy ranking between OpenAI and Anthropic models is preserved across generators from the other family.Figure 8 reports this cross-family generator robustness result.Source: C Generator Robustness · p. 13

D Prompt Library

The prompt library specifies runtime-filled workflows for transforming federal summary-judgment records into structured, evidence-grounded facts, arguments, questions, narratives, and synthetic factual alternatives. It also defines document-grounded factual-QA and investigation harnesses for evaluating agents’ verdicts on competing hypotheses.

D Prompt Library

  • The appendix presents load-bearing, stage-organized Jinja2 templates whose case-specific tokens and includes are filled at runtime.These templates support the INVESTIGATIONWORLDS pipeline.Source: D Prompt Library · p. 14

D.1 Ground-Truth Extraction

  • The ground-truth extraction prompts transform the opinion, party fact statements and responses, and briefs into a fact table, party hypothesis DAGs, and a judge’s ruling annotation.Source: §D.1 · p. 14
  • Opinion Pass 1 extracts an argument map, merits-relevant facts relied on by the court, and atomic factual question stubs from the judge’s opinion.Source: §D.1 · p. 14
  • Extracted facts must be atomic, merits-relevant underlying events, excluding procedural facts, background, and litigation-stage expert characterizations.The prompt tests whether a fact would change if the ruling changed and requires facts to describe what happened before the lawsuit.Source: §D.1 · p. 15
  • Each fact must include an exact 10–30-word opinion quote as its downstream anchor, along with a unique fact ID, opinion page, and cited exhibit when available.Source: §D.1 · pp. 14–15
  • The argument map organizes each resolved legal question into branches, distinguishes evidentiary factual analysis from precedent-based procedural defenses, and lists cited evidentiary documents by branch.Arguments resolved purely on precedent may have no branches and instead include a note explaining why.Source: §D.1 · p. 15
  • Question stubs must ask one factual question per discrete claim, avoid legal or compound questions, and avoid synthesis, contention framing, or post-hoc expert characterizations.Questions should not disclose where to find the answer and should be answerable from the underlying documents rather than requiring the judge’s opinion.Source: §D.1 · pp. 15–16
  • Opinion Pass 2 applies a stricter lawsuit-events-only filter and synthesizes the structured opinion artifact used downstream.It seeks individual facts about the events at issue while excluding facts about the lawsuit itself, including procedural facts, legal interpretations, expert opinions, and synthesized conclusions.Source: §D.1 · pp. 17–18
  • Opinion extraction targets compact, atomic merits facts from the opinion’s factual section, excluding procedural matters, legal interpretations, synthesis, and litigation-stage expert characterizations.Facts should describe underlying events or states relevant to the substantive question the court resolves.Source: §D.1 · pp. 18–19
  • Each extracted fact receives an exact 10–30-word opinion quote, a unique ID, its opinion page, and any cited exhibit; question stubs ask about individual facts.Stubs use descriptive unique IDs and one-line answer sketches, and most reference exactly one fact.Source: §D.1 · pp. 18–19
  • The opinion is decomposed into top-level legal questions, classified as factual analysis or procedural defense, then divided into branches containing cited evidentiary sources.Branches omit legal-authority citations; precedent-only arguments have no branches and receive an explanatory note.Source: §D.1 · p. 19
  • Question stubs must ask one factual, exhibit-answerable question without legal framing, answer cues, party-contention framing, expert characterizations, or cross-source synthesis.Questions should avoid naming a document or clause when possible and should not require reading the judge’s opinion to answer.Source: §D.1 · pp. 19–20
  • SOUF/RSOUF extraction records each numbered fact’s text, admitted-or-disputed status, dispute reason, cited exhibit, page, and a verbatim highlight quote.The response determines whether the opposing party admits, does not contest, denies, or qualifies the fact.Source: §D.1 · p. 20
  • Ground facts reconcile opinions, party fact statements, and briefs into one deduplicated, citation-bearing table of atomic factual propositions underlying the hypotheses.Facts are rewritten as observable events or states, not legal standards or conclusions, and should be verifiable from documents or video without legal knowledge.Source: §D.1 · pp. 20–21
  • Each ground fact is assigned a preferred embedding of direct, pattern, or either, reflecting whether one document, multiple documents, or either presentation best supports it.A direct fact is explicit in a single document; a pattern fact emerges by synthesizing across multiple documents.Source: §D.1 · p. 21
  • Ground-fact extraction targets 30–80 facts per typical case, labels source polarity, and uses the ground type; injected supporting or refuting facts come from a separate pass.Facts should be more atomic rather than fewer and broader.Source: §D.1 · p. 22
  • Party hypothesis DAGs represent each side’s factual narrative as a top-level claim plus sub-claim groups organized by brief Point headings, with each group referencing fact IDs from the provided facts table.Source: §D.1 · pp. 22–23
  • The DAG prompts prohibit inventing facts and require factual, non-legal claims that describe observable conduct rather than legal conclusions.If a needed fact is absent from the canonical table, it must be omitted.Source: §D.1 · pp. 22–24
  • When a party’s brief is missing or lacks clear Point headings, the prompt uses a flat-list fallback; “defense” means the side opposing the movant.The fallback is one group with a null sub-claim and the relevant fact IDs.Source: §D.1 · pp. 22–24
  • A single-pass, opinion-only prompt annotates the existing party DAGs with overall disposition and per-sub-claim rulings, without re-extracting facts or DAGs.It identifies the movant and distinguishes accepted, rejected, and not-reached sub-claims; split rulings are also represented.Source: §D.1 · pp. 24–25
  • For orders without written reasoning, the prompt assigns the overall ruling from the docket disposition, marks every sub-claim not reached, and flags the case as likely to fail quality filtering.The rationale notes that the opinion contains no written reasoning.Source: §D.1 · p. 25

D.2 Corpus Augmentation

  • Each case receives two distinct, factual alternative hypotheses that reuse at least three ground facts, add noncontradictory supporting facts, and include a refuter.The alternatives should avoid relitigating documented facts and explore different angles; a refuter may be an injected fact or an existing ground fact.Source: §D.2 · pp. 25–26
  • A multi-document shard plan embeds one fact across 3–5 partial documents, so readers must synthesize the bundle to recover the proposition.Shards should use varied document types while remaining temporally and entity-consistent; no single shard should reveal the full fact.Source: §D.2 · p. 27
  • Fact-anchored signal prompts generate realistic, standalone exhibits that clearly assert or refute a target proposition without awkwardly repeating it verbatim.A post-generation grader checks the document alone for whether it asserts or refutes the proposition as intended.Source: §D.2 · pp. 27–28
  • Shared document-type rules enforce appropriate formats, such as email headers, deposition Q/A, or tabular invoices, and exclude expert reports from the synthetic corpus.General rules also require plausible document lengths and visible page numbers.Source: §D.2 · pp. 28–29

D.3 Factual-QA Harness

  • The factual-QA harness runs an agent on each case’s question set and grades free-form responses against per-question rubric items.Its system prompt confines the agent to the case exhibit directory and requires an ANSWER: terminator for grading.Source: §D.3 · p. 29
  • The factual-QA system prompt is loaded by bench_utils/run_single.py.It pins the agent to the case exhibit directory and enforces the ANSWER: terminator the grader keys on.Source: §D.3 · p. 29
  • The factual-QA harness restricts answers to the supplied documents and uses a rubric judge to assess responses against the gold answer with per-item Boolean scores.The answer prompt requires the final response to end in the format “ANSWER: <your complete answer to the question>.”Source: §D.3 · pp. 29–30

D.4 Investigation Harness

  • The investigation harness uses two turns: agents first investigate the corpus with tools, then judge shuffled hypotheses without tools.The verdict turn relies only on the self-contained summary from the investigation turn.Source: §D.4 · p. 30
  • In Turn 1, agents summarize the most likely account, alternative scenarios the evidence supports, and evidence inconsistent with the leading account, citing specific documents.The prompt asks agents to identify both the likely account and other evidence-supported possibilities.Source: §D.4 · p. 30
  • The Turn 1 summary must be self-contained because agents cannot reread documents during follow-up questions.The prompt also asks for a thorough understanding of what happened and the possibilities supported by the evidence.Source: §D.4 · p. 30
  • Turn 2 presents shuffled hypotheses and requires a believe, don’t-believe, or don’t-know verdict for each, relying only on the investigation summary.Agents cannot search the documents again during this turn.Source: §D.4 · p. 30
  • The verdict prompt requires a JSON array with one object per hypothesis, including its label, verdict, and a one-sentence reason.The objects must follow the hypotheses’ listed order.Source: §D.4 · p. 30

Related papers

Loading 2610.04129v1…