PaperlayerView exact revision on arXiv

Source-linked AI summary

ARGUS: Role-Aware Event Knowledge Graphs for U.S. Employment-Discrimination Complaints

Sriram Kannan, Swetha Saseendran, Vishnu Vardhan Reddy Kandi, Leslie Barrett, Madhavan Seshadri, Enrico Santus

arXiv:2609.30184v1cs.CL

TL;DR

Legal retrieval can find semantically relevant passages, but does not explicitly represent the participants, temporal structure, and causal relations that matter for comparing legal events. ARGUS builds source-grounded event graphs from employment-discrimination complaints; its graph classifier and document-scoped retrieval improve downstream results, while open-retrieval gains depend on relevant evidence entering the candidate pool.

  • Problem

    Lexical and embedding-based retrieval return largely unstructured text and do not explicitly represent participants, temporal relations, or causal relations relevant to comparing legal events.

    Source: §1 · p. 1

  • Method

    ARGUS converts employment-discrimination complaints into source-grounded EKGs that represent events with participants, legal roles, and supported temporal and causal relations.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    The graph-structured classifier outperforms raw and linearized baselines on held-out claim classification, and EKG-only document-scoped retrieval improves FAQ answer quality, while open-retrieval gains occur only when relevant passages enter the first-stage candidate pool.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    EKGs support organizing and reasoning over relevant legal evidence rather than replacing first-stage retrieval.

    Source: §7 · p. 9

    The 153-complaint corpus from five districts does not establish generalization to other claims, jurisdictions, or legal genres.

    Source: Limitations · p. 9

Abstract

from arXiv · show

U.S. employment-discrimination complaints describe complex event sequences that are not explicitly captured by lexical or embedding-based representations alone. We present ARGUS, a source-grounded pipeline that combines a 5W1H-inspired schema, legal-domain models, and LLM-based structured generation to construct document-level Event Knowledge Graphs (EKGs) from CourtListener complaints. ARGUS extracts fact-bearing statements, builds chunk-level event graphs with participant, temporal, and causal structure, and merges them into document-level representations. We evaluate graph quality through human and multi-model assessment and test downstream utility on claim classification and legal QA. The graph-structured classifier outperforms raw and linearized baselines on the held-out set, and EKG-only retrieval improves document-scoped QA, while open-retrieval gains remain limited by low first-stage candidate recall. These results suggest that EKGs are most useful for organizing and reasoning over evidence once relevant material has been retrieved.

1 Introduction

ARGUS addresses the need to compare legal cases whose events may be described differently, while representing who did what to whom, when, and why. It constructs source-grounded event graphs from employment-discrimination complaints and evaluates their quality and downstream utility.

1 Introduction

  • Lexical and embedding-based retrieval can find relevant passages but do not explicitly represent participants, temporal relations, or causal relations needed to compare legal events.Those structures can determine whether two cases are meaningfully comparable.Source: §1 · p. 1
  • ARGUS converts complaint narratives into source-grounded EKGs that represent events with participants, legal roles, and supported temporal and causal relations.The pipeline identifies fact-bearing statements, groups them into chunks, extracts chunk-level graphs, and merges them into document-level graphs.Source: §1 · p. 1
  • The evaluation combines graph-construction assessment with claim classification and document-scoped FAQ QA, then tests multiple-choice QA and open retrieval in bar-exam experiments.The paper asks whether graphs can be constructed reliably, improve downstream tasks, and generalize to external retrieval settings.Source: §1 · pp. 1–2
  • The graph-structured classifier outperforms raw and linearized baselines on held-out claim classification, while EKG retrieval improves document-scoped QA and open-retrieval gains remain limited by candidate recall.The contribution statement frames these as outcomes of the paper’s downstream evaluations.Source: §1 · p. 2

2 Related Work

Related work spans legal retrieval and representation, event extraction and temporal reasoning, and graph-augmented retrieval. ARGUS applies these strands to source-grounded, document-level event graphs for employment-discrimination complaints.

Legal retrieval and representation

  • Legal retrieval has moved from lexical matching toward neural interaction and learned representations, including paragraph-level case interactions and legal-domain language models.The bar-exam benchmark separates retrieval quality from downstream answer selection.Source: §2 · p. 2

Event extraction and temporal structure

  • General event-extraction systems jointly model entities, relations, or events, while legal work has extracted event participants and temporal information from court decisions and complaints.The 5W1H framework provides a compact representation of core event information.Source: §2 · p. 2

Legal knowledge graphs and graph-augmented retrieval

  • Graph-augmented retrieval includes entity graphs for query-focused summarization, relation-based chunk expansion, and event graphs for multi-event reasoning; ARGUS instead targets source-grounded legal EKGs.Its document-level graphs preserve legal participant roles and temporal and causal relations through staged merging.Source: §2 · p. 2

3 Method

ARGUS builds document-level EKGs through staged fact selection, chunk-level extraction, and graph merging. Its evaluation separates errors across construction stages and assesses extraction and graph quality with automatic, human, and multi-model methods.

3 Method

  • The four pipeline stages are sentence-level fact extraction, semantic chunking, two-stage mini-graph extraction, and deterministic plus LLM-assisted document-level merging.Each component is evaluated separately to distinguish errors from fact selection, event and relation extraction, and merging.Source: §3 · p. 2

Fact extraction

  • Facts are event-bearing statements about concrete actions, communications, decisions, or outcomes, rather than procedural or rhetorical text.LEGAL-BERT or LexLM classifies sentences, and low-confidence cases are passed to an LLM for fact selection.Source: §3 · pp. 2–3

Semantic chunking

  • Semantic chunking groups retained fact sentences into coherent event sequences, topics, or narrative phases while preserving sentence coverage and allowing overlap.Chunk boundaries favor changes in theme, time, or actor and avoid splitting an event across chunks.Source: §3 · p. 3

Event representation

  • The 5W1H-inspired schema records entities, events, legal roles, participant roles, temporal information, source evidence, and confidence; relations are retained only when source-supported.Temporal relations include BEFORE, AFTER, OVERLAP, and SAME-TIME; causal relations include CAUSES, ENABLES, and PREVENTS.Source: §3 · p. 3

Two-stage mini-graph extraction

  • Two-stage extraction first produces grounded entities and events, then adds only source-supported temporal and causal relations using the original chunk and Stage 1 output.Both stages require strict JSON, preserved sentence identifiers, and no unsupported entities, events, dates, or relations.Source: §3 · p. 3

Extraction configuration

  • GEPA optimizes extraction configurations on a quality–cost Pareto frontier using structural checks and LLM judgments; a separate configuration uses FrameNet hints for event typing.Candidate settings include models, prompts, retry instructions, and extraction parameters.Source: §3 · p. 3

Document-level graph merging

  • Deterministic merging normalizes entity names and deduplicates keyed events; LLM resolution judges cross-chunk identity and adds only supported temporal or causal links.Duplicate records accumulate source references, while accepted relations are added after duplicate events are collapsed and edges remapped.Source: §3 · p. 3

Component-level evaluation

  • Fact extraction is tested against a prompted GPT-5 baseline, mini-graphs on 24 chunks receive automatic and human assessment, and document-level graphs are compared with one-shot generation using three LLM judges.The evaluation also reports inter-judge agreement and stability under temperature variation.Source: §3 · pp. 3–4

4 Experimental Setup

The section describes a CourtListener complaint corpus and evaluates ARGUS’s fact extraction, mini-graph quality, and document-level merging.

4.1 Datasets

  • The corpus comprises 153 CourtListener employment-discrimination complaints; their allegations are not adjudicated findings, and preprocessing isolates FACTS narratives from OCR-derived filings.The complaints were collected across five U.S. district courts.Source: §4.1 · p. 4
  • Document-level splits were used to prevent boilerplate and near-duplicate leakage; sentence-level fact extraction inherited its parent document split.Where annotation was complete, district fact rates ranged from 81.7% to 93.0%.Source: §4.1 · p. 4

4.2 Evaluation of Graph Construction

  • 0.7273 Fact-class F1 and 0.8889 precision were highest for LEGAL-BERT on the 100-sentence sample, while GPT-5 recall was 0.9231.On the U.S. holdout, classifier-only and hybrid modes performed similarly; LLM-only extraction was weaker.Source: §4.2 · p. 4, Table 2 · p. 4
  • 0.896 was the Claude Sonnet 4.5 + GEPA mini-graph quality score, with schema validity and coverage both 1.0; open-source models traded semantic and structural strengths.Qwen had higher semantic quality and more events and relations per chunk, while GPT-OSS-120B scored slightly higher on structural and temporal-consistency measures.Source: §4.2 · pp. 4–5, Table 3 · p. 5
  • 85.2% of mini-graphs were aligned for the FrameNet-guided Qwen configuration, versus 78.3% for Claude; model and guidance varied together.Fleiss’ κ was 0.669 for the Qwen configuration and 0.143 for Claude.Source: §4.2 · p. 5, Table 4 · p. 5
  • Only one event was removed across five documents during merging, while entity counts decreased substantially in four cases as repeated mentions were consolidated.This was reported as a structural check before comparison with the one-shot baseline.Source: §4.2 · p. 5, Table 5 · p. 5
  • 93.3% of comparisons favored ARGUS on event grounding, and it was preferred in all comparisons on evidence grounding, edge correctness, and overall quality.Three LLM judges compared staged ARGUS graphs with one-shot graphs.Source: §4.2 · p. 5

5 Experiments on Downstream Tasks

Across classification, clustering, and document-scoped QA, ARGUS’s structured event graphs support stronger downstream results in tasks that use graph structure or event relations. The gains vary by task and question type.

5.1 Classification

  • 0.642 Micro-F1@0.5 for graph-structured EKG-GNN exceeds 0.590 for raw text and 0.583 for linearized EKG on the 22-document held-out test set.The absolute gain over raw text is 0.052 Micro-F1, or 8.8% relative to that baseline.Source: §5.1 · p. 6, Table 8 · p. 6

5.2 Clustering vs. Distance

  • 0.649 silhouette at K = 4 for Core6D exceeds 0.091 for TF-IDF, 0.363 for the full EKG representation, and 0.465 for Legacy7D.The K = 2–40 sweep favors very small K, but K = 4 is used for stronger granularity with high cohesion; validation relies on geometric and LLM-judged diagnostics, not attorney relevance judgments.Source: §5.2 · p. 6

5.3 Document-Scoped Question Answering

  • 0.446 mean Token-F1 for EKG-only retrieval exceeds 0.323 for raw text and 0.363 for hybrid text + EKG; EKG-versus-raw gains are significant across question types with n ≥12.The EKG-only gains are +0.123 F1 over raw text and +0.083 over hybrid; the three-question residual category is not significant (p = 0.25).Source: §5.3 · p. 7, Table 9 · p. 7
  • Temporal-order and causal/enablement FAQ questions show the largest EKG-versus-raw gains, while party-resolution questions show the smallest.The respective gains are +0.157, +0.134, and +0.045; temporal-overlap questions improve by +0.143.Source: §5.3 · p. 7

6 Generalization to External Datasets

On external bar-exam QA, EKG retrieval does not significantly outperform text retrieval in the retrieval-condition comparison, and open-retrieval gains are constrained by first-stage candidate recall. Results support using EKG structure to organize and reason over relevant evidence rather than replace full-corpus retrieval.

6 Generalization to External Datasets

  • The external evaluation tests MBE multiple-choice questions across a retrieval-condition comparison and a large-corpus open-retrieval setting, rather than only employment-complaint FAQs.The two experiments use different retrieval setups and corpus scales and are evaluated separately.Source: §6 · p. 7, §6.1 · p. 8, §6.2 · p. 8

6.2 Experiment B: Open-Retrieval and Oracle-Document Analysis

  • 8% Recall@10 for hybrid Graph-RAG exceeds 1% for BM25 and E5, but relevant passages enter the BM25 top-1,000 pool for only eight of 100 questions.Open Graph-RAG reaches 72% accuracy versus 73% for BM25-RAG and 76% with no passage; the gold-passage condition reaches 79%. The revised oracle-document configuration reaches 82%, but changes retrieval scope, prompting, and context together.Source: §6.2 · p. 8, Table 10 · p. 8

7 Conclusion

ARGUS converts U.S. employment-discrimination complaints into source-grounded Event Knowledge Graphs. Its strongest reported gains are in graph-structured classification and document-scoped QA, while open-retrieval gains depend on relevant passages entering the first-stage candidate pool.

7 Conclusion

  • ARGUS converts U.S. employment-discrimination complaints into source-grounded, document-level Event Knowledge Graphs.The pipeline represents participant, temporal, and causal structure.Source: §7 · pp. 8–9
  • The graph-structured classifier achieves higher held-out Micro-F1 than raw and linearized baselines, and EKG-only document-scoped retrieval improves FAQ answer quality.The reported benefits are strongest on tasks that exploit participant, temporal, or causal structure.Source: §7 · p. 9
  • Open-retrieval gains occur only when relevant passages enter the first-stage candidate pool, while Experiment A shows no significant retrieval-condition difference.The conclusion positions EKGs as tools for organizing and reasoning over retrieved evidence, not replacements for first-stage retrieval.Source: §7 · p. 9

Limitations

The evaluation is limited to U.S. federal employment-discrimination complaints and several small samples, leaving broader legal-domain generalization and end-to-end similar-case retrieval unestablished.

Limitations

  • The 153-complaint corpus covers five districts and does not establish generalization to other claims, jurisdictions, or legal genres.Source: Limitations · p. 9
  • Several evaluations use small samples, including 24 chunks for mini-graph quality, five documents for merger comparison, and 100–117 questions for bar-exam studies.Source: Limitations · p. 9
  • The document-scoped FAQ benchmark and clustering analysis without attorney relevance judgments do not establish end-to-end similar-case retrieval.Human graph evaluation involved four project members rather than legal practitioners, and LLM judges may share system biases.Source: Limitations · p. 9
  • OCR, extraction, coreference, and inferred temporal or causal errors can propagate through the system.The 82% oracle-document configuration also changes retrieval scope, prompting, and context together, so it is not a graph-only ablation.Source: Limitations · p. 9

Ethical Considerations

The ethical discussion cautions against treating automatically extracted legal information as findings or advice. The appendix describes implementation figures, evaluation procedures, model architecture, and fact-extraction reporting.

Ethical Considerations

  • Automatically extracted roles, events, and causal relations can be incomplete or wrong and require attorney review rather than treatment as legal findings.The paper also describes ARGUS as a research prototype for retrieval and organization, not legal advice.Source: Ethical Considerations · p. 9

A Additional Implementation Figures

B.1 One-Shot Event-Graph Prompt

  • The one-shot baseline generates a document-level event graph directly from the full FACTS narrative, without semantic chunking, intermediate mini-graphs, or staged merging.Its prompt requests entities, events, temporal edges, and causal edges in strict JSON.Source: §B.1 · p. 11

B.2 Intra-Model Judge Stability

  • Table 11 reports the fraction of document-level graph judgments unchanged under temperature variation and the corresponding flip rate.Source: §B.2 · p. 11, Table 11 · p. 11

B.3 Full GNN Architecture

  • The EKG-GNN represents merged EKGs with entity and event nodes, participant links, and temporal or causal event links, then applies two mean-neighbor graph-convolution layers.Node descriptions are encoded with LEGAL-BERT; after pooling, a linear classifier predicts 23 labels.Source: §B.3 · p. 11

B.4 Fact Extraction Details

C Detailed Clustering Analysis

The exploratory clustering analysis compares feature configurations and validation patterns, while the FAQ evaluation reports question-type differences between EKG and raw retrieval.

C Detailed Clustering Analysis

  • Run C's added SALI tags and kNN cross-document graph-neighbor feature yield only small differences from Run B, so the analysis establishes no clear advantage for those added features.Run B uses Core6D with a 16-dimensional PCA-compressed EKG-context embedding; Run C adds the SALI and kNN components.Source: C Detailed Clustering Analysis · p. 12
  • Run A produces one dominant cluster and a weak distance–membership association, whereas Runs B and C produce more balanced clusters with negative correlations.Run A's cluster sizes are 111/14/6/2 and its Pearson correlation is approximately −0.06; Runs B and C report Pearson correlations of approximately −0.28 and −0.29, respectively.Source: C Detailed Clustering Analysis · p. 12, Table 14 · p. 12
  • Run B's distance–membership correlation remains similar when validation increases from five to ten documents per cluster (n = 20 versus n = 40).Source: C Detailed Clustering Analysis · p. 12

D.1 FAQ Evaluation

  • For EKG-versus-raw FAQ retrieval, all question categories with at least 12 examples show a significant difference; the three-example temporal-relation category supports no conclusion.The paired comparison covers 299 questions scored under all three retrieval conditions.Source: §D.1 · p. 12

D.2 Bar-Exam Experiment A

The supplied passages report different significance outcomes across FAQ and Bar Exam evaluations: FAQ comparisons are significant, while MBE Experiment A comparisons are not. They also describe Run C’s fused clustering representation and validation analyses.

D.2 Bar-Exam Experiment A

  • Table 19 is described as reporting Experiment A multiple-choice accuracy by retrieval condition (n = 117; Claude Sonnet 5).The supplied passages identify the table’s scope but give no accuracy values.Source: §D.2 · p. 12, Table 19 · p. 13
  • Run C combines Core6D, hybrid PCA context, type-Jaccard graph neighbors, and LLM-assigned SALI tags into one vector for K-means clustering (K = 4).Its validation compares centroid distance with LLM-judged cluster membership (n = 40; Pearson r ≈−0.29, Spearman ρ ≈−0.37).Source: Figure 5 · p. 13, Figure 6 · p. 13

D.3 Bar-Exam Experiment B

  • The Experiment B captions identify full-corpus retrieval Recall@10 and downstream multiple-choice accuracy under open-retrieval, gold-passage, and oracle-document conditions.Table 21 reports Experiment B accuracy using Claude Sonnet 4.5; the supplied passages do not provide its values.Source: §D.3 · p. 13, Table 21 · p. 14
  • Figure 7’s pairwise cluster validation for Core6D and Run A preceded the three-configuration comparison in Table 14.Run A is identified as SALI + Core6D.Source: Figure 7 · p. 13, §D.3 · p. 13
  • All FAQ benchmark comparisons are significant at α = 0.05, with sign-flip permutation tests agreeing at p < 10^-3.For EKG-versus-raw results, categories with n ≥12 are significant; no conclusion is drawn for the temporal-relation category (n = 3).Source: Table 17 · p. 13, Table 18 · p. 13

D.2 Bar-Exam Experiment A

  • No pairwise comparison of MBE accuracy in Experiment A is significant at α = 0.05.The analysis uses exact McNemar tests, paired bootstrap confidence intervals, and Wilcoxon signed-rank tests.Source: §D.2 · p. 12, Table 20 · p. 14
Loading 2609.30184v1…