ABAI at COLIEE 2026 Task 1: Multi-Stage Retrieval with GraphRAG-Enhanced Meta-Learning, and a Post-Hoc Study of the Cross-Validation-to-Test Gap

• The task suppresses cited cases in a 9,556-document corpus, sharply reducing direct query-to-gold lexical overlap.Source: §1 · p. 1
• The four-stage pipeline uses citation-context multi-view BM25, neural reranking, graph-based features, and a LightGBM meta-learner over 34 features.Source: §1 · p. 1
• Cross-validated F1 was 0.311 versus official-test F1 of 0.177; the cross-encoder was the largest single ablation contributor, at 24% relative.Source: §1 · pp. 1–2
• Under leakage-free internal protocols, threshold transfer costs 0.007 F1, decision quality is flat across chronological quartiles, and test queries are no farther from the training manifold than training queries are from one another.Source: §1 · p. 2
• Missed citations split exactly evenly between candidates never retrieved and candidates retrieved but ranked below the cut, making retrieval and reranking equally worth fixing.Source: §1 · p. 2
• BM25 length-normalisation tuning, event-triple retrieval, and full-document embeddings each lift top-200 recall by three to seven points; leakage-corrected citation-graph features add 0.014 F1 across eight seeds.Source: §1 · p. 2
• Per-query cutoff rules, an off-the-shelf reranker swap, and a date filter do not improve the measured results.Source: §1 · p. 2

2 Related Work

• Earlier approaches use lexical features from document parts [22], SAILER's asymmetric fact encoder [12], and segment embeddings with charge relations in SLR.Source: §2 · p. 2
• GraphRAG [20], citation-graph propagation for statutory retrieval [14], and counterfactual graph learning for case retrieval represent related graph-based directions.Source: §2 · p. 2
• The framework uses citation-context windows rather than legal-function segments, with regex-extracted entities, Leiden communities [21], and Graph Attention Network features [23].Source: §2 · p. 2
• The study tests query-adaptive cutoffs, graph-neighbour candidate expansion, and event-triple retrieval [9], reporting that event-based retrieval reproduces on its corpus.Source: §2 · p. 2
• A label-free query-side embedding-neighbourhood probe adapts out-of-distribution detection for dense retrievers [11], since official test labels were unavailable.Source: §2 · p. 2

3.1 Stage 1: Retrieval and Fusion

• Stage 1 fuses BM25 rankings of full documents and ±150-word citation-context windows with Reciprocal Rank Fusion, then forwards the top 200 candidates.It also computes lexical and citation-context features for the meta-learner.Source: §3.1 · p. 3

3.2 Stage 2: Neural Reranking

• Stage 2 combines a contrastively trained bi-encoder, untuned BGE-M3 signals, and a cross-encoder reranker with selective truncation for long legal documents.The cross-encoder input prioritizes document head and tail, citation windows, and TF-IDF-selected paragraphs within a 500-word budget.Source: §3.2 · p. 3

3.3 Stage 3: Graph-Based Reranking

• Because citation links are suppressed, Stage 3 builds proxy structures from shared legal entities and uses independent GraphRAG Lite and GNN modules for complementary features.The modules operate on different graphs.Source: §3.3 · pp. 3–4

3.3.1 GraphRAG Lite.

• GraphRAG Lite extracts statutes, judges, legal domains, and outcomes, projects shared entities into a document graph, and derives features from three Leiden community resolutions.The feature set includes community indicators, shared-entity counts, and weighted entity overlap.Source: §3.3.1 · p. 4

3.3.2 GNN Reranker.

• The separate GNN reranker applies a two-layer graph attention network to a corpus graph combining bi-encoder nearest neighbors and entity-overlap edges.It produces relevance scores and within-query ranks, using five-fold out-of-fold predictions to avoid information leakage.Source: §3.3.2 · p. 4

3.4 Stage 4: Meta-Learner

• Stage 4 uses LightGBM to combine 34 features from preceding stages, with query-grouped cross-validation, gold-positive injection, stratified negative sampling, and threshold re-optimization.The training positives include 3,484 gold cases outside the BM25 top 200; several hyperparameters were fixed without ablation, and a systematic sensitivity sweep is left for future work.Source: §3.4 · pp. 4–5

4.1 Internal Validation

• Internal validation used both a recent-query chronological holdout for baseline comparisons and five-fold grouped cross-validation for ablations.The reported system F1=0.311 was the full-training-set out-of-fold result, not the chronological holdout result.Source: §4.1 · pp. 5–6

4.2 Official Test Results

• F1=0.177 on the official test set, versus cross-validated F1=0.311; the best run ranked 35th of 54 submissions.The recall-favouring run was ABAI's best official run.Source: Table 3 · p. 6, §4.2 · p. 6

4.3 Per-Stage Retrieval Quality

• Recall@200 is fixed by BM25-RRF, which retrieved 57.8% of training gold positives and therefore bounds every downstream stage's recall.The official test labels were unavailable for test-set per-stage recall or NDCG analysis.Source: §4.3 · p. 6

4.4 Ablation Study

• Removing the cross-encoder caused the largest ablation drop, −24% relative F1; lexical features followed at −7.3%.GNN + BGE-M3 contributed −3.8%, GraphRAG −2.0%, and the bi-encoder −0.3%.Source: Table 4 · p. 6, §4.4 · p. 6
• Graph features were enriched in gold pairs by 1.43–1.72× across four categories, and 470 deeper-pool gold pairs retained a positive aggregate graph signal.A representative pair ranked 63rd by BM25-RRF shared three Leiden communities, three statute citations, and two judge names.Source: Table 5 · p. 7, §4.4 · p. 7

4.5 Analysis: CV-to-Test Gap

• The authors considered a 57.8% first-stage recall ceiling, temporal effects, and threshold miscalibration as candidate explanations for the CV-to-test drop.Within training data, top-200 recall was 53.7%, 57.6%, 59.7%, and 60.2% across chronological quartiles; the training-optimized threshold was 0.547.Source: §4.5 · p. 7

5.1 Protocol

• The analysis uses a chronological 1,400/200/401-query split for its three-way tests and cautions that repeated selection on the newest-20% slice can inflate apparent wins.Differences smaller than the seed spread are treated as unresolved.Source: §5.1 · pp. 7–8

5.2 Re-testing the Three Diagnosed Causes

• Threshold transfer costs 0.007 F1, while decision F1 stays between 0.327 and 0.349 across chronological quartiles.The authors report that neither threshold miscalibration nor within-training-window temporal drift reproduces the test gap under these protocols.Source: §5.2 · p. 8, Table 6 · p. 8
• Two embedding probes find official test queries no farther from the training manifold, but these internal proxies cannot identify the actual test-set shift.The probes compare query text; official test labels were not released.Source: §5.2 · p. 8, Table 6 · p. 8

5.3 Where the Pipeline Actually Fails

• Exactly half of the 1,274 missed citations never entered the pool and half entered but ranked below the cut; error patterns also vary by legal area.Immigration misses are ceiling-dominated, while intellectual-property misses are reranking-dominated.Source: §5.3 · pp. 8–9

5.4 Lifting the Recall Ceiling

• Top-200 recall rises from 0.570 to 0.630 with BM25 tuning, from 0.578 to 0.608 with event triples, and reaches 0.649 with full-content dense fusion.The BM25 result is measured against its own run’s baseline; the levers were measured individually, with no combined result reported.Source: §5.4 · p. 9, Table 7 · p. 9

5.5 Reranking and Decision Levers

• Leakage-corrected citation-graph features add 0.0141 F1 on average across eight seeds, positive on every seed, while naive computation costs 0.18 F1.The correction recomputes citation indegree leave-one-query-out; this is the only reranking-side change reported to survive a seed-paired test.Source: §5.5 · p. 9, Table 8 · p. 10
• Six per-query decision rules lose to the calibrated global threshold, a zero-shot reranker swap costs 0.059 F1, and the full date filter costs 0.015 F1.The selective date-filter variant is neutral at +0.0002 F1; the general-purpose reranker scores NDCG@10 of 0.057 versus 0.287 for the fine-tuned cross-encoder.Source: §5.5 · p. 10, Table 8 · p. 10

5.6 Evaluation Artifacts

• Four findings reversed after protocol corrections: threshold-transfer cost fell from 0.188 to 0.007, dense retrieval improved after bug and truncation fixes, graph features changed from −0.18 to +0.014 F1, and the shift claim was refuted.The authors recommend matching models when fitting and applying parameters, checking positive-class feature distributions, and testing cheaply before scaling.Source: §5.6 · pp. 10–11

6 Limitations

• Official test labels were not released, so the study’s experiments use internal proxies; failure to reproduce a cause on those proxies is weaker than refuting it.The authors therefore cannot directly measure the 43% drop they investigate.Source: §6 · p. 11
• The official candidate corpus had 1,848 documents versus 9,556 in training, a difference internal splits cannot reproduce or audit without labels.The authors identify the smaller, differently composed candidate corpus as a leading untested suspect because precision depends on the candidate pool.Source: §6 · p. 11
• The query-text distribution probe cannot detect shifts in the candidate corpus, citation labels, or properties its embedding spaces do not encode.Using two embedding spaces addresses the single-encoder objection, not the probe’s narrow scope.Source: §6 · p. 11
• Recall levers were measured separately, and enlarging the pool may affect reranking and precision; the combined gain and graph expansion’s precision cost remain unmeasured.The corpus-graph lever expands the pool by a factor of 2.5.Source: §6 · p. 11

• The four-stage hybrid pipeline combines lexical retrieval, neural reranking, graph-based community features, and gradient-boosted meta-learning; removing the cross-encoder costs 24% relative F1.GraphRAG Lite features and the GNN reranker provide smaller gains of 2.0% and 3.8%, respectively.Source: §7 · p. 11
• F1=0.177 on the COLIEE 2026 test set, versus a cross-validated 0.311, ranked the best run 35th of 54 official runs and 15th of 22 teams.Source: §7 · p. 11
• Threshold transfer costs 0.007 F1, decision quality is flat across four chronological quartiles, and test queries are no farther from the training manifold in two embedding spaces.The three proposed explanations do not reproduce on the internal proxies tested.Source: §7 · pp. 11–12
• Misses split evenly between candidates never retrieved and candidates ranked below the cut; three retrieval changes raise top-200 recall by three to seven points, while graph features add 0.014 F1 across eight seeds.Source: §7 · p. 12
• The authors recommend combining recall levers while measuring precision costs, training a domain-specific reranker, and determining whether in-pool misses reflect coverage or quality.A third of surfaced golds are never scored by the cross-encoder.Source: §7 · p. 12
• Four results reversed after protocol corrections; three had inflated expected conclusions, while the graph-feature artifact required a distribution check to detect.The table distinguishes the artifacts, protocol faults, corrections, and before-and-after results.Source: §7 · p. 12, Table 9 · p. 12