PaperlayerView exact revision on arXiv

Source-linked AI summary

Annotated Surrogate Retrieval for Polish Statutory Law

Orkun Yiğit Cengiz

arXiv:2608.30929v1cs.CL

TL;DR

The paper addresses retrieval of a single governing Polish statutory article, a setting with limited prior evidence over a full national corpus. It compares surrogate-based cascades with deterministic lexical–dense fusion under controlled examination benchmarks. ASCR-H leads at rank one, while DTF catches up at depth with much lower latency; however, answer accuracy is saturated and surrogate coverage is asymmetric.

  • Problem

    Polish statutory article retrieval lacks evidence on full-corpus, single-reference examination questions, where rank position matters for downstream generation.

  • Method

    The paper evaluates three retrieval designs using index-time document surrogates, reranking, dense and lexical fusion, and deterministic rescoring against baselines and controls.

  • Results

    ASCR-H achieves 72.3% rank-one accuracy and leads at the head, while DTF leads from cutoff twenty at one ninth the latency.

  • Takeaways & Limitations

    The designs occupy complementary cost–quality positions: ASCR-H favors head precision, whereas DTF favors depth coverage under a tight budget.

  • Takeaways & Limitations

    Surrogates cover 27.0% of corpus articles but 100% of benchmark reference provisions, so some rank-one advantage may reflect coverage asymmetry.

Abstract

from arXiv · show

We present a family of retrieval methods for Polish statutory law built on document surrogates: language-model annotations attached to statutory articles at index time. Three designs occupy different points on the cost-quality frontier. ASCR is a surrogate cascade with reranking; ASCR-H fuses a dense list into that cascade; and DTF replaces both language-model stages with three lexical and dense retrievers, weighted reciprocal rank fusion, and a deterministic re-scoring prior, using no model call before generation. We evaluate all three against fourteen lexical, dense, fused and ablated baselines plus four controls, on 300 questions from the 2024 and 2025 Polish bar and legal counsel entrance examinations (264 with their reference article in the corpus), over 82,508 articles from 1,133 acts. On paired McNemar tests, ASCR-H places the reference provision at rank one significantly more often than every other non-oracle configuration except one of its own ablations (eighteen of twenty comparisons significant in its favour at p < 0.005), reaching 72.3% against 61.7% for BM25 and 52.3% for dense retrieval. The advantage is concentrated at the head and does not survive depth: it is significant at cutoffs of one and five, disappears by ten, and by twenty DTF leads on point estimate (86.0% versus 84.5%) at one ninth the latency and less than half the cost. Ablation attributes 27.6 points of rank-one accuracy to the reranking stage alone. We further report that the ranking advantage does not extend to citation accuracy, where DTF matches the oracle ceiling, and three negative results on lemmatisation, pseudo-relevance feedback and query rewriting. Surrogate annotation covers 27.0% of the corpus but every reference provision in the benchmark, an asymmetry we disclose and discuss. Benchmark, per-question outputs and paired significance tests are publicly available.

1 Introduction

The paper studies rank-sensitive retrieval of Polish statutory provisions, where a single governing article must appear near the ranking head. It introduces document-surrogate methods and finds a depth-dependent cost–quality trade-off between reranking precision and deterministic fusion coverage.

  • Motivation: A single governing article must be retrieved from tens of thousands, making rank position especially consequential for downstream generation.Systems equivalent at cutoff ten can differ by more than twenty points at rank one.
  • Motivation: Because nearly every question has one reference provision, rank-one accuracy is a meaningful and generator-sensitive metric.The labels name a single governing provision in all but six of 300 cases.
  • Approach: Document surrogates attach index-time language-model summaries, themes, concepts, and hypothetical questions to articles for retrieval matching.The paper evaluates three surrogate-based designs against fourteen baselines and four controls.
  • Results: 72.3%: ASCR-H places the reference provision first and significantly outperforms nearly every tested non-oracle configuration.ASCR-H fuses a dense list into the surrogate cascade before reranking.
  • Results: From cutoff twenty, DTF leads on point estimate at one ninth the latency, while ASCR-H retains the head-precision advantage.The designs are presented as complements: ASCR-H for head precision and DTF for depth coverage under a tight budget.
  • Results: Lemmatisation, pseudo-relevance feedback, and query rewriting each fail to improve retrieval.The negative results are reported alongside an ablation isolating individual components.

2 Related Work

Prior legal retrieval work covers several jurisdictions and related Polish tasks, but Polish statutory article retrieval over the full corpus remains distinct. The paper positions its benchmark between passage retrieval and legal question answering while building on lexical, dense, fusion, reranking, and representation-expansion methods.

  • Statutory article retrieval: BSARD established non-English statutory retrieval for Belgian law, whereas this work targets Polish statutory articles with examination-based reference provisions.The Polish setting uses a corpus 3.6 times larger than BSARD and mostly single-reference questions.
  • Statutory article retrieval: The target differs from related question-answering benchmarks because it retrieves a governing statutory article rather than gold explanations, caselaw, encyclopedia passages, or binary-question statutes.The task therefore occupies an intermediate position between passage retrieval and legal QA.
  • Legal retrieval benchmarks: Existing multilingual legal benchmarks include Polish, but their tasks focus on classification, named entity recognition, or norm extraction rather than statutory search.Polish national legislation is not indexed as a retrieval target in those listed benchmarks.
  • Polish retrieval: PIRB and Polish-specific dense retrieval provide relevant language resources, but prior Legal Questions retrieval targets lawyer-written answer texts rather than law articles.That resource covers 26,000 passages extracted from more than 1,000 acts.
  • Polish retrieval: Prior evidence predicts weak generalisation for small Polish rerankers, whereas this paper reports the opposite in its reranking ablation.The paper discusses why its small general-purpose model benefits despite that related finding.
  • Retrieval and fusion: The paper combines standard lexical and dense retrieval with reciprocal-rank fusion, reranking, document expansion, hypothetical documents, and pseudo-relevance feedback.These methods address retrieval and vocabulary mismatch at different points in the index–query pipeline.
  • Legal question answering: Retrieval improvements need not improve generated answers, and this benchmark exhibits the same bound more sharply because answer accuracy is saturated.The paper reports answer accuracy for completeness rather than as a retrieval metric.

3 Benchmark and Metrics

The benchmark pairs 300 Polish entrance-examination questions with article-level references in a full statutory corpus, and evaluates retrieval under controlled conditions. Its metrics emphasize rank and citation, while answer accuracy is shown to be saturated for this generator and examination.

  • Benchmark: 300: the benchmark contains 150 joint advocate and legal counsel examination questions from each of 2024 and 2025.The Ministry of Justice publishes the questions with official answer keys.
  • Benchmark: 264: the retrievable subset remains after excluding nine unresolved legal bases and 27 reference articles absent from the corpus.The subset contains 133 questions from 2024 and 131 from 2025.
  • Corpus: 82,508 articles: the corpus spans 1,133 acts and 159,434 embedded chunks, with chunk scores max-pooled to article scores.Max-pooling gives longer articles more scoring opportunities, though the reported effect is small and uncorrected.
  • Corpus and surrogates: Document surrogates annotate 22,241 articles with summaries, themes, concept sets, and hypothetical questions used as derived matching representations.The paper defines a surrogate as a representation standing in for the document during matching.
  • Benchmark scope: The full statutory corpus is 22.6 times larger than the answer-key corpus used by LQuAD-PL, making this benchmark correspondingly harder.The paper notes that its absolute figures therefore sit below the prior resource’s.
  • Metrics: Citation accuracy scores whether the model names the reference provision, while retrieval percentages, MRR, and nDCG@10 are computed over the 264 retrievable questions.Only the first cited article is scored, and article-level matching ignores subdivisions.
  • Protocol: Retrieval is the only variable across configurations; corpus, prompts, generator, temperature, and context size are held fixed.The generator context is capped at 6,000 tokens and individual articles at 6,000 characters.
  • Answer accuracy: 92.7%: the closed-book control answers 278/300 questions correctly, compared with 96.3% for ASCR-H and 94.3% for the oracle.Neither ASCR-H versus closed-book nor ASCR-H versus oracle is statistically significant.

4 Methods

The methods use surrogate annotations and multi-stage retrieval to balance head accuracy, recall, latency, and cost. ASCR and ASCR-H add analysis and reranking, whereas DTF uses deterministic fusion and act-based priors without pre-generation model calls.

  • ASCR: surrogate cascade: ASCR scores surrogate annotations in document and article stages before listwise reranking of the top forty articles.The document stage retains forty acts; the article stage then ranks articles within those acts, followed by one language-model reranking call.
  • ASCR-H: hybrid cascade: ASCR-H adds a dense candidate list before reranking, exposing articles from acts discarded by the surrogate cascade.Its dense branch runs concurrently with query analysis, so it adds no wall-clock time.
  • DTF: deterministic tri-signal fusion: DTF fuses dense, verbatim-text, and surrogate-question retrieval, then applies deterministic rescoring using the act named in the question stem.The three signals address complementary failures involving paraphrase, inflection, statutory terminology, and differences between question and statute wording.
  • DTF: deterministic tri-signal fusion: DTF’s act prior effectively makes retrieval act-scoped: matching non-repealed articles in the scoped act outrank all out-of-scope articles, while repeal can reverse that ordering.The prior demotes rather than deletes out-of-scope articles, preserving fused ordering within and outside the scoped act.
  • DTF: deterministic tri-signal fusion: Act extraction uses five-character token prefixes to collapse Polish case endings without lemmatisation, but the scope prior only rescored retrieved candidates.A scoped act receives no benefit when none of its articles appears in any of DTF’s three retrieval lists; multiple matching acts can also receive the full boost.
  • Baselines and controls: The evaluation compares proposed systems with lexical, dense, fused, surrogate, ablated, and control configurations, including ASCR-H ablations and oracle controls.The configurations include no-rerank, no-analysis, no-covers, no-concepts, closed-book, random, oracle, and oracle-doc controls.

5 Results

ASCR-H delivers the strongest rank-one retrieval, while DTF converges by depth and leads on deeper point estimates at much lower latency. The systems trade head precision against depth coverage through different retrieval mechanisms, and the ranking advantage does not extend to citation accuracy.

  • Rank one: 72.3% of retrievable questions place the reference provision first with ASCR-H, versus 61.7% for BM25 and 52.3% for dense retrieval.ASCR-H records 191 correct rank-one placements among 264 retrievable questions.
  • Depth: 86.0 versus 84.5 at k = 20 marks DTF’s point-estimate lead, which continues at every deeper cutoff and reaches 89.0% at k = 100.The paired difference at k = 20 is not significant, so the authors characterize the pattern as convergence rather than a crossover.
  • Depth: ASCR-H’s rank-one advantage over DTF is significant at k = 1 and k = 5, but not at k = 10.The paired tests report p < 0.001 at k = 1, p = 0.007 at k = 5, and p = 0.52 at k = 10.
  • Beyond rank one: DTF reaches 92.0% document-level hit rate at k=10 and matches the oracle at 70.3% citation accuracy.These metrics show that deeper retrieval and citation accuracy do not follow the same ranking pattern as rank-one performance.
  • Mechanisms and cost: ASCR-H uses two sequential model calls for precise ordering, whereas DTF uses three retrievers, fusion, and a deterministic prior without pre-generation model calls.The designs therefore obtain head precision and depth coverage through mechanisms with different costs.
  • Controls: Oracle-doc attains 37.7% citation accuracy, 9.3 points below the no-context control’s 47.0%.The reported comparison isolates the effect of supplying the correct act while randomizing articles within it.

6 Ablations and Negative Results

The ablations show that reranking drives ASCR-H’s rank-one advantage, while generated representations help mainly when fused with verbatim text. Lemmatisation, pseudo-relevance feedback, and query rewriting do not improve standalone retrieval, and ranking gains do not transfer to citation accuracy.

  • 6.1 Reranking is the dominant component: 27.6 points of Hit@1 disappear when reranking is removed, reducing ASCR-H from 72.3% to 44.7%.MRR also falls from 0.764 to 0.524, while Hit@20 drops by only 9.5 points, indicating an ordering effect.
  • 6.1 Reranking is the dominant component: Reranking is nearly three times the next-largest ASCR-H component effect and has no cheaper substitute.The component ablation identifies it as the largest effect in the study.
  • 6.2 Generated representations help only in fusion: 9.4 points of Hit@1 come from the surrogate Qa term inside ASCR-H, although standalone surrogate fields underperform verbatim text.The surrogate index is therefore useful as a complementary fused signal, not as a replacement for statutory text; its DTF contribution is untested.
  • 6.3 Three negative results: 60.2% versus 61.7% Hit@1 shows that lemmatisation does not improve BM25 on these examination questions.It leads only at k=10 by 0.4 points, plausibly because the questions quote statutory language closely; transfer to free-form practitioner queries is uncertain.
  • 6.3 Three negative results: 6.1 points of Hit@1 are lost by dense pseudo-relevance feedback, despite gains of 6.1 points at k=20 and 9.9 at k=100.Expansion widens the candidate pool while disordering the head, which is unfavorable when rank one matters.
  • 6.3 Three negative results: Query rewriting fails to improve standalone retrieval: dense-rephrased reaches 50.0% versus 52.3% for dense, and bm25-expanded 50.4% versus 61.7% for bm25-raw.Both add roughly 1.1 seconds of latency, whereas query-side generation is useful inside ASCR-H’s cascade.
  • 6.4 Ranking gains do not transfer to citation accuracy: 67.3% citation accuracy for ASCR-H is statistically indistinguishable from BM25, RRF, BM25-lemma, and DTF despite ASCR-H leading ranking metrics.The evidence indicates sensitivity to whether the provision reaches context, but tolerance to its position within that context.

7 Cost and Latency

Cost and latency separate the systems into distinct operating points: multi-call cascades buy head precision at substantial delay, while DTF offers fast, inexpensive deeper retrieval. The preferable configuration therefore depends on whether deployment prioritizes rank-one accuracy or throughput and downstream coverage.

  • Latency: 726–830 ms is the latency range for single-call configurations, versus 7.8–8.0 s for three-call configurations.The gap arises because pre-generation calls are sequential round trips; ASCR-H’s dense branch runs concurrently and adds no wall-clock time.
  • Latency and cost: Two-call systems span 1.8–9.1 s because structured analysis and reranking prompts are much longer than one-line rewrites.Removing a call does not necessarily reduce latency: no-analysis is slower than full ASCR-H, while cost still falls with tokens.
  • Deployment trade-off: 14-second 95th-percentile latency is disqualifying for interactive search but immaterial for asynchronous review or batch enrichment.ASCR-H costs $0.0023 per query, while its 10.6-point rank-one premium over the best call-free configuration may suit non-interactive workloads.
  • DTF trade-off: 820 ms and $0.108 per hundred queries define DTF’s fast, low-cost operating point, with the highest non-oracle Hit@20 at 86.0%.DTF also reaches 89.0% Hit@100, 92.0% document hit rate, and 70.3% citation accuracy, but forfeits 20.4 points at rank one.

8 Conclusion

The paper presents three surrogate-based retrieval designs for Polish statutory law and evaluates them against fourteen baselines and four controls. ASCR-H is strongest at rank one, while DTF matches it at deeper cutoffs with much lower latency.

  • Three surrogate-based designs were evaluated against fourteen baselines and four controls on 300 ministry-published examination questions.
  • ASCR-H places the reference provision at rank one significantly more often than every non-oracle configuration except one of its own ablations.
  • DTF matches ASCR-H from cutoff ten onward at one ninth the latency, placing the designs at opposite ends of a cost–quality frontier.
  • Reranking contributes 27.6 points of rank-one accuracy, while lemmatisation, pseudo-relevance feedback and query rewriting do not improve retrieval.
  • None of the rank-one advantage reaches the generated answer.

Limitations

The benchmark and evaluation leave several important boundaries: surrogate coverage is asymmetric, examination questions are easier than practitioner queries, and some comparisons and outcomes remain untested. Citation scoring, model choice, statistical power and manual settings further limit interpretation.

  • 27.0% of articles have surrogate annotations, but all 264 retrievable reference provisions are covered, confounding surrogate-based comparisons.
  • Examination questions are multiple-choice, closely quote statutes, and name their governing act, making the task easier than practitioner queries.
  • Citation accuracy is scored only at article level, although 69% of reference provisions specify an unscored subdivision, making reported figures an upper bound.
  • Hand-set hyperparameters and an unreported BM25 expansion baseline leave the comparative effects of tuning and document expansion unresolved.
  • Results use one generator, encoder and run, so intervals capture question-sampling variation but not run-to-run variance.
  • The benchmark does not test temporal correctness, complete multi-provision recall, dependent provisions, cross-references or delegated regulations.

Ethics Statement

The benchmark uses publicly available examination materials and statutory texts without personal data or commissioned human annotation. The systems retrieve provisions rather than establish legal correctness, so practical use requires qualified human review.

  • The benchmark uses Ministry-published examinations, official answer keys and statutory texts that are public by law, with no personal data or commissioned human annotation.
  • The systems retrieve provisions rather than interpret or apply them, and none of the metrics measures legal correctness.
  • Practical use belongs behind qualified human review because citation accuracy peaks at 70.3% even with the reference provision in context.
Loading 2608.30929v1…