PaperlayerView exact revision on arXiv

Source-linked AI summary

GreekBarRetrieval: A Benchmark for Greek Statutory Retrieval

Ernest Beta, Odysseas S. Chlapanis, Dimitrios Galanis, Ion Androutsopoulos

arXiv:2608.18752v1cs.IRcs.CL

TL;DR

Greek statutory retrieval must bridge everyday-language questions and facts to specialized legal terminology, but evidence for Greek remains limited. GreekBarRetrieval benchmarks sparse and dense retrievers with LLM query reformulation, finding that iterative reformulation makes BM25 the strongest method on ranking metrics.

  • Problem

    Greek statutory retrieval is underexplored, and everyday-language questions must be mapped to specialized statutory terminology and question-specific legal issues.

  • Method

    GreekBarRetrieval links 283 Greek bar-exam questions and case facts to 6,308 candidate statutory articles, evaluating BM25, dense retrievers, and LLM query reformulation.

  • Results

    0.67 Recall@100: ten-round REACT-like LLM reformulation raised BM25 to this value and produced the highest nDCG and MAP among tested retrievers.

  • Takeaways & Limitations

    Query reformulation is a practical default for retrieval when evidence coverage and computational cost matter, while iterative BM25 suits constrained downstream contexts.

  • Takeaways & Limitations

    Evaluation ground truth is not exhaustive, because additional relevant statutory articles may exist without receiving credit.

Abstract

from arXiv · show

Statutory retrieval is necessary for citation-grounded legal question answering, but remains underexplored for Greek. We introduce GreekBarRetrieval, a public retrieval benchmark derived from, and complementing GreekBarBench, which did not include retrieval. The new benchmark comprises 283 bar-exam questions, each accompanied by the facts of the case it refers to, and 6,308 candidate statutory articles to retrieve from. Questions and facts are stated in everyday language, but need to be mapped to the formal terminology of statutes and their abstract legal concepts. A further complication is that not all of the case facts are relevant to each question of a case. Experimenting with three BM25 variants and nine dense retrievers, we find that vanilla dense retrieval far outperforms vanilla sparse retrieval in Recall@100. However, LLM-based query reformulation helps BM25 close that gap, while also improving dense retrieval. With a ten-round ReAct-like LLM reformulation loop that we introduce, BM25 improves further in Recall@100 and obtains the best nDCG and MAP scores of all tested retrievers. Query reformulation also outperforms pseudo-relevance feedback, sparse-dense fusion, and English translation.

1 Introduction

GreekBarRetrieval addresses the need for citation-grounded Greek statutory retrieval by introducing a public benchmark based on Greek Bar exam questions. Experiments show that dense retrieval and LLM query reformulation substantially improve retrieval, with a ReAct-like reformulation loop making BM25 especially competitive.

  • Retrieval challenge: Everyday-language questions must be mapped to statutes’ specialized terminology and abstract legal concepts, while Greek morphology increases surface variation.These mismatches make exact matching less reliable for BM25, whereas dense retrievers can bridge some semantic gaps but may underweight exact lexical cues.
  • Results: 0.77 Recall@100 was achieved by Gemini-001 versus 0.36 for BM25 in vanilla retrieval, while vanilla reformulation improved BM25 to 0.60 Recall@100.LLM-based query reformulation also improved dense retrieval and largely closed the vanilla sparse–dense gap.
  • Experiments: The study compares BM25 with three Greek-specific preprocessing variants and nine dense retrievers, alongside reformulation, pseudo-relevance feedback, sparse-dense fusion, and English translation.The experiments establish retrieval baselines and assess multiple methods for addressing the language and terminology mismatch.
  • Benchmark: GreekBarRetrieval is a public benchmark derived from and complementary to GreekBarBench, which did not include true retrieval.It evaluates retrieval of relevant statutory articles for Greek Bar exam questions.
  • Results: A ReAct-like LLM query reformulation loop vastly improves BM25 and helps it obtain the highest nDCG and MAP scores among the tested retrievers.The paper presents this loop as a main contribution, while reporting that query reformulation also benefits dense retrieval.

2 GreekBarRetrieval

GreekBarRetrieval is a Greek statutory-retrieval benchmark derived from GreekBarBench, containing 283 queries and 6,308 candidate articles from 23 legal sources. It is challenging because everyday-language case descriptions must be mapped to statutory terminology, while some case facts are irrelevant to individual questions.

  • Retrieval pool: The retrieval pool contains 6,308 complete statutory articles from 23 legal sources, identified by SOURCE::ARTICLE labels.Evaluation operates at the article level, without paragraph or subsection numbers.
  • Dataset construction: GreekBarRetrieval retains 283 queries after filtering GreekBarBench questions by non-empty, fully covered citation sets.The source data contain 310 questions; questions without ground truth or with citations absent from the retrieval pool are excluded.
  • Task challenge: Queries concatenate legal questions with case facts, requiring retrieval from short statutory articles using much longer descriptions of legal cases.Questions and especially facts use everyday language, so retrieval must map them to statutory terminology and abstract legal concepts.
  • Relevance structure: The benchmark contains 775 relevance judgments covering 465 distinct articles, with 2.74 gold articles per query on average.The 465 distinct relevant articles represent 7.37% of the retrieval pool; 56.9% of queries have more than one relevant article.
  • Task challenge: Some facts in a case file are irrelevant to particular questions, adding noise to the retrieval task.Each bar-exam file presents one case followed by multiple questions about that case.

3 Experimental Setup and Retrievers

The experiments evaluate sparse and dense retrievers using ranking and coverage metrics, alongside translation, pseudo-relevance feedback, fusion, and LLM-based reformulation. They also introduce an iterative REACT-BM25 procedure that repeatedly searches, filters evidence, and reformulates queries.

  • Evaluation metrics: Recall@100 is the primary coverage metric, complemented by nDCG@10, nDCG@100, MAP@100, and Recall@10.Recall measures whether required articles are available among retrieved evidence for answer generation or reranking.
  • Baselines and retrievers: The study compares three Greek-text BM25 variants with nine dense embedding models spanning general-purpose, legal-domain, local, and API-served systems.The sparse variants are BM25-GREEKSTEMMER, BM25-SPACY, and BM25-GR-NLP-TOOLKIT; dense models include Gemini-001, Qwen3-8B, Euler-Legal-V1, and others.
  • Additional retrieval strategies: The setup tests English translation, pseudo-relevance feedback, and Reciprocal Rank Fusion to assess alternative or complementary retrieval strategies.Translation uses gpt-oss-120b; PRF expands sparse queries with TF–IDF terms or adjusts dense vectors with Rocchio feedback; fusion uses k = 60.
  • LLM query reformulation: LLM query reformulation removes potentially irrelevant facts and maps everyday-language questions toward the legal terminology and concepts used by statutes.Sparse reformulation produces compact legal keywords, while dense reformulation produces concise rewritten queries.
  • REACT-BM25: REACT-BM25 runs up to 10 BM25 rounds in which an LLM plans queries, observes retrieved articles, retains useful evidence, and explores alternative legal terminology.Retained articles are ranked by retention frequency, earliest retention round, and best BM25 rank, in that order.

4 Experimental Results

Dense retrievers generally outperform vanilla BM25 in Recall@100, but LLM query reformulation—especially the ten-round REACT loop—substantially strengthens sparse retrieval and yields the strongest ranking metrics. The experiments also show tradeoffs between retrieval quality, inference cost, and infrastructure requirements.

  • Main retrieval results: Gemini-001 achieves the highest Recall@100 at 0.77, versus 0.36 for BM25, while Euler-Legal-V1 reaches 0.68 and Qwen3-8B reaches 0.67.Within Qwen3, Recall@100 is 0.67 for 8B, 0.62 for 4B, and 0.45 for 0.6B.
  • Query reformulation: LLM reformulation raises BM25 Recall@100 from 0.36 to 0.60, compared with an increase from 0.67 to 0.73 for Qwen3-8B.The improvements are statistically significant across all five metrics after correction for multiple comparisons.
  • ReAct reformulation: REACT-BM25 reaches 0.67 Recall@100 and records the highest nDCG@10 at 0.43, nDCG@100 at 0.47, Recall@10 at 0.52, and MAP@100 at 0.37.Its feedback loop reviews retrieved articles to guide subsequent queries, improving ranking of relevant articles near the top.
  • ReAct reformulation: REACT-BM25 does not surpass Reform-Qwen3-8B in Recall@100, reaching 0.67 versus 0.73, but significantly outperforms it on nDCG@10, nDCG@100, and MAP@100.The Recall@100 difference was not statistically significant, whereas the ranking-metric differences were significant.
  • ReAct reformulation: REACT-BM25 Recall@100 improves from 0.45 at round 1 to 0.68 at round 10, with most gains occurring in the first round and later rounds continuing to improve coverage.nDCG@10, nDCG@100, Recall@10, and MAP@100 follow the same pattern.
  • Alternative improvements: Reformulation improves BM25 Recall@100 by +0.23, compared with +0.02 for translation, while dense retrieval gains are +0.06 and +0.04, respectively.Parameter tuning, pseudo-relevance feedback, fusion, and translation otherwise have negligible effects on the compared systems.
  • Cost and infrastructure: Reform-Qwen3-8B reaches 0.73 Recall@100 and 0.33 nDCG@10 with two calls and 2.6k rewrite-generate tokens plus 0.3k embedding tokens per query.The experiments explicitly frame retrieval quality against inference cost using nDCG@10, Recall@100, calls, tokens, and wallclock time.
  • Cost and infrastructure: BM25 requires no GPU for retrieval-pool encoding, vector database, or re-indexing after embedding-model replacement, whereas dense systems pay more to build indexes and less per query.REACT-BM25 preserves these BM25 infrastructure advantages at the expense of increased inference time.

5 Related Work

GreekBarRetrieval extends legal retrieval benchmarking to article-level statutory retrieval for Greek bar-exam questions, distinct from GreekBarBench’s reasoning-and-citation evaluation. It builds on sparse, dense, hybrid, reformulation, translation, iterative, and pseudo-relevance-feedback retrieval paradigms while introducing REACT-BM25 for multi-round reformulation.

  • Legal retrieval and RAG benchmarks: Unlike prior legal retrieval benchmarks, GreekBarRetrieval targets article-level statutory retrieval for Greek bar-exam questions, using articles cited in official solutions as gold labels.Related benchmarks cover legally relevant snippets, multiple jurisdictions and document types, Thai legal QA, and U.S. precedent retrieval.
  • GreekBarBench and Greek NLP: Derived from GreekBarBench, GreekBarRetrieval isolates whether systems recover needed statutory articles before answer generation rather than conflating retrieval with reasoning or generation failures.GreekBarBench provides gold relevant articles and distractors but does not evaluate true retrieval.
  • GreekBarBench and Greek NLP: Greek-specific retrieval is motivated by lower NLP resources and morphology that makes lexical matching harder, requiring normalization and stemming alongside Greek-aware BM25.The study also evaluates dense embedding models and Reciprocal Rank Fusion for combining retrievers without score calibration.
  • Query reformulation: The work adapts query-reformulation research to Greek bar-exam questions, facts, and statutory articles rather than legal cases or passages.Prior work addresses generative rewriting, selection of legally salient content from noisy long queries, and rewrite-retrieve-read pipelines.
  • Iterative and agentic retrieval: REACT-BM25 extends iterative retrieval by using retrieved articles to guide later query reformulations across multiple search rounds, while the study also considers translation and pseudo-relevance feedback as alternatives.Translation is used only as a baseline because both queries and articles are originally Greek; PRF updates queries from top-ranked documents treated as relevant.

6 Discussion

Dense retrieval outperforms vanilla sparse retrieval, but query reformulation improves both approaches by aligning equivalent terms and removing irrelevant facts. Iterative REACT-BM25 achieves the strongest ranking metrics, while reformulated dense retrieval is the practical default when evidence coverage and computational cost both matter.

  • Retrieval trade-offs: Dense retrieval outperforms vanilla sparse retrieval, while sparse retrieval remains useful for exact matches such as article references, statute names, and doctrinal expressions.The two approaches rely on different retrieval signals, and simply merging their rankings is not presented as sufficient.
  • Query reformulation: LLM query reformulation improves both sparse and dense retrievers by matching semantically equivalent terms and removing irrelevant case facts.The improvement is particularly strong for BM25 because reformulation supplies some benefits associated with dense retrieval.
  • Iterative sparse retrieval: REACT-BM25 achieves the strongest nDCG@10, nDCG@100, Recall@10, and MAP@100 results, placing relevant articles more consistently near the ranking top.It enables use of conventional inverted term indices rather than embedding models, but does not exceed LLM-reformulated dense retrieval in Recall@100.
  • Computational trade-offs: REACT-BM25 improves top-position ranking at substantial computational cost, making it more useful for small relevant-article sets than for maximizing overall evidence coverage.Rewrite-Qwen3-8B uses one reformulation and one embedding call, processing approx. 2.6k LLM tokens and 0.3k rewritten-query embedding tokens per query.
  • Practical recommendations: Reformulation followed by dense retrieval is the practical default when evidence coverage and computational cost matter, whereas REACT-BM25 suits constrained or unusually difficult queries.Iterative sparse retrieval is useful when only a few articles can be passed downstream, a conventional sparse retriever is required, or extra computation is justified.

7 Conclusions

GreekBarRetrieval is a public benchmark linking Greek bar-exam questions to cited statutory articles, complementing GreekBarBench with retrieval evaluation. Experiments show vanilla dense retrieval outperforms vanilla sparse retrieval, while LLM query reformulation helps BM25 close the gap.

  • Benchmark contribution: GreekBarRetrieval links 283 Greek bar-exam questions to cited statutory articles within a retrieval pool of 6,308 candidates, complementing GreekBarBench.GreekBarBench did not include retrieval.
  • Retrieval findings: Vanilla dense retrieval far outperforms vanilla sparse retrieval in overall evidence coverage, while LLM query reformulation helps BM25 close the gap.The experiments compare three BM25 variants and nine dense retrievers.

Limitations … C.1 Detailed Results of Vanilla Retrievers

The paper identifies important evaluation, methodological, scope, and generalization limitations, then supplements the main experiments with legal-area statistics, source mappings, and detailed vanilla-retriever results. The appendix reports that Qwen3 performance increases with model size, Nomic-v1.5 underperforms sharply, and two BM25 variants tie after rounding.

  • Limitations: The benchmark’s ground truth is limited to articles cited in official Greek bar-exam solutions and may omit other relevant articles.Uncredited relevant articles are possible because the reference set is not exhaustive.
  • Limitations: 283 queries and a mean of 2.74 relevant articles per query make small statistically significant differences warrant caution.LLM-dependent methods were each run once, preventing run-to-run variance estimates.
  • Limitations: BM25 tuning on the full benchmark provides only an upper bound because GreekBarRetrieval lacks a development subset.The authors identify adding a development subset as future work for hyper-parameter tuning.
  • Limitations: Claims about reformulation transferring dense-retrieval benefits to sparse retrieval and filtering irrelevant facts rely mainly on indirect evidence.The authors call for broader analysis and provide only an indicative example in Appendix C.3.
  • Limitations: The study omits re-ranker baselines, whose inclusion could diminish REACT-BM25’s apparent advantages in top-ranked performance and sparse retrieval.Comparisons with sparse retrievers coupled to re-rankers are identified as particularly interesting.
  • Limitations: The benchmark covers only Greek statutory retrieval for bar-exam-style questions, excluding case law, secondary sources, dynamic corpora, and real user sessions.Retrieval is also evaluated independently of end-to-end legal QA, leaving answer-quality effects for future work.
  • A Legal Area Statistics: Table 6 summarizes GreekBarRetrieval by legal area, with multi-label article counts that can exceed the 6,308 unique-article retrieval pool.The table reports queries, gold articles, relevant articles per query, and median article length.
  • B Legal Source (Prefix) to Area Mapping: The appendix maps PREFIX::ARTICLE identifiers to legal sources and areas, and reports all vanilla-retriever configurations beyond the representative systems in Table 3.Within Qwen3, performance rises with parameter count; Nomic-v1.5 obtains 0.08 Recall@100, while BM25-GREEKSTEMMER and BM25-SPACY tie after rounding.

C.2 Results for Translation, Query Reformulation, PRF, and RRF … C.5 Statistical Significance Tests

Query reformulation is the most consistently effective enhancement, substantially improving sparse retrieval and powering REACT-BM25’s gains across rounds. Statistical tests confirm significant improvements over key baselines, with some remaining metric-specific differences.

  • C.2 Results for Translation, Query Reformulation, PRF, and RRF: Query reformulation is the only tested addition that consistently helps both sparse and dense retrievers.It adds 0.24 to BM25-GREEKSTEMMER and 0.20 to BM25-SPACY, versus 0.03 to 0.09 for Greek-capable dense encoders.
  • C.2 Results for Translation, Query Reformulation, PRF, and RRF: PRF changes Recall@100 by at most 0.02, while fusion harms five of eight dense encoders and gives Nomic-v1.5 a +0.24 gain.PRF improves each sparse retriever by +0.01, decreases Qwen3-8B by −0.01 and EmbGemma-300M by −0.02, and fusion benefits Nomic-v1.5 because BM25 exact matching helps.
  • C.3 Query Reformulation Example: A reformulated legal query moves BM25-GREEKSTEMMER’s gold article from outside the top 100 to rank 18 by adding the term παράβαση καθήκοντος.The reformulation matches, after stemming, παραβαίνει τα καθήκοντα in the statutory text.
  • C.4 REACT-BM25 Results: REACT-BM25 improves all five reported metrics at every round, with Recall@100 rising from 0.45 after round 1 to 0.67 after round 10.Most ranking-metric improvement occurs during the first two rounds.
  • C.4 REACT-BM25 Results: Candidate-pool Recall@100 rises from 0.59 after round 1 to 0.77 after round 10, while observer-kept Recall@100 rises from 0.45 to 0.67.By round 10, 98 planner-retrieved relevant query-article pairs affecting 64 queries are absent from the observer-kept output.
  • C.5 Statistical Significance Tests: Query reformulation significantly improves BM25 and Qwen3-8B on nDCG@10, nDCG@100, Recall@100, and MAP@100, while REACT-BM25 significantly improves over Reform-BM25 on all four metrics.The tests use two-sided paired randomization over the same 283 queries, with 100K label swaps and Holm correction across 20 tests.
  • C.5 Statistical Significance Tests: Compared with Reform-Qwen3-8B, REACT-BM25 is significantly better on nDCG@10, nDCG@100, and MAP@100, but not Recall@100.Compared with Gemini-001, REACT-BM25 has significantly lower Recall@100, while the other three metric differences are not statistically significant.

D BM25 Parameter Tuning

BM25 systems use rank_bm25 defaults, while a sensitivity sweep tuned k1 and b for BM25-GREEKSTEMMER on original and reformulated queries. Tuning yields modest Recall@100 gains, especially after reformulation, and can worsen other ranking metrics.

  • Parameter settings: All main-text BM25 systems use rank_bm25 defaults of k1 = 1.5 and b = 0.75.The sensitivity analysis swept k1 from 0.2 to 3.0 and b from 0 to 1, selecting values maximizing Recall@100.
  • Recall@100 gains: +0.05 Recall@100 comes from tuning original queries, compared with +0.01 Recall@100 for reformulated queries.Tuned values were selected on the full benchmark, making these tuned results oracle upper bounds.
  • Reformulation versus tuning: +0.23 Recall@100 comes from adding query reformulation to vanilla BM25, exceeding the gain from parameter tuning.With reformulated queries, tuning for Recall@100 leads to inferior nDCG@10, nDCG@100, Recall@10, and MAP@100 scores.

E Retriever Implementation Details

The retrievers were evaluated with standardized implementations: sparse systems used rank_bm25 with default parameters, while dense systems used released checkpoints without Greek legal-text fine-tuning. Articles were encoded as single passages, and the Greek BM25 variants differed in preprocessing.

  • Sparse retrievers: All sparse systems use rank_bm25 with default parameters, with tuning changes reported separately in Appendix D.The implementation tables provide each retriever’s full identifier and settings.
  • Dense retrievers: All dense systems use released checkpoints without fine-tuning on Greek legal text.Each article is encoded as a single passage.
  • Sparse retrievers: The three Greek BM25 variants differ in preprocessing, while BM25-ENGLISH-SPACY is used only for the English translation baseline.Unless otherwise indicated, systems use default BM25Okapi parameters.

F Further Inference Cost Analysis · G Prompts

The appendix analyzes per-query inference costs and documents the Greek prompt templates used for translation, reformulation, and REACT-BM25. It specifies how prompts, retrieval, observation, and planner feedback interact while supporting reproducibility.

  • F Further Inference Cost Analysis: REACT-BM25’s ten-round cost is dominated by observer-prompt input because the observer receives retrieved article text.LLM token counts include both input and output tokens.
  • G Prompts: The appendix preserves the structure and content of the original Greek prompts, whose English translations support reproducibility without benchmark-result tuning.The actual Greek prompts are included in the code repository.
  • G.1 Translation and Reformulation: The translation baseline translates queries and articles into English, whereas sparse reformulation outputs legal keywords and headings and dense reformulation outputs a prose legal-issue statement.The sparse format is designed for direct BM25 matching.
  • G.2 REACT-BM25: REACT-BM25 uses a fixed planner system prompt, a first-round prompt over Question + Facts, and a follow-up prompt for later rounds.The first-round planner produces a Greek legal keyword query for BM25-GREEKSTEMMER.
  • G.2 REACT-BM25: Each REACT-BM25 round searches the full retrieval pool and returns the top 100 candidate articles for subsequent observation.The observer selects candidates to retain as evidence rather than processing the entire retrieval pool.
  • G.2 REACT-BM25: REACT-BM25 cycles information from planner to retriever to observer and back to the planner through retained articles and newly generated keyword queries.The observer sees the original query and that round’s candidate articles, while later planners also receive prior queries and retained articles.
  • F Further Inference Cost Analysis: Table 17 separates embedding and LLM tokens, reports online calls, runtime, API-equivalent cost, and one-time retrieval-pool embedding cost.Retrieval-pool embedding is treated as a reusable one-time expense rather than a per-query cost.
Loading 2608.18752v1…