PaperlayerView exact revision on arXiv

Source-linked AI summary

What the Query Remembers: Query Abstraction Governs Lexical and Dense Retrieval of Classical Arabic Poetry

Omar Farouk Zouak, Selma Mani, Larbi Said Chikh, Raid Abderrezak Ouahioune

arXiv:2610.31821v1cs.CLcs.AIcs.IR

TL;DR

Retrieval for Classical Arabic poetry is largely unstudied, although it matters for digital archives, education, and literary scholarship. The paper benchmarks dense encoders and BM25 across corpora and query registers, finding that query wording changes which retriever wins and whether explanation augmentation helps.

  • Problem

    Arabic poetry retrieval has received little study, despite its relevance to digital archives, education, and literary scholarship.

    Source: §1 · p. 1

  • Method

    The study benchmarks nine dense encoders and BM25 across two poetry corpora, three indexing modes, and two evaluation protocols, and edits query wording to test its role.

    Source: §10 · p. 9

  • Results

    Retriever rankings and explanation-augmentation effects vary with query register; on tip-of-the-tongue queries BM25 reaches 0.746 vs. 0.717 Recall@5 against the 8B encoder.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Known-item and thematic evaluation protocols disagree on the same systems, so reporting only one protocol or query register can misrepresent retrieval performance.

    Source: §9 · p. 9

    The queries were written by the authors rather than collected from real users, so the study cannot establish how often each query register occurs in practice.

    Source: Limitations · p. 10

Abstract

from arXiv · show

How well a poetry archive can be searched depends on what the user still remembers, not only on what the model understands. We show this for Classical Arabic poetry, benchmarking nine dense encoders and BM25 over two corpora, three indexing modes and two evaluation protocols. Which retriever wins is governed by how much of a poem's wording the query still carries. On tip-of-the-tongue queries BM25 leads every dense encoder, including an 8B state-of-the-art model (0.746 vs. 0.717 Recall@5); on abstract queries it falls to eighth out of ten (0.392 vs. 0.748). Editing the wording of the same queries shows that this ranking largely reflects query composition: adding one remembered half-verse raises BM25 by 38 points, rewriting the quoted words by hand leaves BM25 level with the 8B encoder, and BM25's sensitivity to remembered wording is significantly steeper than that of the strongest dense encoder. The same axis controls explanation augmentation, whose effect swings from +37 to -39 points. The mechanism splits by system type: lexical containment predicts BM25's behaviour, while embedding proximity predicts dense encoders. Finally, the field's two standard protocols, known-item retrieval and thematic relevance, dissociate on the same systems, a result reproduced by two blind human annotators and by two open-weights judges, one of them under a rubric with no literal-match criterion. A benchmark that reports one protocol, or one query register, can give a materially wrong picture. Code and data are available at https://github.com/Ellzo/what-the-query-remembers.

1 Introduction

Retrieval for Classical Arabic poetry is largely unstudied, despite its relevance to archives, education, and scholarship. The paper finds that query register shapes which retrieval system wins and whether explanation augmentation helps or hurts.

1 Introduction

  • Classical Arabic poetry combines morphological variation, omitted diacritics, and figurative language, while retrieval for Arabic poetry remains largely unstudied despite its role in archives, education, and scholarship.The paper presents poetry as a stress test for models intended to retrieve by meaning rather than wording.Source: §1 · p. 1
  • BM25 beats all nine dense encoders on tip-of-the-tongue queries (0.746 vs. 0.717 Recall@5), but ranks eighth of ten on abstract queries, where the 8B encoder leads (0.748 vs. 0.392).The abstract-query comparison also reports p < 10−20.Source: §1 · p. 1
  • Adding a remembered half-verse raises BM25 by 37.8 points, versus 12.8 to 23.2 for the 8B encoder; rewriting quoted words lowers BM25 from 1.000 to 0.646, level with that encoder.These edits manipulate the wording of the same queries.Source: §1 · p. 2
  • Explanation augmentation swings from +37 points on abstract queries to −39 points on queries carrying remembered wording, with the largest swing for a lexical retriever.The effect varies with the same query-abstraction axis that governs retrieval rankings.Source: §1 · p. 2

2 Related Work

  • Prior Arabic poetry NLP emphasizes classification and generation, while poetry retrieval remains an unaddressed capability in related work.General-purpose and Arabic-oriented encoders are often trained on prose or similarity data dominated by prose; Arabic-centric benchmarks do not focus on figurative classical verse.Source: §2 · p. 2
  • Prior IR work documents lexical and dense retrievers’ complementary failure modes; this study adds query editing within one corpus and tests when explanation expansion works.Explanation augmentation is document expansion with an interpretive gloss; related work also identifies known-item tip-of-the-tongue retrieval and cautions that LLM judgments require validation.Source: §2 · pp. 2–3

3 Experimental Setup

The task retrieves the best-matching poem for an Arabic natural-language query, using overlapping verse chunks. The experimental setup contrasts query registers and corpora, then evaluates known-item retrieval and thematic relevance under complementary protocols.

3.1 Task Definition

  • Arabic natural-language queries retrieve poems by scoring overlapping three-verse chunks independently and max-pooling their scores to the poem level.Source: §3.1 · p. 3

3.2 Data

  • The corpora are Fann or Flop poems of at most 16 verses and a Diwan sample of 3,500 poems; each verse receives a short AI-generated explanation.A co-author audited 50 generated explanations: 60% were faithful, 32% had a minor error, and 8% misread the verse.Source: §3.2 · p. 3
  • The benchmark includes known-item queries written to elicit remembered wording or abstract descriptions, plus targetless thematic queries written before corpus inspection.Fann or Flop queries simulate half-remembered searches; Diwan queries were deliberately abstract, while thematic queries name a topic only.Source: §3.2 · p. 3
  • 32.8% of Fann or Flop queries share a verbatim three-token span with their target, versus 2.7% for Diwan; mean containment is 0.368 versus 0.151.The query-abstraction spectrum is quantified with verbatim token overlap and lexical containment.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Evaluation

  • Track A measures known-item retrieval with ranking metrics and paired tests; Track B scores thematic result sets on seven aspects using an LLM judge validated against human judgments and replicated with two open-weights judges.Dense encoders receive diacritic and tatweel stripping, while BM25 additionally uses Arabic orthographic unification.Source: §3.3 · p. 4

4 Track A: Known-Item Retrieval

Track A’s winner changes with query register, and within one corpus, remembered wording explains much of that shift; augmentation and overlap effects follow the same axis.

4 Track A: Known-Item Retrieval

  • On Fann or Flop, BM25 ranks first with 0.746 vs. Qwen3-8B’s 0.717 Recall@5; on Diwan, it ranks eighth while Qwen3-8B leads by 36 points.The Fann or Flop gap is not significant (p=.47); the Diwan gap is significant (p < 10−20).Source: §4 · p. 4, Table 2 · p. 5
  • AI explanations raise Diwan Recall@5 by up to +37 points but lower Fann or Flop BM25 from 0.746 to 0.357, a −39-point change.Richer FoF explanations help on Fann or Flop where AI explanations hurt.Source: §4 · p. 4

5.1 Stratifying the existing queries

  • BM25’s Recall@5 falls 37.8 points without a verbatim three-token match, and its loss exceeds Qwen3-8B’s by 16.2 points (Holm-corrected p=.045).On non-quoting queries, BM25 and Qwen3-8B are statistically indistinguishable: 0.622 vs. 0.646 (p=.69).Source: §5.1 · p. 5

5.2 Editing the same queries

  • Appending a target half-verse raises BM25 by 37.8 points at every position, while hand-rewriting quoted words lowers it from 1.000 to 0.646, level with Qwen3-8B.The rewriting comparison preserves mean query length and leaves the information need and proper names intact.Source: §5.2 · pp. 5–6, Table 3 · p. 6

5.3 A graded mechanism

  • Lexical containment better predicts BM25’s retrieval changes, while embedding proximity better predicts all nine dense encoders on Diwan and eight of nine on Fann or Flop.For BM25 on Diwan, lexical containment has ρ = 0.61 vs. 0.35 for the embedding measure.Source: §5.3 · p. 6, Table 4 · p. 6

6 Track B: Thematic Retrieval

Track B evaluates whether retrieved poems are thematically relevant, and its ranking differs sharply from known-item retrieval: BM25 is near the bottom, a pattern also found in blind human judgments and additional judges.

6 Track B: Thematic Retrieval

  • Blind human judgments reproduce the judge’s system ordering, even though annotators were told that lexical overlap is not evidence of relevance.Two authors independently graded pooled results; their ratings align with the judge’s overall-quality scores.Source: §6 · p. 7
  • BM25 ranks last on Track B on both corpora, scoring 1.90 on Fann or Flop and 1.70 on Diwan, and is significantly below eight of nine dense encoders.LaBSE is the exception; it is statistically indistinguishable from BM25 on both corpora.Source: §6 · p. 7, Table 5 · p. 8
  • Two open-weights judges also place BM25 ninth or tenth, and removing the literal-match rubric aspect barely changes the ordering.The neutral rubric leaves Kendall τ at 0.90 on Fann or Flop and 0.97 on Diwan against the original rubric.Source: §6 · p. 7
  • BM25 ranks first on direct textual relevance but last on aspects requiring knowledge of what a poem is about.The contrast distinguishes matching query wording from thematic aboutness.Source: §6 · pp. 7–8

7 The Dissociation

Known-item retrieval and thematic relevance disagree on the same systems, especially for BM25 on Fann or Flop; the protocols assess different retrieval goals.

7 The Dissociation

  • On Fann or Flop, BM25 shifts from first of ten on Track A to last on Track B, a nine-place change; no other system moves more than four.On Diwan, where queries carry no remembered wording, BM25 performs poorly under both protocols and ranks broadly agree.Source: §7 · p. 8
  • The protocol rankings correlate at τ=+ 0.24 (p=0.38) on Fann or Flop and τ= + 0.51 (p=0.047) on Diwan, but removing BM25 restores Fann or Flop agreement.Among dense encoders the protocols largely concur; the dissociation centers on BM25 when remembered wording enables surface matching.Source: §7 · p. 8
  • Track A rewards recovery of a specific remembered poem, whereas Track B tests whether retrieved poems are about a stated theme; both are valid but measure different things.A score under one protocol carries little information about performance under the other.Source: §7 · p. 8

8 Fusion and Deployment

Reciprocal-rank fusion benefits from explanations across query registers, but a lexical arm helps when queries retain remembered wording and hurts on abstract queries. The paper therefore cautions that retrieval rankings and deployment choices depend on query register and evaluation protocol.

8 Fusion and Deployment

  • +6.1 points on Fann or Flop and −9.7 on Diwan distinguish the best fusion configuration with a lexical arm from the best without one.The best Fann or Flop configuration reaches 0.885 versus 0.746 for the best single system; explanation fusion helps everywhere, while the lexical arm helps only where queries carry remembered wording.Source: §8 · p. 8, Table 6 · p. 9
  • An archive serving readers who half-remember lines should keep a lexical arm; one serving thematic browsing should not.Source: §8 · p. 8

9 Discussion

  • A single Recall@5 score depends jointly on the system and query register, while the two standard protocols disagree on the same systems.Source: §9 · p. 9
  • Known-item retrieval and thematic relevance should be reported together because either alone supports a defensible but opposite claim about dense retrieval’s value.Source: §9 · p. 9
  • Because the queries were written by the authors, the results do not establish how often each register occurs among real poetry-archive users.The paper identifies collecting real-user queries as the next step for estimating that mix and informing deployment.Source: §9 · p. 9
  • Dense encoders do worse on older poetry as era moves toward the present, while BM25 shows no such trend; the authors treat this as exploratory.On Fann or Flop, the reported dense-encoder association is +0.27 log-odds of success per era step towards the present (p=.010); the analysis rests on one corpus and four coarse era bins.Source: §9 · p. 9
  • For non-quoting, most figurative queries, the best system reaches roughly 0.65, leaving this slice an open modelling problem.Source: §9 · p. 9

10 Conclusion

The conclusion is that retrieval performance and explanation augmentation depend on how much poem wording a query retains, and that the two evaluation protocols can dissociate. The paper also identifies limits in query representativeness, annotation, and corpus coverage that constrain interpretation.

10 Conclusion

  • Retriever rankings and explanation augmentation are governed by how much of a poem’s wording the query still carries, and editing queries confirms that remembered wording particularly benefits the lexical retriever.A corpus-level ranking therefore reports its query mix as much as its systems.Source: §10 · p. 9

Limitations

  • Because the queries were written by the authors rather than collected from real users, the study cannot establish how often each register occurs in practice; hand rewrites also came from one co-author.Human relevance annotation covers 126 items with moderate agreement, and open-weights judges cover half of the thematic queries.Source: Limitations · p. 10

Ethical Considerations

  • The Diwan sample over-represents frequently anthologised poets and may underperform on marginal traditions; annotation used two authors and no external annotators.The paper cautions that automated retrieval and generated explanations can misread metaphorical, religious, or culturally sensitive language, and should aid rather than replace human literary expertise.Source: Ethical Considerations · p. 10

AI Usage Disclosure

  • All experiments were designed and run by the authors, and released scripts recompute every number from per-query logs.Source: AI Usage Disclosure · p. 10

A Reproducibility

The ablations show that normalization choices and query–document overlap measures behave differently for lexical and dense retrieval. The section also reports a reproducible pipeline and multiple-comparison corrections.

A Reproducibility

  • The pipeline releases ranked lists and derives metrics, tests, stratifications, fusion variants and figures from per-system, corpus, indexing-mode and query-set files.All models were used off the shelf; dense retrieval took 1,568 seconds of GPU time in total, while BM25 ran on CPU in under a minute.Source: A Reproducibility · p. 12

B Normalisation Ablation

  • Removing diacritics raises dense-encoder Recall@5 by 27–47 points; light normalisation is used for dense encoders, while BM25 receives full normalisation.Further orthographic unification costs 8.9 points on jina-v3, 3.5 on LaBSE and 1.9 on gte, and is not uniformly harmful across metrics.Source: B Normalisation Ablation · p. 12

C Which Overlap Measure Predicts

  • BM25 is the only system for which lexical overlap predicts retrieval changes better than embedding overlap; among Diwan dense encoders, higher Recall@5 associates with lower lexical ρ.The dense-encoder association is absent on Fann or Flop, and the fixed-gte embedding measure makes gte’s own correlation non-independent; conclusions persist when gte is excluded.Source: C Which Overlap Measure Predicts · pp. 12–13, Figure 3 · p. 13

D Multiple Comparisons

  • 36 of 39 nominally significant tests remain significant after Holm-Bonferroni correction within test families; the three exceptions are Fann or Flop AI-explanation contrasts.No paper claim depends on the three contrasts that do not survive correction.Source: D Multiple Comparisons · p. 13

E Threshold Robustness of the Stratification

The query-stratification pattern remains robust across overlap thresholds, while edited-query tests probe how wording changes retrieval. These tests include appended half-verses, human and model rewrites, and a deletion control.

E Threshold Robustness of the Stratification

  • BM25 retrieves every quoting query in the top five at thresholds of three to five tokens, and its quotation slope remains significantly steeper than Qwen3-8B’s at thresholds two to five.At the two-token threshold, Qwen3-8B leads the most abstract non-quoting subset, 0.571 vs. 0.429; the gap closes toward parity as the threshold rises.Source: E Threshold Robustness of the… · p. 13, Table 7 · p. 13

F Construction of the Edited Queries

  • The edited-query tests append a target half-verse to 164 non-quoting queries, rewrite quoted wording by hand for 79 queries, and include a token-deletion control.The appended half-verse is drawn from the opening, middle or final verse; hand rewrites preserve the information need and proper names while removing verbatim three-token spans.Source: F Construction of the Edited Queries · pp. 13–14
  • On 55 model rewrites that retained at least part of the information need and left at most part of the quoted wording, BM25 loses 0.473 and Qwen3-8B 0.255 (p=.006).Source: F Construction of the Edited Queries · p. 14
  • Deleting quoted tokens, which removes wording and information while shortening queries from 10.9 to 6.4 tokens on average, costs BM25 0.600 and Qwen3-8B 0.413 (p=.008).Source: F Construction of the Edited Queries · p. 14

G Metric Robustness and Query Covariates

The section tests whether query-overlap findings hold across metrics and covariates, documents the full Track A result reporting, and examines judge agreement and aspect-level scores in Track B.

G Metric Robustness and Query Covariates

  • BM25 has the steepest quotation slope on all four metrics; after Holm correction, Recall@5 and MRR comparisons remain at or below .05, while Recall@1 and nDCG@10 are about .08.Table 8 reports quotation slopes and BM25’s difference against Qwen3-8B.Source: G Metric Robustness and Query Covariates · p. 14, Table 8 · p. 14
  • Overlap features explain 59–85% of each system’s explained variance, and containment’s interaction with BM25 is +2.99 log-odds per standard deviation (p < 10^-15).The containment coefficient remains significant for all ten systems after controls for query and target characteristics.Source: G Metric Robustness and Query Covariates · p. 14

H Full Track A Results

I The Second and Third Judges

  • The second judge agrees strongly with the primary judge on aboutness aspects (ρ=0.65–0.71), but less on metaphor understanding (0.26) and literal match (0.29).The third judge also evaluates a neutral rubric that removes literal match and its implicit-meaning sentence.Source: I The Second and Third Judges · p. 14

J Track B by Judge Aspect

  • BM25 ranks last on nearly all aboutness aspects but highest on direct textual relevance, scoring 4.26 on Fann or Flop and 4.48 on Diwan.The aspect rubric treats high literal-match scores as repeated query wording, and the aboutness-only ordering matches the overall ordering.Source: J Track B by Judge Aspect · pp. 14–15

K Human Annotation

Human judgments support the thematic-relevance ordering, while a known-item spot check estimates how often missed gold poems still yield acceptable results. Additional corpus checks examine quotation locations, era, duplicates and explanation quality.

K Human Annotation

  • Both human raters rank BM25 below jina-v3 on thematic queries (p=.005 and p=.028), and their scores correlate with the judge’s overall-quality scores on shared cells.They independently graded 126 pooled results; quadratic-weighted κ=0.53, with 90% of items within one point.Source: K Human Annotation · p. 15
  • When the strongest system missed the gold poem, its top result was fully acceptable in 34% of 50 cases and at least partly acceptable in 72%.Using corpus-specific acceptability rates raises best-system Recall@5 from 0.717 to 0.813 on Fann or Flop and from 0.748 to 0.834 on Diwan.Source: K Human Annotation · pp. 15–16

L Corpus Checks

  • Of 80 quoted spans, 77 occur in no other poem, and BM25 retrieves the gold poem for all of them; 37 spans are from opening verses, 41 middle verses and 2 final verses.Source: L Corpus Checks · p. 16
  • Dense-encoder success rises by +0.27 log-odds per step toward the present (p=.010), while BM25 shows no such trend (−0.36, p=.14).The difference between these trends is significant (p=.015).Source: L Corpus Checks · pp. 16–17
  • Poet frequency is unrelated to retrieval success for every system, and gold poems do not differ from the remaining corpus in the tested era, genre, length or poet characteristics.Source: L Corpus Checks · p. 17
  • Diwan duplicates affect verse-mode Recall@5 by 0.4 points for one system and leave the other nine unchanged; Fann or Flop has no duplicates.In the explanation audit, 60% of 50 verse explanations were faithful, 32% had a minor error and 8% misread the verse; 20% added substantive content not in the verse.Source: L Corpus Checks · p. 17

M Annotation Guidelines

The evaluation materials distinguish thematic relevance from whether a retrieved poem would satisfy a query, and specify constrained verse paraphrases and multidimensional judge scoring.

M Annotation Guidelines

  • Thematic judgments reject lexical overlap as evidence of relevance, while known-item judgments score whether a retrieved poem would satisfy the query, even if it was not the intended gold poem.Both protocols use 0–2 ratings, and their guidelines and completed judgments are released.Source: M Annotation Guidelines · p. 17

N LLM Prompts and Rubric

  • Verse explanations must be concise, faithful Arabic paraphrases that add no information, while judges score retrieved poems on seven aspects using their first six verses.Each aspect receives a 1–5 score and a one-line justification.Source: N LLM Prompts and Rubric · p. 17
  • The neutral rubric omits direct textual relevance, whose scale treats least literal matching as best.Exact prompt strings are released with the code.Source: N LLM Prompts and Rubric · p. 17
Loading 2610.31821v1…