Source-linked AI summary
LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents
Omar El Bachyr, Fred Philippy, Laura Maria Bernardy, Saad Ezzini, Jacques Klein, Tegawende Bissyande
TL;DR
LëtzCross addresses limited evidence on cross-lingual retrieval for visually rich PDFs in low-resource languages. It benchmarks OCR-based text-only and ColPali-style page-image retrievers over Luxembourgish documents and studies fine-tuning across query languages. Page-image retrievers perform better in the system-level comparison, while fine-tuning transfers across languages and benefits Luxembourgish queries when Luxembourgish is included.
Problem
Cross-lingual retrieval is insufficiently studied for visually rich PDF documents and low-resource languages such as Luxembourgish.
Method
LëtzCross represents Luxembourgish PDF pages as images, combines text-focused and visually grounded QA pairs, and evaluates page-image retrievers against OCR-based text-only systems.
Results
Late-interaction page-image retrievers outperform evaluated OCR-based text-only baselines, while fine-tuning transfers across query languages and Luxembourgish-inclusive multilingual training improves Luxembourgish retrieval.
Takeaways & Limitations
LëtzCross provides a benchmark for studying cross-lingual page-level retrieval over Luxembourgish documents in page-image RAG settings.
Takeaways & Limitations
The benchmark is relatively small and dominated by text-focused queries, while text-only versus page-image comparisons are system-level and non-English queries are translated from English.
Abstract
from arXiv · showhide
Recent page-image retrievers such as ColPali have improved retrieval over visually rich documents, yet little is known about how they behave in cross-lingual, low-resource settings. We introduce LëtzCross, a benchmark for cross-lingual page-level retrieval over Luxembourgish PDF documents, with document pages indexed as images and queries provided in English, French, German, and Luxembourgish. The benchmark combines text-focused QA pairs with visually grounded QA pairs, covering both textual and visual retrieval needs in PDF-based RAG. We use LëtzCross to compare OCR-based text-only retrievers with ColPali-style page-image retrievers and find that the latter perform better across query languages in this system-level comparison. We also examine single-language and multilingual fine-tuning. Fine-tuning transfers across query languages, with French yielding the highest mean performance on Luxembourgish queries among the single-language settings. In the multilingual setting, including Luxembourgish gives the strongest results and substantially improves retrieval for Luxembourgish queries.
1 Introduction
LëtzCross addresses limited evaluation of visually rich PDF retrieval in multilingual, low-resource settings by benchmarking Luxembourgish page-image retrieval. It compares retrievers and fine-tuning strategies for cross-lingual queries.
- Current cross-lingual retrieval evaluations emphasize text-centric, high-resource scenarios rather than visually rich PDFs.
- LëtzCross represents Luxembourgish PDF pages as images and combines text-focused with visually grounded QA pairs for multilingual RAG retrieval.
- The benchmark evaluates multilingual text-only retrievers and late-interaction page-image retrievers, alongside query-language-specific and multilingual fine-tuning.
- Late-interaction page-image retrievers outperform the evaluated OCR-based text-only baselines in a system-level comparison.
- Fine-tuning transfers across query languages, with multilingual training most effective when it includes the target low-resource language.
2 Related Work
Related work establishes multilingual dense retrieval and recent vision-language retrieval as foundations, while motivating Luxembourgish as a low-resource setting with limited resources.
- Luxembourgish NLP Resources: Luxembourgish NLP resources and applications are developing through manual efforts, data collection, annotation, and language-model construction.
- Cross-lingual retrieval: Multilingual encoders and dense retrieval architectures enable semantic matching across languages in cross-lingual information retrieval.
- Cross-lingual retrieval: Transformer models including BERT, XLM-R, and DPR underpin contextual multilingual representations and dense retrieval systems.
- Vision Language Retrieval: Vision-language retrieval models encode document screenshots or pages to preserve textual and visual information beyond conventional text-only systems.
3 L¨etzCross Benchmark Construction
LëtzCross is a Luxembourgish page-image QA retrieval benchmark built from validated PDFs, combining manually authored visual questions with automatically generated and validated text-focused questions across four query languages.
- LëtzCross contains Luxembourgish documents represented as page images, with queries in English, French, German, and Luxembourgish.
- The benchmark combines manually authored visually grounded QA pairs for visually rich pages with automatically generated text-focused QA pairs for text-dominant pages.
- Automatic QA generation processes each page as an image and validates answers using exact and semantic matching before manual question-formulation review.
- 90.28% of QA pairs were judged semantically equivalent, while 71.50% contained the exact literal gold answer.
- Finalized English queries are translated into German, French, and Luxembourgish with prompts intended to preserve retrieval intent.
- The final benchmark contains 579 QA pairs over 908 document pages.
4 Methods
The methods construct training data from Luxembourgish Wikipedia PDFs and fine-tune a ColPali-based late-interaction retriever that matches query tokens with page-image patches.
- Training Data: Training data comes from Luxembourgish Wikipedia articles converted to PDFs, filtered by page-count thresholds, and expanded with translated queries.
- Retriever: Each document page is encoded as visual patches and each query as token embeddings in a shared latent space for late-interaction relevance scoring.
- Retriever: The score compares query-token embeddings with document-page patch embeddings through late interaction.
- Training Objective: Training uses the original ColBERT contrastive objective with in-batch negatives, pairing each query with its relevant page image.
- Fine-Tuning: Fine-tuning compares separate English, French, and German models with mixed-language models trained on EN+FR+DE or EN+FR+DE+LB.
5 Experimental Setup
The experiments compare multilingual text-only and late-interaction page-image retrievers for page-level retrieval over Luxembourgish PDFs, using a shared extraction pipeline for text baselines and multilingual evaluation.
- Models: The study evaluates multilingual text-only retrievers and late-interaction page-image retrievers, including bge-m3, Qwen3-Embedding-0.6B, multilingual-e5-large, jina-embeddings-v4, and five page-image models.jina-embeddings-v4 is used in a text-only setting despite being multimodal.
- Text Extraction Pipeline: Text-only retrievers use page-level text extracted with layout detection, Luxembourgish Tesseract OCR, table-structure extraction, and existing PDF text when available.The pipeline uses unstructured[pdf] v0.18.15 under the hi res strategy and combines Poppler rendering with YOLOX layout detection.
- Fine-Tuning Details: Fine-tuning uses ColBERT late-interaction contrastive loss with LoRA adapters, three seeds, three epochs, and an effective batch size of 64.The text-component-only setting adapts textual modules while the text+visual setting also adapts visual transformer blocks; untargeted parameters remain frozen.
- Evaluation Setup and Metrics: Retrieval is evaluated at the page level for English, French, German, and Luxembourgish queries, with nDCG@10 as the main metric and nDCG@k for k ∈{1, 3, 5, 10}.Indexing time per page and query latency per query are aggregated across the four query languages.
6 Results and Analysis
The experiments compare OCR-based text-only retrievers with page-image retrievers, then examine efficiency, cross-lingual fine-tuning transfer, visual adaptation, and multilingual fine-tuning. Strong page-image models outperform text-only baselines in the system-level comparison, while fine-tuning benefits depend on language pairing and multilingual supervision.
- Page-image versus text-only retrieval: Strong page-image retrievers outperform evaluated OCR-based text-only baselines across all query languages in a practical system-level comparison.The retriever families differ in input representation, extraction pipeline, and model architecture, so this is not a controlled modality ablation.
- Page-image versus text-only retrieval: 78.83 nDCG@10 is achieved by the best page-image retriever, improving by 3.07 points over the strongest text-only baseline.The paper notes that a more customized OCR and layout-detection pipeline could affect this gap.
- Page-image versus text-only retrieval: Page-image retrievers outperform the strongest text-only baseline on both text-focused and visually grounded questions, with larger gains on the visually grounded subset.The visually grounded subset contains only 49 QA pairs, so these results are descriptive and should be interpreted cautiously.
- Efficiency: Retriever efficiency depends on implementation choices and visual-token count, while precomputed page representations keep query latency similar across most page-image models.For text-only retrieval, parsing takes 5.382 ± 0.054 s/page and dominates offline preprocessing cost.
- Cross-lingual fine-tuning transfer: Query-language-specific fine-tuning improves every evaluation language; FT (FR) has the highest overall average at 80.96 and the strongest Luxembourgish score at 79.45 (+1.57).FT (EN) transfers strongly to German, reaching 81.84 (+2.24), while FT (DE) has the lowest overall average at 80.22.
- Cross-lingual fine-tuning transfer: Fine-tuning gains persist across retrieval cutoffs, although overlapping uncertainty bands mean small differences between fine-tuning languages should not be over-interpreted.The improvements are therefore not limited to nDCG@10.
- Visual adaptation: Visual-component updates add positive gains beyond text-only adaptation for every training and evaluation language combination, ranging from 0.21 to 2.53 nDCG@10 points.The gain depends on both languages, and larger gains in some German fine-tuning settings show higher run-to-run variability.
- Multilingual fine-tuning: Adding Luxembourgish to multilingual fine-tuning raises Luxembourgish retrieval from 77.48 to 80.02 (+2.54), while the four-language setting reaches 80.94 overall, comparable to FT (FR) at 80.96.The multilingual setting improves all query languages relative to EN+FR+DE and provides the strongest Luxembourgish retrieval performance.
7 Conclusion
LëtzCross provides a first benchmark and empirical study of cross-lingual page-level retrieval over Luxembourgish document pages. The results favor page-image retrievers and show that fine-tuning transfers across languages, with Luxembourgish inclusion especially beneficial for Luxembourgish queries.
- LëtzCross combines automatically generated text-focused QA pairs with manually authored visually grounded QA pairs for Luxembourgish PDF page retrieval.
- Late-interaction page-image retrievers outperform the evaluated OCR-based text-only baselines in a system-level comparison.
- Fine-tuning improves retrieval, with gains transferring across query languages.
- French fine-tuning yields the highest mean performance on Luxembourgish queries among the evaluated single-language settings.
- Including Luxembourgish in multilingual fine-tuning produces the strongest results and substantially improves Luxembourgish-query retrieval.
Limitations
The benchmark has limited coverage of manually authored visually grounded queries, and several design choices constrain how broadly its retrieval comparisons can be interpreted.
- LëtzCross is relatively small, with a limited manually authored visually grounded subset compared with the automatically generated text-focused QA pairs.
- The evaluation is dominated by text-focused queries, making visually grounded results complementary rather than representative of the benchmark.
- The text-only versus page-image comparison is system-level because the pipelines use different document-processing stages.
- Non-English queries are translated from English, which may introduce translation artifacts or English-oriented formulations.
- The findings are specific to Luxembourgish and require validation on other low-resource languages before broader generalization.