Cross-Lingual Information Retrieval in Tetun for Ad-Hoc Search

• Timor-Leste’s multilingual landscape makes cross-language information access important, while Tetun has few supporting algorithms and tools.Tetun and Portuguese are official languages; English and Indonesian are working languages.Source: §1 · pp. 1–2
• The study investigates CLIR for retrieving Tetun documents from Portuguese, English, and Indonesian queries, comparing results with existing monolingual Tetun baselines.Source: §1 · p. 2
• The study manually translates Tetun queries into Portuguese, English, and Indonesian, then automatically translates them back into Tetun for retrieval.Source: §1 · p. 2
• It examines translation quality with BLEU and retrieval effectiveness with P@10, MAP@10, and NDCG@10 across the three source languages.The original Tetun queries serve as BLEU references, and retrieval is compared with established monolingual baselines.Source: §1 · p. 2
• The findings provide initial evidence about Tetun CLIR feasibility and the relationship between translation quality and retrieval effectiveness.Source: §1 · p. 2

• Tetun IR resources, including a text dataset and an ad-hoc retrieval test collection with baselines, support investigation of Tetun CLIR.IR and NLP development for Tetun began in 2022.Source: §2 · p. 3
• Studies have also evaluated LLM-based query translation, including zero-shot models for African languages and translation models for English-to-French retrieval [26].For the Tetun study, recent LLM advances and Google Translate’s addition of Tetun motivate testing both approaches against monolingual baselines.Source: §2 · pp. 3–4
• Prior CLIR studies found that higher BLEU scores did not consistently yield better retrieval, while Google Translate improved or approached baseline performance in some settings.Pecina et al. and Lignos et al. reported BLEU-retrieval discrepancies; CLEF 2009 and English-to-Chinese work reported retrieval benefits or near-monolingual performance with Google Translate.Source: §2 · pp. 3–4

• The study uses Labadain-Avaliadór, containing 33,500 documents, 59 queries, and 5,900 relevance judgments, as its Tetun ad-hoc retrieval collection.Source: §3 · p. 4
• Three native Tetun-speaking volunteer students manually translated the Tetun queries into Portuguese, English, and Indonesian, following guidelines to preserve abbreviations and named entities.Source: §3 · p. 4
• Google Translate and four LLMs translated multilingual queries back into Tetun; BLEU used the original Tetun queries as references.Retrieval used BM25, DFR BM25, and the Hiemstra LM, evaluated with Precision, MAP, and NDCG at rank ten for each source language and compared with monolingual baselines.Source: §3 · pp. 4–5

4 Experimental Setup

• The experimental setup evaluates Google Translate, zero-shot LLM prompting, and few-shot LLM prompting with one example.Source: §4 · p. 5

4.1 Query Translations

• Google Translate uses the API’s default basic-translation settings, while LLMs share a temperature-zero prompt for deterministic output; few-shot prompts add one example query and translation per target language.Source: §4.1 · p. 5

4.2 Cross-Lingual Retrieval

• Cross-lingual retrieval uses PyTerrier and the monolingual Tetun framework, indexing document titles and applying uniform lowercasing and punctuation, special-character, apostrophe, and hyphen removal.Queries translated from Portuguese, English, and Indonesian are evaluated using Labadain-Avaliadór.Source: §4.2 · p. 6

Evaluation and Results

• Translation quality was measured with BLEU, and retrieval effectiveness with P@10, MAP@10, and NDCG@10 for each source language.Table 1 reports BLEU scores for Portuguese, English, and Indonesian translated into Tetun.Source: §5 · p. 6, Table 1 · p. 6
• Under zero-shot translation, Google Translate led Portuguese-to-Tetun BLEU, GPT-4.5 led English-to-Tetun, and Claude 3.7 Sonnet led Indonesian-to-Tetun; Claude led all languages with few-shot prompting.Source: §5 · p. 6, Table 1 · p. 6
• Few-shot BLEU improved across all Portuguese-to-Tetun LLM translations by up to +9.79 percentage points, but English and Indonesian results varied by model.For English, changes ranged from -7.25 to +7.63 percentage points; for Indonesian, only Claude 3.7 Sonnet improved (+1.69), while other models fell by up to -14.22.Source: §5 · p. 6
• Google Translate followed by Hiemstra LM generally outperformed LLM approaches in zero-shot retrieval, except that LLMs scored higher for Portuguese-to-Tetun P@10 and MAP@10.Retrieval effectiveness was assessed using P@10, MAP@10, and NDCG@10.Source: §5 · p. 7
• -26.95 percentage points (Portuguese-to-Tetun), -25.93 (English-to-Tetun), and -23.22 (Indonesian-to-Tetun) were the maximum CLIR deficits against monolingual baselines.Source: §5 · p. 7
• With one-example prompting, Claude 3.7 Sonnet outperformed other models across all languages and retrieval metrics.Compared with zero-shot, Portuguese-to-Tetun retrieval improved by up to +0.93 percentage points; English-to-Tetun dropped by up to -2.21, while Indonesian changes were marginal.Source: §5 · p. 7

Translation quality and retrieval

• Higher BLEU scores did not consistently yield stronger retrieval: Google Translate led Portuguese BLEU while Claude 3.7 Sonnet retrieved better, and GPT-4-preview led English BLEU while Google Translate retrieved better.The discussion attributes the mismatch in part to query-document alignment and ranking behavior that BLEU does not capture.Source: §6 · pp. 7–9

Zero-shot retrieval

• Google Translate paired with the Hiemstra LM achieved the highest scores among evaluated zero-shot configurations, generally outperforming LLM approaches.The authors note this pattern was particularly strong for Indonesian-to-Tetun queries.Source: §6 · p. 7, Table 2 · p. 8

Few-shot retrieval

• Claude 3.7 Sonnet led few-shot retrieval across languages, but one example improved Portuguese results by up to +0.93 percentage points and did not let English or Indonesian LLMs surpass Google Translate.The reported improvement is for Portuguese-to-Tetun; the discussion says English and Indonesian results remained below Google Translate.Source: §6 · p. 7, Table 3 · p. 9

Comparison with monolingual retrieval

• CLIR results remained significantly below monolingual Tetun retrieval, and limited Tetun web documents as of 2023 further constrained LLM processing.The text identifies limitations in translation-model capabilities as a possible explanation for the retrieval gap.Source: §6 · pp. 9–10

Limitations and potential biases

• The study’s short, three-to-five-word queries, lack of systematic translation validation, and student volunteer translators may introduce errors or bias.Not all volunteer translators had formal linguistic training.Source: §6 · p. 10

7 Conclusions and Future Work

• Tetun CLIR performance remains significantly below monolingual retrieval, despite the study’s exploration of Google Translate and large language models.The work explores CLIR feasibility by automatically translating queries from Portuguese, English, and Indonesian into Tetun.Source: §7 · p. 10
• LLM translation effectiveness depends on source language and familiarity with Tetun, while Google Translate can be more reliable for a newly supported language.The authors report that traditional machine translation tools prove more reliable in some cases.Source: §7 · p. 10
• BLEU-measured translation quality does not correlate with retrieval effectiveness, which also depends on query alignment with document representations and retrieval models.The authors argue that CLIR evaluation should move beyond translation-centric metrics.Source: §7 · p. 10
• Future work will expand Tetun’s digital footprint, adapt retrieval strategies to its linguistic characteristics and resource constraints, and examine additional translation metrics.The proposed metrics include ROUGE and ME-TEOR, whose correlation with retrieval effectiveness will also be examined.Source: §7 · p. 10