Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains

Ayoub Kirouane; Christos Petrocheilos

Source-linked AI summary

Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains

Ayoub Kirouane, Christos Petrocheilos

arXiv:2608.05138v1Aug 5, 2026eess.AScs.AIcs.CL

TL;DR

Modern Greek is missing from major retrieval models and benchmarks, limiting evidence about specialist Greek RAG. The paper builds Greek data and adapts retrieval, reranking, and grounded-reading components, finding domain-dependent retrieval behavior and large gains in judged reader quality. It also identifies unresolved evaluation and data limitations.

Problem
Modern Greek is absent from Nemotron retrieval models and major multilingual retrieval benchmarks, despite existing Greek work targeting general generation rather than retrieval and grounded reading.
Method
The paper mines and audits Greek-domain data, adapts retrieval and reranking models, fine-tunes a grounded reader, and constructs the HERA benchmark with independently verified generated items.
Results
The adapted system improves specialist retrieval and reader quality, while combined lexical and dense retrieval outperforms either system alone in both reported regimes.
Takeaways & Limitations
Greek RAG evaluation should measure lexical and dense retrieval together and assess grounding, citation quality, and abstention rather than answer correctness alone.
Takeaways & Limitations
Reader results are single-run LLM-judged estimates, retrieval uses synthetic queries, and abstention remains weak despite improving to 30.5%.

Abstract

from arXiv · show

Modern Greek is absent from NVIDIA's Nemotron retrieval models and from major multilingual retrieval benchmarks, despite being important for retrieval-augmented generation (RAG) in legal, energy, financial, and medical applications. We present an end-to-end adaptation of the Nemotron retrieval stack for Modern Greek, including corpus mining, synthetic supervision, retrieval model training, reranker adaptation, reader fine-tuning, and a new benchmark called HERA. Our study shows that a parameter-free BM25 baseline outperforms several off-the-shelf multilingual dense retrieval models on specialist Greek corpora. After fine-tuning on 65,773 Greek retrieval pairs, a Nemotron 1B embedder improves nDCG@10 from 0.362 to 0.835 and substantially outperforms its unadapted counterpart. The learned language competence transfers to general-domain Greek, although the advantage over BM25 remains domain-dependent. We further adapt a cross-encoder reranker and demonstrate consistent improvements across specialist domains. Finally, we LoRA-tune a Nemotron 30B-A3B mixture-of-experts reader for grounded generation, increasing judged answer correctness from 29.4% to 66.9% while significantly improving faithfulness and citation quality. We also introduce HERA, the first large-scale Greek benchmark for retrieval-augmented generation, and release our adapted models and benchmark to support future research on Greek-language RAG systems.

1. Greek falls off the map

Modern Greek is missing from the retrieval models and benchmarks that shape multilingual evaluation, leaving specialist Greek retrieval and RAG performance unmeasured. The paper addresses this gap by building Greek data and adapting retrieval and grounded-reading components.

The gap: Greek is absent from Nemotron retrieval-model language lists and from BEIR and MIRACL, so its multilingual retrieval performance is not measured.These omissions affect both the models and the evaluation yardsticks used by the field.
The gap: Greek-focused retrieval work differs from existing general-generation efforts by targeting retrieval and grounded reading rather than general Greek generation.The paper explicitly notes that it does not compare against Greek-adapted readers, leaving the base-model contribution unresolved.
Corpus construction: The corpus starts from 407,053 raw pairs and yields 65,773 clean per-query records across energy, legal, finance, medical, and general domains.Most reduction comes from regrouping flat pairs by query; positive-wins cleanup removes mined negatives that are actually the query’s positives.
Corpus construction: Synthetic queries are constrained toward long, formal, fact-seeking questions because unconstrained instruct models produce short keyword-like queries unlike real user requests.The training data therefore emphasizes realistic specialist information needs rather than only easy keyword matching.
Corpus construction: Relevance labels assume that the chunk generating a query is positive, while no human marks relevance and mined negatives provide the main contrastive signal.An audit checks whether queries ask for specifics appearing in their source chunks, supporting the construction-by-design labeling assumption.
Corpus construction: Training uses max_len 4,096 because max_len 512 truncates roughly 87% of pairs and specialist Greek positives have a median length of approximately 1,034 tokens.The passage warns that reproducing the setup at 512 would train on truncated positives.
Reader supervision: Reader supervision combines gold passages with shuffled hard negatives, random gold positions, and explicit abstention examples to design robustness into training.About 20% of examples are abstentions, and the audit reports 95.2% grounded faithfulness with 3 wrong citations in 30,903 examples.

4. A Greek benchmark, because none ex isted

The paper builds HERA, a Greek Wikipedia long-retrieval benchmark with citations, and designs it to test multi-passage retrieval-augmented generation, including abstention. Its evaluation uses model-generated items with independent verification, while acknowledging important validation limits.

Benchmark construction: HERA is a 4,946-item Greek Wikipedia long-retrieval benchmark with citations.It was created because existing Greek resources lacked multi-passage context, mined distractors, citation targets, and unanswerable items.
Benchmark construction: The benchmark is released from Greek Wikipedia under CC-BY-SA-4.0 with attribution and revision identifiers.The benchmark is intended as a reproducible resource for Greek long-retrieval evaluation.
Benchmark construction: The benchmark measures abstention explicitly through a difficulty ladder and a deliberate 25% share of unanswerable items.This targets cases where retrieved context does not contain an answer.
Benchmark construction: Qwen3.5-122B generates questions, answers, and citations, while GLM-5.2 independently verifies answerability, correctness, and form.The generator was selected for fluent Greek so benchmark text would not read as translationese.
Validation and scope: Judged answer correctness is exposed to model-authored references, whereas faithfulness, citation, and abstention use benchmark-anchored structures.The authors state that answer correctness is the most exposed number in the evaluation.
Validation and scope: Human validation is limited to consensus labels from two strong models and an unquantified random hand spot-check, not systematic coverage.The authors identify a sized, logged human audit as the most valuable addition to this evaluation.
Validation and scope: Reader evaluation uses an out-of-corpus benchmark against training data from energy, legal, finance, and medical documents, measuring transfer rather than in-domain fit.A 13-gram overlap check between the benchmark and the instruction pool found zero overlapping rows.

5. The uncomfortable baseline

In-domain Greek specialist retrieval exposes an uncomfortable baseline: BM25 outperforms every tested off-the-shelf dense embedder, while parameter scaling does not close the gap. Out of domain, however, larger models separate cleanly and the 8B model becomes statistically indistinguishable from BM25.

In-domain results: BM25 beats the multilingual dense baseline on energy (+0.170), finance (+0.134), and general validation (+0.029), losing only legal (−0.026).Legal is uniformly easiest: BM25 reaches 0.8840, unadapted Nemotron 0.5257, and adapted Nemotron 1B 0.9497.
Scale and domain: Across a 13× parameter range, every off-the-shelf embedder remains below BM25 in-domain, with 4B and 8B differing by only −0.004.The reported in-domain family values are 0.6801 versus 0.6843 for 8B and 4B, while 0.6B scores 0.6140.
Scale and domain: Out of domain, the same family rises monotonically from 0.5268 to 0.6504 across the 13× range, while the 8B model catches BM25 with a nonsignificant +0.0062.The 95% CI for the 8B comparison is [−0.0057, +0.0177].
Scale and domain: The results indicate that the in-domain flatness is corpus-specific: capacity helps on general Greek text but not on these specialist corpora.The paper attributes this contrast to exposure rather than capacity, within the tested family and two query distributions.

6. A 1B model, adapted, wins

Fine-tuning Nemotron for Greek turns a weak 1B retriever into a strong specialist-domain system, while BM25 remains complementary and the transfer advantage depends on domain.

0.362 to 0.835: fine-tuning raises macro nDCG@10 for the Nemotron 1B embedder, with every domain gaining at least +0.42.The largest gain is in finance, +0.53 from a base score of 0.19.
+0.080: the adapted 1B beats BM25 on specialist retrieval, with 95% CI [+0.072, +0.089].This comparison uses paired bootstrap over queries.
BM25 still beats the adapted models out of domain: it scores 0.6442, exceeding the adapted 1B by +0.081 and the adapted 0.6B by +0.069.The out-of-domain evaluation uses 4,946 queries over 300,000 Greek Wikipedia passages.
+0.027: reciprocal-rank fusion beats BM25 out of domain, while +0.013 fusion gain appears where the adapted dense model leads in domain.The fused system combines dense and lexical rankings rather than replacing either component.
512 dimensions retain 0.823 retrieval quality, making the index 4 × smaller and preserving +0.066 over BM25.Quality remains flat down to 512 dimensions and collapses below BM25 only at 128.

7. The second stage, measured against its floor

Reranking must be measured against the first-stage ranking itself: Greek adaptation consistently makes the second stage useful, while the unadapted reranker’s benefit varies across evaluations and domains.

The no-rerank floor is the necessary control for deciding whether either reranker improves the first-stage ranking.Comparing rerankers only establishes which is better, not whether reranking belongs in the pipeline.
+0.047: the adapted Nemotron reranker improves over the no-rerank floor, while the off-the-shelf reranker changes it by −0.006.On the 750-query evaluation, the adapted gain is significant whereas the off-the-shelf contribution is statistically indistinguishable from zero.
+0.029: on the larger evaluation, the adapted Nemotron reranker beats its off-the-shelf counterpart, with 95% CI [+0.022, +0.037].This result is consistent with the +0.053 gap on the smaller evaluation.
+0.032: the off-the-shelf reranker clears the floor on the larger evaluation, reversing the earlier conclusion that it contributed nothing.Its contribution remains inconsistent across domains and evaluation slices.
The adapted 0.6B reranker scores 0.8383 versus 0.8453 for adapted Nemotron on 750 queries, a difference that is not separable.On 2,580 queries, the larger model leads by +0.020, so adaptation separates the adapted arms from the unadapted model more than parameter count does.

8. Chained: what the reader actually re ceives

Replacing both retrieval stages raises end-to-end retrieval quality and, more importantly, increases the chance that the reader receives an answer-bearing passage. However, these retrieval ceilings are not equivalent to end-to-end answering accuracy.

0.848 nDCG@10 replaces 0.559 when both retrieval stages are adapted, a 52% relative gain concentrated in finance and medical domains.The relative gains are finance +0.41 and medical +0.30.
0.955 Recall@10 means the adapted stack supplies an answer-bearing passage for nearly all queries, versus 0.624 for the off-the-shelf stack.The off-the-shelf stack loses the answer for about 38% of queries; the adapted stack loses it for under 5%.
Recall@10 bounds grounded answering because a reader cannot ground an answer in a passage it never receives.It measures whether the answer is present in the context delivered to the reader, not whether the reader answers correctly.
The retrieval ceiling rises from 0.624 to 0.955, but Section 9 evaluates reading on gold-provided context rather than retrieved context.Ungrounded correctness and the difference between conditional and end-to-end evaluation remain separate issues.
Figure 10 uses identical citation-and-abstention prompts for base and adapted models, so its rows measure compliance with the instruction.The comparison does not test whether one model was asked to cite or abstain while the other was not.

9. A MoE reader that cites and abstains

The adapted mixture-of-experts reader is trained for grounded answering and abstention, improving judged correctness and faithfulness while exposing trade-offs in conditional scoring, context length, and false abstention.

The reader is LoRA-tuned from Nemotron-3-Nano-30B-A3B using a 246,909-row Greek instruction blend containing 40,000 cite-and-abstain examples.The model has 3B active parameters, and the blend includes a 7.2% English replay slice.
66.9% judged answer correctness follows 29.4% after LoRA tuning, while faithfulness rises from 25.2% to 84.5%.Faithfulness improves 3.4×, compared with 2.3× for correctness.
Across thirteen auxiliary benchmarks, Greek improves on six of eight tasks, English regresses on three of five, and the mean change is −0.004.Without a zero-replay control, the study cannot determine what the replay slice prevented.
28.7% → 64.3% correctness over all 3,712 answerable items is lower than the conditional 29.4% → 66.9% figure because answerable refusals count as wrong.The conditional metric credits the adapted arm with about 2.6 points it earns through selective attempts.
False abstention increases from 0.4% to 3.4%, making abstention the adapted reader’s reported regression.The corresponding trade is roughly 361 additional correct refusals against 112 additional wrong ones.

10. Four ways our instruments lied

The paper documents evaluation and training failures that produced plausible but wrong conclusions, then argues for baseline-first measurement, benchmark construction, and domain-bounded adaptation.

Four ways our instruments lied: Attention-only LoRA fell below the untuned base, whereas adapting all linear layers improved substantially on the internal Greek evaluation.The passage attributes the mismatch to capacity outside attention blocks in hybrid Mamba-Transformer stacks.
Four ways our instruments lied: Averaging distributed loss incorrectly can produce a plausible published number that survives inspection but fails when a second instrument disagrees.The passage presents independent auditing as necessary for detecting such failures.
Four ways our instruments lied: A Latin-only MCQ scorer misread Greek-letter answers as random guessing, creating a false catastrophic-forgetting result.The evaluator later accepted Greek and Latin answer letters on both arms.
Four ways our instruments lied: A missing prompt prefix made validation quality appear to fall even while true held-out quality rose.The evaluator omitted a prefix used during training, reversing the apparent validation trend.
What to take away: A lexical baseline remains necessary: on specialist Greek domains an 8B embedder loses to BM25, while on general Greek the 8B model catches BM25.The passage concludes that scale helps conditionally, but language exposure and corpus behavior must be measured first.
Availability: The released collection includes an embedder, reranker, grounded reader, and HERA benchmark, but not the retrieval training corpus.Results can be reproduced against the released models and benchmark, not retrained from scratch.
What we are not claiming: Reader point estimates use an LLM judge with limited human verification and generator-family overlap, while retrieval results use synthetic queries and may overstate production lift.The paper also reports that the retrieval margin is in-domain only and reverses on general-domain Greek.

A Metric definitions

The section defines ranking quality, coverage, lexical retrieval, citation matching, and model-judged answering metrics, while specifying how retrieval aggregates are reported.

Ranking quality: nDCG@k normalizes discounted cumulative gain against the ideal ranking, so nDCG@k lies between 0 and 1.The ranking uses binary relevance values for the top-k documents.
Ranking quality: Retrieval figures are macro means of nDCG@10, reported both as equal-weighted evaluation sets and equal-weighted queries.The macro-over-queries difference is +0.080, from 0.8575 − 0.7772; the macro-over-sets difference is +0.078.
Out-of-domain runs: The HERA retrieval evaluation truncates inputs at max_seq_length 4,096, above the corpus’s roughly 1,870-token 99th percentile.This choice matches training length and makes truncation negligible.
Coverage: Recall@k is the fraction of gold passages present among the top-k retrieved set, and it is the quantity that bounds the reader.The coverage notation uses the retrieved set and gold set for each query.
Lexical baseline: BM25 scores query–document pairs using term frequency, document length, mean document length, and k1 = 1.2, b = 0.75.Greek-aware tokenization applies NFD normalization, strips combining marks, and folds final sigma.
Grounded reading: Exact citation-set match requires the emitted citation set to equal the gold set, making it stricter than F1.A single spurious citation index causes the item to fail outright.
Grounded reading: Faithfulness is judged against retrieved context, whereas correctness is judged against the reference answer on grounded-and-answered items.Both metrics are produced by a served LLM rather than a closed-form calculation.

B What each number was measured on

The evaluations use different query sets, first-stage retrievers, candidate corpora, and scoring conditions, so their absolute results are not interchangeable. HERA separates retrieval, reading, and chained end-to-end failure modes.

Embedder evaluation: Table 2 and Figures 1–6 evaluate embedders on 5,830 held-out queries with all positives as the corpus.Models use their configured maximum lengths and family-specific prompting conventions during scoring.
Reranker evaluation: The reranker’s second evaluation uses 2,580 queries, a fixed fine-tuned Qwen3-0.6B first stage, top-50 candidates, and matched scoring conditions.Results are scored against separate positives-plus-negatives corpora, so paired differences against the floor—not absolute levels—carry across runs.
HERA tracks: HERA provides separable tracks for retrieval, reading, and chained retrieval-plus-reading, allowing failures to be distinguished by stage.The reported analysis exercises the reader track and its robustness slice.
Retrieval metrics: Recall@k and Hit@k test whether a gold passage appears in the top k, while nDCG@k and MRR measure its ranking quality after retrieval.nDCG rewards all relevant passages with rank discounting, whereas MRR focuses on the first relevant hit.

C.2 Reader track

The reader track isolates reading by supplying gold context, then separately measures answer correctness, faithfulness, citations, abstention, and deployment-like end-to-end behavior. Positional robustness tests sensitivity to document count and the gold passage’s location.

Evaluation setup: Reader evaluation uses gold-provided context, separating reading quality from retrieval errors.End-to-end evaluation instead reruns reader metrics on retrieved context, where retrieval and reading errors compound.
Reader metrics: Answer correctness measures semantic agreement with the gold answer using an LLM judge rather than exact match.This accommodates free-form Greek answers and correct paraphrases with different morphology.
Reader metrics: Faithfulness checks whether every response claim is supported by the supplied context and remains independent of correctness.The two measures distinguish unsupported correct answers from grounded but mistaken answers.
Reader metrics: Citation F1 compares cited context positions with gold citation positions, balancing precision against indiscriminate citation and recall against uncited answers.Precision penalizes citing everything, while recall penalizes answering without attribution.
Reader metrics: Abstention is scored separately on unanswerable and answerable items through refusal recall and false-abstention rate.Both measures are needed because always refusing or never refusing can game either metric alone.
Robustness: Positional robustness reports accuracy by document count and by the gold passage’s context position to expose lost-in-the-middle effects.These analyses underpin Figure 11 and the position breakdown in Section 9.

D Generation and judging configuration

Generation and judging use controlled decoding, concurrency, and JSON-constrained evaluation, with identical Greek reader prompts for base and adapted models. Separate judge prompts score abstention, correctness, and faithfulness.

Generation configuration: Generation uses max_new_tokens 256, 512 for a few-shot domain prompt subset, seed 42, two or three queries per chunk, and concurrency 32.Requests target an OpenAI-compatible vLLM endpoint.
Judging configuration: Judging uses temperature 0, one judge call per axis, and a single JSON object output for parseable verdicts.The judge endpoint is probed at startup, and thinking must be disabled at the request layer.
Reader prompting: The evaluation prompts are translated into English for reporting but run in Greek, with identical reader system prompts for base and adapted models.The reader prompt requires answers in Greek, document-only grounding, bracketed citations, and explicit acknowledgment when information is absent.
Judge axes: The abstention judge classifies whether a response refuses because the information is absent or attempts a substantive answer.Its output is restricted to the JSON verdict abstain or answer.
Judge axes: The correctness judge evaluates substantive agreement with the source passage, while the faithfulness judge checks that every response claim is document-supported.Each axis returns a separate Boolean JSON field.