PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian

Claudiu Creanga, Liviu P. Dinu

arXiv:2610.00406v1cs.CL

TL;DR

Low-resource-language RAG evaluation lacks reliable benchmarks and reference-based metrics are a poor fit. This paper benchmarks Romanian evaluation strategies against human judgments using AdminRo-Eval and Ragas-style decomposition. The best-performing strategy depends on the metric: decomposition leads for Faithfulness, while ranking leads for Answer Relevance.

  • Problem

    Reliable evaluation benchmarks are scarce for low-resource languages, where the signal needed to distinguish system improvement from regression is often absent or noisy.

    Source: §1 · p. 1

  • Method

    The study evaluates Romanian RAG with AdminRo-Eval and native-speaker judgments, including Ragas-style verification of response statements against context.

    Source: §1 · p. 2, §3.3 · p. 4, §5.3 · p. 5

  • Results

    Faithfulness favors granular decomposition (96% with Gemini 2.5 Pro), while Answer Relevance favors comparative ranking (90% with Gemini 2.5 Pro).

    Source: §7 · p. 8

  • Takeaways & Limitations

    English-centric metrics should not be blindly transferred to Romanian, and Gemini 2.5 Pro provides the most robust and consistent baseline reported.

    Source: §8 · p. 9

    The dataset covers formal public-administration documents and 100 manually annotated samples, and Gemini-generated synthetic test data may introduce self-enhancement bias.

    Source: §9 · p. 9

Abstract

from arXiv · show

Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenge for Low-Resource Languages (LRLs), where standard reference-based metrics fall short. This paper investigates the viability of the "LLM-as-a-Judge" paradigm for Romanian by adapting the Ragas framework using next-generation models (Gemini 2.5 and Gemini 3). We introduce AdminRo-Eval, a curated dataset of Romanian administrative documents annotated by native speakers, to serve as a ground truth for benchmarking automated evaluators. We compare three evaluation methodologies - direct scoring, comparative ranking, and granular decomposition - across metrics for Faithfulness, Answer Relevance, and Context Relevance. Our findings reveal that evaluation strategies must be metric-specific: granular decomposition achieves the highest human alignment for Faithfulness (96% with Gemini 2.5 Pro), while comparative ranking outperforms in Answer Relevance (90%). Furthermore, we demonstrate that while lightweight models struggle with complex reasoning in LRLs, the Gemini 2.5 Pro architecture establishes a robust, transferable baseline for automated Romanian RAG evaluation.

1 Introduction

Low-resource-language evaluation lacks reliable benchmarks, leaving the signal needed to distinguish improvement from regression scarce or noisy. The paper adapts Ragas for Romanian with newer Gemini models and introduces AdminRo-Eval.

1 Introduction

  • Reliable benchmarks are scarce for low-resource languages, making it harder to obtain the signal needed to distinguish system improvement from regression.Resources for Romanian complex-reasoning pipelines remain limited.Source: §1 · p. 1
  • Ragas provides reference-free RAG metrics, but its original validation was English-centric and used the dated GPT-3.5 architecture.The paper applies the framework to Romanian with Gemini 2.5 and Gemini 3 models.Source: §1 · p. 1
  • The paper presents the first rigorous non-English evaluation of Ragas and reports cross-lingual portability on Romanian data.It frames the work as a precedent for reference-free evaluation in other low-resource languages.Source: §1 · p. 2
  • The paper benchmarks Gemini 2.5 Pro, Gemini 3 Pro, and Gemini 3 Flash, reporting that some variants align with human judgments better than legacy baselines.The authors argue that model selection should not assume a single evaluator suits every case.Source: §1 · p. 2
  • AdminRo-Eval is a domain-specific benchmark curated from Romanian public-administration documents for evaluating RAG systems in a formal, knowledge-intensive setting.The dataset is intended to support public-sector AI agents answering citizen queries from official documentation.Source: §1 · p. 2

2 Related Work

The paper motivates reference-free evaluation for Romanian RAG, constructs a corpus and contrastive test pairs from official documents, and uses native-speaker judgments as ground truth.

2 Related Work

  • Reference-based metrics are poorly suited to RAG and low-resource settings, while LLM judges and Ragas offer reference-free alternatives for assessing reasoning.The cited limitations include lexical-overlap dependence and sparse or outdated knowledge bases.Source: §2 · p. 2

3.1 Data Source

  • The corpus began with 84 official Romanian PDFs, was filtered to 27 usable documents, and was divided into contiguous three-page blocks yielding 100 chunks.The source institutions include the Romanian Health Ministry, the Senate, and official publishing houses.Source: §3.1 · p. 2

3.2 Data Generation in Romanian

  • For each chunk, Romanian prompts generated a text-supported question and contrastive examples for Faithfulness, Answer Relevance, and Context Relevance, producing 100 pairs per metric.The pairs contrast grounded with ungrounded answers, complete with evasive answers, and original with noisy contexts.Source: §3.2 · p. 4

3.3 Human Annotation

  • Two native speakers used Romanian guidelines and forced-choice comparisons to judge the synthetic examples, providing ground truth for testing automated evaluation.The annotation set comprised 100 generated questions and their corresponding metric-specific pairs.Source: §3.3 · p. 4

4 Framework for Romanian RAG Evaluation

The Romanian evaluation protocol follows Ragas and defines three metrics that distinguish response grounding, answer fit, and retrieval quality.

4.1 Ragas Metrics

  • Faithfulness measures whether every assertion in a generated response can be inferred from the retrieved context.Source: §4.1 · p. 4
  • Answer Relevance measures how well the generated response aligns with the user’s initial inquiry.Source: §4.1 · p. 4
  • Context Relevance assesses whether retrieved context contains the information needed to answer the query while excluding superfluous content.Source: §4.1 · p. 4

5 Automated Assessment Strategies

The study compares three Gemini-based evaluation strategies with human-annotated ground truth. Results differ by metric: granular decomposition leads for Faithfulness, while pairwise ranking leads for Answer Relevance.

5.1 Method 1: Direct Scoring (LLM-as-Judge)

  • Direct scoring asks the model to assign independent 0–10 ratings; Gemini 2.5 Pro aligned with human Faithfulness judgments at 86%, compared with 82% for Gemini 3 Pro.Source: §5.1 · p. 5

5.2 Method 2: Comparative Ranking (LLM-as-Ranker)

  • Comparative ranking replaces absolute scores with a binary preference between two competing answers presented alongside the query.Source: §5.2 · p. 5

5.3 Method 3: Granular Decomposition (Ragas-style)

  • Granular Faithfulness evaluation parses a Romanian response into statements and checks each against context, scoring the ratio of verified statements to total statements.Source: §5.3 · p. 5

5 Automated Assessment Strategies

  • 96% human-annotation agreement for Faithfulness with Gemini 2.5 Pro came from granular decomposition, while comparative ranking reached 90% for Answer Relevance with Gemini 2.5 Pro.Source: §5.3 · p. 5, §5.2 · p. 5

5.3 Method 3: Granular Decomposition (Ragas-style)

  • Granular Answer Relevance achieved 85% agreement with Gemini 2.5 Pro, while Context Relevance scores essential sentences against the chunk total and reached 93% with Gemini 3 Pro.Source: §5.3 · p. 5

6 Results

Across Romanian data, the best-performing evaluation approach varied by metric; English results were higher for the reported Faithfulness and Context Relevancy comparisons.

6 Results

  • The comparison covers three judge models across Romanian low-resource and English high-resource datasets, measuring Faithfulness, Answer Relevancy, and Context Relevancy.Source: §6 · p. 5

6.1 Performance on Romanian Data

  • 96%: Gemini 2.5 Pro led Romanian Faithfulness with Ragas-style decomposition; Comparative Ranking led Romanian Answer Relevancy at 90%.For Faithfulness, Gemini 3 Pro and Gemini 3 Flash scored 84% and 87% with the same decomposition method.Source: §6.1 · p. 5
  • 93%: Gemini 3 Pro with Ragas achieved the highest Romanian Context Relevancy score, while Gemini 3 Flash scored 0%–5% across methods.Source: §6.1 · p. 5

6.2 Performance on English Data

  • 100%: Gemini 3 Pro reached near-perfect English Context Relevancy alignment; Gemini 3 Flash scored 98% on English Faithfulness with Ragas, 11 points above its Romanian result.Source: §6.2 · pp. 5, 8

7 Discussion

The discussion reports weaker evaluation alignment in Romanian than English and argues that judge selection should depend on both language and metric. Gemini 2.5 Pro was strongest across several Romanian benchmarks, while Gemini 3 Pro led Context Relevancy.

7 Discussion

  • Gemini 3 Pro Faithfulness alignment fell from 94% in English to 84% in Romanian, while Gemini 3 Flash Answer Relevancy fell from 84% to 76%.The authors conclude that LRL pipelines cannot rely blindly on English-centric prompts and thresholds.Source: §7 · p. 8
  • 96%: Ragas-style decomposition led Romanian Faithfulness, while Comparative Ranking led Answer Relevancy at 90% with Gemini 2.5 Pro.The discussion associates atomic-claim decomposition with objective metrics and pairwise comparison with holistic relevance judgments.Source: §7 · p. 8
  • Gemini 2.5 Pro outperformed newer Gemini variants on several Romanian benchmarks, particularly Faithfulness, while Gemini 3 Pro led Context Relevancy at 93%.The authors describe Gemini 2.5 Pro as the most consistent all-around Romanian performer.Source: §7 · p. 8
  • Gemini 3 Flash scored near 0% on Romanian Context Relevancy and was judged unsuitable for this metric in the tested setting.The authors attribute the shortfall to insufficient reasoning depth for discerning subtle noise in Romanian retrieved contexts.Source: §7 · pp. 8–9

8 Conclusion and Future Work

The conclusion presents a Romanian RAG evaluation benchmark and finds that automated judge strategies should be tailored to the metric. It identifies dataset expansion and open-weight judges as future directions.

8 Conclusion and Future Work

  • The study introduces AdminRo-Eval and reports that English-centric evaluation approaches cannot be transferred to Romanian without performance degradation.Source: §8 · p. 9
  • Ragas-style decomposition achieved 96% Faithfulness alignment with Gemini 2.5 Pro, while Comparative Ranking achieved 90% for Answer Relevance.The conclusion also cautions against lightweight Flash models for complex LRL reasoning, citing their poor Context Relevance assessment.Source: §8 · p. 9
  • Future work proposes extending AdminRo-Eval beyond public administration into medical and legal domains and evaluating open-weight judges such as Llama 3 and Mistral.The authors note that reliance on proprietary Gemini and GPT models limits accessibility for the LRL community.Source: §8 · p. 9

9 Limitations

The study’s conclusions are bounded by the formal public-administration domain, a 100-sample human ground truth, and possible bias from Gemini-generated synthetic test data.

  • The dataset covers only formal public-administration documents, so findings may not generalize to open-domain or informal settings with higher semantic ambiguity.Source: §9 · p. 9
  • The manually annotated ground truth contains 100 samples, limiting statistical power.Source: §9 · p. 9
  • Gemini generated the synthetic test data, so high agreement scores may partly reflect self-enhancement bias favoring the model’s own generation style.Source: §9 · p. 9

10 Ethics Statement

The study uses official public Romanian documents while emphasizing privacy safeguards and human oversight in public-sector automation.

  • The AdminRo-Eval corpus uses public documents from official Romanian government websites, with manual checks for private citizens’ sensitive PII.Source: §10 · p. 9
  • Human evaluators were volunteer native speakers, and the Gemini judge models may retain biases from their pre-training data.Source: §10 · p. 9
  • Automated public-sector evaluation tools are intended to assist, not replace, human decision-making in administrative processes.Source: §10 · p. 9

A Annotation Pipeline

The annotation pipeline used separate interfaces for human evaluation of Faithfulness, Context Relevance, and Answer Relevance.

  • Faithfulness annotators compared two candidate answers against the question and context, selecting the response strictly supported by the source.Source: Figure 10 · p. 11
  • Context Relevance annotators chose between original context and context modified with extraneous text based on usefulness for answering the prompt.Source: Figure 11 · p. 11
  • Answer Relevance annotators compared a complete, direct answer with an intentionally partial or evasive response.Source: Figure 12 · p. 11
Loading 2610.00406v1…