PaperlayerView exact revision on arXiv

Source-linked AI summary

RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

Ivan Bondarenko, Roman Derunets, Oleg Sedukhin, Mikhail Komarov, Ivan Chernov, Mikhail Kulakov

arXiv:2605.04523v1cs.CLcs.AIcs.LG

TL;DR

Multi-turn retrieval-augmented generation requires systems to combine dialogue context, retrieved evidence, abstention, and faithfulness. This paper uses a judge-selected ensemble of seven diverse LLMs and ranks first among 26 teams with a conditioned harmonic mean of 0.7827, versus 0.6390 for the strongest baseline.

  • Problem

    Multi-turn RAG requires combining dialogue context, retrieved passages, abstention behaviour, and faithfulness, making it substantially harder than standard RAG.

  • Method

    A heterogeneous ensemble generates candidates from seven diverse LLMs under two prompting variants, while a lightweight GPT-4o-mini judge selects one per instance.

  • Results

    0.7827 conditioned harmonic mean ranked the system 1st out of 26 teams, outperforming the strongest baseline, gpt-oss-120b, at 0.6390.

  • Takeaways & Limitations

    Diversity matters more than scale: complementary failure modes let the ensemble outperform the best single model despite including models 5–50× smaller.

  • Takeaways & Limitations

    MTRAGEval’s empty-context shortcut lets detecting empty reference passages and returning “I don’t know” achieve perfect conditioned scores on unanswerable questions.

Abstract

from arXiv · show

We present our winning system for Task~B (generation with reference passages) in SemEval-2026 Task~8: MTRAGEval. Our method is a heterogeneous ensemble of seven LLMs with two prompting variants, where a GPT-4o-mini judge selects the best candidate per instance. We ranked 1st out of 26 teams, achieving a conditioned harmonic mean of 0.7827 and outperforming the strongest baseline (gpt-oss-120b, 0.6390). Ablations show that diversity in model families, scales, and prompting strategies is essential, with the ensemble consistently beating any single model. We also introduce Meno-Lite-0.1, a 7B domain-adapted model with a strong cost--performance trade-off, and analyse MTRAGEval, highlighting annotation limitations and directions for improvement. Our code is publicly available: https://github.com/RaguTeam/ragu_mtrag_semeval

1 Introduction

MTRAGEval Task B targets faithful answers to final queries in multi-turn dialogues grounded in reference passages, where systems must integrate dialogue context, retrieval, and abstention. RaguTeam addresses this with a judge-selected ensemble of seven diverse LLMs and two prompting variants, ranking first among 26 teams.

  • Task motivation: Task B requires answering the final user query in a multi-turn dialogue using provided reference passages.The task combines dialogue context, retrieved evidence, abstention behaviour, and faithfulness.
  • Judge-selected ensemble: 0.7827 vs. 0.6390 best baseline: the judge-selected ensemble ranked 1st out of 26 teams.GPT-4o-mini selects the best candidate for each instance.
  • Diversity ablations: Diversity ablations show gains over any single model across model families, scales, and prompting strategies.The ensemble uses seven diverse LLMs under two prompting variants.
  • Meno-Lite-0.1: 0.681 HM3 on answerable instances: Meno-Lite-0.1 is a 7B domain-adapted model with a strong cost–performance trade-off.It also provides occasional high-quality selections.
  • Benchmark analysis: Benchmark analysis identifies annotation and metric limitations and proposes directions for improvement.The work’s code is publicly available in the RaguTeam repository.

2 Background

The background frames retrieval-augmented generation as grounding language models in external evidence and highlights the added demands of multi-turn dialogue. It also motivates heterogeneous LLM ensembles with model-based selection for comparing candidate responses when reference metrics are insufficient.

  • Retrieval-Augmented Generation: RAG combines parametric knowledge with retrieved non-parametric memory to ground language models in external evidence.Recent work extends RAG beyond single-turn, text-based question answering to heterogeneous settings, including multimodal evidence.
  • Multi-Turn RAG: Multi-turn RAG requires interpreting the current query alongside dialogue history while remaining faithful to retrieved evidence.Core challenges include resolving coreferences, tracking intent, handling underspecified or unanswerable requests, and avoiding unsupported continuation.
  • Multi-Turn RAG: The mtRAG benchmark contains 110 conversations and 842 tasks across four domains, yet strong RAG systems still struggle with late-turn coherence and faithful generation.MTRAG-UN further targets unanswerable, underspecified, non-standalone, and unclear conversational turns.
  • Model Ensembles and LLM-Based Selection: LLM ensembles exploit complementary strengths across model families, scales, and prompting strategies, while LLM-based evaluation compares candidates when reference metrics miss faithfulness or appropriateness.The paper combines these ideas by generating diverse candidates for selection.

3 System Description

The system combines two prompting strategies with parallel generation from seven heterogeneous LLMs, then uses GPT-4o-mini to select the most faithful response. Category-aware few-shot examples target diverse context conditions, while empty-reference cases trigger a fixed abstention.

  • Pipeline: The three-stage pipeline constructs prompts, generates candidates in parallel from seven heterogeneous LLMs, and selects the final response with a judge.The two prompting strategies are an iteratively refined system prompt and a category-aware few-shot variant.
  • Prompting strategies: Category-aware few-shot prompting uses four medoid-selected exemplars spanning full, empty, and first-turn context patterns, improving robustness especially for unanswerable cases.The examples comprise one from each of the first two categories and two from the empty-history category.
  • Ensemble composition: The ensemble contains four system-prompt-only models and three few-shot models, including Meno-Lite-0.12, a 7B model adapted from Qwen2.5-7B-Instruct.Meno-Lite-0.12 was continued-pretrained on Russian–English educational data and supervised fine-tuned.
  • Judge selection: GPT-4o-mini scores each of seven candidates for faithfulness on a 0 to 1 scale and selects the top-ranked response, breaking ties with validation-set scores.Faithfulness means that all claims are supported by the provided passages.
  • Post-processing: For empty reference contexts, post-processing replaces the ensemble output with “I don’t have an answer,” ensuring reliable abstention.This fixed response is applied to all instances where the reference context is empty.

4 Experimental Setup

The experiments use the 507-instance MTRAGEval Task B evaluation set and author-disjoint dialogue splits, with performance assessed by three metrics combined into the official HM3 score. Validation uses GPT-4o-mini as a single judge, while models run with greedy decoding on A100 GPUs or proprietary APIs.

  • Dataset: The Task B evaluation set contains 507 instances from FiQA, IBMCloud, CLAPnq, and Govt, while training has 842 samples from 110 conversations averaging 7.7 turns.The collections contribute 77, 131, 142, and 157 instances, respectively.
  • Metrics: The evaluation uses RB_alg, RB_llm, and RL_F, with conditioned variants for unanswerable questions and HM3 as the official harmonic-mean score.RB_alg combines BERTScore Recall, BERT-K-Precision, and ROUGE-L; RB_llm is adapted from RAD-Bench, and RL_F comes from RAGAS.
  • Validation: Author_id-disjoint splits allocate 288 training instances to 6 authors, 180 validation instances to 2 authors, and 374 test instances to 5 authors.Rapid iteration uses a stratified 96-instance subset.
  • Validation: GPT-4o-mini agrees strongly with organiser multi-judge scores, reaching Spearman ρ=0.89 unconditioned and ρ=0.95 conditioned.These correlations are reported for the validation procedure.
  • Implementation: Open-weight models run with vLLM on NVIDIA A100 GPUs using greedy decoding, whereas proprietary models are queried through APIs with default settings.The implementation uses different execution paths for open-weight and proprietary models.

5 Results and Analysis

The judge-selected ensemble ranked first on MTRAGEval Task B, outperforming individual models and the strongest baseline, while ablations show that diversity, informed selection, and few-shot prompting drive its gains. Meno-Lite-0.1 contributes niche complementarity at low cost despite limited final-ensemble impact.

  • Main results: 0.7827 conditioned HM3 placed the ensemble 1st among 26 teams, beating the strongest baseline, gpt-oss-120b (0.6390), by +14.4 absolute points.These results are reported on the MTRAGEval Task B final test set.
  • Standalone performance: GLM-4.6 was the strongest individual model at HM3=0.748, whereas the ensemble reached 0.783 with organisers’ judges and 0.819 with the single judge.GLM-4.6 excels on RB_llm, while Gemini-3-Pro-Preview leads on RL_F, providing complementary strengths for judge-based selection.
  • Ensemble versus individual models: 0.785 versus 0.782 showed a small ensemble advantage on Answerable instances, but GLM-4.6 with few-shot exceeded the ensemble on Underspecified instances, 0.381 versus 0.367.The judge can select confident-sounding but non-clarifying responses, while GLM-4.6 with few-shot more consistently requests clarification; across categories, the ensemble still wins overall.
  • Judge versus random selection: +2.5 p.p. on Answerable and +1.9 p.p. on Underspecified were the judge-selection gains over random selection, while RL_F reached 0.998 versus 0.922 on Answerable.Even random selection outperformed most individual models, indicating that candidate diversity itself has value.
  • Meno-Lite-0.1 contribution: Removing Meno-Lite-0.1 changed Answerable and Underspecified performance by only 0.2 and 0.4 percentage points, and the judge selected it in only 2 of 424 instances.Its selected responses nevertheless achieved HM3 ≈0.707, while standalone Unanswerable performance reached an IDK rate of 0.54 despite fewer parameters than Qwen2.5-32B-Instruct.
  • Prompting ablation: Few-shot prompting outperformed iterative prompt refinement, with Underspecified gains of +3.7 p.p. for GLM and +2.1 p.p. for Llama.Category-aware demonstrations were especially effective for edge cases requiring clarification or abstention; ensemble diversity and informed selection were complementary.

6 Critical Analysis of the Benchmark

The benchmark has a major target-leakage flaw: all 97 unanswerable questions use empty reference passages, making a trivial “I don’t know” strategy perfectly conditioned-scoring. It also has annotation coverage gaps, possible metric leakage, and no shared generator–evaluator guidelines; the authors recommend distractors for unanswerable instances.

  • Target leakage: 97 unanswerable questions have empty reference passages, so detecting empty context and returning “I don’t know” can yield perfect conditioned scores.This creates a trivial target-leakage shortcut.
  • Target leakage: In realistic deployments, unanswerable questions include irrelevant retrieved passages rather than empty context.The paper recommends adding distractor passages to unanswerable benchmark instances.
  • Additional limitations: The benchmark also exhibits annotation coverage gaps, possible metric leakage from reference-less submission-set scores, and no shared generator–evaluator guidelines.These limitations are identified through manual examination and detailed in Appendix K.

7 Discussion and Conclusion

The heterogeneous ensemble’s success is attributed to model and prompting diversity, while Meno-Lite-0.1 offers a cost-efficient alternative. The discussion also identifies computational, judging, and benchmark limitations and proposes more adaptive, multi-objective evaluation and routing.

  • Key Insights: HM3=0.783 secured 1st place in MTRAGEval Task B, surpassing the strongest baseline at 0.639.The result came from heterogeneous ensemble generation with judge-based selection.
  • Key Insights: HM3 = 0.783 vs. 0.748 shows diversity can outperform the best single model despite including models 5–50× smaller.The comparison is against GLM-4.6, a 357B model; complementary failure modes are presented as more important than raw parameter count.
  • Key Insights: +3.7 p.p. improvement shows category-aware few-shot exemplars outperform an iteratively refined system prompt for underspecified questions.The finding suggests concrete behavioural demonstrations handle edge cases more effectively than abstract prescriptive instructions.
  • Key Insights: 0.555 on MERA shows Meno-Lite-0.1, a domain-adapted 7B model, tied Llama-3.3-70B performance on Russian benchmarks.It is presented as a strong cost–performance trade-off for resource-constrained settings.
  • System Limitations: Seven generators plus a judge are computationally expensive, sequential generation and judging add unsuitable latency, and proprietary APIs remain partly necessary.GPT-4o-mini also occasionally favours longer responses.
  • System Limitations: Spearman ρ=0.95 conditioned indicates strong judge correlation, but judge dependence may fail across different model failure modes.The paper motivates open-weight judges and multi-judge panels, while noting a fully open-weight pipeline remains underexplored.
  • Benchmark Limitations: Empty references for all 97 unanswerable questions incentivise answerability detection, while reference-less faithfulness computation enables partial metric leakage.The proposed remedies are distractor passages for unanswerable questions and restricting reference-less metric computation to evaluation.

A Meno-Lite-0.1 Details … E Grouped Analysis

The paper develops Meno-Lite-0.1 as a compact, context-grounded model and designs prompting strategies through behaviour analysis, LLM synthesis, and category-aware exemplars. Grouped results indicate that the system-prompt-only strategy outperforms the system-plus-few-shot strategy across the evaluated metrics.

  • A Meno-Lite-0.1 Details: Meno-Lite-0.1 is a 7B model optimized for context-grounded tasks, including RAG question answering, extraction, multi-hop reasoning, and knowledge graph construction.Its design prioritizes cultivated language skills over broad factual recall and targets deployment on a single consumer GPU.
  • A Meno-Lite-0.1 Details: 3.77 characters per token on Russian text represents a 47% improvement over Qwen2.5’s 2.57, while English efficiency remains 4.13 characters per token.The inherited extended vocabulary is intended to reduce inference time and serving costs, especially for Russian workloads.
  • A Meno-Lite-0.1 Details: 0.555 on MERA matches Llama-3.3-70B-Instruct and exceeds Qwen2.5-7B-Instruct’s 0.482, while MultiQ reaches 0.536/0.403 versus GPT-4o’s 0.572/0.431.The lower CheGeKa score of 0.346/0.293 reflects the model’s intended trade-off toward context-grounded skills.
  • B LLM-Based Design of System Prompt for Group 1: The system prompt is refined by analysing agent behaviour on 96 validation instances, synthesizing characteristics with Gemini, and applying a second self-critique iteration.The analysis checks document-only grounding and incorrect or controversial claims, producing numbered behavioural characteristics for prompt synthesis.
  • B.1 LLM-Based Analysis of Results: The behavioural analysis evaluates whether responses use only retrieved documents and whether claims are incorrect or controversial, then records characteristics such as neutral synthesis and context grounding.These analyses are generated per sampled instance before being consolidated into a system prompt for a 7B-scale agent.
  • C Final System Prompt for Group 1: Group 1’s final prompt encodes strict context adherence, extractive phrasing, depersonalized synthesis, concise direct delivery, and fidelity to source structure.The prompt was selected for Group 1 models over the self-critiqued revision P2 based on Table 5.
  • D Few-Shot Exemplars for Group 2: Group 2 augments a simpler base prompt with four category-aware few-shot exemplars selected from empty-context, empty-history, and non-empty-context/non-empty-history cases.The exemplars demonstrate abstention without documents, document-only answering, and multi-turn coreference resolution.
  • E Grouped Analysis: Group 1 outperforms Group 2 across all three evaluated metrics when models are aggregated by prompting strategy and licensing.Different base models were intentionally assigned to the groups to increase architectural and behavioural diversity and enable per-instance complementarity.

F Analysis by Question Category

Performance varies substantially by answerability category: large models are relatively homogeneous on answerable questions, while unanswerable and underspecified cases expose major differences in abstention and clarification behavior. GLM-4.6 leads on partially answerable questions, whereas Qwen3-235B-A22B-Instruct-2507 leads on underspecified ones.

  • Answerable questions: 0.750–0.779: large models perform relatively homogeneously on answerable questions, while Meno-Lite-0.1 scores 0.681 but exceeds the organiser baseline.Claude 4.5 Haiku and Qwen2.5-32B-Instruct match GLM-4.6 in this category.
  • Unanswerable questions: 0.794: Claude 4.5 Haiku achieves the strongest reported unanswerable-question performance, with a 79.4% IDK rate.Qwen2.5-32B-Instruct scores 0.237 with a 23.7% IDK rate and frequently fabricates answers without evidence.
  • Underspecified questions: 0.566: Qwen3-235B-A22B-Instruct-2507 performs best on underspecified questions, while all other models score below 0.374.The passage attributes this result to clarification-seeking behavior and identifies underspecification detection and clarification generation as open challenges.
  • Partially answerable questions: 0.649: GLM-4.6 leads on partially answerable questions, suggesting better calibration between providing available information and acknowledging gaps.This category requires models to balance answering with hedging.

G Per-Model Contribution Analysis · H Additional Preprocessing Steps

The faithfulness judge’s selections are dominated by Gemini-3-Pro-Preview on ANSWERABLE cases and an IDK fallback on empty-context UNANSWERABLE cases, while PARTIAL cases remain the main weakness. Two optional preprocessing components were explored but excluded from the final submission because query rewriting did not improve generation and relevance filtering was redundant in Task B.

  • G Per-Model Contribution Analysis: 253/285 selected rows in ANSWERABLE instances used Gemini-3-Pro-Preview, while the deterministic IDK fallback covered 97 UNANSWERABLE empty-context rows with perfect conditioned scores.UNDERSPECIFIED instances were excluded because organisers did not include that subset in the analysis.
  • G Per-Model Contribution Analysis: PARTIAL instances showed the sharpest degradation, with incomplete evidence coverage driving residual errors under the organiser metrics.Gemini-3-Pro-Preview’s harmonic mean dropped substantially on PARTIAL compared with ANSWERABLE instances.
  • G Per-Model Contribution Analysis: HM ≈0.697 on 8 PARTIAL rows made Claude 4.5 Haiku stronger than Gemini-3-Pro-Preview’s HM ≈0.465 on 32 PARTIAL rows among selected rows.Haiku was selected infrequently overall, appearing in 28 rows, but behaved as a PARTIAL specialist.
  • G Per-Model Contribution Analysis: HM ≈0.707 on 2 selected rows indicates that Meno-Lite-0.1 supplied potentially complementary candidates despite its low selection rate.The authors caution that this small sample is insufficient for firm statistical conclusions.
  • G Per-Model Contribution Analysis: The authors identify PARTIAL handling as the most promising improvement direction, proposing explicit separation of supported sub-claims from unknowns or greater Haiku weighting for low-evidence cases.They suggest these changes may improve RB_alg_idk without sacrificing RL_F_idk.
  • G Per-Model Contribution Analysis: The analysis weights each model’s harmonic mean by its selection frequency, defining contribution as w_i·HM_i with w_i=N_i/Σ_jN_j.Figure 2 visualises these contribution-weighted harmonic means; Llama-3.3-70B-Instruct and Qwen variants were omitted for clarity after showing no stable signal.
  • H Additional Preprocessing Steps: Query rewriting resolved relative dates and dialogue references into self-contained queries, but Qwen3-8B achieved conditioned HM: 0.615 versus 0.619 without rewriting.Because dialogue history and reference passages already disambiguated Task B inputs, rewriting was excluded from the final submission, though it remained relevant to retrieval-stage improvements.
  • H Additional Preprocessing Steps: Relevance filtering classified retrieved documents as relevant or irrelevant, but Task B’s reference chunks were already relevant, making the component largely redundant.Both preprocessing components were optional development experiments and were not used in the final submission.

I Error Analysis · J Qualitative Error Examples

Manual analysis identified five dominant failure patterns in the final submission, including missing clarification, pragmatic incompleteness, topic drift, ungrounded elaboration, and metric disagreement. Qualitative examples show how these failures produce grounded but non-responsive answers, or apparently strong scores despite incomplete or unsupported reasoning.

  • I Error Analysis: Manual analysis identified five dominant failure patterns in the final submission.The patterns were missing clarification, pragmatic incompleteness, topic drift, ungrounded elaboration, and metric disagreement.
  • I Error Analysis: Ambiguous questions sometimes receive arbitrary interpretations instead of clarification, producing passage-grounded but non-responsive answers and low underspecified-question scores.Examples involve multiple seasons, animal species, or clinics where the intended referent is unclear.
  • I Error Analysis: Grounded responses can remain pragmatically incomplete by omitting explicit confirmations, contradictions, or responses to prior-turn confusion.Such cases score well on faithfulness but poorly on reference-based metrics.
  • I Error Analysis: Noisy evidence can induce topic drift toward salient tangential details, yielding grounded but non-responsive answers.This failure occurs when secondary information in the reference passages distracts from the user’s actual intent.
  • I Error Analysis: Ungrounded elaboration can increase reference-based scores while reducing faithfulness when models extrapolate beyond the provided evidence.The outputs retain a style similar to the reference answer despite unsupported additions.
  • I Error Analysis: Metric disagreement arises when faithful answers miss a key sentence or faithfulness judges overestimate internally consistent but off-topic responses.The analysis therefore recommends interpreting multiple signals jointly rather than relying on any single metric.
  • J Qualitative Error Examples: A qualitative example shows that stating availability in Sao Paulo without explicitly connecting it to South America can produce an incomplete correction and inflated judge-based scores.The evidence supports an explicit contradiction of the user’s claim, not merely a location-specific statement.

K Detailed Benchmark Analysis

The benchmark has several validity limitations, including target leakage, incomplete or misleading annotations, and potentially unreliable metrics. More explicit shared guidelines and stronger validation procedures could improve evaluation robustness.

  • Target leakage through empty context: All 97 unanswerable Task B questions have empty reference passages, so returning “I don’t know” whenever context is empty yields a perfect conditioned score.Realistic deployments usually include irrelevant retrieved passages for unanswerable questions, making answerability detection harder.
  • Annotation coverage limitations: Reference answers may be incomplete because retrieval and manual search can miss scattered or paraphrased evidence, especially for broad questions.The paper recommends corpus-level domain experts validate answer completeness.
  • Possible leakage through reference-less metrics: Reference-less faithfulness scoring can enable per-instance model selection, raising concern that system design partially incorporates evaluation information rather than genuine quality improvement.The paper notes that this potential metric leakage remains unclear.
  • Metric robustness: All metrics occasionally rank models incorrectly, with ROUGE-L particularly noisy and metric–human correlations potentially inflated by comparing very strong and very weak systems.Correlations among similar-quality systems may be substantially lower and are more practically relevant.
  • Guidelines alignment: Evaluation lacks shared guidelines for external knowledge, contradictions, response goals, and passage citation versus synthesis, introducing uncontrolled variance into generation and judging.LLM-assisted validation can flag partially misleading or incorrect gold responses, but it may also introduce systematic biases.
Loading 2605.04523v1…