Source-linked AI summary
The Impact of Backbone Evolution on LLM-Based Relevance Assessments
Chuting Yu, Guido Zuccon, Teerapong Leelanupab
TL;DR
This paper examines whether newer versions of the same LLM backbone make better relevance judges under fixed prompts. It evaluates UMBRELA and EXAM across successive model versions and finds that aggregate gains can mask regressions in earlier correct judgments.
Problem
It is unclear whether newer versions of the same LLM family preserve correct relevance judgments, despite the cost of redesigning and re-validating prompts for each backbone update.
Source: §1 · pp. 1–2Method
The study fixes UMBRELA and EXAM prompts and evaluates successive Gemini, GPT, Qwen, and Llama versions on two TREC datasets, measuring aggregate performance and instance-level regressions.
Source: §1 · p. 2, §3 · pp. 4–5Results
Newer versions do not consistently improve relevance judgments, and aggregate accuracy gains can mask instance-level volatility, with regression severity shaped by prompt structure.
Source: §1 · p. 2, §5 · p. 12Takeaways & Limitations
Backbone updates call for re-validating prompt–model alignment against human baselines rather than treating evaluation pipelines as “set and forget.”
Source: §5 · p. 12The evaluation is restricted to TREC DL and UMBRELA and EXAM; EXAM’s question generator was held static to isolate grader evolution.
Source: §5 · p. 12
Abstract
from arXiv · showhide
LLMs are evolving rapidly, with newer models offering stronger capabilities. This suggests that in LLM-based relevance judging, more capable models will achieve higher agreement with human judgements under the same prompt. We challenge this understanding by investigating the behavior of LLM-based relevance judges under backbone evolution. Keeping the prompts fixed, we evaluate a representative single-prompt (UMBRELA) and a rubric-based prompt (EXAM) across sequential model versions of commercial (Gemini, GPT) and open-weight (Qwen, Llama) models. Overall, we find no consistent evidence that newer versions lead to better relevance judges. Crucially, similar or improved aggregate performance does not imply judgment stability: correct judgements made by an earlier version of an LLM backbone are not necessarily preserved by later versions. We investigate the potential drivers of these regressions. Our findings caution against the assumption that judging prompts designed and validated for one backbone version will perform equivalently or better when the model is updated, even within the same family.
1 Introduction
This study asks whether newer versions of the same LLM family make better relevance judges when prompts remain fixed. It finds no consistent improvement, and aggregate results can mask regressions in judgments that earlier versions got right.
1 Introduction
- Prior work found that judging frameworks can produce different assessment effectiveness with different model backbones.The introduction situates this study’s focus on successive versions within a family against prior comparisons across different LLMs.Source: §1 · p. 2
- The motivation is that prompt design and validation can be costly, while a prompt may behave differently when run by a newer backbone in the same family.The authors note that redesigning and re-validating prompts for each update can undermine a main attraction of LLM-based evaluation.Source: §1 · pp. 1–2
- The study tests whether successive Gemini, GPT, Qwen, and Llama versions preserve correct labels across two TREC datasets using fixed UMBRELA and EXAM prompts.It defines a regression as an earlier version predicting the exact ground-truth label when a later version does not.Source: §1 · p. 2
- Under fixed prompts, newer versions do not consistently improve relevance judgments; aggregate gains can coexist with regressions on earlier correct judgments.Regressions are more substantial for EXAM than for UMBRELA, and the analysis identifies overweighting, overly strict relevance criteria, and superficial keyword matching.Source: §1 · p. 2
2 Related Work
Prior work examines variability from prompt formulation and backbone choice, while model evolution has received less empirical attention. This study addresses whether successive versions preserve their predecessors’ correct relevance judgments.
2 Related Work
- Prompt engineering studies find that prompt configurations can substantially vary in agreement with human judgments; UMBRELA adapts structured prompting with explicit relevance criteria.Thomas et al. report the largest improvement from aspect-based assessment, and Upadhyay et al. [22] introduced UMBRELA as an open-source reproduction.Source: §2 · p. 3
- Prompt sensitivity research finds that changes in framing, formatting, or instruction order can affect agreement with human assessments.Arabzadeh&Clarke compared human- and LLM-authored prompts across binary, graded, and pairwise relevance settings.Source: §2 · p. 3
- Studies also find that relevance-judging performance varies across backbones, including lower UMBRELA effectiveness for LLaMA-3.3-70B and smaller models than GPT-4o or DeepSeek V3.Farzi&Dietz reported comparable performance for GPT-4o and DeepSeek V3.Source: §2 · p. 3
- Model updates have been associated with behavioral drift and changed evaluation results, including altered rankings in toxicity evaluation.Pozzobon et al. showed that commercial evaluation API updates could change previously obtained toxicity results and relative model rankings.Source: §2 · p. 3
- Although LLM evolution was identified as a threat to relevance-assessment validity, its effects on preserving correct judgments had not been empirically quantified.Dietz et al. advocated periodic re-evaluation; this study investigates stability across subsequent versions of the same backbone.Source: §2 · p. 3
3 Experimental Methodology
The methodology holds prompts fixed while varying model versions within families, then evaluates both aggregate agreement with human labels and instance-level changes. It compares two TREC passage-ranking datasets and two relevance-assessment frameworks.
3 Experimental Methodology
- The controlled design fixes the evaluation prompt and varies only model version within a family to assess aggregate changes and instance-level regressions.The analysis addresses all three research questions through these two levels of comparison.Source: §3 · p. 4
- The evaluation uses TREC Deep Learning passage-ranking tasks from 2019 and 2020, with human relevance labels from 0–3 as ground truth.The graded labels support assessment of both exact agreement and error magnitude.Source: §3 · p. 4
- UMBRELA uses zero-shot DNA prompting for graded relevance, while EXAM generates query-specific questions and uses their answerability ratings to derive passage grades.EXAM ratings range from 0 to 5 before being mapped to the 0–3 TREC relevance scale.Source: §3 · p. 4
- For EXAM, the earliest family version generates the query-specific questions, which are then fixed across later versions to isolate changes in grading behavior.The authors explain that evolving both question generation and grading would conflate those changes.Source: §3 · p. 4
- The study compares successive versions from commercial Gemini and GPT and open-weight Qwen and Llama lineages.Figure 1 presents release dates and key innovations of the evaluated LLMs; Qwen and Llama sizes were selected to be as close as available across generations.Source: §3 · p. 4, Figure 1 · p. 5
- Aggregate performance is evaluated against human judgments, while instance-level analysis tracks changes across versions with a focus on regressions.A regression occurs when the baseline predicts the correct relevance grade for a query–document pair but a later version does not.Source: §3 · p. 5
- The evaluation reports Exact Match, Mean Absolute Error, and Pearson correlation to measure exact agreement, grading-error magnitude, and linear association with human grades.The metrics are defined across the supplied passages.Source: §3 · p. 5
4 Results
Across the evaluated model families and prompts, newer versions do not consistently improve aggregate relevance-judging performance, and aggregate gains can coexist with regressions on previously correct judgments. Regression patterns and failure modes vary with the judging framework and the scale of the model update.
4.1 RQ1: Do newer LLM versions lead to better relevance judges?
- Newer versions show no consistent aggregate gains, and improved aggregate performance can mask regressions on previously correct judgments.The reported regression analysis also finds substantial churn across evaluated families.Source: §4.1 · p. 5, §4.2 · pp. 7–8
- Gemini Flash Lite and Llama improve more consistently, while Gemini Flash and Qwen fluctuate across releases and judging frameworks.Qwen EXAM EM falls from 47.87% to 15.15% on DL 2019 and from 55.32% to 19.30% on DL 2020 between Qwen2.5-7B and Qwen3-8B, then recovers with Qwen3.5-9B.Source: §4.1 · pp. 5–7
- EM, MAE, and Pearson r can move inconsistently across versions, and their patterns vary by judging framework.The authors conclude that fixed frameworks may need re-evaluation as backbones evolve.Source: §4.1 · p. 7
4.2 RQ2: Are there regressions when switching to a newer version?
- EXAM generally raises regression rates over UMBRELA for minor updates but can buffer severe degradation; Figure 2 maps rates by baseline and newer model.Gemini 2.5-to-3.5 regressions are 4.8% under UMBRELA and 11.5% under EXAM, while GPT-4o Mini-to-GPT-5 Mini rates are 28.0% and 11.5%, respectively.Source: Figure 2 · p. 8, §4.2 · p. 9
- For Gemini 3.5 Flash Lite and Llama 3.1, aggregate gains coincide with regressions of 5.7% and 6.7%, respectively, on previously correct instances.The cited Llama comparison reports a near-doubling in EM alongside its regression rate.Source: §4.2 · p. 8
- Downgrading can lose more of a newer model’s correct assessments than the forward upgrade loses; Llama 3.1-to-3 downgrading regresses 21.8% versus 6.7% for the upgrade.The authors note this matters when older models become cheaper after newer releases.Source: §4.2 · pp. 9–10
4.3 RQ3: What are the drivers beyond the regressions?
- Failure patterns differ by prompt: UMBRELA’s largest reported driver is Over-Weighted Specific Aspect at 31.3%, while EXAM’s is Overly Strict Relevance Threshold at 33.3%.EXAM also exhibits 14.2% Overcrediting Tangential Relevance; under UMBRELA, Superficial Keyword Matching is 6.4% and Overly Strict Interpretation is 13.9%.Source: §4.3 · pp. 10–12
- The Gemini Flash analysis found 18.3% EXAM Score/Reasoning Inconsistency and hallucinations in 4.2% of UMBRELA and 7.9% of EXAM regressions.The meta-evaluation focused on Gemini Flash and TREC DL 2019, using Gemini 3.8 Flash to classify regression reasoning; a manual audit covered 50 cases.Source: §4.3 · pp. 10, 12
5 Conclusion
Aggregate accuracy gains can mask severe instance-level volatility in relevance judgments, and regression severity depends on prompt structure. The authors recommend revalidating prompt–backbone alignment after updates while noting that the study’s scope is limited.
5 Conclusion
- Aggregate accuracy gains can mask severe instance-level volatility, with regression severity depending on prompt structure.The rigid EXAM rubric exacerbated volatility during minor updates but buffered severe generational degradation; UMBRELA regressions involved semantic hyper-fixation, while EXAM regressions involved scoring inconsistencies despite sound reasoning.Source: §5 · p. 12
- High cross-version volatility means evaluation pipelines require prompt–backbone revalidation against human baselines after each backbone update.The authors also warn that older, discounted models may disproportionately sacrifice capabilities because regression rates are asymmetric.Source: §5 · p. 12
- The evaluation is restricted to TREC DL and UMBRELA and EXAM, while EXAM’s question generator remains static to isolate grader evolution.The authors call for testing generalization to other tasks and multilingual corpora, exploring other frameworks, and assessing joint generator and grader upgrades.Source: §5 · p. 12
- Future work should explore regression-aware prompt optimization to minimize capability loss across model versions.The authors suggest adapting dynamic prompt-tuning frameworks to adjust constraints across versions.Source: §5 · p. 13