Source-linked AI summary
Does task decomposition improve automatic NLG evaluation?
Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
TL;DR
NLG evaluation needs alternatives to costly and weakly human-aligned reference-based metrics, motivating LLMaJ and task decomposition. This paper systematically compares decomposition-based methods with direct prediction across NLG datasets. It finds no evidence that decomposition improves performance, while human-label training explains prior gains and enables direct prediction to approach human annotators on some tasks.
Problem
Reference-based NLG metrics require expensive human annotations and correlate poorly with human judgments, motivating evaluation of reference-free LLMaJ and proposed task decomposition.
Method
The paper systematically compares decomposition-based LLMaJ with a fair direct-prediction baseline using matched human-label access across multiple NLG datasets and LLMs.
Results
No evidence shows decomposition outperforming direct prediction; reported decomposition gains stem from human-label training, while direct prediction with labels reaches human-level performance on some tasks.
Takeaways & Limitations
Direct Prediction without decomposition can match or exceed CheckEval, and with human labels can perform comparably to human annotators on some evaluation criteria.
Takeaways & Limitations
The study covers a limited number of NLG tasks and datasets, and its conclusions may not extend to complicated multi-step reasoning tasks.
Abstract
from arXiv · showhide
The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.
1 Introduction
NLG evaluation is difficult because common reference-based metrics depend on costly annotations and correlate poorly with human judgments. This paper tests whether decomposing LLMaJ criteria improves evaluation beyond direct prediction.
- Reference-based metrics such as ROUGE and BERTScore rely on expensive human annotations and correlate poorly with human judgments.
- LLMaJ offers a reference-free alternative for evaluating automatic NLG output quality.
- Prior work decomposes evaluation criteria into simpler subcriteria to improve agreement with human annotations and reduce variance.
- The paper systematically compares decomposition-based LLMaJ with direct prediction and strengthens the baseline using the same human labels.
- Decomposition-based LLMaJ shows no evidence of superiority over direct prediction, while reported gains are attributed to human-label training rather than decomposition.
2 Problem Statement
The paper formalizes LLMaJ as predicting a score for a text under an evaluation criterion and aligning it with the corresponding human score. It contrasts decomposition-based scoring with direct prediction of the criterion.
- For text t and criterion d, LLMaJ produces annotation score s = f(t, d), while the target is the human score s∗ = human(t, d).
- Human-rated criteria are represented by L1, while Ln denotes subcriteria after n −1 decomposition steps.
- Decomposition-based evaluation breaks an L1 criterion into subcriteria and aggregates their scores into one L1 prediction.
- Direct prediction estimates the L1 criterion directly without decomposition.
3 Decomposition-based LLMaJ
Decomposition-based LLMaJ expands an L1 criterion into subcriteria, scores them, and aggregates the results. The paper compares representative designs differing in decomposition structure, scoring scale, and aggregation.
- The decomposition pipeline generates L2 subcriteria, scores each with an LLM, and aggregates the scores into a single output.Human annotations are available only for the original L1 criteria, not the decomposed subcriteria.
- Existing decomposition methods vary by flat or hierarchical structure, binary or ordinal scoring, and label-free or learned aggregation.
- The study focuses on HD-Eval and CheckEval because they use general subcriteria and share NLG benchmarks for direct comparison.
- HD-Eval hierarchically decomposes criteria into L2 and L3 subcriteria, scores them on an ordinal scale, and uses a human-label-trained regressor.
- CheckEval decomposes criteria into binary yes/no questions and averages positive answers without human-labeled aggregation.This approach is described as equivalent to TICK.
- The paper extends HD-Eval with AOI decomposition and in-context learning to seek more independent, observable, and human-aligned subcriteria.
4 Direct Prediction LLMaJ
Direct Prediction LLMaJ estimates an L1 criterion score without decomposition. Its prompt uses criterion guidance, a rating scale, reasoning encouragement, and human-evaluation examples, with optional learned calibration.
- Direct Prediction estimates the score for an L1 criterion without decomposing it into subcriteria.
- Its prompt includes the criterion name and definition, rating scale, a chain-of-thought-inducing statement, and five human-evaluation examples.
- Because the target L1 score is predicted directly, Direct Prediction requires no aggregation step.
- For fair comparison with HD-Eval, the study also trains a regressor on the one-dimensional LLM output using human labels.
5 Experimental Setup
The study evaluates LLMaJ methods across multiple NLG datasets using human-label alignment metrics, comparing decomposition-based approaches with direct prediction.
- The experiments use SummEval, TopicalChat, and Seahorse with multiple evaluation criteria.
- Direct Prediction is evaluated with and without human labels, while decomposition-based methods may use human labels to train an aggregator.
- The main evaluation reports Spearman’s ρ, Advantage Probability, and Win-Rate, while Seahorse additionally uses accuracy, Krippendorf’s α, and inter-annotator agreement.
- The primary experiments use Claude-4 with temperature t = 0, with additional results for Qwen3-32B and GPT-OSS-120B.
- The main results average performance across criteria and distinguish whether decomposition and human labels are used.
6 Results and Discussion
Across the evaluated datasets, decomposition-based LLMaJ does not consistently outperform Direct Prediction. Reported gains are instead associated with human-label access, while Direct Prediction can approach or reach human-level agreement on several criteria.
- 6.1 Decomposition VS Direct Prediction: Direct Prediction performs better or only slightly worse than the best decomposition-based approaches, and one combination achieves the best overall AP on both datasets.
- 6.1 Decomposition VS Direct Prediction: Decomposition extensions using ICL and AOI provide marginal improvements but do not consistently outperform Direct Prediction.
- 6.1 Decomposition VS Direct Prediction: The L1-agnostic HD-Eval variant performs on par with standard decomposition and Direct Prediction, indicating that its gains stem from learned aggregation rather than decomposition itself.
- 6.2 How far is Direct Prediction LLMaJ from human-level performance: WR ≥0.5 without human labels makes Direct Prediction as good as human annotators by the study’s threshold on both datasets.
- 6.2 How far is Direct Prediction LLMaJ from human-level performance: With human labels, Direct Prediction reaches WR of 100% on both datasets, meaning it is closer to average human ratings than individual humans.
- 6.2 How far is Direct Prediction LLMaJ from human-level performance: The remaining gap to humans is mostly concentrated in Grammar and Main Ideas, while Direct Prediction is close enough on the other three criteria to be considered a valid alternative.
7 Conclusion
The paper concludes that decomposition does not reliably improve LLMaJ over a fair Direct Prediction baseline across the studied NLG settings. Direct Prediction can achieve human-level performance on some tasks when human labels are available.
- Across three datasets and multiple LLMs, decomposition-based methods do not consistently outperform the fair Direct Prediction baseline.
- HD-Eval’s reported gains stem from access to human labels rather than task decomposition.
- Direct Prediction with human labels reaches human-level performance on some evaluation tasks in SummEval and TopicalChat.
- Future work could examine decomposition-based evaluation when human raters assess both original and decomposed criteria.
Limitations
The study’s conclusions are bounded by its limited set of NLG tasks and datasets and do not necessarily extend to complicated, multi-step reasoning tasks.
- The number of NLG tasks and datasets is limited, although the authors consider it diverse enough to support their conclusions.
- The finding that decomposition is not beneficial for typical NLG tasks does not necessarily extend to complicated, multi-step reasoning tasks.
- Bias, fine-tuning, and human–LLM collaboration are outside the study’s scope.
- The study cannot determine whether the evaluated LLMs saw the benchmark datasets during pre-training.
A Results on Seahorse
On Seahorse, decomposition does not improve over direct prediction and is often worse; direct methods remain close to humans on most criteria, though Grammar and Main Ideas account for much of the remaining gap.
- Results: Decomposition does not improve above the direct-prediction baseline on Seahorse and is often worse.The authors suggest this may reflect criteria that are already relatively atomic.
- Evaluation: Krippendorf’s α is the main metric because Seahorse’s binary criteria make accuracy less meaningful through chance agreement and class imbalance.Only one annotation per sample is available, so the study compares against published inter-annotator agreement rather than using the alt-test.
- Results: LLM performance remains below human agreement on average, with most of the difference concentrated in Grammar and Main Ideas.On the other three criteria, the LLM is at minimum close enough to humans to be considered a valid alternative.
- Evaluation: The study uses accuracy and Krippendorf’s α for Seahorse, with all reported results based on vertical aggregation.Accuracy is retained to enable comparison with prior results, while Krippendorf’s α is prioritized for interpretation.
- Aggregation: Vertical aggregation trains each regressor only on the children of its corresponding L1 criterion, unlike horizontal aggregation, which uses all sub-criteria.Horizontal aggregation generally performs better because it supplies the regressor with more features, but vertical aggregation better matches task decomposition.
- Evaluation: Floating-point outputs can remove ties and create numerical advantages against floating-point ground truth, so the study reports rounded and unrounded results and applies rounding throughout.Rounding generally improves alt-test AP and WR but slightly worsens correlations.
D.4 Regressor Implementation Details
The aggregators are implemented with scikit-learn using default configurations for linear regression, decision trees, random forests, and multilayer perceptrons.
- Implementation: The aggregators are implemented with scikit-learn using its default parameters.The listed implementations include LinearRegression, DecisionTreeRegressor, RandomForestRegressor, and MLPRegressor.
- Regressors: The implementation includes a DecisionTreeRegressor with squared-error criterion and unrestricted maximum depth.
- Regressors: The implementation includes a RandomForestRegressor with 100 estimators and unrestricted maximum depth.
- Regressors: The implementation includes an MLPRegressor with a 100-unit hidden layer, ReLU activation, Adam solver, and squared-error loss.
E Results with other Models
Across Qwen3-32B and GPT-OSS-120B, decomposition still shows no significant performance improvement, although ICL and AOI perform best on SummEval. The results are averaged across criteria and reported with multiple evaluation metrics.
- Decomposition does not yield significant performance improvements across Qwen3-32B and GPT-OSS-120B.This observation is especially clear on TopicalChat, while the two models perform comparably to Claude-4.
- ICL and AOI decomposition achieve the best scores on SummEval.Their benefit is smaller on AP than on the correlation metrics.
- Table 6 averages results across all criteria and reports Spearman’s ρ, Kendall’s τ, AP, and WR.The ICL column refers to evaluation-phase ICL rather than decomposition-phase ICL.
- Per-criterion results are presented separately for SummEval and TopicalChat.