PaperlayerView exact revision on arXiv

Source-linked AI summary

Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Text

Haoyuan Li, Snigdha Chaturvedi

arXiv:2608.23783v1cs.CL

TL;DR

LLM judges may depend on non-target dimensions when evaluating open-ended text, undermining independent multi-dimensional assessment. The paper introduces CorrGap to quantify this dependence and DimCheck to remove unrelated chain-of-thought evidence. CorrGap reveals pervasive dependence, while DimCheck mitigates it and outperforms strong baselines across three LLMs and four tasks.

  • Problem

    LLM judges may use non-target dimensions when evaluating a target dimension, despite the need for independent multi-dimensional assessment.

  • Method

    CorrGap compares predicted-to-ground-truth score correlations across text groups, while DimCheck iteratively removes unrelated evidence from judge-generated chain-of-thought.

  • Results

    Inter-dimension dependence is pervasive across LLM judges, and DimCheck mitigates it while outperforming strong baselines across three LLMs and four tasks.

  • Takeaways & Limitations

    Smaller trained LLMs can approximate larger LLMs in DimCheck with reduced inference cost.

  • Takeaways & Limitations

    The work focuses on pointwise evaluation and DimCheck requires judges that explicitly generate chain-of-thought before scoring.

Abstract

from arXiv · show

LLM-as-a-judge methods are widely used for evaluating the quality of generated open-ended text. Such evaluations are generally multi-dimensional, since the error patterns in texts can be different for different dimensions. Therefore, reliable LLM judges should evaluate each target dimension independently. To quantify the extent to which LLM judges depend on non-target dimensions when evaluating a target dimension, i.e., inter-dimension dependence, we propose CorrGap. To measure this, CorrGap uses the difference in correlations between LLM-predicted scores and ground truth scores across different groups of texts. Using CorrGap, we show that inter-dimension dependence is pervasive across LLM judges in open-ended text evaluation tasks. To mitigate inter-dimension dependence, we propose DimCheck, a method that iteratively removes unrelated evidence from COTs generated by LLM judges in a step-wise way. We show that DimCheck mitigates inter-dimension dependence and outperforms strong baselines across three LLMs and four tasks. We also show that smaller trained LLMs can approximate larger LLMs in DimCheck, with much lower inference costs.

1 Introduction

Multi-dimensional evaluation captures distinct error patterns, but LLM judges may rely on non-target dimensions when scoring a target dimension. The paper introduces CorrGap to measure this dependence and DimCheck to mitigate it by removing unrelated reasoning evidence.

  • Motivation: Multi-dimensional evaluation assesses specific aspects separately because open-ended texts can contain different, non-interchangeable error patterns.A fluent but factually incorrect summary and a factual but incoherent summary can receive the same overall score while exhibiting different deficiencies.
  • Problem: LLM judges may incorrectly use evidence about non-target dimensions when evaluating a target dimension.An example judge evaluating relevance relies on coherence-related evidence in its chain-of-thought.
  • Approach: CorrGap quantifies inter-dimension dependence using differences in correlations between predicted and ground-truth scores across text groups.The metric is intended to reveal when target-dimension scores are influenced by other dimensions.
  • Findings: Across most settings, CorrGap reveals pervasive inter-dimension dependence among LLM judges in open-ended text evaluation.The study measures dependence across multiple evaluation frameworks, LLMs, and tasks.
  • Mitigation: DimCheck iteratively removes evidence unrelated to the target dimension from judge-generated chain-of-thought before rescoring.The modified reasoning is used as the basis for a new evaluation.
  • Results: DimCheck outperforms strong baselines across three LLMs and four tasks, while smaller trained LLMs can approximate larger models at lower inference cost.These results are reported from comprehensive evaluations of the method.

2 Related Work

Related work establishes multi-dimensional evaluation as a response to varied text errors and surveys methods for improving LLM-judge agreement. It also frames judge biases as non-target dimensions while distinguishing this work’s CorrGap-based analysis from perturbation-based evaluations.

  • Multi-dimensional evaluation: Multi-dimensional evaluation is used because open-ended texts exhibit varied error patterns that a single score may not capture.It also supports comprehensive reward signals in multi-objective alignment and detailed rubrics.
  • LLM judges: Prior LLM-judge methods improve predicted-to-ground-truth correlation through uncertainty modeling, pre-evaluation analysis, agent interaction, or detailed rubrics.These methods address pointwise or pairwise text evaluation.
  • Biases: Existing studies measure biases such as self-enhancement and verbosity using perturbation-based methods, whereas this work treats such biases as non-target dimensions.The passage positions these biases within the paper’s inter-dimension-dependence setting.

3 CorrGap

CorrGap measures whether an LLM judge evaluates a target dimension independently by comparing score correlations across groups whose target and non-target quality differ to different extents. Distribution control makes these correlations comparable.

  • Definition: Multi-dimensional evaluation assigns predicted and human ground-truth scores to each text across multiple dimensions, enabling analysis of target-versus-non-target influence.Inter-dimension dependence is defined as non-target influence on the predicted score for a target dimension.
  • Group construction: CorrGap compares predicted-to-ground-truth score correlations between low-variance and high-variance text groups.Low-variance texts have similar target and non-target scores, whereas high-variance texts have different scores.
  • Distribution control: Percentile differences rather than raw scores accommodate differing score distributions across dimensions, while distribution control prevents spurious correlation gaps.Without control, group correlation differences may reflect distribution changes rather than genuine inter-dimension dependence.
  • Group construction: Texts are split by percentile differences between target-dimension and non-target-dimension ground-truth scores, with the top 50% assigned to the high-variance group.Continuous ground-truth scores can be binned before applying the split.
  • Metric: CorrGap is the difference between predicted-versus-ground-truth correlations across the two groups.The metric uses correlation as its accuracy measure.
  • Interpretation: Independent evaluation should produce similar correlations across groups and a CorrGap near zero; dependence lowers high-variance correlation and increases CorrGap.The paper reports CorrGap as an accurate measure and observes pervasive dependence in LLM judges.

4 DimCheck

DimCheck reduces inter-dimension dependence by iteratively removing evidence tied to non-target dimensions from an LLM judge’s chain of thought, then rescoring when the reasoning changes. The process proceeds dimension by dimension and preserves the original score when no modification occurs.

  • Method: DimCheck modifies an LLM judge’s COT by removing evidence unrelated to the target dimension before assigning a new score.The judge first generates a COT and score, then evaluates the text again conditioned on the modified COT.
  • Iterative removal: DimCheck removes evidence related to one non-target dimension at a time because dimension boundaries can be unclear and overlapping.Each iteration removes evidence related to d′ but unrelated to d_i without changing other parts of the COT.
  • Iterative removal: The process repeats sequentially across non-target dimensions, producing a final COT intended to contain only target-dimension evidence.The final modified COT is denoted rnew_i,t.
  • Rescoring: When a COT is modified, DimCheck regenerates only the target-dimension score using the modified COT; otherwise, it keeps the original score.The evaluation prompt remains unchanged for modified COTs.
  • Algorithm: The algorithm initializes the original COT and score, iterates over non-target dimensions, and returns the resulting score.Its inputs are the text, target dimension, and set of non-target dimensions.

5 Experimental Setup and Dataset

The experiments assess inter-dimension dependence and DimCheck across diverse open-ended text tasks and multiple LLM judges. The setup uses four datasets with multidimensional human annotations and evaluates CorrGap broadly while testing DimCheck on three LLMs.

  • Datasets: The evaluation uses SummEval, Topical-Chat, Hanna, and OpinSummEval across news summarization, knowledge-grounded conversation, narrative generation, and opinion summarization.These datasets provide human annotations on multiple dimensions in diverse domains.
  • DimCheck evaluation: DimCheck is evaluated on Llama3.3-70b-Instruct, Qwen3-32B, and gemma-3-27b-it using the same inference hyperparameters as the preceding experiments.The same LLM modifies the reasoning process, with model-specific thinking configurations for COT modification.

6 Experiment

The experiments validate CorrGap as a measure of inter-dimension dependence, show that this dependence is widespread, and evaluate DimCheck as a mitigation method that preserves judge accuracy while improving performance.

  • Evaluation of CorrGap: CorrGap correctly distinguishes judges with low versus high inter-dimension dependence in both constructed-judge and synthetic-score evaluations.FactScore is designed to minimize dependence on other dimensions, while synthetic perturbations vary the degree of dependence.
  • Evaluation of CorrGap: CorrGap identifies FactScore as having low inter-dimension dependence, while Analyze-Rate shows statistically significant dependence across all three evaluated LLMs.Controlling the ground-truth score distribution is necessary to avoid spurious correlation differences.
  • Evaluation of CorrGap: CorrGap is statistically nonsignificant when synthetic dependence is absent and generally significant when dependence is introduced, demonstrating sensitivity to controlled perturbations.The exception for nonzero dependence occurs on OpinSummEval; controlling the ground-truth distribution improves discrimination.
  • Evaluating Inter-dimension Dependence: Inter-dimension dependence is pervasive across evaluation frameworks and LLMs, with statistically significant dimensions consistently showing lower high-variance-group correlations.Correlation and CorrGap provide complementary views of LLM-judge performance, so both should be considered.
  • Evaluating Inter-dimension Dependence: Dependence varies with text quality: it is more severe for high-quality texts on SummEval and Hanna, but for low-quality texts on Topical Chat.The analysis splits texts into low- and high-quality groups using ground-truth scores.
  • Evaluation of DimCheck: DimCheck consistently lowers CorrGap, maintains comparable correlations with Analyze-Rate, and achieves a significantly higher overall score than the second-best method.DimCheck also performs best overall in the ablation comparison, and its COT-editing identification accuracy reaches 95 percent for Llama3.3-70b and 85 percent for Gemma3-27b and Qwen3-32b.
  • Training Smaller LLMs for Efficient COT Editing: A trained smaller LLM achieves performance close to DimCheck while outperforming one-step and no-thinking variants with reduced inference costs.This result supports using smaller trained models for COT editing as an efficient approximation of larger-model DimCheck.

7 Conclusion

The paper introduces CorrGap to quantify inter-dimension dependence and DimCheck to mitigate it, finding improvements across LLM judges and tasks.

  • CorrGap measures inter-dimension dependence using differences in correlations across groups of texts.
  • Inter-dimension dependence is pervasive across nine LLMs and four tasks.
  • DimCheck iteratively removes unrelated evidence from LLM judges’ COTs to mitigate inter-dimension dependence.
  • DimCheck mitigates inter-dimension dependence and outperforms strong baselines across three LLMs and four tasks.

8 Limitation

The work has two scope limitations: it studies pointwise evaluation and DimCheck requires LLM judges that explicitly generate COTs.

  • The analysis focuses on pointwise evaluation, while pairwise and listwise paradigms remain beyond the work’s scope.
  • Extending the analysis to broader evaluation paradigms and non-COT settings is left for future research.
  • DimCheck is not directly applicable to evaluation methods that do not explicitly generate COTs.

9 Ethics Statement

The ethics statement describes the study’s public resources, computing setup, writing assistance, and human-evaluation safeguards.

  • All datasets and models used in the paper are publicly available or publicly accessible.
  • Inference and finetuning were performed on four Nvidia A6000 or Nvidia L40 GPUs without annotating new data.
  • Amazon Mechanical Turk annotators were compensated at $20 per hour and were not exposed to sensitive or explicit content.
  • LLMs were used to polish the paper’s writing.

A.1 Implementation Details of DimCheck

DimCheck removes unrelated evidence from COTs sequentially, remains broadly stable under random removal orders, and can reduce its inference overhead through a trained smaller model.

  • Implementation Details of DimCheck: DimCheck’s prompt is provided in Table 11.
  • Implementation Details of DimCheck: DimCheck caps generation at 2000 tokens and restarts generation when no answer is produced within that limit.
  • Implementation Details of DimCheck: Non-target dimensions are removed sequentially according to the order introduced in each dataset’s paper.
  • Implementation Details of DimCheck: DimCheck achieves the highest average correlation among methods on high-variance groups.
  • Implementation Details of DimCheck: Changing removal order slightly affects individual datasets, but overall performance remains stable and random-order DimCheck still improves consistently.
  • Implementation Details of DimCheck: 120 minutes versus 20 minutes: trained Llama-3.1-8B reduces DimCheck’s editing time while maintaining its benefits.

A.2 COTs Editted by DimCheck

DimCheck edits LLM judges’ reasoning by identifying and removing evidence tied to non-target dimensions. Experiments examine its editing behavior, stability, quality-group variation, and efficient training with smaller models.

  • Editing procedure: DimCheck checks reasoning for evidence related to non-target dimensions before revising the reasoning process.The procedure instructs the editor to focus on the target and specified non-target dimensions while identifying unrelated evidence.
  • Editing behavior: A large proportion of reasoning processes contains unrelated evidence, and DimCheck can accurately identify such cases.
  • Prompt stability: DimCheck’s performance remains close across prompt variants, with the difference from the best other method statistically significant at p < 0.05.The variant reverses some instruction order and changes phrasing, yet the reported performance remains very close to the original.
  • Quality-group analysis: Inter-dimension dependence is more severe for high-quality texts on SummEval and Hanna, but for low-quality texts on Topical Chat.Texts are split into low- and high-quality groups using dataset-specific ground-truth score thresholds.
  • Efficient training: Trained smaller models achieve performance close to larger inference-based DimCheck models while reducing inference cost.Llama-3.1-8B-Instruct on one GPU reduces inference time by approximately 85% compared with Llama-3.3-70B-Instruct across four GPUs.
Loading 2608.23783v1…