PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Language Models in Realistic Conversational Contexts

Ilija Subasic, Andrew Rabinovich, Zhao Chen

arXiv:2608.26131v1cs.CL

TL;DR

Existing evaluation frameworks do not adequately measure the nuanced quality of open-ended, human-scale dialogue. The paper introduces UPHELD, a professionally authored and human-annotated benchmark, and finds that ensemble evaluation improves alignment with human judgments by approximately 30%.

  • Problem

    Existing metrics and benchmarks focus on factual correctness, lexical overlap, or synthetic sources, providing limited evidence for evaluating sustained human-scale conversational quality.

  • Method

    UPHELD pairs professionally written human-to-human dialogues with model-generated continuations and per-turn human evaluations of content, style, and reasonableness.

  • Results

    A Mixture-of-Judges ensemble correlates more highly with human judgments than individual LLM judges and established syntactic and semantic metrics, improving prediction by approximately 30%.

  • Takeaways & Limitations

    UPHELD provides a human-grounded foundation for developing evaluation frameworks aligned with human perceptions of effective conversation.

  • Takeaways & Limitations

    The verification datasets are deficient in freeform human-to-human interaction and limited to predefined topics, so they are not replacements for UPHELD.

Abstract

from arXiv · show

As Large Language Models (LLMs) are increasingly deployed to serve open-ended, multi-turn interactions, evaluating conversational quality at human scale has become a central challenge. Existing evaluation frameworks built for summarization, translation, or short-form QA tasks fall short of adequately measuring the consistency of human-scale dialogue, especially when derivation and validation of these metrics themselves often rely on synthetic rather than human sources. We fill the gap by introducing UPHELD (UPwork Human-Scale Evaluated Long Dialogues), a large, reference-full benchmark for evaluating human-scale conversational ability beyond factual correctness. UPHELD consists of hundreds of complete human-to-human dialogues authored by professional script writers, with realistic turn densities and 36,000+ per-turn human annotations across 30,000+ expert-generated dialogue turns. Using UPHELD, we systematically evaluate classical automatic metrics and reference-free LLM-as-a-judge approaches, and find them unreliable when correlated with expert human judgment. Building off this analysis, we use UPHELD to develop a Mixture-of-Judges framework that combines multiple evaluative signals and improves correlation with human assessments by approximately 30%. Overall, UPHELD provides a robust, human-grounded foundation for evaluating human-scale conversational intelligence that fills a crucial gap in the pre-existing LLM dataset landscape.

1. Introduction

UPHELD addresses the difficulty of evaluating sustained, natural, useful conversation with a human-grounded benchmark designed around realistic task-oriented dialogues. It also shows that existing metrics underperform, while a Mixture-of-Judges ensemble better predicts expert judgment.

  • Existing metrics inherited from translation, summarization, and question answering inadequately assess sustained coherence, context management, and open-ended conversational quality.
  • UPHELD contains over 30,000 human-written dialogue turns across hundreds of conversations, with multiple human labels for coherence criteria.
  • Professional script writers created natural, casual task-oriented dialogues in which writers could determine how tasks were navigated.
  • UPHELD targets realistic conversational turn distributions, with human-collected datasets averaging 9.3 turns and rarely exceeding 20 turns.
  • The benchmark emphasizes conversational fluency and everyday task utility rather than complex mathematics, coding, or knowledge retrieval.
  • A Mixture-of-Judges ensemble correlates more highly with human judgments than individual LLM judges and established syntactic and semantic methods.

2. Related Work

Existing LLM benchmarks commonly evaluate discrete tasks, factual extraction, or reference-free judgments using input-output scores. These approaches do not adequately capture sustained, subjective, open-ended conversation, motivating a curated, expert-authored, human-annotated dataset.

  • Current benchmarks emphasize instruction following, discrete task performance, factual question answering, exact matches, and precision, recall, or F1 scores.
  • LLM-as-a-judge methods evaluate outputs without requiring a ground-truth or reference answer.
  • Long non-topical conversations require sustained coherence, context management, engaging dialogue, and handling of discussions without concise correct answers.
  • The paper responds by introducing a dataset characterized by comprehensive curation, expert authorship, and human annotation.

3. Datasets

UPHELD combines professionally authored conversations with model-generated continuations and blinded human comparisons across content, style, and reasonableness. The study also constructs verification datasets, while noting their narrower conversational scope.

  • Professional Upwork writers created open-ended, naturalistic dialogues across topics such as customer service and education.
  • LLMs generated predicted next turns from conversation history, alongside a GPT-4o-rephrased human completion and a random baseline.
  • Annotators compared blinded human and model continuations on content consistency, style consistency, and general reasonableness.
  • Each model-generated turn was evaluated by five independent human annotators.
  • Annotation dimensions: The annotation dimensions used distinct scales for content equivalence, style equivalence, and utterance reasonableness.
  • Annotation dimensions: Pilot studies motivated a three-level style scale and a binary reasonableness judgment because annotators found finer rankings less consistent.
  • The study collected 30,715 label sets, totaling 153,578 individual labels, across 800 conversations and 9,203 turns.
  • Dataset quality: Quality procedures included diverse writer recruitment, professional review, filtering and revision, informed consent, and removal of personally identifiable information.

4. Metrics

The section compares token-, semantic-, and LLM-based metrics for conversational evaluation, while addressing how reference-full evaluation handles multiple valid responses. UPHELD uses task-oriented dialogue structure and semantic-consistency evidence to support its ground-truth labels.

  • Metric Groups: 12 candidate metrics were grouped into token-based, semantic-based, and LLM-based evaluation approaches.Token-based metrics measure exact token overlap, semantic-based metrics measure semantic overlap, and LLM-based metrics use LLM-as-a-judge paradigms.
  • Classical Metrics: ROUGE compares generated text with reference text using n-gram overlap in ROUGE-N and longest-common-subsequence similarity in ROUGE-L.ROUGE is a reference-based metric originally used for summaries and translations.
  • Classical Metrics: Cosine similarity between message embeddings and BERTScore were tested as semantic-similarity metrics against reference text.BERTScore matches contextual word embeddings and computes precision, recall, and F1.
  • LLM-Based Metrics: LLM-as-a-judge metrics scored outputs for criteria including coherence, relevance, and overall quality using binary and Likert judgments.The experiments varied whether judges supplied explanations and whether they used prompts given to human raters.
  • Reference Grounding: Reference-free preference evaluation has reported reliability, but its judgment quality is limited for long task-oriented dialogues and out-of-distribution tasks.UPHELD instead uses ground-truth references to generate human labels.
  • Reference Grounding: 93% versus 74% semantic consistency shows that UPHELD dialogues have less conversational multiplicity than freeform dialogues.UPHELD reduces ambiguity through style and reasonableness labels and task-oriented settings where content accuracy is well defined.

5. Experiments

Experiments show that UPHELD exposes weaknesses in single-metric and LLM-as-a-judge evaluation while supporting fine-tuning and learned ensemble metrics that better align with human judgments.

  • UPHELD as a Fine-Tuning Dataset: UPHELD evaluation uses 756 label sets across 4,777 unique utterances, averaging 6.3 user-assistant turn pairs, while excluding fine-tuning conversations to avoid leakage.Each label set was rated by up to five annotators.
  • UPHELD as a Fine-Tuning Dataset: Approximately 40% higher content and style accuracy is achieved by the held-out fine-tuned custom model versus base models, reaching scores comparable to GPT-5 and Deepseek v3.The model retained the core functionality of Llama-3.1-70b with minimal observed overfitting.
  • Traditional Metrics: Traditional metrics show weak to moderate correlations with human ratings, so no single metric adequately captures conversational content and style quality.Semantic metrics performed best for content and style, while token-based metrics exceeded LLM-as-a-judge approaches.
  • Ensemble Metrics: 30-40% relative improvement over the best individual metrics is obtained by the random-forest ensemble, while SVM ensembles produce consistently higher correlations across datasets.The ensembles were trained on UPHELD and applied to LLM Arena and Topical-Chat; random-forest transferability was poor.
  • Annotator Agreement: Moderate human-human agreement of Cohen’s κ = 0.33 accompanies strong dependence of LLM-as-a-judge performance on annotator agreement.Approximately 25% of data points received full agreement across all five judges.

6. Conclusion

The conclusion presents UPHELD as a human-annotated benchmark for human-scale conversational evaluation and reports that learned ensembles align better with human evaluations than existing metrics.

  • UPHELD combines tens of thousands of human-annotated labels on conversation consistency with high-quality human-written dialogues.The dataset targets human-scale conversational settings.
  • Existing evaluation metrics do not effectively capture the nuances of human judgment for conversational quality on UPHELD.
  • Simple-to-learn ensemble metrics substantially improve correlation with human evaluations.The conclusion frames these findings as supporting more robust evaluation frameworks aligned with human perceptions of effective conversation.

Impact statement

The authors identify UPHELD as a dataset of interest for language-model research and suggest extending its use to additional conversational domains, including enterprise applications.

  • UPHELD is positioned as a dataset for investigating more conversational verticals, including targeted customer service and technical dialogues.
  • The benchmark’s dimensions describe data provenance, scale, task focus, and evaluation rigor as criteria for assessing suitability for LLM training and evaluation.

A.1. Data Statistics

The statistics framework characterizes dialogue benchmarks by resource density, provenance, task objective, context source, conversational format, and evaluation methodology.

  • Density Metrics: Total Conversations counts distinct conversational threads or sessions in a dataset.
  • Density Metrics: Total Utterances counts individual speech acts or turns produced by all participants across the dataset.
  • Density Metrics: Total Comparison Labels counts human or machine judgments used to compare model-response quality, including preferences, rankings, and score annotations.The metric indicates utility for pairwise evaluation and RLHF.
  • Data Provenance: Human indicates whether dialogue turns were composed by human actors rather than generated synthetically, while Direct indicates whether conversations were obtained raw from their source.
  • Data Provenance: Verified indicates whether humans performed explicit quality checks, validation, or filtering for coherence, safety, or task adherence.
  • Task Focus: Task categories distinguish factual QA, open dialogue, summarization, specialized tasks, and context retrieval by their functional objectives.
  • Context Source: External context is classified as RAG, grounded knowledge, or none, depending on how information enters the dialogue.
  • Evaluation Rigor: Evaluation methodology records whether benchmarks use explicit references, reference-free judgment, human annotators, correlation with human scores, and creator-selection details.

B. Annotation Materials and Instructions

UPHELD provides structured materials and instructions for comparing human and generated dialogue continuations on content, style, and reasonableness. The appendix also reports how instruction-based LLM judging performed and why those metrics were excluded from later analysis.

  • B.1. Materials: Annotators used a three-sheet spreadsheet containing instructions, examples, and a six-column annotation table.The table included chat history, two continuation options, and three task-specific score fields.
  • B.2. Instructions: The annotation setup presented a chat history and two possible continuations, Option A and Option B, for comparative judgment.Annotators entered scores in columns corresponding to the relevant task.
  • B.2.1. TASK 1 – CONTENT EQUIVALENCE: Content equivalence measured whether both options conveyed roughly the same information and preserved the conversation’s flow.Instructions emphasized considering the chat history and evaluating each option as written, including incomplete continuations.
  • B.2.2. TASK 2 - STYLE EQUIVALENCE: Style equivalence judged whether both options sounded like the same person in the same mood, considering vocabulary, tone, syntax, sentiment, and delivery.Annotators were encouraged to read both options aloud before assigning a score.
  • B.2.3. TASK 3 – REASONABLENESS: Reasonableness applied only to Option B and asked whether it was an appropriate continuation of the chat history, independently of Option A.Annotators were told to consider conversational flow, readability, topicality, tone, helpfulness, and whether a user would be satisfied with the response.
  • B.2.1. TASK 1 – CONTENT EQUIVALENCE: Content scores were lowered when Option B added excessive, irrelevant, or non-expansive detail, while AI self-identification phrases alone did not reduce scores.Unreadable or incoherent Option B responses received a score of 3 under the stated guidance.
  • B.2.2. TASK 2 - STYLE EQUIVALENCE: Style instructions allowed different lengths when expression remained similar, but penalized options that sounded like they came from different people.Content differences did not necessarily lower a style-equivalence score if voice and mood remained aligned.
  • B.3. Instructions as LLM judge prompts: 0.36, 0.21, and 0.08 correlations for content, style, and reasonableness respectively led the authors to remove instruction-prompted LLM-judge metrics from further analysis.These values were lower than the corresponding free-instruction correlations of 0.4, 0.24, and 0.12; a separate qualitative check examined cases where humans preferred GPT-3.5 over GPT-4o.

H. Annotator agreements levels

The benchmark measures annotator agreement at perfect, majority, and plurality levels. Agreement analysis indicates that substantial data remains usable even under strict agreement criteria, while stronger human agreement aligns with higher metric correlation.

  • Agreement levels: Agreement was quantified at three levels: perfect, majority, and lead/plurality.Perfect means all annotators agree, majority means more than half agree, and lead/plurality means one score occurs more often than the others.
  • Agreement levels: Approximately 70% to 90% of data could be retained when requiring some positive agreement, depending on the score.The dataset also retained a substantial amount of data where all five annotators agreed.
  • Agreement and metric correlation: Higher human agreement corresponded to higher correlation between human labels and automatic metrics.Table 9 reports content and style scores with relative increases between successive agreement levels.

I. Additional Linear Regression Analysis

Additional regression analyses examined which metrics contributed most to the ensemble while controlling for correlations among predictors. Cosine-similarity BERT metrics emerged as the main significant predictors, whereas LLM-as-a-judge metrics generally contributed less.

  • Metric significance: The authors used interpretable linear regression to identify the most significant metrics within the ensemble.Single-variable regressions were used to control for correlation effects among mutually correlated metrics.
  • Metric significance: Cosine similarity BERT metrics were the main significant predictors, while LLM-as-a-judge metrics generally ranked behind them.The significance analysis was conducted for metrics included in the ensemble.
  • Coefficient analysis: All metrics showed strong positive correlations with UPHELD labels except the reasonableness LLM-as-a-judge metric, which performed poorly and was omitted from most analyses.The conclusion is based on coefficient values reported for content consistency.

J. The Reasonableness Label

The reasonableness label is overwhelmingly positive and shows weak or insignificant metric correlations, so the paper treats it primarily as a sanity check and out-of-distribution filter.

  • Reasonableness correlations with various metrics are fairly weak and/or statistically insignificant.
  • Most models receive reasonableness scores substantially greater than 80%, except the random model baseline.
  • The positive skew arises because models often produce reasonable outputs even when inconsistent with the conversation history.
  • The reasonableness score is used mainly as a sanity-check label and a filter for out-of-distribution data.
  • The paper does not analyze reasonableness labels at length and identifies uncertainty or out-of-distribution detection as potential follow-up uses.
Loading 2608.26131v1…