PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models

Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez

arXiv:2410.12851v7cs.CLcs.AI

TL;DR

Correctness-focused evaluations do not capture the subjective, task-dependent preferences reflected in model outputs. VibeCheck discovers and measures differentiating output traits, finding that they predict model identity and user preference in real-world comparisons.

  • Problem

    Correctness-focused evaluation and predefined criteria do not capture the open-ended, task-dependent role of subjective user preferences.

    Source: §1 · p. 1

  • Method

    VibeCheck iteratively discovers model-output traits and uses LLM judges to score them, measuring their agreement, model separability, and relation to user preference.

    Source: §1 · p. 2, Vibe discovery · p. 4

  • Results

    80% model-identity accuracy and 61% user-preference accuracy were achieved on real-world comparison data using discovered vibes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VibeCheck adds measurable qualitative properties that influence human judgments to evaluations centered on correctness.

    Source: §8 · p. 10

    Vibe-based preference associations may reflect confounding factors, discovery may miss relevant differences, especially when model accuracy differs substantially, and validation is costly.

    Source: §7 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) often exhibit subtle yet distinctive characteristics in their outputs that users intuitively recognize, but struggle to quantify. These "vibes" -- such as tone, formatting, or writing style -- influence user preferences, yet traditional evaluations focus primarily on the singular axis of correctness. We introduce VibeCheck, a system for automatically comparing a pair of LLMs by discovering identifying traits of a model (vibes) that are well-defined, differentiating, and user-aligned. VibeCheck iteratively discovers vibes from model outputs and then utilizes a panel of LLM judges to quantitatively measure the utility of each vibe. We validate that the vibes generated by VibeCheck align with those found in human discovery and run VibeCheck on pairwise preference data from real-world user conversations with Llama-3-70b vs GPT-4. VibeCheck reveals that Llama has a friendly, funny, and somewhat controversial vibe. These vibes predict model identity with 80% accuracy and human preference with 61% accuracy. Lastly, we run VibeCheck on a variety of models and tasks including summarization, math, and captioning to provide insight into differences in model behavior. VibeCheck discovers vibes like Command X prefers to add concrete intros and conclusions when summarizing in comparison to TNGL, Llama-405b often overexplains its thought process on math problems compared to GPT-4o, and GPT-4 prefers to focus on the mood and emotions of the scene when captioning compared to Gemini-1.5-Flash. Code and vibe visualizer found at https://bench-mark.org/

1 INTRO

VibeCheck addresses the limits of correctness-focused evaluation by discovering model traits that matter to users and measuring their usefulness. The paper reports that discovered vibes predict model identity and preference, and reveal differences across tasks.

1 INTRO

  • Correctness-focused evaluations and predefined criteria miss the open-ended, task-dependent role of subjective user preferences.Tone and creativity may matter in creative writing, while efficiency and readability may matter in coding.Source: §1 · p. 1
  • Users can recognize models by output traits such as friendliness; the paper gives Llama-3 as friendlier than GPT-4 and Claude, which tend to be more formal.The authors suggest these differences could inform model selection for customer service or coding tasks.Source: §1 · p. 1
  • VibeCheck discovers and measures pairwise model differences through traits that are well-defined, differentiating, and user-aligned.A panel of LLM judges scores where each model's output falls on each vibe, and those scores measure the three criteria.Source: §1 · p. 2
  • 80% model-identity accuracy and 61% user-preference accuracy were achieved using discovered vibes on real-world comparison data.Llama-70b was described as using more typographic emphasis and examples and as funnier than GPT-4 and Claude-3-Opus; GPT-4 and Claude commented more on ethics and limitations than Llama.Source: §1 · p. 2
  • On summarization, math, and captioning tasks, VibeCheck finds differences including Command X's concrete introductions and conclusions, Llama-405b's detailed math explanations, and GPT-4's focus on scene mood.These examples compare Command X with TNGL, Llama-405b with GPT-4o, and GPT-4 with Gemini-1.5-Flash, respectively.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates outputs with automatic metrics, predefined criteria, human-powered comparisons, or LLM-based preference predictions. VibeCheck builds on these approaches by discovering interpretable differences and measuring their utility.

2 RELATED WORK

  • Aspect-based evaluation improves on automatic metrics with criteria such as factual accuracy, fluency, and conciseness, while broader benchmarks standardize evaluation across tasks.The cited approaches include BLEU and ROUGE improvements and multi-metric benchmarks.Source: §2 · p. 2
  • LLM-judge evaluations cover open-ended tasks but generally define evaluation axes in advance, whereas VibeCheck discovers axes and measures their correlation with human preference.The figure illustrates scoring outputs along a vibe and using those scores with preference labels to compute utility.Source: §2 · p. 2, Figure 1 · p. 3
  • HCI tools use side-by-side outputs and LLM rationales to support qualitative comparison, but focus more on software tools than quantitatively verified pipelines.These findings inform VibeCheck's discovery mechanism, which is intended to align with human-powered qualitative analysis.Source: §2 · p. 2
  • LLM-based preference prediction can align with human judgments, but does not provide an interpretable account of why users prefer one output.VibeCheck aims to supply that account through interpretable differences between outputs.Source: §2 · pp. 2–3
  • Automated qualitative-analysis methods propose and validate candidate differences in unstructured text or images, while VibeCheck extends this line to pairwise inputs and measures separability, consistency, and alignment.Related work also uses evolutionary text descriptions to separate image classes for zero-shot classification.Source: §2 · p. 3

3 VIBE-BASED EVALUATIONS

This section defines vibes as perceptible axes of difference between model outputs and describes how their usefulness is evaluated. It measures agreement, model separability, and prediction of user preferences.

3 VIBE-BASED EVALUATIONS

  • A vibe is a human-perceptible axis along which two texts differ, described with definitions of what counts as high or low on that axis.For example, the axis can be defined as low = formal and high = friendly.Source: §3 · p. 3
  • For each prompt and pair of model outputs, a judge assigns -1, 0, or 1 to indicate whether model A scores lower, similarly, or higher on a vibe than model B.These scores impose an ordering on the outputs.Source: §3 · p. 3
  • A useful vibe should be well-defined, differentiating, and user-aligned, with agreement between evaluators measured using Cohen's Kappa.Differentiating means one model's outputs consistently rank higher across prompts, measured by a separability score.Source: §3 · p. 4
  • Model separability is measured by how accurately vibe scores classify which model produced an output, individually or as a feature vector for a linear classifier.The multi-vibe classifier reports held-out model-matching accuracy.Source: §3 · p. 4
  • Vibe scores are used as features in logistic regression to predict annotated user preferences, with held-out accuracy measuring preference prediction.Coefficients and p-values estimate the influence of an individual vibe; a differentiating trait such as frequent use of the letter “e” may not predict preference.Source: §3 · p. 4
  • The iterative process focuses on prompt-output tuples where existing vibes fail to distinguish the models, seeking more distinguishing vibes.The supplied description identifies this as an iteration step in optimizing for the three criteria.Source: §3 · p. 4

4 VIBECHECK

VibeCheck discovers candidate axes from model outputs, validates their agreement and predictive value, and iteratively adds vibes for examples that remain misclassified.

4 VIBECHECK

  • VibeCheck has three stages: vibe discovery, vibe validation, and vibe iteration.The paper directs readers to an appendix for implementation and prompt details.Source: §4 · p. 4
  • For discovery, GPT-4o examines batches of random prompt triplets and proposes human-interpretable, informative differences between the models.It generates multiple axes per sample, then recurring vibes are summarized across samples into a final set.Source: Vibe discovery · p. 4
  • For each vibe from discovery, authors apply it to validation tuples, compute agreement, model-matching accuracy, and preference-prediction accuracy, and filter out vibes with low scores.Vibes with Cohen’s Kappa below 0.2 or separability below 0.05 are removed. Judges score each sample twice with output order swapped; position-dependent decisions receive a score of 0 for that judge.Source: Vibe validation · pp. 4–5
  • Iteration uses tuples misclassified by the model-matching stage to prompt an LLM for new, currently unrepresented axes, then validates and appends the resulting vibes.The process repeats for a fixed number of iterations; after 3–5 iterations, discovery generally finds no additional vibes that significantly reduce model-matching error.Source: Vibe iteration · p. 5

5 RESULTS

VibeCheck is evaluated against human-discovered traits and predefined qualitative criteria, then applied to Chatbot Arena preferences across prompt types. It largely recovers human findings, outperforms predefined vibes on model matching, and reveals that preference prediction varies by topic.

5 RESULTS

  • The evaluation compares VibeCheck’s discovered traits with human-annotated differences, then measures agreement, model matching, and held-out preference prediction on pairwise user–LLM data.The human comparison uses prior work collecting human and GPT-3.5 responses to the same questions and 200 annotators reviewing prompt-output triples.Source: §5 · p. 5, §5.1 · p. 6
  • Unless otherwise stated, runs use three iterations, a proposer batch size of five, and 20 discovery samples per iteration; MATH, CNN/DailyMail, and COCO preferences are simulated with LLM judges.The simulated judgments ensemble GPT-4o and Claude 3.5 Sonnet, with ties removed; the top 10 vibes are selected using least-angle regression to predict model identity and sorted by separability score.Source: §5 · p. 5

5.1 MEASURING VIBECHECK’S ALIGNMENT WITH HUMAN DISCOVERY

  • VibeCheck recovers most human-discovered vibes, but misses GPT fact fabrication and its literal interpretation of questions; the authors suspect a weakness in the GPT proposer.Table 1 summarizes the top 10 vibes alongside corresponding human insights.Source: §5.1 · p. 6, Table 1 · p. 6

5.2 DESCRIBING USER PREFERENCE ON CHATBOT ARENA

  • VibeCheck achieves higher model-matching accuracy than predefined vibes across categories, and more iterations improve model matching and preference prediction.The paper also reports that its discovered vibes are more fine-grained.Source: §5.2 · p. 6
  • Model-matching and preference-prediction accuracy are lower for STEM prompts than writing or overall prompts, while preference prediction is best for writing prompts.The authors suspect stylistic traits matter less on objective questions, where factual accuracy may matter more; style is often more important for open-ended writing tasks.Source: §5.2 · pp. 6–7
  • On Chatbot Arena, Llama is described as more humorous, more formatted, and more example-rich, and as commenting less on ethics than GPT and Claude; these traits correlate positively with human preference.The figure caption explains that negative separability scores align Llama with the low (red) description, while negative preference coefficients indicate low descriptions are preferred.Source: Figure 2 · p. 7

6 APPLICATIONS

VibeCheck is applied to summarization, math problem-solving, and image captioning, comparing models selected for similar correctness performance or using a random subset of COCO images. Across these tasks, the analyses identify style differences associated with preference.

6 APPLICATIONS

  • Across overall, STEM, and writing Chatbot Arena data, VibeCheck achieves higher model-matching accuracy than predefined vibes, similar preference-prediction accuracy, and its largest improvements in writing.The table reports model-matching accuracy, preference-prediction accuracy, and average Cohen’s Kappa.Source: Table 2 · p. 8
  • Writing favors detailed explanations, humor, and expressive emotion, while STEM favors logical rigor and shows less separability for humor and emotional tone.The corresponding preference correlations for detailed explanations, humor, and expressive emotion are positive in writing and negative in STEM.Source: Figure 3 · p. 8

6.1 WHAT DO DIFFERENT MODELS FOCUS ON WHEN SUMMARIZING?

  • 71.12% Command X win-rate on CNN/DailyMail accompanied 71.29% model-matching accuracy and 61.42% preference-prediction accuracy, with summaries favoring introductions, conclusions, examples, and multiple viewpoints over TNLG.Command X’s identified qualities were positively correlated with human preference.Source: §6.1 · p. 8

6.2 HOW DO DIFFERENT LLMS SOLVE MATH PROBLEMS?

  • 97.09% model-matching accuracy and 72.79% preference-prediction accuracy accompanied preference correlations favoring formal tone and notation over Llama-405b’s overexplaining.The analysis used only questions both models answered correctly; Llama-405b also used markdown headings and a conversational tone.Source: §6.2 · p. 9, Figure 4 · p. 9

6.3 WHAT ARE VLM’S CAPTIONING STYLE?

  • 99.13% model-matching accuracy and 89.02% preference-prediction accuracy accompanied GPT-4V’s poetic, mood-focused captions versus Gemini-1.5-Flash’s literal, object-focused descriptions.Captions were compared without the images, though the framework can be adapted to multimodal evaluation.Source: §6.3 · pp. 9–10

7 LIMITATIONS

VibeCheck measures how vibes relate to model identity and user preference, but its estimates have interpretive, evaluation, and reproducibility limits.

7 LIMITATIONS

  • Vibe-preference relationships may reflect confounding factors, such as factual accuracy, rather than a direct influence of the vibe itself.For example, accurate outputs may incidentally seem more engaging because of their clarity or relevance.Source: §7 · p. 10
  • When model accuracy differs substantially, discovered vibes may focus primarily on accuracy-related differences and miss other relevant traits.Source: §7 · p. 10
  • Validation is costly, inexpensive LLM judges can be incorrect or biased, and repeated runs can yield different vibes and results.The passage notes that judges may favor their own outputs.Source: §7 · p. 10

8 CONCLUSION

VibeCheck adds measurable qualitative properties to model evaluation, complementing correctness metrics because those properties affect how people judge models.

8 CONCLUSION

  • VibeCheck is presented as a complement to correctness metrics for capturing qualitative properties that measurably affect how people judge models.Source: §8 · p. 10
  • The framework can be extended to audio or visual content and applied to compare pairwise sets of texts.Source: §8 · p. 10

A EXPERIMENTAL DETAILS & DATASET STATISTICS

This section lists dataset-statistics, model-win-rate, and hyperparameter tables, while describing HC3 human annotations used as gold-standard labels. The annotations contrast ChatGPT’s response traits with human responses.

A EXPERIMENTAL DETAILS & DATASET STATISTICS

  • The appendix lists dataset statistics, model win rates, and VibeCheck hyperparameters, but the supplied table captions provide no cell values.The hyperparameter descriptions include iteration count, evaluation-vibe count, discovery sample count, batch size, and final-vibe count.Source: Table 3 · p. 15, Table 4 · p. 15, Table 5 · p. 15

B GOLD STANDARD LABELS

  • The listed ChatGPT characteristics include organized, stepwise, detailed answers, bias minimization, scope refusals, and occasional incorrect or fabricated information.The supplied passages say responses often begin with definitions and conclude with a summary.Source: B GOLD STANDARD LABELS · p. 16
  • Human evaluators’ HC3 differences include ChatGPT staying on topic and giving objective, fact-based answers, unlike humans’ related-topic shifts and personal opinions.The HC3 passage introduces these as key differences reported by its human evaluators.Source: B GOLD STANDARD LABELS · p. 16
  • ChatGPT is described as formal and structured, while human speech is more conversational and may express emotion through punctuation or tone.Source: B GOLD STANDARD LABELS · p. 16

C GENERATING PRESET VIBES

The appendix describes prompting GPT-4o to propose interpretable axes, then discovering, reducing, validating, and iteratively expanding vibes. It also specifies a separate prompt for judging response preference.

C GENERATING PRESET VIBES

  • GPT-4o generates 10 preset axes intended to capture common, mutually exclusive, human-interpretable differences useful for choosing between language models.The prompt asks for clearly defined high and low scores and differences that may appear in unseen data.Source: Table 6 · p. 16, C GENERATING PRESET VIBES · pp. 16–17

D.1 VIBE DISCOVERY

  • Vibe discovery prompts an LLM to identify clear, interpretable differences between two models’ outputs, then clusters and reduces similar axes.The appendix describes embedding-based agglomerative clustering followed by GPT-4o reduction to fewer than 20 vibes.Source: §D.1 · pp. 17–18

D.2 VIBE VALIDATION

  • Vibe validation asks judges to compare two outputs on a specified axis and choose which output scores higher, allowing “N/A” for inapplicable or roughly equal cases.Source: §D.2 · p. 19

D.3 VIBE ITERATION

  • When prior model-identity predictions misclassify prompts, iteration discovers additional uncovered axes on those examples, reduces redundancy, and appends resulting vibes.Source: §D.3 · p. 20

D.4 GENERATING PREFERENCE LABELS

  • Preference labels come from a judge prompt comparing two assistants’ response quality across helpfulness, relevance, accuracy, depth, creativity, and detail.The prompt requests a comparison and explanation before assigning A, B, or tie.Source: §D.4 · p. 21

E FURTHER RELATED WORKS

  • Prior automatic evaluation work includes metrics such as BLEU and ROUGE and efforts to assess factual accuracy, fluency, conciseness, and standardized evaluation.Source: E FURTHER RELATED WORKS · p. 21

F LIMITATIONS

The supplied limitation-related figures flag weaknesses in the GPT-4o-mini judge and instability in some ranker evaluations. These concerns are tied to mathematical ability and question timing or model knowledge cutoffs.

F LIMITATIONS

  • Figure 5 identifies weaknesses in the mathematical abilities of the GPT-4o-mini LLM judge.Source: Figure 5 · p. 22
  • Figure 6 says answers to some questions vary with asking time and model or ranker knowledge cutoffs, reducing evaluation reliability and inter-annotator agreement.Source: Figure 6 · p. 22

G VIBES FROM EACH APPLICATION

The appendix presents vibe visualizations for several datasets and model pairs, including Chatbot Arena overall, STEM, and writing comparisons. It also shows application examples and additional Llama-versus-GPT examples.

G VIBES FROM EACH APPLICATION

  • The figures compare preset and VibeCheck vibes on Chatbot Arena overall, STEM, and writing data, though their captions do not state comparative outcomes.Source: Figure 9 · p. 23, Figure 8 · p. 23, Figure 11 · p. 24
  • On MATH CoT, VibeCheck finds five distinct vibes that can almost perfectly separate GPT-4o and Llama-3-405B outputs.The caption attributes the five-vibe count to the reduction step not requiring at most 10 vibes.Source: Figure 15 · p. 26

H MORE LLAMA VS GPT EXAMPLES

Related papers

Loading 2410.12851v7…