Source-linked AI summary
NoveltyBench: Evaluating Language Models for Humanlike Diversity
Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, Daphne Ippolito
TL;DR
Language models are rarely evaluated for the diversity of their possible responses, despite mode collapse limiting varied and customizable generation. NOVELTYBENCH measures novelty and quality across curated and real-user prompts, finding low distributional diversity—including weaker diversity among larger same-family models—and motivating diversity-aware evaluation and training.
Problem
Existing language-model evaluations focus mainly on knowledge, reasoning, or the single most likely generation, leaving response diversity insufficiently assessed.
Method
NOVELTYBENCH evaluates sets of generations from 1,100 prompts using functional-equivalence classes and metrics that combine novelty with quality.
Results
Current language models lack distributional diversity; they produce fewer than 3 distinct responses in 10 queries, while larger same-family models tend to be less diverse than smaller ones.
Takeaways & Limitations
Prompting can elicit some novel generations, but direct methods for improving distributional diversity remain valuable for producing varied outputs across generation settings.
Takeaways & Limitations
The benchmark does not determine how much diversity users want, since some prompts may favor variation while others may favor consistent responses.
Abstract
from arXiv · showhide
Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel outputs. Our work introduces NoveltyBench, a benchmark specifically designed to evaluate the ability of language models to produce multiple distinct and high-quality outputs. NoveltyBench utilizes prompts curated to elicit diverse answers and filtered real-world user queries. Evaluating 20 leading language models, we find that current state-of-the-art systems generate significantly less diversity than human writers. Notably, larger models within a family often exhibit less diversity than their smaller counterparts, challenging the notion that capability on standard benchmarks translates directly to generative utility. While prompting strategies like in-context regeneration can elicit diversity, our findings highlight a fundamental lack of distributional diversity in current models, reducing their utility for users seeking varied responses and suggesting the need for new training and evaluation paradigms that prioritize diversity alongside quality.
1 Introduction
NOVELTYBENCH addresses the under-evaluation of response diversity by measuring whether language models can produce multiple differing, correct, and high-quality answers to prompts where humans would vary. It complements quality benchmarks by targeting the diversity of the output distribution.
- Human responses to subjective, creative, and preference-based prompts are typically diverse, whereas language models often produce near-duplicates known as mode collapse.This can reduce usefulness for users with differing preferences and needs.
- Most existing evaluations assess the quality of a single likely generation rather than meaningful alternatives across the output distribution.This creates incentives to improve the best response without measuring distributional diversity.
- The benchmark introduces metrics that quantify unique functional alternatives and combine novelty with response quality.distinct_k counts unique equivalence classes among k samples, while utility_k incorporates novelty, quality, and user patience at p = 0.8.
- NOVELTYBENCH measures whether models generate multiple differing but still correct answers for prompts involving subjectivity, randomness, and creativity.The benchmark is intended to complement existing quality-based evaluations while retaining generation quality as part of the assessment.
- Across 1,100 prompts and twenty frontier models, the evaluation finds that current models lack diversity, with larger same-family models tending to produce less diverse outputs.The findings motivate evaluation and training approaches that consider diversity alongside quality.
2 Related Work
Related work documents biases and homogenization in language-model outputs, emerging methods for increasing diversity, and prior evaluations that consider multiple valid generations. However, diversity remains less central than single-generation quality in standard evaluation.
- Prior studies report output biases toward particular demographic groups, genders, countries, and personas, which can limit downstream utility.
- An analysis of 67 COLM 2024 and ICLR 2025 benchmark papers found that over 90% evaluate a single or best generation rather than the full output distribution.
- Research has explored decoding changes such as higher temperature and modified training objectives to diversify language-model outputs.
- Earlier work includes the Duolingo Shared Task for diverse correct translations and probabilistic evaluation of commonsense knowledge through open-ended generation.
3 Benchmark
The benchmark combines curated and real-user prompts with human-informed functional-equivalence modeling to evaluate both novelty and quality. Its metrics count meaningfully distinct generations and estimate their cumulative utility for users with finite patience.
- 3.1 Dataset Curation: NOVELTYBENCH contains NB-CURATED with 100 author-designed prompts and NB-WILDCHAT with 1,000 prompts filtered from real ChatGPT interactions.The prompts are selected to elicit multiple valid or diverse responses.
- 3.1 Dataset Curation: NB-CURATED covers randomness, underspecified factual knowledge, creative writing, and subjectivity as categories expected to produce diverse answers.
- 3.1 Dataset Curation: Eight human responses are collected per curated prompt, although the annotators’ institutional homogeneity makes this an approximate lower bound on human diversity.Even this group produced more diverse responses than most state-of-the-art language models.
- 3.2 Evaluating Novelty: The unified utility metric discounts repeated functional equivalents and weights novel generations by a user-patience model, with p set to 0.8.The framework combines diversity and individual-generation quality to estimate cumulative user utility.
- 3.2 Evaluating Novelty: The benchmark partitions generations into functional-equivalence classes learned from human annotations, treating surface-level variation as insufficient when it adds little user utility.The classifier was trained on 1,000 annotated pairs and achieved 79% accuracy and F1 0.811 on 100 held-out pairs.
- 3.2 Evaluating Novelty: distinct_k counts the equivalence classes among k sampled generations, so higher values indicate more meaningful alternatives to the most likely response.
4 Evaluation Results on NOVELTYBENCH
NOVELTYBENCH evaluates whether models can produce multiple distinct, useful responses, revealing substantial diversity deficits across frontier systems. Alternative prompting can improve novelty, while alignment stages progressively reduce diversity despite some utility gains.
- Frontier models struggle to produce simultaneously diverse and useful responses on NOVELTYBENCH.The benchmark evaluates 20 state-of-the-art models, including systems from Anthropic, OpenAI, Google, and Meta.
- Below 4 out of a maximum utility score of 10, Claude 3, Gemini, and GPT-4o score in cumulative utility evaluations.The metric captures both overall response diversity and individual generation quality.
- As users demand more diverse generations, larger models degrade in utility more quickly than smaller models.Larger models perform better when only the best generation is considered, but can underperform smaller models when users require diverse, creative outputs.
- In-context regeneration is the most successful prompting strategy, roughly matching human-writer diversity across the evaluated models.Paraphrasing and system prompting are only marginally effective, while in-context regeneration explicitly uses previous answers to elicit new ones.
- Each alignment stage progressively reduces model diversity, with significant drops during DPO, while transitioning from SFT to DPO improves utility.The results indicate a trade-off between alignment objectives and output diversity.
5 Discussion and Future Work
The discussion identifies distributional diversity as a persistent weakness, while noting that user needs vary between diversity and consistency. It calls for training, evaluation, and user studies that better balance these objectives.
- Discussion: Prompting strategies can elicit novel generations, but they are cumbersome and do not resolve models’ low-entropy output distributions.The authors argue that methods directly improving distributional diversity would provide diverse outputs across generation settings.
- Discussion: The benchmark does not determine how much diversity users want, because some tasks favor varied answers while others favor consistency.The paper contrasts dice-rolling prompts with recommendations and highlights consistency requirements in healthcare and legal assistance.
- Future Work: Current models struggle to represent diverse valid human responses while also maintaining perfect consistency across similar prompts.The authors propose studying how models might modulate between diversity and consistency according to task requirements and user expectations.
- Future Work: Diversity-promoting post-training methods remain an emerging area, and the authors call for broader approaches that preserve response quality.They note that modified supervised fine-tuning and preference optimization are initial steps.
Contributions
The contributions span experiment design, benchmark engineering, data annotation, visualization, website development, and project supervision. Authors also contributed collectively to experiments, annotation, and writing.
- Contributions: YZ conceived evaluation beyond most-likely outputs and ran the experiments.
- Contributions: HD, HL, and VS contributed to the NOVELTYBENCH codebase.
- Contributions: XL established the benchmark website, while BW helped collect NB-CURATED and created the paper’s visualizations.
- Contributions: SH annotated data for training and evaluating the equivalence classifier and verified prompts in the NB-WILDCHAT subset.
- Contributions: DI supervised the project, proposed evaluating creativity against humans, and suggested alternative diversity-inducing prompts.
- Contributions: All authors contributed to experiment design, data annotation, and paper writing.
A.1 Diversity of Reasoning Models
The reasoning-model analysis extends the benchmark to models with controllable inference-time reasoning effort. OpenAI models show reduced diversity and utility with more reasoning, whereas Gemini shows minimal change.
- A.1 Diversity of Reasoning Models: OpenAI’s reasoning models show a trend toward reduced diversity and utility as reasoning effort increases.
- A.1 Diversity of Reasoning Models: The core analysis focuses on 20 representative non-reasoning models because reasoning capabilities were not yet broadly available across model families.
- A.1 Diversity of Reasoning Models: Gemini’s reasoning budget had minimal effect on the evaluation, unlike the trend observed for OpenAI models.
A.2 Prompt for Selecting Diverse Prompts
The WildChat subset used a GPT-4o prompt to select prompts expected to support diverse responses.
- A.2 Prompt for Selecting Diverse Prompts: GPT-4o was used to select WildChat prompts that allow for diverse responses.
A.3 Example Prompts from NB-WILDCHAT
This section presents sample prompts from the NB-WILDCHAT dataset, selected using a GPT-4o classifier.
- The section presents a sample of prompts from the NB-WILDCHAT dataset.
- The prompts were selected using a classifier.
- GPT-4o was used as the classifier for selecting the sample prompts.
A.4 Evaluation of equivalence classification methods
The study compares heuristic and learned methods for judging whether two language-model generations are equivalent. A fine-tuned DeBERTa model performed best on validation, achieving the highest AUC and strong accuracy and F1 scores.
- Methods: The evaluation compared BERTScore, BLEU, ROUGE, fine-tuned GPT-3.5-turbo and GPT-4o-mini, and fine-tuned DeBERTa.The models were trained or evaluated using a manually labeled dataset of 1,000 generated-text pairs.
- Results: 0.81 AUC was achieved by DeBERTa on a validation set of 100 pairs, outperforming fine-tuned OpenAI models and heuristic-based methods.The passage reports that the improvement was statistically significant.
- Results: 71.0% accuracy and 0.811 F1 score were obtained by the fine-tuned DeBERTa model after selecting an appropriate confidence interval.These metrics summarize its classification performance under the chosen confidence setting.