Source-linked AI summary
Evaluating the Diversity and Quality of LLM Generated Content
Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, Osbert Bastani
TL;DR
LLM evaluation has paid limited attention to measuring quality and diversity together, even though varied, high-quality outputs matter for open-ended applications. The paper introduces effective semantic diversity, which measures semantic variation among valid outputs using automated evaluation. It finds that preference-tuned models can achieve greater effective semantic diversity than SFT or base models, while smaller models are often more parameter-efficient for generating unique content.
Problem
Existing diversity metrics may not capture meaningful semantic content, and the field lacks systematic evaluation of quality and diversity together across model families and training methods.
Method
The paper introduces effective semantic diversity and evaluates it with automated quality checks, code execution for programs, and LLM judges for natural language.
Results
Preference-tuned models show greater effective semantic diversity than SFT or base models, while smaller models are often more parameter-efficient for unique content within a fixed sampling budget.
Takeaways & Limitations
Evaluating diversity together with output quality changes conclusions about preference tuning and highlights practical trade-offs between model scale and unique-content efficiency.
Takeaways & Limitations
The original effective-diversity formulation can be confounded by the number of samples in variable-sized valid-only subsets, motivating a pairwise alternative.
Abstract
from arXiv · showhide
Recent work suggests that preference-tuning techniques -- such as Reinforcement Learning from Human Feedback (RLHF) methods like PPO and GRPO, as well as alternatives like DPO -- reduce diversity, creating a dilemma given that these models are widely deployed in applications requiring varied outputs. We argue that diversity without consideration of quality has limited practical value. To address this issue, we introduce a framework for measuring effective semantic diversity -- diversity among outputs that meet quality thresholds -- which better reflects the practical utility of large language models (LLMs). Using open-ended tasks that require no human intervention, we find counterintuitive results: when using diversity metrics that do not explicitly consider quality, preference-tuned models -- particularly those trained via RL -- often produce outputs with lower diversity; however, these same preference-tuned models generate greater effective semantic diversity than supervised fine-tuned (SFT) or base models. Our analysis further shows another trend: while larger models may exhibit greater effective semantic diversity than smaller models, the smaller models are consistently more parameter-efficient at producing unique content within a fixed sampling budget. These findings have practical implications for applications that require diverse yet high-quality outputs, from creative assistance to synthetic data generation.
1 Introduction
The paper argues that useful diversity requires outputs to be both semantically varied and sufficiently high-quality, then introduces a framework to measure this trade-off without inference-time human evaluation. Experiments show that preference tuning can increase effective semantic diversity despite reducing diversity under quality-blind metrics, while smaller models can be more parameter-efficient for unique content.
- Motivation and framework: Effective semantic diversity counts meaningful variation among outputs that meet quality thresholds, rather than rewarding unusable diversity.The framework is designed to reflect practical utility in open-ended generation.
- Motivation and framework: The framework requires no human evaluation at inference time and supports comparisons across model families and training techniques.
- Motivation and framework: Quality-blind diversity metrics may make base models appear most diverse because they can generate many invalid outputs.In Figure 1, effective semantic diversity penalizes excessively low or high temperatures and incoherent generations.
- Findings: Preference-tuned models exhibit greater effective semantic diversity than SFT models in code generation and creative writing, despite possible reductions under quality-blind metrics.The higher proportion of high-quality outputs outweighs lower diversity per high-quality sample in the reported code-generation analysis.
- Findings: Smaller models, down to around 500 million parameters, are often more efficient at generating unique programs within a fixed sampling budget.
2 Background and Related Work
Prior work measures LLM diversity mainly with lexical or neural metrics because human evaluation is expensive, but it remains unclear whether these metrics capture meaningful semantic content across model distributions. The paper situates its contribution against research on preference tuning, quality-diversity trade-offs, novelty, and synthetic-data diversity.
- Preference tuning: PPO, DPO, rejection sampling, and GRPO are prominent approaches for aligning language models with human preferences.Prior work also reports that over-optimizing reward models can produce incoherent or undesirable outputs.
- Diversity measurement: Lexical diversity metrics commonly summarize n-gram statistics, including Distinct-N and Self-BLEU, while neural approaches use learned representations.
- Diversity measurement: Existing diversity metrics have not established whether neural diversity reflects effective semantic content across models with different capabilities, safety attributes, and styles.Many evaluation models are trained on human-authored text, leaving their robustness for LLM-generated distributions unclear.
- Related directions: Related studies examine linguistic novelty, quality-diversity trade-offs, population-level associations, and diversity in synthetic training data.
3 Measuring Effective Semantic Diversity
The framework measures effective semantic diversity as the prevalence of outputs that are both valid and semantically unique. It combines validity and semantic-equivalence functions with set-based and pairwise formulations, and applies the framework to open-ended programming and natural-language tasks.
- Measuring Effective Semantic Diversity: The framework defines validity over generations and semantic equivalence through a semantic function mapping outputs into a semantic space.Two generations are equivalent when their semantic-function values match.
- Measuring Effective Semantic Diversity: Effective semantic diversity counts valid generations that are semantically unique within each prompt’s output set.The metric is normalized by the total number of generations.
- Measuring Effective Semantic Diversity: A pairwise formulation normalizes diversity by all possible generation pairs, making it robust to variation in the number of valid outputs.For natural language, the semantic distance uses a soft similarity-based formulation.
- Measuring Effective Semantic Diversity: The set-based formulation can be confounded by the number of samples when analysis uses variable-sized valid-only subsets.The authors therefore adopt a pairwise metric to address this issue.
- Dataset and Evaluation Tasks: The evaluation uses open-ended programming tasks adapted from competitive-programming problems and creative-writing prompts, with validity checked differently across domains.Programs are assessed using executable behavior, while natural-language quality is judged against criteria by an LLM judge.
4 Experimental Setup
The experiments compare diversity across post-training methods, model sizes, and sampling settings using code and natural-language tasks. They define domain-specific validity and semantic measures, evaluate multiple diversity metrics, and use paired nonparametric statistical tests.
- Diversity Metrics: The evaluation includes effective semantic, lexical, and syntactic diversity metrics, including EAD with n=4 and AST-based code diversity.The effective semantic metric depends on the validity and semantic functions.
- Validity and Semantic Equivalence: For code, a generation is valid when it runs without errors and produces non-null outputs on every test case.Program semantics are represented by the complete output trace across the fixed test set.
- Validity and Semantic Equivalence: For natural language, an LLM judge scores creative-generation quality and normalizes the result to the [0, 1] interval.The judge also evaluates conceptual and thematic overlap for the soft semantic-similarity measure.
- Models and Comparisons: The study compares SFT, DPO, PPO, GRPO, instruction tuning, preference tuning, and model size across several model families.Additional models are included to assess parameter efficiency for generating unique examples.
- Statistical Analysis: Model comparisons report Wilcoxon signed-rank p-values and Cohen’s D effect sizes for average diversity across the dataset.Models are paired within families while prompt templates vary across zero-shot, two-shot, and chain-of-thought settings.
5 Experimental Results
Across programming and natural-language experiments, post-training improves effective semantic diversity and validity relative to base models, while preference tuning changes lexical and syntactic diversity by domain. Larger models generally increase semantic diversity, but smaller models are more parameter-efficient for generating unique programs under a fixed sampling budget.
- Post-training increases effective semantic diversity and validity relative to base models, with reinforcement-learning methods substantially outperforming supervised fine-tuning in effective semantic diversity.
- Preference tuning reduces lexical and syntactic diversity in programming tasks but increases these metrics in natural-language creative writing.
- Within valid code generations, preference-tuned models have more semantic duplicates than supervised fine-tuned models, but increased validity more than compensates overall.
- Larger models generally exhibit higher semantic diversity in code and natural language, without reducing code lexical or syntactic diversity.
- For 32 generations per prompt, smaller programming models are consistently more parameter-efficient than larger models at producing unique examples.
6 Discussion and Conclusion
The paper studies effective semantic diversity through methods that jointly assess output quality and diversity, using code execution for programming and LLM judges for natural language. This framework supports empirical analysis of how post-training and model size influence diversity.
- The study measures effective semantic diversity by jointly evaluating quality and diversity, using code execution for programming and LLM judges as a proxy for human evaluation in natural language.
A.1 Additional Neural Diversity Metrics
The paper supplements its primary measures with neural diversity metrics across temperature settings and models. These analyses use ICESCORE and cosine diversity from CODELLAMA-7B-INSTRUCT embeddings.
- Neural diversity is evaluated across temperature sweeps using ICESCORE and cosine diversity of CODELLAMA-7B-INSTRUCT embeddings.
- The additional metrics are intended to compare how temperature modulation affects neural diversity across models.
A.2 Sample Size Confounding Diversity and Analysis of Pairwise Diversity Metric
The analysis shows that sample size can confound pairwise diversity measurements, motivating a new metric whose value converges to a constant as sampling grows.
- Varying subset size can dramatically change Equation (2), with the metric decreasing as samples become larger.This is problematic when comparing valid-only program subsets whose sizes differ across models.
- Equation (3) is therefore necessary when valid-only subsets vary in size across models.
- Analysis of Pairwise Diversity Metric: The original semantic diversity metric converges to zero as the number of sampled responses tends to infinity.
- Analysis of Pairwise Diversity Metric: The new Equation (3) metric instead converges to a constant as n →∞ under a finite set of semantic meanings.The analysis represents each meaning by its sampling proportion π_k and weights pairwise semantic distances accordingly.
A.3 Dataset Creation and Additional Details
The dataset was built from CodeNet and AlphaCode materials through language-model-assisted expansion, manual validation, test-case integration, and final experimental checks.
- Dataset Creation: The dataset construction began with CodeNet and AlphaCode problems and used language-model assistance followed by manual validation.
- Dataset Creation: Each selected problem received a canonicalized description, a wrapper function, and a property-based testing function.
- Dataset Creation: More than 300 CodeNet programs were initially processed, and 108 validated examples were ultimately used.
- Validation: Manual editing corrected language-model mistakes in problem components before further processing.
- Validation: Problems were required to have at least 10 test cases, with 100 additional property-based cases generated for three under-covered problems.
- Validation: Final validation fixed faulty problem-description argument parsers before producing the final dataset.
- Additional Details: For each generation, the pipeline extracts relevant code, serializes outputs and types, and executes test cases to capture results.Execution occurs inside an isolated Docker container to reduce risks from blindly running generated programs.
A.4 Raw Experimental Results for All Experiments
The appendix provides the raw results used in the paper’s analysis.
- Tables 5 and 6 contain the raw results used for the analysis.
A.5 Additional Information on the Syntactic Diversity Metric
The appendix documents syntactic-diversity measurement, prompting, constrained generation, natural-language evaluation, and model-comparison results across code and language tasks.
- Syntactic Diversity: Syntactic diversity is computed from AST subtrees of height 4 and only applies to syntactically correct generations.
- Prompts: Figures 7–9 provide the zero-shot, two-shot, and two-shot chain-of-thought prompt templates used in code experiments.
- Constrained Generation: The code validity oracle accepts integer lists with maximum length 1000, and prompts were modified to enforce these constraints.
- Syntactic Diversity: Program identifiers are canonicalized so AST comparisons emphasize structural differences rather than variable names.
- Constrained Generation: Constrained-generation results were broadly consistent with earlier findings, with stronger semantic-diversity differences between DPO- and RL-tuned models in code.
- Natural Language Experiments: Natural-language experiments used creative writing, argumentative writing, and brainstorming tasks evaluated with pairwise effective semantic diversity.
- Natural Language Experiments: Post-training was consistently associated with higher effective semantic diversity than base models, while raw neural diversity often favored less aggressive post-training.
- Natural Language Experiments: Larger models tended to achieve higher effective semantic diversity in argumentative and creative writing, but this pattern was less consistent for brainstorming.