Do Large Language Models Capture the Diversity in their Training Data?

Youqi Wu; Farzan Farnia

Source-linked AI summary

Do Large Language Models Capture the Diversity in their Training Data?

Youqi Wu, Farzan Farnia

arXiv:2609.02275v1Sep 2, 2026cs.CLcs.AIcs.LG

TL;DR

The paper investigates whether conditional generative models capture the full diversity of valid training outputs. It measures conditional diversity from paired samples, proves the relevant matrix entropy is concave, and applies entropy-constrained reweighting. Across open language and image-generation settings, generated outputs consistently have lower conditional entropy than training data, while the proposed projection preserves the input marginal and remains close to the original distribution.

Problem
Existing evaluations do not directly establish whether conditional generative models capture the full range of valid outputs in their training distributions.
Method
The paper compares conditional entropy and matrix-based von Neumann entropy on paired samples, then reweights multiple generated candidates through entropy-constrained projection with scalable mirror descent.
Results
Across OLMo, Pythia, GPT-Neo, ImageNet, and MS-COCO settings, model-generated outputs consistently exhibit lower conditional entropy than corresponding training data.
Takeaways & Limitations
The work provides an information-theoretic framework for diagnosing and mitigating conditional diversity gaps while preserving the input marginal.
Takeaways & Limitations
Direct training-data comparison is constrained to models whose relevant training data are publicly accessible or reconstructable.

Abstract

from arXiv · show

Large language models are trained to model conditional distributions over text, yet it remains inadequately understood whether they capture the full diversity of plausible outputs present in their training data. We study this question through an information-theoretic lens by comparing the conditional entropy of model-generated outputs with that of the corresponding training data. Given paired input-output samples, we use conditional entropy and its matrix-based analogue based on von Neumann entropy to measure output variability beyond what is explained by the conditioning input, without requiring multiple reference outputs for the same prompt. Across LLM families with publicly available training data, including OLMo, Pythia, and GPT-Neo, we consistently find that model-generated outputs exhibit lower conditional entropy than their training data, across different model scales, sequence lengths, and decoding strategies. We observe a similar conditional diversity gap beyond language modeling, including class-conditioned ImageNet generators and text-conditioned models trained on MS-COCO. To address this gap, we propose a post-hoc correction mechanism that generates multiple outputs for each input and reweights them through a matrix-entropy projection, increasing conditional diversity while remaining close to the original model distribution. We prove the concavity of the matrix-based conditional entropy functional, which makes the resulting entropy-constrained projection a convex optimization problem, and develop a scalable mirror-descent algorithm for its implementation. Our results reveal a systematic conditional diversity gap between modern generative models and their training data, and provide an information-theoretic framework for measuring and mitigating this gap.

1 Introduction

The paper asks whether conditional generative models capture the full diversity of valid training outputs, a question existing predictive and generation metrics do not directly answer. It introduces entropy-based measurement and post-hoc reweighting, finding a consistent conditional output-range gap across modalities.

Existing likelihood, alignment, and semantic metrics can miss whether models concentrate probability mass on a narrower subset of valid outputs.The paper motivates direct comparison with training-data diversity.
Paired input-output samples enable conditional entropy estimates without requiring repeated reference outputs for every condition.The method uses Shannon conditional entropy and a matrix-based analogue built from product kernels.
Model-generated continuations have lower conditional entropy than corresponding training data across OLMo, Pythia, GPT-Neo, ImageNet, and MS-COCO settings.The reported pattern is consistent across language and image-conditioned generation.
The matrix-based conditional entropy is concave, making entropy-constrained projection toward higher conditional diversity a convex optimization problem.This provides the structural basis for the proposed correction method.
The post-hoc method reweights multiple generated candidates while preserving each prompt’s total weight and uses mirror descent with sketched joint features for scalability.The implementation operates over a product of simplices and avoids explicitly forming full tensor-product representations.

2 Preliminaries

The preliminaries define conditional generation through joint and marginal distributions, then construct kernel-based conditional entropy from paired samples. The framework isolates output uncertainty beyond variability explained by the conditioning inputs.

A conditional model specifies Q_Y|X, which combined with the data input marginal P_X induces the joint model distribution Q_XY = P_X Q_Y|X.
The product kernel on paired inputs and outputs is k_XY((x,y),(x′,y′)) = k_X(x,x′) · k_Y(y,y′).
For paired samples, the product-kernel Gram matrix is K_XY = K_X ⊙ K_Y, where ⊙ denotes the Hadamard product.
Matrix-based conditional entropy measures joint input-output entropy minus input entropy, isolating uncertainty contributed by outputs after accounting for conditioning variability.
The framework also uses order-2 matrix entropy H2(A) = −log Tr(A^2) as a computationally simpler counterpart.

3 Conditional output-range gaps in open language models

The open-language-model experiments compare paired corpus continuations with model continuations under multiple prefixes and decoding strategies. Across all tested families and methods, the model’s measured conditional output range is smaller than the training data’s.

The study evaluates OLMo, Pythia, and GPT-Neo using prefixes of 3–5 tokens and continuations of 2–4 tokens.Model outputs use greedy, nucleus, or ancestral sampling; training outputs are corpus continuations.
Reported diversity measures are exponentiated conditional VNE, conditional RKE, and lexical Distinct-2, interpreted as effective numbers of conditionally distinguishable outputs for entropy measures.
Positive gaps occur across all three model families and all decoding methods, meaning model-generated continuations have smaller measured conditional output ranges than training data.The gap is defined as Δ = Metric_train − Metric_model.
Distinct-n scores also show positive gaps in every setting, with the largest deficits under greedy decoding.Distinct-n is supporting evidence because it does not explicitly condition on the prefix.
These findings motivate testing both the structural basis of conditional von Neumann entropy and candidate reweighting for higher-entropy conditional distributions.

4 Theoretical Features of Kernel-induced Conditional von Neumann Entropy

This section establishes concavity for the kernel-induced conditional von Neumann entropy, a property not inherited directly from the classical conditional-entropy identity. Consequently, entropy-threshold constraints become convex and support tractable post-hoc projection.

Classical conditional Shannon entropy is concave in the joint distribution P_XY for finite alphabets.
The kernel-induced conditional von Neumann entropy requires a separate proof because it lacks the classical identity H(Y|X) = E_X[H(Y|X = x)].The proof uses strong subadditivity of quantum entropy.
For normalized positive semidefinite kernels, conditional von Neumann entropy is a concave functional of the joint distribution P_XY.
Because superlevel sets of concave functions are convex, requiring conditional entropy at least ρ defines a convex constraint over reweighted empirical distributions.
The resulting entropy-based procedure is suitable as both a diagnostic and the basis for tractable post-hoc reweighting.

5 Kernel-based Entropy-Constrained Projection for Conditional Generators

The paper formulates entropy-constrained projection as a convex optimization problem that reweights generated candidates toward higher conditional entropy while preserving the input marginal. A block mirror-descent algorithm makes this projection scalable.

Entropy-constrained projection: The projection reweights multiple generated outputs per input toward higher conditional entropy while remaining close to the original model distribution.Each input retains total mass 1/N, so only the conditional output distribution changes.
Optimization formulation: The feasible set fixes the input marginal, making the input-entropy term constant across candidate reweightings.The objective combines conditional entropy with a finite-sample conditional MMD discrepancy from uniformly weighted model samples.
Convexity: The entropy-constrained projection is a convex program because matrix-based conditional entropy is concave and the relevant feasible set is convex.The finite-support and covariance-space formulations are both identified as convex programs.
Scalable algorithm: CEP-BEG solves the covariance-space problem using block mirror descent with exponentiated-gradient normalization separately for each input.Block normalization preserves the empirical input marginal throughout optimization.
Scalable algorithm: Sketched joint features reduce the computational burden of tensor-product representations, with each iteration costing O(Nmr^2 + r^3) under full eigendecomposition.Random projections support finite-dimensional embeddings, while random Fourier features support shift-invariant kernels.

6 Numerical Results

Experiments across open language models and conditional image generators consistently find higher conditional diversity in training data than in generated samples. Entropy projection and guidance increase measured diversity, while diversity-oriented decoding preserves standard summarization quality within reported statistical comparisons.

Conditional diversity gaps: The conditional diversity gap generally grows with sample size because training-data conditional VNE increases faster than generated-sample VNE.This pattern is reported for language and image-generation experiments.
Conditional diversity gaps: Training data exhibit higher conditional entropy than generated outputs across OLMo, Pythia, GPT-Neo, ImageNet generation, and MS-COCO image generation.The language-model comparison spans multiple decoding strategies and model scales.
Downstream application: CVS-MBR achieves the highest conditional VNE and Distinct-2 and the lowest Self-BLEU-2 across three summarization settings, without significant ROUGE-L or BERTScore degradation versus MBMBR-L.The comparison reflects a reported quality–diversity trade-off.

7 Conclusion and Limitations

The paper concludes that conditional diversity gaps persist across open language and image-generation models and that matrix-based conditional entropy supports both diagnosis and correction. Its direct training-data comparisons are constrained to models whose training data are publicly available or reconstructable.

Conclusion: The study finds a consistent conditional output-range gap across open language models and conditional image-generation settings.It uses conditional entropy and matrix-based von Neumann entropy as the diagnostic framework.
Conclusion: Concavity of kernel-induced conditional von Neumann entropy supports entropy-constrained projection as a convex optimization problem.The same structure underlies the proposed post-hoc reweighting method.
Conclusion: The proposed reweighting preserves the input marginal while redistributing probability across generated candidates.Its implementation uses product-simplex mirror descent with sketched joint features.
Limitations: Direct training-data comparisons focus on models whose relevant training data are publicly available or reconstructable.Many widely used models lack publicly accessible training data, making equivalent analysis more difficult.
Scope: The entropy-based evaluation complements assessments of factuality, alignment, and semantic correctness rather than replacing them.The paper identifies alternative feature choices and extensions to closed-source systems as natural directions.

C.3 Additional Theoretical Results on Finite-Sample Conditional Entropy

Theoretical results establish finite-sample underestimation and monotonicity for both Shannon and matrix-based conditional entropy. They also show that entropy-constrained conditional reweighting is a convex projection problem with scalable optimization structure.

Finite-sample Shannon entropy: E[H(Y | X; bPn)] ≤ H(Y | X; P), so empirical Shannon conditional entropy underestimates the population quantity.
Finite-sample Shannon entropy: E[H(Y | X; bPm)] ≤ E[H(Y | X; bPn)], establishing monotonicity as the paired-sample size increases.
Finite-sample matrix entropy: E[HvN(Y | X; bPn)] ≤ HvN(Y | X; P), and E[HvN(Y | X; bPm)] ≤ E[HvN(Y | X; bPn)] for matrix-based conditional entropy.
Convex projection: The conditional entropy superlevel set Cρ(PX) is convex, supporting projection toward higher conditional entropy under a fixed input marginal.
Scalable implementation: The implementation reweights multiple generated continuations per prompt on a product of simplices while preserving each prompt’s total weight.
Convex projection: The finite-support objective is a convex program because the MMD term is convex, constraints are linear, and matrix entropy is concave.

D.3 Evaluation Settings and Hyperparameters

The evaluation uses matched conditional representations and kernels across language, image, and decoding experiments. Reweighting and MBR settings preserve prompt-level comparability while testing quality–diversity trade-offs.

LLM evaluation: LLM evaluations separately embed prefixes and continuations with Qwen3-Embedding and use Gaussian product kernels with median-heuristic bandwidths.
Image evaluation: Image evaluations use DINOv2 for images and CLIP for condition text within the same normalized product-kernel construction.
Entropy projection: Entropy-projected reweighting assigns 10 generated candidates to each prefix and optimizes one simplex per prefix, keeping the empirical prefix marginal fixed.
Diffusion guidance: Diffusion guidance applies conditional VNE during SDXL denoising on MS-COCO with guidance scale η = 0.03 and matched caption sets.
MBR decoding: MBR comparisons use nucleus sampling with p = 0.9 across XSum and CNN/DailyMail settings and compare Monte Carlo, model-based, and conditional-Vendi weighting.
MBR decoding: CVS-MBR consistently attains the highest conditional VNE, while 95% confidence intervals for ROUGE-L and BERTScore differences contain zero in all settings.

E.1 Additional LLM Diversity Gap Results

Across OLMo, Pythia, and GPT-Neo, generated continuations have lower conditional diversity than matched training continuations. The gap varies with decoding, sample size, prefix length, and continuation length, and is not fully captured by lexical metrics.

Cross-model results: Training continuations consistently achieve higher conditional VNE and conditional RKE than generated continuations across OLMo, Pythia, and GPT-Neo.
Decoding strategies: Greedy decoding has the largest gap, nucleus sampling reduces it, and ancestral sampling is usually closest to the training reference.
Lexical versus kernel diversity: Distinct-2 and Distinct-3 are lower for greedy decoding, but conditional VNE and conditional RKE gaps remain positive even when lexical scores approach the reference.
Sample-size effects: The measured gap becomes more pronounced with larger evaluation sets because training references gain conditional VNE faster than generated groups.

E.2 Additional Image Diversity Gap Results

Class-conditioned ImageNet and text-conditioned MS-COCO experiments reproduce the conditional diversity gap seen in language models. Real-data references remain most diverse, although stronger generators approach them more closely.

ImageNet: 54.15, 49.95, 47.86, 22.47, and 17.04 are the 20K conditional VNE values for ImageNet, DiT-XL-2, LDM, ADM, and BigGAN.
ImageNet: ImageNet has the highest conditional VNE at every sample size, while DiT-XL-2 and LDM are closest among the evaluated generators.
MS-COCO: 40.52, 30.25, 29.94, 29.50, and 25.25 are the 20K conditional VNE values for MS-COCO, SDXL, U-ViT Deep S/2, U-ViT S/2, and PixArt-Σ.
MS-COCO: The MS-COCO reference has the highest conditional VNE across sample sizes, with SDXL strongest and PixArt-Σ showing the largest gap.
Cross-domain pattern: Across image experiments, real-data references grow faster with sample size than generated groups under matched labels or captions.

E.3 Additional Reweighting Results

Post-hoc entropy projection increases conditional diversity by reweighting existing generated candidates while preserving each prompt’s total mass. Conditional VNE guidance similarly raises diversity during image sampling, though guided samples remain below the reference data.

Additional Reweighting Results: At 20K samples, conditional VNE increases from 366.35 to 376.77 for OLMo, from 371.72 to 382.74 for Pythia, and from 340.90 to 353.68 for GPT-Neo.The gains come from redistributing probability mass over already generated candidates.
Additional Reweighting Results: At 20K samples, conditional RKE increases from 241.14 to 284.92 for OLMo, from 236.24 to 285.05 for Pythia, and from 246.13 to 296.02 for GPT-Neo.These improvements require neither retraining nor new generated continuations.
Additional Reweighting Results: The projection preserves each prefix’s total mass, changing only the conditional distribution over available continuations.This keeps the input marginal fixed while adjusting output probabilities.
Additional Conditional VNE Guidance Results: Conditional VNE guidance raises diversity during SDXL sampling while keeping the text condition fixed.At 20K samples, conditional VNE rises from 30.25 without guidance to 32.27 with guidance, versus 40.52 for reference MS-COCO data.

E.5 Ablation Study Analysis

Ablation studies show that the conditional diversity gap persists across embeddings, kernels, prefix and continuation lengths, chunk sizes, and larger models. The proposed projection redistributes mass toward underrepresented alternatives, while temperature scaling can match entropy without matching quality measures.

Robustness to Representations: Training data retain the highest conditional diversity, while decoding consistently orders diversity as ancestral > nucleus > greedy across embedding and kernel choices.The qualitative conclusion is unchanged despite substantial changes in absolute score scale.
Prefix-Length Ablation: With a 4-token prefix, training references still exceed all generated groups in conditional VNE and conditional RKE, although gaps are generally smaller than with 3-token prefixes.The decoding order remains greedy, then nucleus, then ancestral.
Sequence-Length Ablation: The average conditional VNE gap remains positive across tested prefix–continuation length configurations, often increasing with shorter prefixes and longer continuations.Increasing continuation length raises absolute conditional VNE for both training and generated continuations.
Chunk-Size Ablation: Across chunk sizes 1, 2, and 4, GPT-Neo preserves the ordering training > ancestral > nucleus > greedy and positive conditional VNE gaps.Changing chunk size alters absolute scale but not the qualitative long-sequence result.
Temperature Baseline: Temperature T ≈1.9 approximately matches reference entropy but yields worse precision and external conditional NLL than the training data.This contrasts entropy matching with the proposed projection’s quality-oriented evaluation.
Mass Redistribution: The projection produces a de-peaking pattern by moving mass from dominant modes toward equally valid, underrepresented alternatives.The probe evaluates 1,000 continuations for each of two prompts.
Larger Language Models: At 20K samples, OLMo-3-7B shows relative reductions of approximately 33.0%, 16.2%, and 12.0% under greedy, nucleus, and ancestral decoding, respectively.Substantial gaps also remain for Pythia-2.8B and Pythia-6.9B.