PaperlayerView exact revision on arXiv

Source-linked AI summary

Forcing Diffuse Distributions out of Language Models

Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter, Daphne Ippolito

arXiv:2404.10859v2cs.CLcs.LG

TL;DR

Instruction-tuned language models often produce low-entropy outputs despite requests for randomness, limiting applications that depend on diverse valid choices. The paper fine-tunes models to produce diffuse distributions, reporting transferable diversity gains in synthetic biographies without complex prompt engineering or manual rewriting.

  • Problem

    Instruction-tuned models often produce low-entropy distributions over valid options despite prompts encouraging randomness, while diversity is important for tasks such as synthetic dataset construction.

    Source: §1 · p. 1

  • Method

    The authors fine-tune language models to produce diffuse probabilities over valid outputs, using small samples as a proxy when the ground-truth distribution is unknown.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Fine-tuning yields four times as many unique first names as the untuned baseline in synthetic biographies, alongside reported gains in birth-place and career diversity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method generalizes across tasks and supports more diverse synthetic dataset generation without complex prompt engineering, decoding changes, or manual rewriting.

    Source: §1 · p. 2

    The method assumes a structured output space with accessible valid output sequences, so it is not directly applicable to open-ended generation with practically infinite targets.

    Source: §8 · p. 10

Abstract

from arXiv · show

Despite being trained specifically to follow user instructions, today's instructiontuned language models perform poorly when instructed to produce random outputs. For example, when prompted to pick a number uniformly between one and ten Llama-2-13B-chat disproportionately favors the number five, and when tasked with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times more often than we would expect based on the U.S. population. When these language models are used for real-world tasks where diversity of outputs is crucial, such as language model assisted dataset construction, their inability to produce diffuse distributions over valid choices is a major hurdle. In this work, we propose a fine-tuning method that encourages language models to output distributions that are diffuse over valid outcomes. The methods we introduce generalize across a variety of tasks and distributions and make large language models practical for synthetic dataset generation with little human intervention.

1 Introduction

Instruction-tuned models can remain highly biased even when asked to produce random outputs, limiting tasks that depend on diversity across valid choices. The paper proposes fine-tuning to diffuse output distributions and reports gains in synthetic biography diversity.

1 Introduction

  • Examples include repeated suggestions of “Anya” nearly 40% of the time and twenty-sided die rolls of 14 over 60% of the time.Source: §1 · p. 1
  • Instruction-tuned models often produce low-entropy distributions over valid options despite prompts encouraging randomness.The paper identifies multiple-choice and Likert-scale answers and synthetic dataset construction as settings where diversity matters.Source: §1 · p. 1
  • The authors examine deviations from expected randomness and report that findings extend beyond instruction-tuned models to pretrained language models.They propose fine-tuning that diffuses probabilities over valid options without sacrificing generation quality and generalizes across tasks.Source: §1 · pp. 1–2
  • The method produces four times as many unique first names, three times as many unique birth places, and 1.5 times as many unique careers as the untuned baseline in synthetic biographies.The gains required no complex prompt engineering, decoding-strategy tweaking, or manual rewriting.Source: §1 · p. 2

2 Preliminaries

The preliminaries define sequence probabilities for autoregressive models and describe how to measure output distributions. They distinguish enumerable target spaces, assessed against a ground truth, from complex settings where empirical diversity measures are used.

2.1 Problem setting

  • An autoregressive model assigns a probability to a multi-token target by multiplying the conditional probabilities of its successive tokens.The input is a token sequence, and each token probability conditions on the input and previously generated tokens.Source: §2.1 · pp. 2–3

2.2 Measuring output distributions

  • For enumerable targets, output distributions can be compared with a ground-truth distribution using KL-divergence, which is zero exactly when the distributions match.A uniform distribution is the expected target for a fair twenty-sided die.Source: §2.2 · p. 3
  • When the ground-truth support cannot feasibly be enumerated, empirical entropy and Coverage-N measure diversity from sampled generations.Coverage-N counts unique outputs in N samples, indicating how many distinct generations a user could practically observe when N is sufficiently large.Source: §2.2 · p. 3

3 Forcing Diffuse Probabilities

The paper fine-tunes language models to match a desired distribution over target sequences, with the aim of encouraging diffuse outputs that generalize beyond training tasks. It describes a prefix-free target-set condition and uses LoRA for parameter-efficient tuning.

3 Forcing Diffuse Probabilities

  • Fine-tuning for diffuse probabilities on one task can generalize to tasks unseen during training.Source: §3 · p. 3

3.1 Distribution matching as language model fine-tuning

  • When valid targets and their ground-truth probabilities are known, the method maximizes model log-likelihood across targets, weighted by their ground-truth likelihoods.This is cross-entropy extended to multiple target sequences and is equivalent to minimizing KL-divergence from the model distribution to the target distribution.Source: §3.1 · pp. 3–4
  • The target set must be prefix-free; when the ground truth is unknown, the method uses about 200 samples as an empirical proxy.The authors report that minimizing the loss to zero did not empirically cause the model to produce only sampled targets.Source: §3.1 · p. 4

3.2 Parameter-efficient Fine-tuning

  • LoRA fine-tuning optimizes low-rank additions to model weight matrices instead of updating the original weights.The authors report that this makes tuning models with at least 7B parameters practical with only a small fraction of the memory overhead.Source: §3.2 · p. 4

4 Results on Simple Tasks

Untuned models produce imbalanced outputs for simple name and number tasks, while fine-tuning aligns outputs with target distributions and generalizes across prompt formats and number ranges.

4.1 Language models fail to follow instructions for randomness

  • 40% probability went to Anya in Gemma, while Gemma and Llama-2 put over 60% on five and Mistral over 25% on eight; none covered all ten numbers.The untuned name and number distributions were strongly imbalanced despite prompts requesting randomness.Source: §4.1 · p. 5

4.2 Optimizing for BABYNAMES and RNG diversifies generation

  • After tuning on ranges 1–10 and 1–100, Llama-2 remained near-uniform on unseen prompt formats and ranges such as 1–45, and more entropic for ranges 154–204.Figure 2 presents generalization across prompt formats and number ranges.Source: Figure 2 · p. 6, §4.2 · p. 6
  • RNG tuning targeted a uniform distribution over numbers 1–10, while BABYNAMES tuning fit the empirical distribution of a 200-name sample.Source: §4.2 · p. 6
  • RNG tuning yielded near-uniform choices with full coverage and near-perfect entropy; BABYNAMES coverage exceeded twice baseline, with most outputs absent from fine-tuning targets.For BABYNAMES, the fine-tuning set covered only a small portion of possible outputs.Source: §4.2 · p. 6

5 Generalization Across Tasks

Leave-one-out experiments find that tuning on diffuse-probability tasks raises entropy on held-out tasks across three models. Task-specific gaps remain in some settings, while general model capabilities are largely preserved.

5 Generalization Across Tasks

  • Tuning substantially improved entropy for all three models on both included and held-out tasks, bringing their tuned entropies to comparable levels.Baseline Mistral was more diffuse than baseline Gemma and Llama-2; Figure 3 reports the in-distribution and out-of-distribution results.Source: Figure 3 · p. 7, §5 · p. 7
  • Leave-one-out tests tuned on five of six tasks and evaluated on the remaining task to measure transfer to tasks unseen during tuning.The six tasks were BABYNAMES, COUNTRIES, FRUITS, DAYS AND DATES, NUMBERS, and OCCUPATIONS.Source: §5 · p. 7
  • Countries and fruits showed sizable in-distribution versus out-of-distribution entropy gaps, but tuning did not substantially change general writing and reasoning capabilities.For occupations, Llama-2 and Mistral generalized beyond the 17 provided professions.Source: §5 · pp. 7–8

6 Constructing More Diverse Synthetic Datasets

Fine-tuning on other tasks improves synthetic biography diversity, and tuning on categorical attributes can balance those attributes while preserving diverse achievements. The results show both the value and remaining limits of transferring diffuse-output training to dataset generation.

6 Constructing More Diverse Synthetic Datasets

  • Leave-one-out experiments indicate that diffuse-output fine-tuning generalizes across tasks, which matters when synthetic data cannot be tuned on identically formatted examples.Source: §6 · p. 8

6.1 Out-of-distribution training improves synthetic dataset diversity

  • The out-of-domain biography evaluation compares 1,000 Tuned-OOD generations with baseline Llama-2 and measures categorical coverage and unigram diversity.The biographies include names, gender, birth year, birthplace, profession, and achievement descriptions.Source: §6.1 · p. 8, §6 · p. 8
  • More than 2X coverage gains for most categorical attributes accompany substantial improvements in achievement and whole-biography diversity after out-of-domain tuning.The tuned model was not trained specifically for biography generation; Table 2 reports frequent attribute values out of 1,000 generations.Source: §6.1 · pp. 8–9, Table 2 · p. 9
  • The frequency of “Evelyn” fell by 25X and that of birth year 1985 by over 3X, though female biographies remained 76.2% and 1985 births 21.1%.The remaining skews could limit dataset utility.Source: §6.1 · p. 9, §6.2 · p. 9

6.2 Diversifying distributions of categorical attributes

  • At comparable generation diversity, categorical-only tuning made gender and birth years roughly uniform, widened birthplace coverage, and preserved diverse open-ended achievements.The Tuned-ID model was trained on categorical attributes and evaluated on another 1,000 biographies.Source: §6.2 · p. 9

7 Related Work

Prior text-generation work addresses a quality-diversity tension through decoding strategies and training methods. Language-model dataset creation has also relied on substantial human intervention to ensure diverse, high-quality data.

Diversity in Text Generation

  • Prior work addresses the quality-diversity tension through decoding methods and diversity-promoting constraints during decoding or training.The paper distinguishes its setting by assuming temperature-one sampling while instruction-tuned models still fail to generate diversely.Source: §7 · p. 10

Language Model for Dataset Creation

  • Language-model dataset creation has targeted specialized NLP data, including semantic similarity, relation extraction, understanding, and instruction following.Source: §7 · p. 10
  • Dataset creation pipelines are often iterative and arduous, requiring substantial human intervention to ensure diverse, high-quality outputs.This work aims to reduce that intervention by improving generation diversity.Source: §7 · p. 10

8 Concluding Discussion and Future Directions

The paper proposes fine-tuning for diffuse output distributions and reports transferable gains in generation diversity, including synthetic dataset generation. Its approach assumes a structured output space, leaving open-ended generation as a future direction.

8 Concluding Discussion and Future Directions

  • Fine-tuning for diffuse distributions yields sizable, transferable improvements in generation diversity and improves synthetic data with or without task-specific tuning.Source: §8 · p. 10
  • The authors suggest exploring distribution matching against representative-instance distributions as a possible route to reducing model bias.They present this as plausible future work, not a demonstrated result.Source: §8 · p. 10
  • The method requires valid output sequences and therefore assumes a structured output space; it is not directly applicable when valid targets are practically infinite.The paper identifies diversity in open-ended generation as an area for future work.Source: §8 · p. 10

A Training details

Experiments use LoRA fine-tuning for up to 50 steps, with early stopping for tasks whose ground-truth distribution cannot be enumerated.

A Training details

  • All experiments use LoRA for up to 50 steps; for non-enumerable ground-truth tasks, training stops when loss is within 20% of its minimum.The experiments use AdamW at a learning rate of 5 × 10^-5 and batch size 32, totaling approximately 50 GPU hours on one NVIDIA A100.Source: A Training details · p. 14

B Prompts and Qualitative Results

This appendix documents the random-number prompts used for Figure 2 and the synthetic-biography prompt and sample outputs used for qualitative comparison.

B.1 Prompts used for Figures 2

  • Figure 2a averages entropy over five prompts, with low fixed at 1 and high set by the x-axis.Source: §B.1 · p. 14
  • The five Figure 2a prompts ask for an integer from low to high and specify curly-brace output, with some wording variations.Source: §B.1 · p. 14
  • Figure 2b averages entropy over five versions of Prompt 1, with low randomly chosen from 1–900 and high set by the x-axis.Source: §B.1 · p. 14

B.2 Prompt used for synthetic biography generation

  • Table 3 provides the prompt used for synthetic biography generation.Source: Table 3 · p. 15

B.3 Side-by-side comparison of generated biographies across systems

  • In five examples, baseline biographies repeat birth dates, places, and professions and use similar accomplishment descriptions, while fine-tuned biographies are more diverse.Source: §B.3 · p. 16, Table 4 · p. 16
  • Table 4 presents five sampled biographies each from baseline and fine-tuned Llama-2 models.Source: Table 4 · p. 16

C.1 Leave-one-out generalization results

The supplied results cover leave-one-out coverage and entropy, synthetic-biography metrics, birth-place diversity, and MT-Bench utility.

C.1 Leave-one-out generalization results

  • Tables 5 and 6 report mean coverage and mean entropy, respectively, with higher values marked as better and better model-task results bolded.Source: Table 5 · p. 17, Table 6 · p. 17

C.2 Coverage and entropy of generated biographies

C.3 Maps of birth places in generated biographies

  • Fine-tuned models generate more diverse birth places; 500 plotted biographies show fewer baseline points because duplicate cities overlap.Source: Figure 5 · p. 19, §C.3 · p. 19

C.4 MT-Bench

  • MT-Bench results show minimal differences between baseline and fine-tuned models, suggesting no significant impact on general capabilities.Source: §C.4 · p. 20, Figure 6 · p. 20

Related papers

Loading 2404.10859v2…