PaperlayerView exact revision on arXiv

Source-linked AI summary

Base Models Beat Aligned Models at Randomness and Creativity

Peter West, Christopher Potts

arXiv:2505.00047v2cs.CL

TL;DR

The paper asks whether alignment’s gains on common benchmarks extend to tasks requiring unpredictable outputs, where a distribution of responses matters. It examines random-number generation, mixed-strategy games, and poetry, finding that aligned models underperform base models across these tasks. The findings point to practical trade-offs in creative work and settings that require mixed strategies.

  • Problem

    Common benchmarks emphasize tasks with a single correct answer, while random generation, mixed-strategy games, and poetry require unpredictable outputs.

    Source: §1 · p. 1

  • Method

    The paper compares base models with popular alignment recipes on random-number generation, mixed-strategy games, and poetry writing.

    Source: §1 · p. 1

  • Results

    Aligned models perform worse across the tested randomness, mixed-strategy, and poetry tasks despite better performance on common benchmarks.

    Source: §1 · p. 1, §6 · p. 10

  • Takeaways & Limitations

    The findings raise concerns about aligned models’ usefulness for creative tasks and settings requiring mixed strategies, while suggesting possible safety benefits against deception.

    Source: §6 · p. 10

    Whether unpredictability has an inherent trade-off with the capabilities at which aligned models excel remains an open question.

    Source: §6 · p. 10

Abstract

from arXiv · show

Alignment has quickly become a default ingredient in LLM development, with techniques such as reinforcement learning from human feedback making models act safely, follow instructions, and perform ever-better on complex tasks. While these techniques are certainly useful, we propose that they should not be universally applied and demonstrate a range of tasks on which base language models consistently outperform their popular aligned forms. Particularly, we study tasks that require unpredictable outputs, such as random number generation, mixed strategy games (rock-paper-scissors and hide-and-seek), and creative writing. In each case, aligned models tend towards narrow behaviors that result in distinct disadvantages, for instance, preferring to generate "7" over other uniformly random numbers, becoming almost fully predictable in some game states, or prioritizing pleasant writing over creative originality. Across models tested, better performance on common benchmarks tends to correlate with worse performance on our tasks, suggesting an effective trade-off in the required capabilities.

1 Introduction

Common benchmarks may miss capabilities that require unpredictable outputs, including random number generation, mixed-strategy games, and poetry. The paper finds that popular alignment recipes can reduce performance on these tasks even as aligned models perform better on popular benchmarks.

1 Introduction

  • Popular benchmarks may not capture capabilities such as poetry writing or mixed-strategy play, which differ from the tasks they evaluate.The authors note that most users interact with public-facing aligned models, making possible capability losses consequential.Source: §1 · p. 1
  • The paper studies random number generation, mixed-strategy games, and poetry writing as tasks that require unpredictable outputs rather than one correct answer.For random number generation, the desired output is a distribution of answers.Source: §1 · p. 1
  • Aligned models show consistent performance drops across randomness, mixed-strategy games, and poetry, while base models do best on unpredictable tasks.The reported patterns include a preference for “7,” greater determinism in games, and pleasantness taking priority over creative originality.Source: §1 · p. 1, Figure 1 · p. 2
  • Higher performance on popular benchmarks tends to predict lower performance on the paper’s tasks, suggesting a possible trade-off between these skill sets.The figure compares 8B and 70B models on the Open LLM Leaderboard and unpredictable tasks.Source: §1 · p. 2, Figure 1 · p. 2

2 Basic Randomness: Number Generation

The paper tests random-number generation as a diagnostic of unpredictability, using single integers and length-10 sequences. Alignment recipes reduce randomness in these settings, though sequences can look more uniform by frequency while remaining non-random.

2.1.1 Generating Independent Random Numbers

  • The independent-number test asks models to generate uniformly sampled integers from 0 to 10, with 1,500 valid outputs collected per model.The study compares Llama-3.1 base with four alignment recipes at 8B and 70B parameters.Source: §2.1.1 · p. 2

2.1.2 Generating Random Number Sequences

  • A second setting tests whether models can account for their generation history by producing valid sequences of 10 integers in the same range.Sequences shorter than 10 or containing entries outside 0–10 are removed.Source: §2.1.2 · p. 3

2.2 Results

  • Aligned models’ single-number distributions are roughly an order of magnitude more divergent than base models, often favoring “7”; this pattern persists with temperature adjustment.The base models are more uniform across model sizes, while Llama-Instruct is the most divergent aligned model and Tulu-SFT the least.Source: §2.2 · p. 3, Figure 2 · p. 4
  • For length-10 sequences, aligned models can look more uniform in overall frequency yet avoid repeats, unlike true uniform sequences.The most common repetition count is three for base models and the true sample, but zero for aligned models.Source: §2.2 · p. 3, Table 1 · p. 4
  • In single-integer generation, 70B models have higher divergence than 8B models across the board, including base models.This result does not follow the general tendency for performance to improve with scale.Source: §2.2 · p. 4

3 Games Requiring Randomness

This section tests whether aligned models retain the unpredictability needed for mixed-strategy games, where deterministic play can be exploited. Against greedy deterministic adversaries, base models score highest across Rock Paper Scissors and Hide & Seek.

3.1 Background: Mixed Strategy Games

  • In mixed-strategy games, probabilistic play is required when deterministic opponents know move probabilities; pure strategies can be countered.Source: §3.1 · pp. 4–5

3.3 Results

  • Base models achieve the highest scores across both games and model sizes; in Hide & Seek they lead every baseline by at least 40 points.In Rock Paper Scissors, Tulu-SFT is second-best for both sizes and within 7 net points.Source: Table 2 · p. 5, §3.3 · pp. 6–7

3.2.1 Rock Paper Scissors

  • In Rock Paper Scissors, 500 ten-round games pit prompted models against greedy deterministic opponents that exploit estimated next-move probabilities.Source: §3.2.1 · pp. 5–6

3.3 Results

  • In Rock Paper Scissors, base move probabilities cluster near 0.5, while aligned models become more deterministic after ties or wins; base models lack that consistent pattern.Source: Figure 4 · p. 6, §3.3 · p. 7

3.2.2 Hide & Seek

  • Hide & Seek tests unpredictable hiding across spots: equilibrium is uniform, yielding roughly 85% wins (about 70 net points) against an adversarial seeker.Source: §3.2.2 · p. 6

4 Creative Poetry Generation

The poetry evaluation separates originality from preference: base models lead on originality, while instruct models lead on pleasantness. The authors use contest-style human judgments to examine this difference across poems and models.

4.1 Experiments

  • Models were prompted zero-shot to generate fixed-length four-line poems on everyday topics; because 70B parameter models were significantly more proficient at this task, analysis focused on these models, using the same models as earlier sections.GPT-4 supplied a list of everyday topics.Source: §4.1 · p. 7

4.2 Results

  • Base models win on creative originality in every case, but this does not translate to dominance in human preference.Figure 5 also shows that base poems are not always preferred or pleasant.Source: Figure 5 · p. 8, §4.2 · p. 8

4.1 Experiments

  • Human judges compared randomized poem pairs separately for originality, pleasantness, and preference; Bradley–Terry rankings aggregated disagreements.Agreement varied across axes: Cohen’s κ was 0.33 for originality, 0.27 for pleasantness, and 0.67 for preference.Source: §4.1 · p. 8
  • The evaluation tested Tulu-Full, Llama-Instruct, and the base model at 70B across four prompts, with five poems per model and 720 annotated comparisons.Each contest compared 15 poems; the prompts were coffee, sleep, weather, and smartphones.Source: §4.1 · p. 8

4.2 Results

  • Mean rank correlation with preference is ρ = 0.34 for pleasantness and ρ = −0.08 for originality.The authors suggest preference may not clearly signal artistic originality or impact.Source: §4.2 · p. 9
  • 10.5 was the base model’s lowest median human-preference rank among 15 poems, despite one preference-contest win and no pleasantness wins.The authors note that opinions of base-model poems were highly variable.Source: §4.2 · p. 9

5 Related Work

Related work covers alignment’s effects on model capabilities and diversity, creativity evaluation, randomness, and human biases. The authors distinguish their focus on alignment comparisons and unpredictability across tasks from these adjacent studies.

Related work

  • Prior work compares base and aligned models and studies ways to encourage alignment; Li et al. (2024) also report alignment narrowing capabilities beyond task-based improvement.Other work examines diversity loss, while Shypula et al. (2025) find higher semantic but lower surface diversity for aligned models.Source: §5 · p. 9
  • Creativity studies measure text matching or use human judgments; this work follows human evaluation and focuses on direct model creativity, also relevant to human–model collaboration.The cited collaboration setting includes writing with humans Padmakumar & He (2024); Chakrabarty et al. (2024); Anderson et al. (2024).Source: §5 · p. 9
  • Earlier work studies random-number generation, demographic sampling, and games, but does not aim to examine alignment’s effects on randomness.Koevering & Kleinberg (2024) report repetition avoidance, while Bigelow et al. (2024) find transitions from randomness to formal languages in some settings.Source: §5 · p. 9
  • Related studies examine model biases associated with human cognition, including biases and opinions across varied settings.The authors identify this as related to their study of randomness and mixed-strategy games, which have strong human biases.Source: §5 · p. 9
  • The experiments use integer sampling, unlike prior random-number work on binary or continuous distributions, and focus on aligned-versus-base comparisons in games.The cited game studies do not focus on that model divide.Source: §5 · p. 9

6 Conclusion

The paper reports lower performance by aligned models across the tested tasks despite their stronger common-benchmark performance. It connects these findings to potential uses and trade-offs involving unpredictability, while leaving the nature of that trade-off unresolved.

6 Conclusion

  • Aligned models show lower performance across the tested tasks despite better performance on common benchmarks.The conclusion describes the findings as support for alignment recipes eroding capabilities present in base models.Source: §6 · p. 10
  • The findings raise possible implications for creative assistance and mixed-strategy settings such as bargaining, alongside a possible safety benefit from reduced deception ability.The authors qualify these as potential implications of the tested tasks and argue that safe deployment remains important.Source: §6 · p. 10
  • Whether unpredictability inherently trades off against capabilities at which aligned models excel remains an open question.The authors say resolving it could shed light on underlying mechanisms of model capabilities.Source: §6 · p. 10

Ethics Statement

The authors report that they analyze existing models without training new models or introducing datasets. They state that they advocate only for safe public deployment, not broad release of large, untested base models.

Ethics Statement

  • The work analyzes existing language models and introduces neither newly trained models nor new datasets.The authors also report following IRB guidelines in human evaluations and aiming to pay workers $15 per hour on average.Source: Ethics Statement · p. 10
  • The authors advocate safe public deployment only, not broad release of large, untested base models.They distinguish this from their argument that alignment may erode useful capabilities in base parameters.Source: Ethics Statement · p. 10

A.2 Overall

The section says the suggested prompt formatting was used for each model.

A.2 Overall

  • The experiments use each model’s suggested prompt formatting.The passage does not provide further details about the formatting.Source: §A.2 · p. 17

Tulu

This material describes prompt and inference setup alongside additional randomness experiments. It reports that aligned models remain less uniform than base models under entropy matching and tested decoding changes.

Tulu

  • Tulu prompts may include an infix reflecting its typical introductory text, while base-model instructions may need contextual phrasing rather than direct commands.The setup also describes inference with SGLang, using full precision for 8B models and fp8 for 70B models.Source: Tulu · p. 17

A.2.1 Games Requiring Randomness

  • For games, move probabilities are approximated from next-token probabilities for move-starting tokens, while actual moves are parsed from generated text.The passage says the parsed moves better indicate model behavior but do not permit consistent probability estimation.Source: §A.2.1 · p. 17

A.3.1 Random Number Generation

  • Across a grid over top-p and entropy, no tested decoding setting lets aligned models reach the base model’s default divergence of 137, with temperature and top-p both 1.0, in single-number generation.Higher entropy pushes more probability outside the valid output space rather than smoothing the distribution.Source: §A.3.1 · p. 18
  • Qwen2.5 shows the same larger divergence for aligned models, with no significant improvement from scale; generated-letter biases vary by model, recipe, and scale.Tulu models most often choose “j” at 8B and “f” at 72B.Source: §A.3.1 · p. 18, Table 5 · p. 19, Table 6 · p. 20
  • Across five paraphrased prompts, divergence varies, but prompt wording does not bring aligned models up to base-model performance.The prompts are compared for single random integer generation.Source: §A.3.1 · p. 18, Table 3 · p. 18
  • The Tulu reward model gives “7” the highest reward by a small margin, although rewards are similar for many integers.The passage notes that reward maximization could favor generating only “7.”Source: §A.3.1 · p. 18, Table 7 · p. 20
  • For sequential integer generation, counts become more uniform, but aligned models still follow heuristics such as avoiding repeats.Llama Instruct, Tulu DPO, and Tulu Full also show strong position-dependent structure, unlike Base and SFT.Source: §A.3.1 · p. 18, Figure 6 · p. 20

A.3.2 Poem Examples

Loading 2505.00047v2…