Source-linked AI summary
Is the Judge Its Twin? Post-training makes a model's writing more predictable but barely moves its taste, as a judge, toward predictable writing
Arman Nik Khah, Arvin Bahreini
TL;DR
The paper asks whether post-training that narrows a model’s writing also shifts its judging toward predictable text, and tests both roles across stages of two model families. Writing changes substantially, but judges’ preference for predictability barely moves; training is instead associated with other judging habits and a loss of stable meaning for “creative.”
Problem
Earlier work found model judges favor predictable or AI-typical text, but had not tracked one family’s writing and judging together through training stages.
Source: §1 · p. 1Method
The study follows OLMo-2 and Zephyr checkpoints as short-story writers and judges of story pairs, modeling picks against predictability and length.
Source: §4 · p. 5Results
No trained judge’s estimated preference for predictable stories grows by as much as one point of pick probability; the one-sided 95% upper bound is about 5 points on the widest pairs.
Source: §1 · p. 2, §10 · p. 14Takeaways & Limitations
For these two families, the practical judging hazards are measurable habits such as sensitivity to length and display order, and a creativity criterion that trained judges no longer apply reliably.
Source: §9 · p. 12, §10 · p. 14Pairs cross training stages, confounding preference direction with stage, and the post hoc drift bound is based on ten prompt clusters that may understate uncertainty.
Source: Limitations · p. 13
Abstract
from arXiv · showhide
Language models now routinely grade other language models. If post-training makes a model's writing more predictable, it may also teach the same model, as a judge, to reward predictable writing. We test this directly. We follow two open 7-billion-parameter model families, OLMo-2 and Zephyr, from their base models through supervised fine-tuning (SFT) and direct preference optimization (DPO), and measure every stage as a writer of short stories and as a judge of story pairs. As writers, the models drift as feared. Each family's trained model finds its stages' stories more familiar in the order base, SFT, DPO, in all ten prompts, and a model from the other family finds the trained stories 7.0 to 9.2 percent (OLMo-2) and 24 to 25 percent (Zephyr) less surprising per token. As judges, they barely move toward predictable writing. Asked which story is better, a question every judge passes a competence check on, no trained judge's preference for the more predictable story grows by as much as one point of pick probability at its estimate. A one-sided 95% upper bound on that growth is about 5 points on the widest pairs. Training instead makes judges more swayed by story length and by display order, and it breaks "more creative" as a question. Trained judges no longer reliably prefer a story to its own words in scrambled order. For these two families, the practical hazard in a small trained judge of creative writing lies in these measurable habits.
1. Introduction
The introduction asks whether post-training that narrows a writer also shifts a same-family judge toward predictable writing, a gap not addressed by studies of finished models. Across two model families, writing changes more reliably than judges’ preferences for predictability, while training disrupts the meaning of “creative.”
1. Introduction
- Earlier work finds judges favor predictable or AI-typical text, but no study had followed one model family’s writing and judging stage by stage.Public training checkpoints make that direct comparison possible.Source: §1 · p. 2
- The study follows OLMo-2 and Zephyr checkpoints from base through their available SFT and DPO stages, testing each as a short-story writer and as a judge of the other family’s stories.OLMo-2 also includes a final reinforcement-learning stage.Source: §1 · p. 2
- Training makes stories more familiar in the order base, SFT, DPO across all ten prompts, and other-family readers find trained stories less surprising per token.The reported reductions are 7.0 to 9.2 percent for OLMo-2 and 24 to 25 percent for Zephyr.Source: §1 · p. 2
- No trained judge’s preference for the more predictable story grows by as much as one point of pick probability at its estimate; the one-sided 95% upper bound is about 5 points on the widest pairs.The paper reports no detected predicted drift and claims no direction for the two families’ creativity-specific estimates.Source: §1 · p. 2
- Trained judges fail a basic creativity competence check that both base models pass, and some stop distinguishing stories from scrambled text.Zephyr’s SFT judge prefers scrambled words over the story itself 87% of the time; every stage passes when asked which story is better.Source: §1 · p. 2
- A follow-up could not remove the cross-stage pair confound because Zephyr’s less predictable stories often failed a prespecified quality floor.The authors note that this left no clean contrast.Source: §1 · p. 2
2. Background
Prior studies identify several habits of model judges and post-training-related narrowing of writing, but mostly examine finished models. Predictability is not equivalent to quality, which complicates tests of judges’ taste for novelty.
2. Background
- Studies report that judges favor answer positions, longer answers, their own writing, predictable text, AI-typical style, and generators related by training lineage; none follows a judge through its own training stages.Length-controlled scoring was developed to correct length preference.Source: §2 · p. 3
- Post-training narrows writing, with prior work tracing diversity loss partly to human preference data and identifying supervised fine-tuning as a major site.This paper measures typicality as predictability under a reader model on both the writer and judge sides.Source: §2 · p. 3
- Low predictability is not itself a defect: improbable sampling can be incoherent, text quality falls at both likelihood extremes, and creativity combines novelty with value.Model judges of creative writing also agree poorly with professional writers.Source: §2 · p. 3
3. Measuring how predictable a story is
The paper measures a story’s predictability using a reader model’s average token-level surprise. It frames the prompt to avoid a short-text cold-start artifact and compares scores from trained and base readers.
3. Measuring how predictable a story is
- A story’s typicality is measured by mean negative log-likelihood: average token surprise in nats, with lower scores indicating more predictable text.The reader model assigns probabilities to successive tokens in the story.Source: §3 · p. 3
- Scoring a story without its prompt makes short texts look unpredictable because early tokens incur cold-start cost; the paper therefore scores story tokens with the writing prompt prefixed.The prompt-framed scoring amendment was recorded before re-scoring, and the stated expectations were met.Source: §3 · p. 3
- The stranger’s view tests predictability using the other family’s base model, which shares no training with the writer.The paper also uses the score to describe writers and within the judge analysis.Source: §3 · pp. 3–4
- The family-level view subtracts the family’s base-model score from its fully trained model’s score for the same story, so a negative difference means the trained model finds it easier to predict.Because the same text is scored by both readers, a fixed reader offset cancels when stages are compared.Source: §3 · p. 4
4. Models, stories and judges
The study generates short stories at successive checkpoints and evaluates story-pair choices from each checkpoint as a judge. It checks task competence, measures preferences with a length-adjusted model, and accounts for prompt-level clustering.
4. Models, stories and judges
- Seven checkpoints generated 1,389 stories from ten flash-fiction prompts, using temperature 1.0, top-p 0.95, and a 220-token cap.Eleven empty OLMo-2 generations were dropped; Zephyr’s DPO checkpoint is also its final stage.Source: §4 · p. 4
- The judge prompt asks for the more creative response or substitutes “better,” but the executed substitution reads “The more better response,” which the paper re-checks in its competence analysis.Each judge is scored through next-token probabilities over the A/B answer choices rather than being asked to generate an answer.Source: §4 · p. 4
- Before interpreting taste, the study gates judges on whether they prefer intact stories to shuffled copies in at least 80% of pairs with a Wilson interval lower bound above 50%.The shuffled-text competence check was fixed before it ran; stub and off-prompt pairs were also included in the check.Source: §4 · pp. 4–5
- Judges compare 200 pairs of intact stories from the other family, and their A/B probabilities are averaged across both display orders to remove fixed slot preference from the measurement.The order preference itself remains a judge habit assessed separately.Source: §4 · p. 4
- Story pairs always cross training stages, so a judge’s preference direction is confounded with stage.The study therefore describes the estimated drift as not detected and reports a post hoc bound on its size.Source: §4 · p. 5
- The fitted pick probability uses the difference in surprise score and difference in log length; a negative surprise coefficient indicates preference for the more predictable story.The paper also fits versions controlling for the judge’s “better” probability and using the creativity-minus-better contrast, with prompt-level bootstrap intervals.Source: §4 · p. 5
5. The writer drifts
Across both families, post-training made stories more familiar to the family’s trained model and less surprising to a model from the other family. The stage ordering held across all ten prompts, though endpoint self-scoring partly affects the family-specific comparison.
5. The writer drifts
- From base to DPO, surprise fell by 1.000 nats per token for OLMo-2 (95% interval −1.06 to −0.95) and 0.569 for Zephyr (−0.59 to −0.55).Within both families, base, SFT, then DPO ordered from most foreign to most familiar in all ten prompts; OLMo-2’s final stage was indistinguishable from DPO.Source: §5 · p. 5
- The family-specific ordering is partly qualified by endpoint self-scoring, and Zephyr has only one middle stage because DPO is also its final stage.OLMo-2’s SFT and DPO stories, written by neither scorer, still fall in order; Zephyr’s ordering leans more on the endpoints.Source: §5 · p. 6
- 0.692 nats per token for OLMo-2 and 0.261 for Zephyr was the family-specific spread between the least and most familiar stages.The other family’s trained model agreed story by story with the family’s model (correlations 0.68 on OLMo-2 stories and 0.80 on Zephyr’s), so the analysis subtracts its view to isolate family-specific change.Source: §5 · p. 6
- 24.9% and 24.0%: OLMo-2’s base model found Zephyr’s SFT and DPO stories less surprising per token; OLMo-2’s own drift was 9.2%, 7.0% and 7.7% at SFT, DPO and final.The Zephyr drops appeared in all ten prompts; OLMo-2’s appeared in 9, 9 and 8 prompts, respectively.Source: §5 · p. 6
- At DPO, the stranger’s view softened slightly while each family’s trained model continued to find later-stage stories more familiar.Stranger-view changes went from −9.2% to −7.0% for OLMo-2 and −24.9% to −24.0% for Zephyr; family-view scores continued from −0.121 to −0.567 and −0.139 to −0.325, respectively.Source: §5 · p. 6
6. The judge barely moves toward predictable writing
On the competence-checked “better” question, judges showed no consistent training-driven move toward predictable stories, and the estimated changes were small. The paper also examines “creative” judgments, where competence checks and stage confounding limit what the slopes establish.
6. The judge barely moves toward predictable writing
- OLMo-2’s “better” slopes moved from −.057 at base to −.048, −.064 and −.069; Zephyr’s were +.024, +.023 and +.076, with only OLMo-2’s base interval excluding zero.Negative slopes indicate preference for the more predictable story; Zephyr’s DPO estimate pointed away from it, so the stage patterns do not show a consistent trained-judge drift.Source: Table 2 · p. 7, §6 · p. 7
- At the average judged pair, the largest one-sided 95% upper bound on movement toward the predictable story was 2.7 pick-probability points; at p90 it was 4.9 for OLMo-2 and 4.3 for Zephyr.These bounds quantify possible movement, not observed growth; they cover movement toward the predictable story only.Source: §6 · p. 7
- Every trained judge failed the “creative” competence check, unlike both base judges, although picks on real story pairs varied within the preregistered limits.Zephyr’s SFT judge’s scrambled-text failure therefore does not by itself describe how it picked between real stories; the paper treats “creative” as a secondary reading.Source: §6 · pp. 7–8
- A stage-difference control on long stories could flip slope signs, so the design cannot separate preference for less predictable writing from preference for later-stage writing.The control used only stories at least 400 characters; the authors therefore claim no slope sign, only that the predicted drift was not detected and bound how large it could be.Source: §6 · pp. 8–9
7. What training does to a judge instead
Training does not make judges reliably prefer more predictable stories; instead, it undermines their ability to judge creativity and strengthens length and display-order effects.
7. What training does to a judge instead
- Every trained stage fails the shuffled-story competence check under “creative,” while both base models pass; Zephyr’s SFT judge prefers scrambled text 87% of the time.The other trained judges mostly default to a display slot rather than discriminate.Source: §7 · p. 9, Table 3 · p. 9
- Every stage passes the competence check under “better,” so the creative-task failure is not a general inability to distinguish intact stories from scrambled text.Changing the criterion, rather than the chat template, restores SFT judges’ competence.Source: §7 · p. 9
8. Manipulating predictability degrades quality
The attempted same-writer test could not separate predictability from story quality: routes that lowered predictability often damaged stories, while asking for improbability did not shift predictability. Thus, the judge question remains unanswered by these pairs.
8. Manipulating predictability degrades quality
- The experiment used Zephyr’s final model and twelve prompts, pairing finished stories with length ratios no greater than 1.25.Pairs had to pass an outside-reader manipulation check and a blind-read quality floor before any judge could be tested.Source: §8 · p. 10
- Heating and selecting lowered predictability, but each produced at least one broken story and the raters preferred the predictable story in 6 of 8 pairs.The less-predictable stories did not reliably hold together, so the pairs failed the quality floor needed to isolate judges’ taste.Source: §8 · pp. 10, 12
- Asking the writer for a story it would give under 10 percent probability did not reduce predictability; no judge was run because quality and predictability did not come apart cleanly.The question requires less-predictable stories that reliably hold together, potentially from a stronger writer or human-written stories with quality ratings.Source: §8 · pp. 11–12
- At temperature 1.3, the predictability gap was large and consistent, but the blind-read rule failed because one heated story scored 2 against a median of 3.5.That story contradicted itself, and raters preferred the ordinary story as both better and more creative in all four pairs.Source: §8 · pp. 10–11
- Selecting story pairs by predictability on half their tokens yielded a held-out gap of +0.176 nats per token on the other half, positive for 7 of 11 prompts.Across both reads, three of eight less-predictable stories scored 2 or below, all with logic breaks, versus one of eight more-predictable stories; raters chose the more-predictable story as better and more creative in 6 of 8 pairs.Source: §8 · pp. 11–12
9. Discussion
For these two model families, training barely shifts judges toward predictable writing, while making them more susceptible to measurable habits and weakening “more creative” as a criterion. The follow-up also shows that lower predictability can coincide with broken stories, so novelty requires an independent quality check.
9. Discussion
- For these two families, the practical risk is measurable judging habits and a criterion word that no longer means what the user intends, not a hidden taste for predictability.Training makes judges favor longer stories and a display slot, and makes “more creative” unstable.Source: §9 · p. 12
- Both routes that lowered predictability in the follow-up also produced broken stories often enough to fail a preset quality floor.A study manipulating predictability should independently establish that its less predictable items remain good, or judges may be penalizing incoherence rather than unpredictability.Source: §9 · p. 12
Limitations
The study’s claims are bounded by its two model families, prompt-level sample, and judging setup. Several design choices also limit how precisely its results distinguish predictability from other factors or generalize to human judgments.
Limitations
- The study covers two 7B families at one size; a planned third family, Tülu-3, was not run.The main study used ten flash-fiction prompts and 200 judged pairs per judge.Source: Limitations · p. 13
- With only ten prompt clusters, percentile-bootstrap intervals may understate uncertainty, making the post hoc drift bound potentially optimistic.The prompt, not each judged pair, is the unit of inference.Source: Limitations · p. 13
- Stories were cut at 220 tokens without recording why, and unfinished endings confound story length with judges’ length preference.The study did not separate the effects of truncation and length preference.Source: Limitations · p. 13
- All main-study “better” results used the ungrammatical prompt “The more better response is Response”; only the two SFT judges were rerun with grammatical wording.Those two judges passed the competence check with the grammatical version, but the other stages and main measurement were not rerun.Source: Limitations · p. 13
- No judge read its own family’s stories, and the main study had no human creativity judgments, so it does not test self-preference or human-valued creativity.“Better” reflects the judge’s own quality axis, which could be polish rather than value.Source: Limitations · p. 13
- The follow-up used one writer, two author-raters, eight pairs, and half-point scores, with no claimed statistical test.Its localization check covers three stories, and no inter-rater agreement is on record.Source: Limitations · p. 13
10. Conclusion
Across OLMo-2 and Zephyr, writing grows more predictable with training, while judges’ preference for predictable stories barely shifts. Training instead increases other judging habits and makes “creative” unreliable as a question.
10. Conclusion
- For OLMo-2 and Zephyr, every trained stage writes more predictably than its base, and each family comes to find its own writing familiar.The conclusion cautions that pushing Zephyr toward less predictable writing broke too many stories to pass a quality floor.Source: §10 · pp. 13–14
- No trained judge’s preference for predictable stories grows by as much as one point at its estimate, and the one-sided 95% upper bound is about 5 points on the widest pairs.Training instead makes judges more swayed by length and position, while trained judges asked about creativity no longer reliably prefer stories to their own scrambled words.Source: §10 · pp. 13–14
- The cleaner test of whether judges favor less predictable writing requires less predictable stories that reliably remain coherent.Lower predictability alone can reflect broken stories rather than novelty.Source: §10 · p. 14
Appendix A. Where the reader’s surprise sits
The appendix examines where surprise falls within three broken stories, and the accompanying sections report AI assistance and provide study materials.
Appendix A. Where the reader’s surprise sits
- In three logic-broken stories, only one of seven marked break sentences fell in the most-surprising quarter, suggesting surprise was diffuse rather than concentrated at the breaks.Four of the seven break sentences were above their story’s median; openings and odd premises were among the most surprising, but openings have the least context.Source: Appendix A · p. 14
AI use
- AI tools assisted with code development and improving text clarity, while the authors conceived, designed, and directed the study.The first author conducted the experiments and analyses; both authors approved the final manuscript and take responsibility for it.Source: AI use · p. 14
Code and data
- The code, generated stories, predictability scores, judge outputs, and dated design documents are publicly available.The passage provides the project’s GitHub repository.Source: Code and data · p. 14