Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures

1 Introduction

• On objective-focused RewardBench tasks, reward models reach 95% accuracy, but sequence-based models fall to 52.7% when objective quality signals are removed.The authors characterize this as a 42-percentage-point degradation and argue that current preference learning primarily optimizes error detection rather than subjective creative quality.Source: §1 · p. 1
• Existing evaluation resources mix preference with safety, functional writing, or proxy signals, while cross-lingual evaluation—particularly for Chinese rhetorical traditions—remains largely unexplored.Writing tasks account for over 40% of language model interactions.Source: §1 · pp. 1–2
• WritingPreferenceBench contains 1,800 English and Chinese preference pairs with responses matched for grammatical correctness, factual accuracy, and length.It focuses on creativity, stylistic sophistication, and emotional resonance.Source: §1 · p. 2
• 81.8% accuracy for generative reward models with reasoning chains contrasts with 52.7% for sequence-based reward models and 53.9% for zero-shot judges.Across evaluated models, genre instability remains high, and scaling or reasoning-enhanced general-purpose judges provides no consistent advantage.Source: §1 · p. 2
• The paper presents the benchmark as enabling reproducible cross-lingual evaluation through 1,800 validated pairs with systematic signal isolation.Its stated empirical contribution is that reasoning-based generative reward models perform better, while zero-shot judges cannot reliably assess creative quality.Source: §1 · pp. 2–3

2 WritingPreferenceBench

• The benchmark’s construction is presented as a human-in-the-loop process that filters pairs to reflect subjective preference rather than objective differences.The stated aim is to isolate aesthetic and stylistic preference from objective quality signals.Source: §2 · p. 3, §2.1 · p. 3

2.1 Benchmark Construction Pipeline

• The pipeline starts from 51 creative-writing categories, with English queries designed to be culturally neutral and Chinese queries adapted to Mandarin rhetorical conventions while maintaining structural parallelism.Expert-authored blueprints are expanded into queries and independently reviewed by two instructors.Source: §2.1 · pp. 3–4
• Twenty language models generate five sampled responses per query, after which automated screening removes objective deficiencies before expert evaluation.The initial filter removes approximately 15% of raw responses for issues including grammatical errors, factual inconsistencies, and prompt violations.Source: §2.1 · p. 4
• Eleven expert annotators evaluate responses using a calibrated rubric after training, and pairs require directional agreement from at least two of three annotators and a check for confounds such as length disparities.A separate bilingual expert team validates that scoring standards are applied equivalently across languages.Source: §2.1 · pp. 4–5

2.2 Dataset Statistics

• The benchmark comprises 1,800 human-validated pairs—1,200 English and 600 Chinese—designed to isolate stylistic preference from objective correctness.Its coverage spans eight macro-categories, with deliberate oversampling of underrepresented genres such as poetry and scriptwriting.Source: §2.2 · p. 5, Figure 3 · p. 5
• Chosen responses have higher length variance than rejected responses in both languages (English SD=1801.9 vs. 593.4; Chinese SD=1967.5 vs. 1311.0), and show right-skewness.Source: §2.2 · p. 5

3.1 Evaluation Protocols

• Reward models predict a preference by scoring both responses, while language-model judges select a preferred response based on creativity, style, and emotional resonance.Judges use deterministic decoding and are evaluated zero-shot, without specialized preference training.Source: §3.1 · p. 6

3.2 Models

• The evaluation covers seven reward models, including sequence classifiers and generative reward models, alongside 14 language-model judges.The judge set includes both reasoning-enhanced and standard models.Source: §3.2 · p. 6

3.3 Implementation Details

• All models use standardized prompt templates, and evaluation covers the full dataset of 1,200 English and 600 Chinese pairs without subsampling.Reward models use their repositories’ default inference configurations; language-model judges receive a standardized prompt requesting a preference judgment and justification.Source: §3.3 · p. 7

4.1 Reward Model Performance

• 52.7% mean accuracy leaves sequence classifiers statistically indistinguishable from chance; three of four fall below 50% on their weaker language.Individual language-subset accuracy ranges from 46.8% to 62.6%.Source: §4.1 · p. 7
• RM-R1-Qwen2.5-7B reaches 81.8% accuracy in English, highest across all models and 30 percentage points above the sequence-classifier mean; in Chinese, best generative performance reaches 64.5% versus 53.5% for sequence classifiers.Source: §4.1 · p. 7
• RM-R1-DeepSeek improves from 50.3% to 62.6% (ZH) when scaled from 7B to 14B, while variance drops from 9.8 to 5.5; the 27B Skywork-Gemma shows no improvement over 8B variants.The 14B model has the lowest reported variance across genres.Source: §4.1 · p. 7
• 10.1% mean within-model standard deviation among discriminative models accompanies large genre swings, including Nvidia/AceMath-7B’s 18.2%–61.5% range.Generative models’ mean within-model standard deviation is 9.2%.Source: §4.1 · p. 7
• 62.6% (ZH) and 62.5% (EN) for RM-R1-DeepSeek-Qwen-14B contrast with Nvidia/AceMath-7B’s 53.5% (ZH) and 46.8% (EN).The reported cross-language consistency is higher for the generative model.Source: §4.1 · p. 7

4.2 Language Model Judge Performance

• 53.9% mean accuracy for zero-shot language-model judges trails reward models at 58.2%, and the best judge remains 13.1% below the top generative reward model.Table 2 reports pairwise preference-evaluation accuracy for language-model judges.Source: §4.2 · p. 8, Table 2 · p. 9
• Reasoning capability does not consistently improve judge accuracy: the correlation is r=0.08 (p>0.5), and OpenAI-o3-high scores 48.1% (EN).Doubao-1.5-Pro reaches 68.7% (EN) without reasoning, compared with 61.0% (EN) for Claude-4-Opus-thinking.Source: §4.2 · p. 8
• 11.4% mean within-model standard deviation among LLM judges exceeds reward-model variation, with 9 of 14 judges above 10%.Gemini-2.5-Pro ranges from 80.0% on English Poetry to 34.8% on Scriptwriting.Source: §4.2 · p. 8

A Performance Ceiling on Subjectivity

• 62.5% accuracy for the top bilingual reward model suggests a ceiling on current methods for capturing nuanced stylistic and creative preferences.The discussion contrasts this performance with models’ stronger identification of objective flaws, which the benchmark filters out.Source: §5 · p. 9

Explicit Reasoning is No Panacea

• Advanced reasoning does not resolve the gap: the authors attribute weak subjective assessment to missing representations for originality, emotional resonance, and stylistic flair.They characterize the issue as representational rather than logical.Source: §5 · p. 9

Preference Functions are Brittle and Unstable

• Genre swings exceeding 50 percentage points suggest models rely on brittle genre-specific heuristics rather than generalizable principles of good writing.The authors warn that optimizing RLHF against such a volatile reward signal could introduce unpredictable biases favoring stylistic mimicry over genuine quality.Source: §5 · pp. 9–10

Cross-Lingual Performance Reflects Training Data Imbalances

• English–Chinese gaps vary by model, leading the authors to attribute them likely to training-corpus imbalances rather than consistent linguistic or aesthetic differences.They argue robust cross-lingual preference modeling will require approaches that explicitly encourage language-agnostic aesthetic understanding.Source: §5 · p. 10

Preference Learning and Evaluation Benchmarks

• RewardBench, AlpacaEval, and MT-Bench measure conversation, reasoning, or instruction-following, but the authors argue such benchmarks conflate objective correctness with subjective preference.RewardBench’s reported 95% performance on safety does not evaluate aesthetic judgment; MT-Bench measures factual accuracy rather than creative quality.Source: §6 · p. 10

Evaluating Creative and Subjective Writing

• WritingPreferenceBench is presented as addressing prior limitations through human validation, cross-lingual coverage, and isolation of subjective quality discrimination.The cited comparisons include LitBench’s Reddit-upvote proxy and English-only scope, WritingBench’s mixture of creative and functional tasks, and AlignBench’s focus on general Chinese-language capabilities.Source: §6 · p. 10

7 Conclusion

• WritingPreferenceBench isolates subjective writing preference by systematically neutralizing objective quality signals.The benchmark is introduced as the paper’s evaluation resource.Source: §7 · p. 10
• 52.7% accuracy for sequence-based reward models contrasts with 81.8% for generative models incorporating explicit reasoning; evaluated models show genre standard deviations of 10.1–14.0%.The conclusion reports individual genre accuracy ranging from 18.2% to 92% and interprets this variation as reliance on genre-specific heuristics.Source: §7 · p. 10
• The 30 percentage point architecture gap challenges direct preference optimization, while the authors call for hybrid architectures and objectives that encourage genre-invariant preference learning.The conclusion also questions whether scaling laws from objective tasks apply to subjective preference modeling.Source: §7 · p. 10

Contributions and Acknowledgements

• The benchmark materials describe M-A-P as a donation-run, non-profit open-source AI research community.The acknowledgements also list contributors and their affiliations.Source: Contributions and Acknowledgements · p. 11

A Use of Large Language Models

• A large language model was used to polish and refine the manuscript, while the authors developed and verified its arguments and technical content.Source: A Use of Large Language Models · p. 14

B Full Taxonomy of Writing Categories

• The taxonomy groups 51 writing categories into eight high-level domains.Source: B Full Taxonomy of Writing Categories · p. 14

C. Genre-Specific Scoring Guidelines

• The genre-specific guidelines combine a general quality rubric with universal and genre-specific standards.Source: C Genre-Specific Scoring Guidelines · p. 16

C.1 General Scoring Rubric (4-Point Scale)

• The four-point rubric ranges from incoherent or irrelevant writing (0) to creative, professional-quality writing (3).Intermediate levels distinguish formulaic or flawed writing (1) from competent but predictable writing (2).Source: §C.1 · p. 16

C.2 Universal Evaluation Criteria

• Annotators assess content, language, prompt adherence, and structure, including specificity, clarity, genre fit, coherence, pacing, and reading fluency.Source: §C.2 · p. 16

C.3 Genre-Specific Evaluation Criteria

• Genre-specific standards complement universal criteria, with examples addressing fiction, scripts, nonfiction, functional documents, and funny writing.Source: §C.3 · p. 17, §C.3.1 · p. 17, §C.3.2 · p. 17

C.3.1 Fiction (e.g., Sci-Fi, Fantasy, Mystery)

• Fiction criteria include consistent characters and relationships, action-led narration, and originality in premises, characters, plots, and narrative devices.Source: §C.3.1 · p. 17

C.3.2 Scriptwriting

• Scriptwriting criteria cover believable dialogue, staged action and production elements, and character motivations that advance the plot.Source: §C.3.2 · p. 17

C.3.3 Non-Fiction (e.g., Essays, Biographies, Reviews)

• Nonfiction criteria check factual accuracy, credible expression of the author’s experience, and depth beyond surface description.Source: §C.3.3 · p. 17

C.3.4 Functional Documents (e.g., Resumes, Proposals, Memos)

• Functional-document criteria address clarity of purpose, completeness, and conventional format and logic, including support for persuasive arguments.Source: §C.3.4 · p. 17

C.3.5 Funny (e.g., Internet Memes, Copypasta)

• Funny-writing criteria assess niche, often non-literal styles through deliberate absurdity, subculture-specific techniques, and ironic or satirical tone.Source: §C.3.5 · p. 17

• Table 3 reports distributional properties of preference pairs in WritingPreferenceBench.Source: Table 3 · p. 18

E.1 Example 1: Poetry

• The poetry prompt asks for a nostalgic poem portraying a nonfunctioning refrigerator as a guardian of family memories.Source: §E.1 · p. 18

E.2 Example 2: Product Review

• The product-review prompt requests a structured evaluation of a high-end outdoor jacket’s performance, usability, pros, cons, and purchase suitability.Source: §E.2 · p. 18

E.3 Example 3: Funny

• The funny-writing prompt specifies abstract Evangelion fan fiction exploring selfhood and otherness through fragmented, multi-voice stream of consciousness.Source: §E.3 · pp. 18–19

E.4 Example 4: Short Story

• The short-story prompt centers on a neighbor’s late-night noise, a poignant revelation, and eventual reconciliation.Source: §E.4 · p. 19

E.5 Example 5: Argumentative Essay

• The argumentative-essay prompt asks writers to reconcile looking back and moving forward, using historical evidence and a contemporary or personal example.Source: §E.5 · p. 19

E.6 Example 6: Speech

• The graduation-speech prompt requires stories about a teacher and classmates to develop gratitude and responsibility.Source: §E.6 · p. 19

E.7 Example 7: Advertising Copy

• The marketing-email prompt calls for a vivid backpack-user story, a catchy subject line, and a clear call to action.Source: §E.7 · pp. 19–20

E.8 Example 8: Biography

• The biography prompt asks writers to connect Frida Kahlo’s life events and paintings, examining how pain fueled her artistic creation.Source: §E.8 · p. 20