Source-linked AI summary
WritingBench: A Comprehensive Benchmark for Generative Writing
Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, Fei Huang
TL;DR
Generative-writing evaluation lacks broad task coverage and robust assessment of complex, query-specific requirements. WritingBench combines a diverse benchmark with dynamically generated criteria and a fine-tuned critic model. Its evaluation-guided curation also improves smaller writing models, with the framework reaching 84% human agreement and filtered models outperforming larger models in experiments.
Problem
Existing generative-writing benchmarks have limited domain and task coverage, while static evaluation criteria inadequately assess specialized, stylistic, formatting, length, and material-use requirements.
Method
WritingBench combines 1,000 queries across 6 domains and 100 subdomains with query-dependent generation of five criteria and a fine-tuned critic model for scoring.
Results
84% human agreement demonstrates the practical viability of the critic-based framework, while models trained on filtered data outperform larger models in writing experiments.
Takeaways & Limitations
The released benchmark and evaluation resources support systematic assessment of writing across diverse domains and requirements.
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing.
1 Introduction
Existing generative-writing benchmarks inadequately cover diverse real-world tasks and lack robust evaluation for complex requirements. WritingBench addresses these gaps with broad domain coverage, human-AI query construction, query-dependent criteria, and critic-based scoring.
- Limitations of Existing Benchmarks: Existing benchmarks have limited domain and task diversity, often using simplistic prompts, homogeneous materials, or single-domain coverage.They therefore do not adequately capture the diversity and intricacies of practical writing tasks.
- WritingBench: A four-stage human-AI construction pipeline generates and diversifies queries, collects and optimizes materials, and supports broad domain and requirement coverage.The domain taxonomy is grounded in real-world writing scenarios and combines model-generated refinement with human annotation.
- Query-Dependent Evaluation: The query-dependent evaluation framework dynamically generates five instance-specific criteria and uses a fine-tuned critic model for criteria-aware scoring.The framework evaluates writing with criteria tailored to each query rather than relying only on fixed dimensions.
- WritingBench: WritingBench provides 1,000 queries across 6 primary domains and 100 subdomains, covering style, format, and length requirements.The benchmark includes 445 Chinese and 555 English queries and supports varied input lengths from tens to thousands of words.
- Limitations of Existing Benchmarks: Static evaluation criteria inadequately adapt to specialized domains, style, format, length, and reference-material requirements.These limitations motivate instance-specific evaluation criteria for complex writing tasks.
- Evaluation-Guided Data Curation: Framework-filtered data enables smaller fine-tuned models to outperform larger models in writing experiments.Selecting the top 50% of samples per subdomain produced 12K high-quality samples used to fine-tune Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct.
4 Experiment
Experiments evaluate 17 LLMs on WritingBench, analyze domain and requirement performance, test reasoning effects, human alignment, and data curation. Results identify strengths in education and marketing, difficulties with knowledge-intensive and long-output tasks, and benefits from query-dependent criteria and filtered data.
- Experimental setup: 17 LLMs are evaluated on WritingBench using uniform generation settings, with responses capped at 16,000 tokens or the platform maximum.Generation uses temperature 0.7, top-k 20, and top-p 0.8; critic and LLM scoring use temperature 1.0.
- Domain scores: Education (D5) and Advertising & Marketing (D6) are consistently stronger domains, whereas Academic & Engineering (D1) and Finance & Business (D2) are more challenging.Niche tasks such as bid proposals and financial reports remain difficult because they require knowledge, long-text generation, and contextual consistency.
- Domain scores: Reasoning-capable models outperform non-reasoning counterparts in Literature & Art (D4), and CoT-based fine-tuning improves creative-writing performance over corresponding non-CoT variants.The comparison includes Claude-3.7-thinking, Deepseek-R1, and o1-Preview, with further tests on Qwen-32b-Instruct using a 12k SFT dataset.
- Requirement scores: Style scores are generally highest, format follows, and length is weakest, while criteria outside specialized requirements emphasize source integration and writing depth.Length remains difficult for section-specific constraints and extended generation; most models typically cap outputs around 3,000 tokens, while Claude-3.7 reasoning-enhanced and Qwen-Max generate longer outputs effectively.
- Overall findings: Claude-3.7-thinking leads across domain and requirement dimensions, while creative-domain performance varies and cross-lingual inconsistencies appear in Deepseek-R1 and Llama-3.3-70B-Instruct.The analysis also identifies long-form generation as a persistent obstacle for current models.
- Human consistency: Dynamic query-dependent criteria achieve higher human alignment than globally uniform or domain-specific static criteria, and the critic model reaches 84% agreement.The human study uses 300 queries and five professionally trained annotators; domain-specific criteria underperform despite customization.
- Data curation: Models trained on the filtered 12K dataset significantly improve over their previous versions, outperform models trained on the full 24K dataset, and approach advanced-model capabilities.The experiments use Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct on WritingBench and LongBench-Write.
5 Conclusion
WritingBench evaluates generative writing across diverse domains and requirements, while its query-dependent evaluation framework and critic model achieve high human alignment. Filtered-data training further demonstrates improved writing performance for compact models, and the benchmark resources are publicly available.
- Conclusion: WritingBench contains 1,000 queries across 6 primary domains and 100 subdomains, evaluating style, format, and length requirements.The benchmark is designed for generative writing across diverse domains.
- Conclusion: Its query-dependent evaluation framework, supported by a critic model, achieves high human alignment.The framework dynamically supports evaluation of writing requirements across instances.
- Conclusion: Compact models trained on curated data outperform GPT-4o in writing, according to the paper’s conclusion.The benchmark, evaluation tools, and modular framework components are publicly available.
NeurIPS Paper Checklist
The checklist records that the paper’s contributions are supported by methods and experiments, while limitations are discussed in Appendix D. It also emphasizes transparent reporting of assumptions and scope.
- Claims and scope: The paper’s contribution claims are supported by detailed methodologies in Section 3 and experimental validation in Section 4.The checklist asks that claims match the results and reflect their expected generalization scope.
- Limitations: Limitations are discussed in Appendix D, including the need to identify assumptions, robustness boundaries, and practical scope.The checklist emphasizes transparent reporting of limitations and factors affecting performance.
3. Theory assumptions and proofs
The supplied checklist material reports no theoretical results requiring formal assumptions or proofs, and instead documents experimental support and reproducibility information. It points to benchmark construction, prompts, code, data access, and training details as the relevant evidence.
- Theory assumptions and proofs: The paper is treated as having no theoretical results requiring a complete proof and assumption set.The checklist explicitly marks the theoretical-results question as not applicable while referring to experimental analyses.
- Experimental reproducibility: Experimental reproducibility is supported by benchmark-construction details, relevant prompts, and access to the evaluation dataset and code.The checklist cites Section 3.1 and supplementary prompt materials as supporting information.
- Open access: The benchmark dataset and code are open-sourced with detailed README instructions.The checklist identifies the GitHub repository as the access point.
- Training and test details: Training and test details are reported for the benchmark, critic model, and writing model across the cited sections.The checklist identifies Sections 4.1, 3.3, and 3.2 as containing settings, parameters, and datasets.
7. Experiment statistical significance
The paper reports standard deviations for benchmark evaluation and affirms that compute resources and ethical considerations are addressed.
- Standard deviations are provided for benchmark evaluation in Table 3.
- The reported experiments provide information about the compute resources used.
- The research is stated to conform with the NeurIPS Code of Ethics.
- Potential positive and negative societal impacts are discussed in Appendix E.
- Released prompts undergo manual review to eliminate harmful information.
12. Licenses for existing assets
The paper states that existing assets are cited, new assets are documented with usage terms, and LLM involvement in benchmark construction and scoring is described.
- The assets used in the paper have been properly cited.
- New assets are described in the paper and accompanied by instructions and terms of use in the GitHub repository.
- The paper states that annotator instructions and compensation are provided.
- Annotators are employed by a professional annotation company, informed of data usage, and documented as having provided consent.
- LLM involvement in benchmark construction and scoring is described in Sections 3.1 and 3.2.
A Experiment Results
WritingBench experiments examine taxonomy construction, requirement dimensions, creative-writing CoT, human evaluation, length sensitivity, and criteria generation across models.
- Taxonomy and benchmark construction: WritingBench’s taxonomy is refined using over 200K anonymized real-user writing queries under strict data security protocols.
- Requirement dimensions: Style requirements account for approximately 29.4% of analyzed queries, while format requirements comprise about 22.48%.
- Creative writing: CoT-enhanced models outperform non-reasoning models on both WritingBench-D4 and EQBench.
- Human evaluation: Five linguists achieve substantial agreement with κ = 0.69 in randomized human pairwise evaluations.
- Length analysis: Most models produce outputs constrained to approximately 3,000 tokens, whereas Claude-3.7, Qwen-Max, and LongWriter support extended response lengths.
B.2 Overview of 100 Secondary Subdomains
The benchmark’s secondary subdomains span diverse academic, business, legal, creative, technical, educational, and marketing writing scenarios.
- Research writing: Research-oriented subdomains include paper outlines, research proposals, contributions, conclusions, and limitations.
- Academic & Engineering: Academic and engineering subdomains include abstracts, literature reviews, experiment reports, patents, and test reports.
- Organizational writing: Operational and organizational subdomains include meeting minutes, internship reports, event planning, recruitment, and human resource management.
- Finance & Business: Business and finance subdomains include correspondence, market research, risk management, sales reports, contracts, and investment analysis.
- Education and Advertising & Marketing: The listed subdomains also cover education and advertising and marketing.
C Prompts
The classification prompt organizes queries by hierarchical domains and extracts stylistic, formatting, and length requirements in a strict JSON structure.
- C.1 Query Classification Prompt: It asks the evaluator to determine which domains a query belongs to and identify any stylistic, formatting, or length requirements.
- C.1 Query Classification Prompt: Format requirements include document mimicry, outline adherence, or academic-paper conventions.
- C.1 Query Classification Prompt: Style requirements include audience-appropriate language, rigorous wording, or humorous tone.
- C.1 Query Classification Prompt: Length requirements cover word count, duration, and other output-size constraints.
- C.1 Query Classification Prompt: The prompt introduces a classification system with domains spanning Academic & Engineering, Finance & Business, and other categories.
- C.1 Query Classification Prompt: The requested output is strict JSON containing a domain list plus style, format, and length fields.
C.2 Initial Query Generation Prompt
The query-construction prompts generate realistic, detailed writing requests, refine them with targeted guidance, and return modified requirements with material needs.
- C.2 Initial Query Generation Prompt: The initial-generation prompt asks for different writing requests within a specified subdomain and primary domain, in English or Chinese.
- C.2 Initial Query Generation Prompt: Generated requests should be detailed, specific, and reflective of realistic user tone and needs.
- C.2 Initial Query Generation Prompt: The generation step requires a JSON-only response listing the writing requests.
- C.3 Guidance Pool: A guidance pool randomly selects zero to six refinement items for each iteration.
- C.3 Guidance Pool: Guidance can add length constraints, format adherence, audience or occasion-specific style, personalization, and focused content requirements.
- C.3 Guidance Pool: The guidance pool can also require the request to be expressed concisely in one sentence.
- C.4 Query Refine Prompt: The refinement prompt enhances original requirements using the selected guidance and asks whether additional writing materials are needed.
- C.4 Query Refine Prompt: Refinement results must be returned as JSON containing the modified query and material specification.
C.5 Criteria Generation Prompt
The criteria-generation and scoring prompts create detailed, query-specific rubrics and use them to assign strict integer scores supported by concrete response evidence.
- C.5 Criteria Generation Prompt: The criteria-generation prompt asks an expert evaluator to produce five strict criteria for the response to a given query.
- C.5 Criteria Generation Prompt: Each criterion includes a name, description, and five score bands ranging from 1-2 through 9-10.
- C.5 Criteria Generation Prompt: The rubric distinguishes relevance, coherence, depth, specificity, and adherence to the query context.
- C.5 Criteria Generation Prompt: The score descriptions progress from critical deficiencies at 1-2 to exceptional, flaw-free performance at 9-10.
- C.6 Rubric-based Scoring Prompt: The scoring prompt evaluates the response against the query and repeated criteria using explicit scoring rules.
- C.6 Rubric-based Scoring Prompt: Reasons must cite exact response passages, identify strengths or deficiencies, and remain strict beyond superficial format or length.
- C.6 Rubric-based Scoring Prompt: It warns evaluators to detect fabricated substance and responses that provide only an introduction instead of completing the query.
- C.6 Rubric-based Scoring Prompt: The evaluator assigns an integer score from 1 to 10 and returns the score with a specific justification in JSON.
D Limitations
The paper identifies limitations in optimization, complex length evaluation, and preference annotation, while positioning WritingBench as an open resource for broad and targeted assessment.
- Limitations: The writing and critic models rely primarily on conventional supervised fine-tuning, without systematic exploration of enhanced optimization strategies.
- Limitations: Chain-of-thought mechanisms show partial efficacy in creative domains, but their potential remains less explored than in mathematical reasoning.
- Limitations: The evaluation framework has reduced precision for multi-dimensional length requirements such as temporal sequencing and section-specific word counts.
- Limitations: Pairwise preference annotation remains difficult for compositional tasks because evaluators introduce subjective biases between otherwise fair responses.
- Limitations: Consensus procedures mitigate some annotation variability, but cannot achieve complete alignment with diverse user preferences.
- Impact: WritingBench supports evaluation of general-purpose writing and domain-specific expertise across diverse domains and requirements.
- Impact: Its public release of queries, protocols, criteria tools, and critic models supports replication, validation, and further methodological development.
- Impact: The benchmark can evaluate subsets such as legal writing, medical documentation, or marketing for model selection.