Source-linked AI summary
Generating Constructive Feedback on Stories via Reinforcement Learning
Maja Stahl, Timon Ziegenbein, Henning Wachsmuth
TL;DR
LLM-generated writing feedback often lacks specificity, actionability, and prioritization, despite the value of feedback for writers. The paper trains LLMs with GRPO and a multi-component reward function, without ground-truth revisions, and reports stronger performance than baselines across three story corpora and human evaluation. Actionability is identified as the primary driver of constructiveness.
Problem
LLM-generated feedback frequently lacks specific references and actionable guidance and misses critical writing issues, limiting its usefulness for creative-story revision.
Method
The paper uses GRPO with a multi-component reward function targeting priority, specificity, and actionability, without requiring ground-truth feedback or revision data.
Results
The GRPO approach outperforms strong baselines, including Gemini 2.5 Flash, across automatic and human evaluations on three story corpora.
Takeaways & Limitations
Actionability is the primary driver of constructiveness, with actionable feedback preferred by both LLM and human judges.
Takeaways & Limitations
The approach is evaluated only on English creative stories, so its transferability to other genres and languages remains open.
Abstract
from arXiv · showhide
Constructive feedback is crucial for creative writers to refine their storytelling abilities. Since receiving feedback from human experts is often costly and time-intensive, large language models (LLMs) offer a scalable and efficient alternative as automatic writing assistants. Despite their potential, research indicates that LLM-generated feedback is often generic, lacks actionability, and fails to identify which writing issue is most critical. To address these limitations, we present a reinforcement learning approach that steers LLMs to generate constructive feedback without the need for ground-truth feedback. We train our model using group relative policy optimization (GRPO) with a novel multi-component reward function aiming at constructiveness: it prioritizes feedback that is uniquely tailored to the story, helps to improve story quality, and addresses the most critical writing issue. In automatic and human evaluation across three story corpora, our approach outperforms state-of-the-art LLMs (including Gemini) and competitive baselines. We find that providing actionable suggestions is the main driver of feedback constructiveness.
1 Introduction
The paper addresses limitations in LLM-generated writing feedback by training an RL-based system to produce feedback that is prioritized, story-specific, and actionable. Across automatic and human evaluations, the approach outperforms strong baselines, and actionability emerges as the primary driver of constructiveness.
- Constructive feedback is intended to help authors improve current texts and refine their writing skills over time.
- LLM-generated feedback often misses critical issues, hallucinates strengths or errors, and lacks specific references and actionable guidance.
- The approach uses GRPO with a novel multi-component reward function targeting priority, specificity, and actionability.It is designed to generate constructive feedback without ground-truth revisions or feedback.
- The study evaluates the three criteria with automatic metrics on three story corpora and with human evaluation.
- The GRPO approach outperforms strong baselines, including Gemini 2.5 Flash, while actionability is identified as the primary driver of constructiveness.Feedback emphasizing actionability is preferred by both LLM and human judges.
2 Related Work
Prior writing-feedback research spans specialized error correction, comment generation, prompting, and preference optimization. This paper instead uses feedback-quality dimensions as reward signals for holistic, open-ended feedback across diverse story corpora.
- Earlier systems focused on specialized subtasks such as essay scoring, grammatical error detection, correction, or sentence-level comments.
- The paper focuses on holistic, open-ended feedback intended to guide full-text revision rather than assessment or local edits.
- Prompting-based LLM feedback methods can miss critical writing issues or hallucinate strengths and errors.
- Prior revision-as-reward methods model student revision behavior but require simulated behavior and ground-truth revision data.
- This work uses GRPO to evaluate feedback itself and adopts specificity, actionability, and priority as corpus-agnostic reward dimensions.A fourth criterion, positive reinforcement, is omitted because it can only be evaluated on one specific corpus.
3 Approach
The approach trains an LLM policy with GRPO to generate natural-language feedback for a given story. It removes the need for ground-truth feedback or revision data by aligning the policy with a reward function derived from human annotations of feedback quality.
- An LLM policy trained with GRPO generates natural-language feedback for a given story.
- The method replaces ground-truth feedback or revision data with a reward function derived from human annotations of feedback quality.
R Overall reward
The paper aligns an LLM policy to generate constructive story feedback by optimizing a GRPO objective with a reward that equally averages priority, specificity, and actionability. This design avoids requiring ground-truth feedback or revision data while enabling systematic evaluation of the three criteria.
- Constructive feedback identifies the story’s most critical issue, is tailored to that story, and offers suggestions that can improve its quality.
- The approach uses GRPO to sample candidate feedback texts and compute each candidate’s relative advantage from group rewards, without a separate critic model.
- The policy maximizes expected reward while the scoring functions provide proxy quality signals for generated feedback in the context of each story.
- The overall reward is the arithmetic mean of three bounded scoring functions for priority, specificity, and actionability, assigning equal importance to each criterion.
- The policy can be trained on any story corpus without corresponding feedback or quality labels, although the scoring functions may require annotated data.
4 Data
The study aligns and evaluates its approach on three human-written story corpora with different sizes, genres, and construction methods. StoryFeedback additionally supplies human quality annotations and evaluates feedback on encouragement, priority, specificity, and actionability.
- The approach requires stories for alignment and evaluation, plus quality-annotated feedback for training its reward models.
- The three corpora are StoryFeedback, Storal, and WritingPrompts, all containing human-written stories.
- StoryFeedback contains 1,304 stories formed from 326 unique ROCStories and BIG-bench stories plus synthetically corrupted versions.
- Storal comprises 1,779 English moral-value stories averaging 302 words, whereas WritingPrompts contains 300,000 fictional stories averaging 735 words.
- Human evaluators assess StoryFeedback-generated feedback for encouragement, priority, specificity, and actionability.
5 Experiments
The experiments train dedicated reward models, optimize several instruction-tuned LLMs with GRPO, and evaluate feedback across three corpora using automatic metrics, LLM judges, and human ratings. Reward models correlate with human judgments, and the ensemble judge improves reliability through majority voting.
- Reward models: Three independent ModernBERT-large regression models predict human ratings for priority, specificity, and actionability from concatenated stories and feedback.
- Reward models: The reward models are optimized for Pearson correlation over 20 Optuna trials, with outputs clipped to [0, 1] as bounded quality scores.
- Reward models: pcc = .865 for specificity, .537 for priority, and .651 for actionability, while the reward models outperform random and mean baselines across dimensions.
- Reward models: The reward models correlate more strongly with manual judgments than an LLM-as-a-judge ensemble for overall constructiveness: pcc = .43 versus .38.
- Policy training: Training uses DrGRPO with k = 8 sampled completions per story and downsamples Storal and WritingPrompts to match StoryFeedback’s 908 training examples.
- Evaluation: Automatic evaluation uses 200 test instances from each corpus and reports priority, specificity, and actionability scores, with higher scores better.
- Evaluation: Pairwise LLM judgments are aggregated with Bradley-Terry rankings, where lower average rank indicates higher global preference.
6 Results
Across three story corpora, GRPO improves feedback quality and ranking over instruction-tuned baselines, while actionability is the strongest contributor to perceived constructiveness. Human evaluation confirms these gains, including stronger performance than Gemini on original stories.
- Automatic evaluation: GRPO substantially improves mean priority, specificity, and actionability over instruction-tuned base models across datasets and prompts.For Gemma+Base_targeted on StoryFeedback, priority rises from .718 to .952 and specificity from .912 to .993.
- Prompting effects: Open prompting usually achieves better overall rank, whereas targeted prompting often yields higher priority and specificity scores.The authors attribute the open-prompt ranking advantage to fewer constraints and greater flexibility in addressing present issues.
- Ablation study: Actionability is the dominant reward component: GRPO_open_ac achieves the best rank across all three corpora, while priority-only optimization receives the poorest rank.On Storal, GRPO_open_ac reaches rank 2.15, whereas GRPO_open_pr reaches priority .969 but rank 5.94.
- Human versus synthetic errors: On original human-written stories, Gemma+GRPO_open surpasses both Gemini variants, while Gemini is biased toward synthetically corrupted stories.The comparison uses original-story rank on the x-axis and corrupted-story rank on the y-axis.
- Manual evaluation: Human evaluation finds Gemma+GRPO_open_ac best across priority, specificity, actionability, and constructiveness, with constructiveness rated 4.01.Its scores are priority 4.45, specificity 4.90, actionability 4.11, and constructiveness 4.01.
- Manual evaluation: Actionability and priority are the strongest predictors of overall constructiveness, with Kendall’s τ of 0.556 and 0.452, respectively.The remaining criterion correlations are comparatively low to moderate, with τ ≤ .31.
7 Conclusion
The paper introduces GRPO with a multi-component reward targeting priority, specificity, and actionability to improve constructive story feedback without ground-truth revisions. Across three creative story corpora, it reports stronger feedback than state-of-the-art baselines, especially for nuanced errors in original stories, with actionability as the primary driver of perceived constructiveness.
- Contribution: The paper presents the first GRPO-based reinforcement learning approach that optimizes constructive feedback without ground-truth feedback or revisions.Its reward function targets priority, specificity, and actionability.
- Results: Across three creative story corpora, the approach produces significantly more constructive feedback than state-of-the-art baseline models.The reported gains are robust across different base models and corpora.
- Conclusion: Actionability is the primary driver of perceived constructiveness, while the approach is especially effective on nuanced errors in original human-authored stories.The conclusion bases this claim on ablation studies and manual evaluations.
8 Limitations
The paper identifies limitations involving scope, subjectivity, and factual faithfulness in its feedback-generation approach. Its experiments focus on English creative stories, annotator consensus may not match every author, and the reward function lacks a dedicated validity constraint.
- Scope: The experiments and reward models focus exclusively on English creative stories, leaving transfer to other genres and languages uninvestigated.The authors note that high-quality feedback annotations are unavailable for other settings and that generalization to academic or technical writing remains an open question.
- Subjectivity: Creative-writing quality is subjective, so annotator consensus may not match every author’s stylistic vision or cultural context.The authors state that the approach could be retrained on other data if available.
- Faithfulness: The multi-component reward function omits a dedicated validity or faithfulness component for enforcing factual accuracy.The authors distinguish this limitation from their observed reduction in severe quality problems and suggest integrating factuality criteria in future work.
9 Ethical Considerations
The ethical discussion highlights risks that feedback systems may mislead authors, reduce independent learning through overreliance, or homogenize creative expression. These concerns accompany implementation details about reproducible training and the evaluation prompt.
- Author impact: The system cannot fully guarantee that feedback will not occasionally mislead authors, although the authors observed no such cases during inspection.
- Author impact: Overreliance on explicit, ready-to-use AI suggestions may reduce writers’ learning effect and independent development of writing and editing skills.
- Creative diversity: Aligning reward models with specific annotators’ consensus may homogenize creative expression and suppress non-standard, unconventional, or culturally distinct storytelling.The authors state that this bias cannot be fully prevented and was not an optimization target.
- Reproducibility: Training used deterministic seeding, isolated model caches, and bf16 mixed precision on 4× NVIDIA H100 GPUs, with full runs taking approximately 16 to 22 hours.The configuration covered Gemma, LLaMA, and Qwen training runs across the three models and datasets.
- Evaluation: The evaluation prompt asks judges to compare two feedback texts for the same story on specificity, improvement potential, and the biggest issue, returning a structured verdict.
C LLM Judge Bias Analysis
The bias analysis measures whether judge models prefer outputs from their own model family relative to cross-family preferences. The reported biases are small overall, with positive effects for Gemma and LLaMA but a negative effect for Qwen.
- Method: The analysis defines self-preference as a judge’s preference for its own generating model family relative to other judges’ preference for that family.
- Method: A positive ∆self indicates family-specific preference, whereas a near-zero value indicates alignment with other judges’ consensus.The metric uses win rates for the judge model J and generating model G.
- Results: +3.80% and +2.63% are the self-preference deltas reported for Gemma (27B) and LLaMA (72B), respectively.These win-rate increases occur when evaluating their smaller counterparts compared with other judges.
- Results: −2.78% is the self-preference delta for Qwen (72B), indicating that it is more critical of its own family’s outputs than other judges.
E Calibration of Automatic Metrics
The calibration analysis compares reward models and LLM judges with human ratings and with each other across feedback-quality dimensions. Reward models correlate more strongly with human judgments than the LLM-judge ensemble in representative dimensions, while the two automatic metrics capture related but non-identical aspects of quality.
- Reward models and human judgments: Each human rating criterion correlates most strongly with its matching reward model, while the full reward correlates with overall human-rated constructiveness at pcc = .43.
- LLM judgments and human judgments: The majority-vote LLM-judge ensemble achieves the highest correlation with human judgments across all four feedback-quality dimensions.
- Metric overlap: Reward-based and judge-based metrics show moderate correlations, indicating that they capture related but non-identical aspects of feedback quality.
- Comparison with LLM judges: .53 versus .38 for specificity and .42 versus .23 for actionability show stronger human correlations for regression-based reward models than for the LLM-judge ensemble.The comparison also supports using lightweight reward models in the RL loop because the judge ensemble has substantially higher inference cost.
- Evaluation setup: Table 8 evaluates GRPO against instruction-tuned base LLMs and Gemini on priority, specificity, actionability, and Bradley-Terry average rank across original, corrupted, and all StoryFeedback stories.
- Evaluation criteria: The evaluation dimensions define feedback quality through specificity, actionability, priority, and constructiveness, with actionable feedback providing a clear path for improvement.Priority concerns the story’s most critical issue, while constructiveness balances the three preceding criteria.