Source-linked AI summary
User Feedback Provides a Unique Signal that LLMs Can not Detect
Shachar Don-Yehiya, Leshem Choshen, Omri Abend
TL;DR
The paper asks whether naturally occurring user feedback is genuinely useful for improving LLM responses despite evidence that it is noisy. It compares feedback-informed and baseline revisions on synthetic and naturalistic data, finding higher issue resolution with feedback while LLM judges often prefer the lower-quality baseline outputs. The authors argue that evaluation bias masks feedback’s utility and may reflect a signal unavailable through standard model-improvement techniques.
Problem
Naturally occurring feedback may help align LLMs with human intent, but prior work describes it as noisy and difficult to use effectively.
Method
The paper compares model revisions with and without feedback using corrupted synthetic responses with ground truth and feedback extracted from real conversations.
Results
9%–32% higher resolution on synthetic data and 16%–27% higher resolution on real user data were obtained with feedback than without feedback, while judges frequently preferred baseline variants.
Takeaways & Limitations
Evaluation bias can systematically penalize genuine feedback-informed corrections, suggesting feedback provides a signal not captured by current LLM judges.
Takeaways & Limitations
For reliability, the main text focuses on math and code domains, although appendix results and multi-domain naturalistic data indicate broader trends.
Abstract
from arXiv · showhide
Harnessing naturally occurring feedback from user interactions offers a promising learning signal for Large Language Models (LLMs). However, recent studies suggest this feedback is inherently noisy and difficult to leverage effectively. We challenge this conception by demonstrating that user feedback is a highly actionable signal for improvement, and that its perceived ineffectiveness stems from a systematic bias in current evaluation paradigms. To isolate the usefulness of feedback, we construct synthetic data with a definitive ground truth, alongside naturalistic data to validate that our findings hold in real-world scenarios. By comparing model revisions generated with and without access to feedback across both settings, we show that feedback-informed revisions resolve targeted issues at significantly higher rates than baseline revisions. Finally, we expose the root of the evaluation bias: when a model successfully fixes an issue exclusively due to feedback, LLM judges frequently fail to identify the genuinely corrected response, systematically preferring inferior baseline outputs instead.
1 Introduction
The paper argues that naturally occurring feedback is an effective improvement signal, while evaluation methods systematically understate its value. Controlled synthetic experiments and naturalistic user-feedback data show stronger issue resolution with feedback, but LLM judges often prefer inferior baseline revisions.
- Naturally occurring feedback is promising for aligning LLMs with human intent, but prior work characterizes it as noisy and difficult to use effectively.
- Controlled experiments show that feedback is a strong improvement signal whose contribution is masked by systematically biased evaluation paradigms.
- Synthetic data with ground truth and naturalistic data from real conversations enable comparisons of revisions made with and without feedback.
- 9%–32% higher resolution on synthetic data and 16%–27% higher resolution on real user data were obtained with feedback than without feedback.
- LLM judges frequently fail to identify feedback-informed corrections, preferring general stylistic changes over genuinely corrected responses.
2 Method
The method creates controlled corruptions with simulated feedback, extracts and filters naturally occurring feedback, and compares improvement with versus without feedback.
- Synthetic experiments corrupt existing model responses so feedback can be evaluated against a definitive ground truth.
- The corruption process defines four types: causality inversion, crucial omission, entity/subject swap, and logic operator reversal.
- The corrupting model produces an invalid response, change log, difficulty score, and three feedback levels ranging from a solution to a generic error indication.
- Naturalistic feedback is extracted from conversations, classified into a taxonomy, and filtered to retain actionable insights applicable to future users.
- The improver generates paired responses by addressing the corrupted response with feedback and without feedback.
3 Evaluation
The evaluation uses separate judges for issue resolution and direct pairwise preference, allowing correction success to be compared with evaluator preferences.
- An issue-resolution judge assesses whether each improved response addressed the simulated feedback explaining how to fix the corruption.
- A pairwise judge directly compares two responses, randomly orders them to reduce order bias, and outputs a preference or tie.
4 Experimental Setup
Experiments use synthetic Arena-Hard data, multiple improver sizes, a stronger judge, and validation procedures, while pairwise evaluations compare feedback conditions across corruption types.
- The synthetic dataset contains 500 challenging real-world queries and 250 creative-writing queries, with o3 responses used before corruption.
- The improver models are Gemini-3-flash-preview and Qwen3-8B, while Gemini-3.1-Pro-preview serves as the stronger judge.
- Across four corruption types, judges consistently preferred the large improver’s without-feedback variant despite its lower corruption-fix rate.
- 92.2% of hard-prompt corruptions were manually marked correct, whereas 59.4% of creative-writing corruptions were marked correct; main-text results therefore focus on hard prompts.
- Issue-resolution annotations achieved Cohen’s Kappa of 0.81 on synthetic data and 0.24 on naturalistic data after excluding unfamiliar samples.
5 Results
External feedback consistently improves issue resolution, but pairwise LLM judges often prefer lower-quality without-feedback revisions. On cases where feedback uniquely enables correction, judges frequently fail to recognize the valid response, with errors associated with evaluator and improver limitations.
- 5.1 The Value of External Feedback: Feedback improves issue resolution by 9%–35% over withholding feedback across synthetic and naturalistic data.Smaller models benefit more substantially from feedback than larger models.
- 5.1 The Value of External Feedback: In naturalistic data, feedback raises issue resolution from 70% to 89% for large improvers and from 26% to 53% for small improvers.Absolute rates are lower than in synthetic data, but the feedback gain remains robust.
- 5.3 With-Feedback Should Win: Despite lower issue-resolution rates, pairwise judges usually prefer without-feedback variants, including both improvers on naturalistic data.The synthetic trend reverses for the small improver, whose with-feedback revisions are judged superior.
- 5.3 With-Feedback Should Win: On feedback-dependent cases, judges often fail to prefer the with-feedback response even though it is the only variant resolving the corruption.Correct preference rates are 80.9% for the small improver and 54.6% for the large improver on synthetic data, versus 54% and 34% on naturalistic data.
- 5.4 Models Struggle to Evaluate What They Cannot Improve: The large judge prefers feedback-enabled responses at 87.7% on self-improvable cases and 72.2% on feedback-improvable cases, a 15.5-point gap.This supports correlated failure modes between improving independently and evaluating improvements.
6 Further Analysis
Self-judging results connect evaluation failures to model capability, while improvement-type analysis shows that feedback shifts revisions toward substantive content changes. Even weaker feedback signals remain beneficial, though their gains are smaller than full feedback.
- 6.1 Reproducing Correlated Failures in the Small Improver: For the large improver, default and self-judge accuracy is nearly identical: 54.6% and 54.4%, respectively.
- 6.1 Reproducing Correlated Failures in the Small Improver: For the small improver, default judging prefers with-feedback responses in 80.8% of cases, versus 67.6% for self-judging.The 13-point gap appears when the judge lacks a substantial capability advantage.
- 6.2 Improvement Types: Feedback increases content improvements and decreases “no improvement” cases relative to style-focused edits.Content categories include factuality, completeness, and logic; style categories include tone, conciseness, formatting, and structure.
- 6.2 Improvement Types: Judges fail to recognize feedback-enabled revisions’ higher-quality substantive content improvements and remain biased toward stylistic changes.
- 6.3 Weaker Feedback Signal: “What Problem” and “Problem exists” feedback both outperform no feedback, with gains of about 6%–17% and 3%–10%, respectively.Their margins are smaller than full feedback’s 9%–32% gain, and the small improver benefits more.
7 Related Work
Prior work studies natural-language and spontaneously occurring user feedback, including comparisons where regeneration without feedback appeared superior. This paper situates that pattern within broader known biases of LLM-as-a-Judge evaluation.
- Earlier studies elicited free-text feedback, while later work extracted feedback that arose spontaneously in conversations for model training.
- Liu et al. compared feedback-based “regeneration with semantics” against “regeneration from scratch” using a reward model and found the latter superior.The comparison resembles this paper’s with-feedback versus without-feedback setup.
- This paper reports that LLM-as-a-Judge can fail to prefer valid responses over corrupted ones, alongside established position, verbosity, and self-preference biases.
- The work improves models at test time without further training, distinguishing its approach from training-based self-improvement research.
8 Discussion
Across synthetic and naturalistic data, feedback-informed revisions resolve more targeted issues, yet pairwise LLM evaluation often favors without-feedback variants. The authors argue this bias may indicate that naturally occurring feedback supplies a signal not encoded in model weights.
- 9%−32% more cases were resolved with feedback than without feedback across synthetic and naturalistic data.
- Pairwise evaluations frequently favor without-feedback variants despite their lower issue-resolution rates.
- When both variants resolve an error, without-feedback outputs may merit preference because they can introduce broader improvements.
- When only the feedback-informed variant resolves the corruption, preferring the no-feedback variant or a tie is a clear judging error.
- The authors speculate that judges’ failures to recognize feedback-informed improvements may reflect a signal absent from model weights.
- The paper calls for evaluation frameworks and specialized methods that appropriately credit feedback-driven resolutions.
Limitations
The paper limits its main synthetic-data analysis to math and code because corruption was less reliable for creative writing. Human evaluation also could not guarantee that revisions introduced no regressions.
- Creative-writing corruption was less effective than corruption in the other synthetic-data categories.
- The main text therefore focuses on math and code, while appendix results include creative writing and report persistent overall trends.
- Annotators could not conclusively guarantee that revisions introduced no regressions because responses were long and domains diverse.
A Full Results
The full-results analyses evaluate corruption, feedback extraction, response improvement, issue resolution, and pairwise quality judgments across synthetic and real-user data. Results consistently show higher resolution with feedback but frequent judge preference for without-feedback variants.
- Synthetic-data results show a consistent positive resolution delta for with-feedback versus without-feedback variants.
- The evaluation pipeline extracts feedback, filters feedback instances, generates improvements with two prompt variants, and evaluates resolution and pairwise quality.
- Human evaluation was used to assess automated extraction and evaluation processes, but annotators reported unfamiliar cases and uncertainty about regressions.
- Table 4 measures issue-resolution percentages across four corruption types in the synthetic hard-prompt category.
- Full synthetic-data pairwise judgments consistently prefer the without-feedback variant even though it fixes corruption less often.
- Real-user pairwise results show the same preference pattern for without-feedback variants, including for the small improver.
F Re-improvements
The re-improvement experiment tests whether a second feedback-informed step can combine targeted correction with broader improvements from an initial no-feedback revision. The judge prefers feedback in one comparison but reverses preference when comparing two-stage variants.
- F Re-improvements: Iterative improvement applies an initial no-feedback step followed by a feedback-informed step.
- F Re-improvements: The two-stage design aims to combine targeted feedback corrections with broader stylistic enhancements.
- F Re-improvements: The second feedback-informed revision is compared with both the single-stage no-feedback baseline and a second no-feedback revision.
- F Re-improvements: The judge prefers the two-stage feedback-informed response over the single-stage baseline, then reverses preference between the two two-stage methods.
G Initial Results with Another Improver
Initial GPT-OSS-20B experiments compare response improvements made with and without feedback on synthetic samples. The results report consistent positive deltas for feedback-informed issue resolution and evaluate the variants with pairwise judges.
- G Initial Results with Another Improver: About 400 synthetic samples were used for initial GPT-OSS-20B results.These results are reported in Tables 14 and 15.
- G Initial Results with Another Improver: Feedback-informed and context-only revisions were generated using separate improvement prompts.The feedback variant uses specific human feedback, whereas the baseline relies solely on conversation context.
- G Initial Results with Another Improver: Reimprovement results favor two-stage with-feedback improvement over the single-stage baseline, but the preference reverses against two-stage without-feedback improvement.This comparison is reported in Table 13.
- G Initial Results with Another Improver: Table 14 reports a consistent positive delta between without-feedback and with-feedback variants on hard prompts.The table measures the percentage of improved responses successfully addressing the issues.
- G Initial Results with Another Improver: The pairwise judge comparison averages results across corruption types and identifies cases where only the with-feedback variant addressed the issue.The WFShouldWin column denotes this subset of cases.