PaperlayerView exact revision on arXiv

Source-linked AI summary

RefineBench: Evaluating Refinement Capability of Language Models via Checklists

Young-Jun Lee, Seungone Kim, Byung-Kwan Lee, Minkyeong Moon, Yechan Hwang, Jong Myoung Kim, Graham Neubig, Sean Welleck, Ho-Jin Choi

arXiv:2511.22173v1cs.CL

TL;DR

Prior refinement evaluations have largely centered on verifiable tasks, despite refinement requests arising in real-world interactions and varying in how much feedback they provide. REFINEBENCH evaluates self-, guided, and partially guided refinement with checklists across diverse tasks; most models struggle to improve unaided, while guided refinement produces substantially stronger results. The findings support using the benchmark to study and improve refinement, while leaving open how results vary across evaluation conditions.

  • Problem

    Prior refinement studies have largely examined mathematics or code, while real user requests include free-form tasks and both explicit and unspecified correction targets, leaving LM refinement capability unresolved.

    Source: §1 · p. 1

  • Method

    REFINEBENCH evaluates 1,000 problems across 11 domains with checklist-based assessment in self-refinement, guided refinement, and partially guided multi-turn settings.

    Source: §1 · p. 2, §3.2 · p. 5, §6 · p. 11

  • Results

    Most models show little self-refinement improvement across five turns, while most proprietary and 70B-or-larger open-weight models score above 90.0 by guided-refinement turn five.

    Source: §1 · p. 2

  • Takeaways & Limitations

    REFINEBENCH provides a testbed for tracking and improving refinement capabilities, and the tested results indicate substantial room for improvement in self-refinement.

    Source: §1 · p. 3

    The observed trends do not establish that LMs lack self-refinement ability, since results may vary with domain, task difficulty, prompt scaffolding, and inference configuration.

    Source: B LIMITATIONS · p. 17

Abstract

from arXiv · show

Can language models (LMs) self-refine their own responses? This question is increasingly relevant as a wide range of real-world user interactions involve refinement requests. However, prior studies have largely tested LMs' refinement abilities on verifiable tasks such as competition math or symbolic reasoning with simplified scaffolds, whereas users often pose open-ended queries and provide varying degrees of feedback on what they desire. The recent advent of reasoning models that exhibit self-reflection patterns in their chains-of-thought further motivates this question. To analyze this, we introduce RefineBench, a benchmark of 1,000 challenging problems across 11 domains paired with a checklist-based evaluation framework. We evaluate two refinement modes: (1) guided refinement, where an LM is provided natural language feedback, and (2) self-refinement, where LMs attempt to improve without guidance. In the self-refinement setting, even frontier LMs such as Gemini 2.5 Pro and GPT-5 achieve modest baseline scores of 31.3% and 29.1%, respectively, and most models fail to consistently improve across iterations (e.g., Gemini-2.5-Pro gains only +1.8%, while DeepSeek-R1 declines by -0.1%). By contrast, in guided refinement, both proprietary LMs and large open-weight LMs (>70B) can leverage targeted feedback to refine responses to near-perfect levels within five turns. These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses, and that RefineBench provides a valuable testbed for tracking progress.

1 INTRODUCTION

The introduction motivates broader evaluation of language-model refinement and presents REFINEBENCH, a checklist-based benchmark spanning diverse tasks and feedback settings. Its central finding is that self-refinement remains limited, while guided refinement yields much stronger results.

1 INTRODUCTION

  • Prior refinement evaluation has focused largely on mathematics or code, while real-world requests include both explicit correction requests and revisions without specified problems.About 10.24% of 159,134 WildChat queries requested some form of refinement.Source: §1 · p. 1
  • REFINEBENCH covers free-form and correctness-evaluated tasks across 11 domains, using checklist items to assess self-refinement and guided refinement.Guided feedback identifies unfilled checklist items; self-refinement supplies no specific guidance.Source: §1 · p. 2
  • 31.3 points for Gemini 2.5 Pro and 29.1 points for GPT-5 contrast with scores above 90.0 by turn five for most proprietary and 70B-or-larger open-weight models under guided refinement.Claude-Opus-4.1 reaches 98.4% at turn five, a +79.7% improvement.Source: §1 · p. 2
  • Most models do not substantially improve through five-turn self-refinement; DeepSeek-R1 declines by -0.1% across five turns, while proprietary reasoning models range from -0.8% to 2.6%.Reasoning models typically self-refine better than instruction-tuned models, but their capabilities remain limited.Source: §1 · p. 2
  • Self-refinement outcomes vary by domain: the Law domain shows non-trivial improvement for Claude-Opus-4.1 and Gemini-2.5-Pro.Claude-Sonnet-4 shows only +0.8% improvement over five turns on REFINEBENCH despite refining effectively on AIME-24.Source: §1 · pp. 2–3
  • REFINEBENCH is intended as a testbed for improving and evaluating refinement, and the results point to substantial room for improvement in frontier and smaller open-weight models.The introduction reports that smaller open-weight models cannot perform either self-refinement or guided refinement in the tested settings.Source: §1 · p. 3

2 RELATED WORK

The related work distinguishes REFINEBENCH from research that trains refinement methods or uses refinement as a proxy for critique. It positions the benchmark as a checklist-based, multi-turn assessment with controllable feedback and broader task coverage.

2.1 REFINEMENT CAPABILITY IN LMS

  • Earlier work proposes refinement algorithms, trains critic models, or studies task-specific feedback; REFINEBENCH instead assesses refinement capability and lets practitioners control feedback amount, including in free-form tasks.The cited work also notes analyses suggesting that self-refinement may fail without sophisticated feedback.Source: §2.1 · p. 3

2.2 REFINEMENT BENCHMARKS FOR LMS

  • Prior refinement benchmarks largely use refinement as a proxy for critique quality and focus on extrinsic refinement, while none combines checklist-based intrinsic and extrinsic refinement across verifiable and non-verifiable tasks.CriticBench, CriticEval, and RealCritic are described as assessing critique and correction or the downstream effectiveness of critique.Source: §2.2 · p. 3

2.3 MULTI-TURN EVALUATION BENCHMARKS FOR LMS

  • REFINEBENCH focuses multi-turn evaluation on refinement and uses a unified checklist framework to measure progressive improvement, unlike prior benchmarks that do not adequately capture this across turns.The related work also surveys multi-turn benchmarks for follow-up questions, instruction following, and interactive agent or social scenarios.Source: §2.3 · pp. 3–4

3 REFINEBENCH

REFINEBENCH is a multi-turn benchmark for evaluating guided and self-refinement with a unified checklist framework. It combines broad dataset coverage with a protocol that varies feedback availability and analysis of the benchmark’s composition.

3 REFINEBENCH

  • REFINEBENCH provides a unified checklist-based benchmark for evaluating guided and self-refinement across multiple turns.The paper presents it as the first benchmark designed to comprehensively evaluate these capabilities.Source: §3 · p. 4

3.1 DATASET CONSTRUCTION PROCESS

  • Source problems come from academic materials including university humanities prompts, California Bar questions, Stanford math and statistics problems, and Humanity’s Last Exam.Some humanities and social-science problems include plots or tables, which are converted into textual descriptions or markdown and manually checked.Source: §3.1 · pp. 4–5
  • Checklist items are generated with advanced LMs when source problems lack them, then manually reviewed and refined; 1.1% are filtered, and experts judge 96.1% appropriate.Filtering uses backtranslation with reference answers; the expert evaluation covers 854 checklist items across 100 samples.Source: §3.1 · p. 5

3.2 EVALUATION PROTOCOL

  • The protocol repeats refinement, binary checklist evaluation, and feedback for up to five turns, using GPT-4.1 as evaluator.Self-refinement provides no explicit feedback; guided refinement provides failed checklist items, while partially guided refinement reveals only a subset.Source: §3.2 · p. 5

3.3 ANALYSIS OF REFINEBENCH

  • 1,000 problems average 9.9 binary checklist questions each and span 11 domains, 239 subjects, and free-form and exact-match tasks.Table 1 presents basic statistics for the benchmark.Source: §3.3 · p. 6, Table 1 · p. 6
  • Math is the largest domain at 32%, followed by Humanities/Social Science at 19% and Law at 14%.Figure 3 presents the distribution of domain categories.Source: §3.3 · p. 6, Figure 3 · p. 6
  • Compared with existing datasets, REFINEBENCH combines controllable checklist feedback, verifiable and free-form tasks, 11 domains, and multi-turn evaluation.Its free-form tasks include longer responses across varied domains.Source: §3.3 · p. 6, Table 2 · p. 6

4 EXPERIMENTS

The experiments compare 34 frontier LMs and find that self-refinement remains weak, whereas explicit guidance enables near-perfect refinement for many models.

4.1 EXPERIMENTAL SETUP

  • The 34 frontier LMs span open-source and proprietary instruction-tuned and reasoning-model categories.Source: §4.1 · pp. 6–7
  • Evaluation reports Acct as the percentage of correct checklist items and Passt as strict all-items-correct success, averaging scores across instances.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 31.3% was Gemini 2.5 Pro’s best self-refinement score; most models’ average improvement (∆) ranged from -2.5% to 0%, while proprietary reasoning models gained 0–2.6%.Overall self-refinement performance remained below 32%.Source: §4.2 · p. 7
  • With explicit guidance, Claude-Opus-4.1 reached 94.3% at t = 3 and o3-mini reached 98.2% at t = 5, while smaller LMs were an exception.Source: §4.2 · p. 7
  • Qwen3-30B-A3B-Thinking improved +1.4% while its instruction-tuned counterpart declined –1.6%; o1’s –0.2% decline was smaller than GPT-4o’s –1.4%.Source: §4.2 · p. 7
  • DeepSeek-R1 declined –0.1% and DeepSeek-R1 (Qwen-32B) declined –2.5%, against a backdrop of notably low DeepSeek self-refinement.Source: §4.2 · p. 7

5 DISCUSSIONS AND ANALYSIS

The analysis separates difficulty identifying what to revise from difficulty applying a revision, then examines how refinement varies with feedback, turn-taking, domain, and reasoning behavior. Explicit criteria can unlock substantial gains, but unprompted multi-turn refinement remains inconsistent.

5.1 WHY DO LMS STRUGGLE WITH SELF-REFINEMENT?

  • At t = 5, providing evaluation criteria improves LLaMA-3.1-70B-Instruct by +43.6% and Gemini 2.5 Pro by +44.5% over self-refinement without criteria.The criteria identify which elements need revision without specifying how to revise them.Source: §5.1 · p. 8, Table 4 · p. 9
  • With feedback on 50% of checklist items, LMs incorporate the supplied feedback but still struggle to refine aspects left unspecified.This partial-feedback comparison distinguishes using identified targets from independently identifying what else needs revision.Source: §5.1 · p. 8, Figure 4 · p. 9

5.2 IN-DEPTH ANALYSIS

  • Gemini 2.5 Pro refines better when it reasons longer, but this trend remains similar across turns rather than increasing dramatically with more refinement steps.The authors conclude that merely increasing refinement steps is insufficient.Source: §5.2 · p. 9, Figure 5 · p. 9
  • Most models terminate self-refinement around turns 3–4; proprietary reasoning LMs stop earlier, and longer termination is negatively correlated with Pass5 (p-value < 0.01; R2 = −0.477).Open-source reasoning LMs generally perform worse than proprietary ones, despite tending to continue refinement longer.Source: §5.2 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9
  • Self-refinement varies by domain: Claude-Opus-4.1 gains +7.8 and Gemini-2.5-Pro +5.0 in Law, while STEM gains range from -1.2 to +2.5.GPT-5 shows the opposite pattern, refining poorly in Law but performing well in Math & Statistics.Source: §5.2 · pp. 9–10, Figure 5 · p. 9
  • For Gemini-2.5-Pro, evaluation costs per sample are $0.038 for self-refinement and $0.028 for guided refinement, with respective latencies of 51.1 and 22.9 seconds.The authors describe the evaluation protocol as practically usable.Source: §5.2 · p. 10

5.3 WHY DOES THE DEEPSEEK SERIES SHOW DECREASING TRENDS IN SELF-REFINEMENT?

  • DeepSeek-R1’s reasoning tokens drop by 69.7% after the initial turn, while identified reasoning patterns, especially self-correction and verification, diminish from turn two onward.By turn three, its reasoning focuses more on checking the appropriateness and consistency of its prior answer than exploring improved pathways.Source: §5.3 · pp. 10–11, Figure 8 · p. 10
  • For DeepSeek-R1, correct-answer retention falls from 42.7% at 1→2 to 25.0% at 2→3, while incorrect-to-correct transitions rise only from 1.4% to 8.1%.In later transitions, incorrect answers remain dominant at over 64%, while correct-answer retention stays around 32–34%.Source: §5.3 · p. 11, Figure 9 · p. 11

6 CONCLUSION

The conclusion presents REFINEBENCH as a testbed for evaluating refinement and reports a contrast between limited gains from self-refinement and improvement when feedback is provided. It also identifies future research directions and describes the benchmark’s ethical and licensing considerations.

6 CONCLUSION

  • REFINEBENCH evaluates self-refinement and guided refinement on 1,000 problems across 11 domains using checklist-based evaluation.Most LMs do not significantly benefit from five turns of self-refinement, whereas most improve by incorporating provided feedback.Source: §6 · p. 11
  • The authors propose using REFINEBENCH to study self-refinement improvements and exploring training methods for reasoning models that repeatedly fix only initially addressed aspects.They also suggest checklist-based reward functions as a possible direction for helping models identify which items to refine.Source: §6 · p. 11

A BROADER IMPACT

  • The benchmark includes subjective Law and Humanities/Social Sciences tasks, for which the authors secured research-only permissions and released the benchmark under CC-BY-NC-ND.They report that the benchmark generally avoids sensitive social concepts such as social norms, gender, and ethical biases.Source: A BROADER IMPACT · p. 17

B LIMITATIONS

The paper qualifies its findings as specific to the tested benchmark and describes limitations in its analysis of reasoning behavior. It also reports benchmark statistics, category-granularity choices, and a check of whether checklist items capture distinct aspects.

B LIMITATIONS

  • The benchmark’s observed trends do not establish that LMs lack self-refinement; outcomes may vary with domain, task difficulty, prompt scaffolding, or inference configuration.The authors say they checked whether their checklist, LLM-as-a-Judge pipeline, and prompting function as intended.Source: B LIMITATIONS · p. 17
  • The DeepSeek-R1 reasoning analysis relies mainly on keyword searches, and the authors say more precise analysis is needed.They also report a 69.7% reduction in average reasoning tokens between turns one and two.Source: B LIMITATIONS · p. 17

C USE OF LARGE LANGUAGE MODELS

D.1 DETAILED STATISTICS OF REFINEBENCH

  • Humanities problems include 316 extensive passages averaging 413.81 tokens, and often require interpreting descriptions of visual and tabular data.The authors characterize REFINEBENCH as combining verifiable and non-verifiable tasks, long-context passages, and non-textual information.Source: §D.1 · p. 17, Table 5 · p. 18

D.2 DETAILED CATEGORY DISTRIBUTION

  • Mathematics and Statistics use broad categories that follow source classifications; the authors plan finer-grained re-annotation using GPT-4.1.The Humanities/Social Science domain contains substantially more unique subject categories than other domains.Source: §D.2 · pp. 17–18, Table 6 · p. 19

D.3 DOES THE CHECKLIST REALLY EVALUATE REFINEMENT CAPABILITY FROM INDIVIDUAL ASPECTS?

  • ROUGE-L results indicate that checklist items do not overlap, supporting the authors’ claim that they capture diverse, distinct aspects of refinement.The checklists were manually verified but partially created with LLM assistance.Source: §D.3 · p. 18, Figure 10 · p. 18

D.4 IS THERE A CONCERN ABOUT CONTAMINATION ISSUE IN REFINEBENCH?

The authors report negligible contamination rates and describe permissions and licensing for benchmark materials. They also provide experimental settings and an evaluation implementation intended to support researchers’ model evaluations.

D.4 IS THERE A CONCERN ABOUT CONTAMINATION ISSUE IN REFINEBENCH?

  • Min-K% Prob tests on LLaMA2-13B found contamination rates of 0.1% for questions and 0.5% for reference answers.The authors conclude that contamination is negligible.Source: §D.4 · pp. 19–20

D.5 COPYRIGHTS AND LICENSE

  • The benchmark is released under CC-BY-NC-ND for research use, with permissions obtained for specified institutional problem sets; accompanying code and prompts use the MIT license.The benchmark license restricts redistribution and modification.Source: §D.5 · p. 20

E.1 ADDITIONAL EXPLANATION OF EXPERIMENTAL SETUP

  • Experiments used two NVIDIA A100 GPUs, specified generation settings, and vLLM and OpenRouter; the evaluation code also supports LiteLLM and asynchronous evaluation.The authors describe the framework as user-friendly and intend to maintain compatibility with newly released models.Source: §E.1 · p. 20

E.2 FULL RESULTS OF SELF-REFINEMENT PERFORMANCE

The appendix reports full self-refinement and guided-refinement results using Pass_t and Acc_t. For self-refinement, it defines Δ as the average improvement from Pass1 to Pass5.

E.2 FULL RESULTS OF SELF-REFINEMENT PERFORMANCE

  • Table 7 reports full self-refinement results using Pass_t and Acc_t, with Δ defined as average improvement from Pass1 to Pass5.Source: §E.2 · p. 20, Table 7 · p. 21

E.3 FULL RESULTS OF GUIDED REFINEMENT PERFORMANCE

  • Table 8 reports full guided-refinement results using Pass_t and Acc_t.Source: §E.3 · p. 21

E.4 DETAILS OF HUMAN EVALUATION

The human evaluation assessed checklist appropriateness and whether responses satisfied checklist items. Annotators judged 96.1% of checklist items appropriate overall.

Human Evaluation

  • Six Ph.D.-level domain experts per question evaluated 100 samples, judging checklist appropriateness and response satisfaction with selected items.Each sample paired a randomly selected checklist item with a response randomly drawn from 18 models.Source: §E.4 · p. 21
  • The two annotations were intended to validate checklist effectiveness and the reliability of the checklist-based evaluator, respectively.Source: §E.4 · pp. 21–22

Checklist Quality Assessment

  • 96.1% of checklist items (821 out of 854) were judged appropriate overall.Appropriateness ranged from 100% in Chemistry (95 out of 95 items) to 89.1% in Engineering (82 out of 92 items).Source: §E.4 · pp. 22–23, Figure 11 · p. 23

E.5 META-EVALUATION RESULT

GPT-4.1 achieved the highest agreement with human judgments among six evaluator LMs, supporting its use as the primary evaluator.

E.5 META-EVALUATION RESULT

  • 90% agreement with human judgments made GPT-4.1 the top-performing evaluator among six LMs and supported its use as the primary evaluator.Source: §E.5 · p. 23

E.6 ADDITIONAL ANALYSIS

The additional analysis examines the relationship between answer correctness and problem-solving process quality, alongside reasoning-model transitions during self-refinement.

Answer Correctness and Process Quality

  • Correct answers tended to accompany proper problem-solving processes, while incorrect answers still showed partial evidence of valid reasoning.Source: Figure 12 · p. 23, §E.6 · p. 23

Transition Analysis

  • Figure 13 extends transition analysis to four DeepSeek-R1-Distill-Qwen models in the self-refinement setting.The models are the 1.5B, 7B, 14B, and 32B variants.Source: §E.6 · p. 23

E.7 ANALYSIS OF WILDCHAT

The supplementary passages describe how refinement-related WildChat queries are classified, note a gap in vision-and-language evaluation, and provide prompts for refinement and checklist-based evaluation.

E.7 ANALYSIS OF WILDCHAT

  • WildChat analysis uses LLaMA-3.1-8B-Instruct on English instances to classify refinement queries as unguided self-refinement or requests specifying what to revise.Figure 13 presents transition analysis for DeepSeek-R1-Distill-Qwen models at 1.5B, 7B, 14B, and 32B.Source: §E.7 · pp. 23–24, Figure 13 · p. 24

F EXTENDED RELATED WORKS

  • Existing multi-turn vision-and-language benchmarks are fewer than language-model benchmarks, and none explicitly evaluate self-refinement; extending them is proposed as future work.Source: F EXTENDED RELATED WORKS · p. 25

G.1 PROMPT TEMPLATE FOR USER QUERY IN REFINEBENCH

  • Guided refinement converts incorrect checklist items into feedback, while the prompt lets the model terminate when it judges no refinement remains.The response should not be formatted as answers to individual criteria.Source: §G.1 · p. 25

G.2 PROMPT TEMPLATE FOR EVALUATION IN REFINEBENCH

  • Checklist-based evaluation asks a model to answer Yes or No for each item about a response, without additional explanations.Source: §G.2 · pp. 25–26

G.3 PROMPT TEMPLATE FOR CHECKLIST CREATION IN REFINEBENCH

  • Checklist creation derives detailed, specific, minimally correlated questions from a query and reference answer, with item count calibrated to problem difficulty.Items should capture generally applicable elements of a good answer rather than merely check whether reference-answer content appears.Source: §G.3 · p. 26

H REFINEBENCH EXAMPLES

This section presents illustrative RefineBench problems across technical and nontechnical domains, pairing problems with checklists or reference answers. The examples range from proofs, scientific identification, programming, and communications to measurement calculations, evidence integration across passages, and legal analysis.

Statistics

  • If Yn→Y almost surely and νk→∞ almost surely, then Yνk→Y almost surely; but with Yn=1/n→0 almost surely, independent νk can diverge in probability while Yνk fails to converge almost surely.In this counterexample, P(νk=1)=1/k and P(νk=k)=1−1/k; Yνk=1 infinitely often almost surely.Source: H REFINEBENCH EXAMPLES · p. 27

Biology/Medicine

  • The Biology/Medicine example identifies triplets 1, 4, and 8 as the three whose mimicry or signaling terms, species, and traits are related.The reference answer explains the relationships using distraction displays, Gilbertian mimicry, and aposematism.Source: H REFINEBENCH EXAMPLES · pp. 28–29

Chemistry

  • The Chemistry example infers the structure of X as C5H6=CH-CH2-CO-CH2-CHO from combustion, mass-fraction, reaction, and spectroscopic clues.The reference answer reasons from a C10H12O2 formula and ozonolysis yielding two identical fragments.Source: H REFINEBENCH EXAMPLES · p. 30

Computer Science/AI

  • The Computer Science/AI example optimizes an X++ interpreter by reading only the operation character and estimates a worst-case runtime of 18800 ms.The proposed approach processes 121 statements, recognizing one symbol per statement rather than comparing characters.Source: H REFINEBENCH EXAMPLES · pp. 32–33

Engineering

  • The Engineering example applies QPSK mapping, modified Alamouti coding, phase rotation, and channel gain to derive transmitted symbols for both antennas and time slots.The supplied derivation expresses the resulting symbols using the factor 0.8 × e^(j7π/24).Source: H REFINEBENCH EXAMPLES · p. 34

Additional example

  • A further probability example models escape from a random walk with melting and escape sites, deriving that the escape-probability function is linear outside those sites and constant on each outer interval.The supplied excerpt ends before giving the requested conditional probability.Source: H REFINEBENCH EXAMPLES · pp. 35–36

Mathematics

  • The mathematics example’s checklist evaluates defining and modeling escape probability, accounting for jump probabilities, and solving for the probability from bin 0.Criteria include recognizing linearity or constancy away from the portal and torch and solving equations for relevant bin intervals.Source: H REFINEBENCH EXAMPLES · p. 36

Physics

  • The physics example asks for the exposure needed to measure a magnitude-20 star’s B-band magnitude to ±0.01 accuracy; the reference answer gives 108 seconds.The prompt specifies a 1-meter telescope and requests rounding to the nearest integer.Source: H REFINEBENCH EXAMPLES · p. 38

Humanities/Social Science

  • The humanities and social science example asks readers to connect evidence on social mobility, inequality, education, meritocracy, and Korea’s changing prospects.The reference answer links inequality to lower intergenerational mobility, notes unequal access to top universities, and cautions that meritocracy does not necessarily eliminate inequality.Source: H REFINEBENCH EXAMPLES · pp. 40–41

Law

  • The law example’s reference answer analyzes Belle’s equitable remedies for obtaining Parcel 1 and legal damages if she cannot, including $150,000 in expectation damages.It discusses a temporary restraining order, preliminary injunction, contract reformation, and specific performance, then considers expectation, reliance, and restitution damages.Source: H REFINEBENCH EXAMPLES · p. 43

Related papers

Loading 2511.22173v1…