PaperlayerView exact revision on arXiv

Source-linked AI summary

Seq@K: Evaluating Language Models Under Sequential Feedback

Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong

arXiv:2610.37859v1cs.AI

TL;DR

Pass@K does not capture how models use feedback, despite iterative refinement being common in agent workflows. Seq@K measures success within K sequential attempts with feedback after failures, and the paper finds that model rankings and cost-performance frontiers can differ under this evaluation.

  • Problem

    Pass@K is feedback-blind, although agent workflows commonly involve iterative refinement from feedback.

    Source: §1 · p. 1

  • Method

    Seq@K evaluates whether a model solves a task within K attempts, receiving feedback after each failed attempt.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Model rankings, actor gaps, and cost-performance frontiers can differ between Seq@K and Pass@K.

    Source: §6 · p. 11

  • Takeaways & Limitations

    Practitioners building feedback-loop agent systems should choose models using Seq@K and evaluate how feedback and history handling affect refinement.

    Source: §6 · p. 11

    A numerical Seq@K value requires specifying the feedback mode, benchmark split, verifier, prompt builder, decoding settings, and attempt budget.

    Source: §3 · p. 3

Abstract

from arXiv · show

Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively refine their answers in response to feedback, a prevalent mode of operation in modern agentic workflows. However, the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts. We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure. Unlike multi-turn agentic benchmarks that examine long-horizon planning, Seq@K isolates and formalizes the core capability of refinement from feedback. We evaluate Seq@K on a suite of 7 benchmarks and 16 frontier models, and show that Seq@K reveals a materially different model-selection landscape: under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K. We further show that analyzing model trajectories under Seq@K helps characterize iterative refinement behavior, such as the rate at which models address issues raised in feedback, or how rigidly they hold onto their own answers. Finally, we study how the nature of feedback affects models' iterative refinement, showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems. Together, these results establish how Seq@K offers a more faithful metric for intelligent systems that typically operate in such feedback loop, and the necessity to consider Seq@K in evaluating models for these systems.

1 INTRODUCTION

Current evaluations center on feedback-blind Pass@K even though iterative feedback is common in agent workflows. Seq@K measures solving a task within K attempts with feedback after each failure, and the paper examines how this changes model selection, revision analysis, and feedback-loop diagnosis.

1 INTRODUCTION

  • Pass@K does not measure whether agents improve through feedback, a limitation that can hide failure modes and undervalue models that improve across iterations.The introduction frames proposal, targeted critique, and revision as a dominant pattern of agent use.Source: §1 · pp. 1–2
  • Seq@K measures whether a model solves a task within K attempts, receiving feedback after each failed attempt.The paper evaluates the metric across coding, instruction-following, domain-reasoning, and medical benchmarks.Source: §1 · p. 2
  • Model rankings and cost-performance frontiers can differ between Pass@K and Seq@K, and Pass@K can understate or overstate capability differences under sequential feedback.The paper presents Seq@K as a calibration metric for more cost-efficient model selection.Source: §1 · p. 2
  • Revision trajectories reveal when models gain, whether they address feedback, and whether revisions break requirements that earlier attempts met.Some models gain mainly on their first retry, while others gain on later attempts.Source: §1 · p. 2
  • Feedback usefulness depends on the benchmark and model capability; rubric-informed critiques tend to help repair more failed requirements, while summarizing prior attempts can lower success.The paper argues that history handling should be reported alongside Seq@K.Source: §1 · p. 2

2 RELATED WORK

Prior work studies model refinement from critique or execution feedback and evaluates agents in long interactions. Seq@K instead isolates revision from feedback as an evaluation metric on existing benchmarks and settings.

Refinement from feedback

  • Prior refinement work studies models reasoning and revising in response to critique or execution feedback, including Reflexion’s use of binary pass/fail signals or self-generated tests.Other evaluations include RefineBench’s checklist-based tasks and ICPC-Eval’s Refine@K for execution feedback.Source: §2 · p. 2
  • Rather than constructing new tasks, Seq@K applies to existing benchmarks and settings to reveal model-ranking and cost-performance-frontier insights.The paper positions Seq@K as a metric to report alongside Pass@K where iterative refinement is expected in practice.Source: §2 · p. 2

Multi-turn and agentic evaluation

  • Long-horizon agent and multi-turn benchmarks jointly test planning, tool use, state tracking, and feedback use, whereas Seq@K isolates feedback-driven revision rather than environment navigation.The related-work discussion names SWE-bench, WebArena, τ-bench, and MINT as examples.Source: §2 · pp. 2–3

3 SEQ@K EVALUATION PROTOCOL

Pass@K evaluates independent candidates drawn from the original prompt, while Seq@K evaluates trajectories in which attempts can use prior responses and feedback. A Seq@K result depends on the feedback mode and other protocol choices.

Pass@K and Seq@K

  • Both metrics measure success within an attempt budget, but Pass@K uses independent draws whereas Seq@K uses feedback-conditioned revisions.The evaluation therefore distinguishes static-prompt sampling from sequential refinement.Source: §3 · p. 3

Pass@K

  • Pass@K estimates benchmark success when at least one of K independently sampled candidates from the original prompt passes the verifier.Because verifier outcomes are binary, the maximum over attempts indicates whether any candidate succeeded.Source: §3 · p. 3

Seq@K

  • Seq@K generates attempts iteratively, conditioning each revision on the original prompt and the interaction history of prior responses and feedback.A failed attempt returns feedback according to the selected mode; interaction ends early after success.Source: §3 · p. 3

Feedback Modes and Protocol Dependencies

  • A numerical Seq@K value is defined only after specifying the feedback mode, benchmark split, verifier, prompt builder, decoding settings, and attempt budget.Seq@K is a family of evaluations indexed by feedback mode.Source: §3 · p. 3

4 EXPERIMENTAL DESIGN

The experimental design covers seven benchmark variants and compares independent sampling with three sequential feedback modes. Feedback modes differ in the information actors receive after failure, including whether critiques have access to grading rubrics.

Experimental coverage

  • The study evaluates seven benchmark variants spanning instruction following, rubric-graded generation, and executable code synthesis.Actor, task-split, and protocol configurations are drawn from a central collection, with matching criteria and remaining differences specified in the appendix.Source: §4 · p. 4
  • The coverage table summarizes actor coverage, run counts, and the Results sections associated with each benchmark variant.Its Actors column counts distinct models evaluated across benchmarks.Source: Table 1 · p. 4

Feedback modes

  • The design compares independent parallel sampling with binary, raw-verifier, and LLM-critic feedback after failed attempts.LLM critics may be the actor itself or another model, and their access to underlying grading rubrics is varied.Source: §4 · p. 4
  • Feedback modes set the information boundary for revision, from direct test diagnostics to rubric-aware guidance.The diagnostics can include stack traces, failed assertions, or rubric-item breakdowns.Source: §4 · p. 4

5 RESULTS

The results show that sequential feedback changes model selection and cost–success trade-offs, while revealing revision behaviors and feedback conditions that shape iterative success. Across benchmarks, the value of sequential evaluation depends on the actor, feedback setup, and how prior attempts are retained.

5.1.1 MORE PERFORMANT AND EFFICIENT MODEL SELECTION WITH SEQ@K

  • Seq@3 or Seq@4 calibration picks better actors for sequential deployment than Pass@5 at similar or lower cost, while model rankings can also shift between protocols.On AdvancedIF, Sonnet moves from the bottom rank under Pass@5 to the top tied group under sequential feedback.Source: §5.1.1 · pp. 4–5, Figure 1 · p. 5, Figure 2 · p. 6

5.1.2 PASS@K CAN OVERSTATE OR UNDERSTATE PERFORMANCE GAPS

  • Pass@K can understate or overstate actor-performance gaps: on AdvancedIF, the best–worst gap is 33.3 points under Pass@5 but 6.7 under Seq@5.On HumanEval Pro, the gap instead widens from 10.8 to 51.7 points under Seq@5.Source: §5.1.2 · p. 6, Table 3 · p. 6

5.1.3 SEQ@K REVEALS BETTER PARETO-OPTIMAL CHOICES

  • Seq@K often yields a better cost–success Pareto frontier than Pass@K; on DKR, independent sampling solves at most about 14% of tasks, versus 50% with Opus 4.7 at K = 5.The comparison covers AdvancedIF and DKR, with K = 1–5.Source: Figure 3 · p. 7, §5.1.3 · p. 7

5.1.4 SEQ@K TRACKS A DISTINCT CAPABILITY SIGNAL FROM PASS@K

  • Across 26 model–benchmark pairs, Pass@5 and Seq@5 correlate at +0.68, but after removing the first attempt, the correlation between gains falls to −0.08.The authors interpret retry gains from independent sampling and feedback as distinct capability signals.Source: §5.1.4 · p. 7, Figure 4 · p. 8

5.2.1 REVISION TIMING DISTINGUISHES FRONT-LOADED FROM DISTRIBUTED GAINS

  • Revision gains vary by turn: Gemini 3 Flash gains most on the first retry, while non-zero LGS for Opus 4.7 and DeepSeek V4 Pro indicates later gains.EGS measures the share of improvement on the first retry; LGS measures the share on the final allowed attempt.Source: §5.2.1 · p. 8, Table 4 · p. 8

5.2.2 REVISIONS DO NOT ALWAYS ADDRESS THE FEEDBACK

  • Across 842 revisions, 22% were unhelpful, 34% addressed some feedback, and 44% addressed all; even fully addressed feedback can leave prior criteria broken, especially in DKR and AdvancedIF.In coding, Qwen3 Coder Flash repeated an unchanged attempt 48% of the time, while DeepSeek V3.2 did so 6% of the time.Source: §5.2.2 · p. 9, Figure 5 · p. 10

5.3.1 EFFECTIVENESS OF BINARY FEEDBACK DEPENDS ON THE BENCHMARK

  • In DKR, rubric-informed critiques repair more failed criteria than binary feedback or critiques without the rubric, which perform about like independent retries.Binary feedback also varies by model and benchmark: it raises Sonnet 4.6’s Seq@5 by 17–26 points in coding, but by 1 point on average in DKR.Source: §5.3.1 · p. 10, Table 5 · p. 11

5.3.2 SUMMARIZING THE HISTORY CAN LOWER SEQUENTIAL SUCCESS

  • Replacing prior attempts with a summary changes performance little on AdvancedIF and coding but lowers success on other benchmarks, most consistently HealthBench.The authors therefore identify history representation as part of the Seq@K protocol that should be reported.Source: §5.3.2 · pp. 10–11, Table 6 · p. 11

6 DISCUSSION AND CONCLUSION

Seq@K evaluates whether models solve tasks within K attempts while receiving feedback after each failure. The paper concludes that its rankings, trajectories, and feedback-loop analyses provide information relevant to selecting models for systems that revise from feedback.

6 DISCUSSION AND CONCLUSION

  • Model rankings, actor gaps, and cost-performance frontiers can differ between Seq@K and Pass@K, so Pass@K may select a model that uses feedback poorly.The paper recommends choosing models with Seq@K under the feedback the agent will receive.Source: §6 · p. 11
  • Seq@K trajectories reveal when models gain and how revisions can fail by ignoring feedback or breaking previously satisfied requirements.Feedback-loop performance also depends on what feedback says, whether the critic knows the grading criteria, and how prior attempts are retained.Source: §6 · p. 11

A BENCHMARK AND SETUP DETAILS

The appendix documents the benchmarks, feedback and grading setup, coding prompts, and task screening used in the evaluation. It also points to further details of the model-selection test and actor gaps.

A BENCHMARK AND SETUP DETAILS

  • Table 7 lists the benchmarks and their success criteria; the appendix provides benchmark definitions, screening procedures, and decoding parameters.The appendix also identifies which Results sections use the benchmark variants.Source: Table 7 · p. 14, A BENCHMARK AND SETUP DETAILS · p. 14

A.1 FEEDBACK, HISTORY, AND GRADING

  • Sequential-attempt history normally includes the task, attempt number, and all earlier attempts with feedback; summarized-history runs replace those attempts with a summary.AdvancedIF is graded by Sonnet 4.6 and CLBench by GPT-5.4, with GPT-5.2 as fallback; main comparisons use T = 0.7 without temperature tuning.Source: §A.1 · p. 14

A.2 CODING PROMPTS AND FEEDBACK

  • Coding Pass@K uses an unchanged prompt for all K attempts, whereas Seq@K adds attempt numbering and earlier attempts with feedback and stops at the first passing attempt.The task prompt omits reference solutions and tests; test-detail feedback can show up to two failed tests, with fields truncated at 300 characters.Source: §A.2 · pp. 14–15

A.3 EXCLUDED CODING TASKS

  • Fifteen of 100 coding tasks name no function that their tests call, and coding comparisons exclude them, leaving 45 MBPP Pro and 40 HumanEval Pro tasks.The history comparison instead uses all 50 tasks from each benchmark; the screening may leave other kinds of defects undetected.Source: §A.3 · p. 15

B MODEL SELECTION DETAILS

B.1 SELECTION TEST

The appendix details a model-selection test, actor-gap analysis, and protocol cost comparison. Seq@3 and Seq@4 calibration yielded positive gains over Pass@5 in the reported comparisons, with costs varying by benchmark and setting.

B.1 SELECTION TEST

  • Four candidate actors ran three repetitions of Pass@5 and test-detail Seq@5 at T = 0.7 on 45 MBPP Pro and 40 HumanEval Pro tasks.The actor with the highest calibration-task success was evaluated on separate tasks; calibration loss measures the difference from the best actor on those evaluation tasks.Source: §B.1 · p. 16
  • Calibration cost sums API charges across actors, calibration tasks, and repetitions, while task splits and resampling were used to compare calibration metrics on shared tasks.The reported mean cost has no per-calibration-set spending cap; confidence intervals resample tasks and rerun selection over random calibration-task orders.Source: §B.1 · p. 16
  • Seq@3’s gain over Pass@5 remained positive under all 20 alternative task splits, but Seq@3 and Seq@4 runs were prompted as if five attempts were available.The comparison is against Pass@5, not every independent-sampling option with the same budget.Source: §B.1 · p. 16
  • Seq@3 and Seq@4 calibration gains over Pass@5 were positive in all four comparisons; cost ratios were 0.94 and 1.06 on MBPP Pro, and 0.79 and 0.92 on HumanEval Pro.All four gain intervals exclude zero, while three of four cost-ratio intervals include one.Source: §B.1 · p. 16, Table 8 · p. 17

B.2 GAPS BETWEEN ACTORS

  • The appendix reports two actor-gap measures: best-versus-worst performance and the mean absolute difference across actor pairs.The best–worst gap is more sensitive to which actors occupy the endpoints when actors are close.Source: §B.2 · p. 17, Table 9 · p. 17

C.1 COST OF EACH PROTOCOL

  • Figure 3 compares mean cost per task with success rate for each actor and K, using recorded K = 5 runs and stopping at success or attempt K.AdvancedIF cost includes the actor and judge, while DKR cost includes the actor only; the cost axis is logarithmic.Source: §C.1 · p. 17

C.2 CORRELATION BETWEEN PROTOCOLS

Across 26 model–benchmark pairs, Pass@5 and Seq@5 scores were positively correlated, but their respective gains showed little correlation. The pairs excluded coding, and the selected sequential-feedback setting used the highest Seq@5 on the same tasks.

C.2 CORRELATION BETWEEN PROTOCOLS

  • The 26 pairs covered five non-coding benchmarks; each protocol pair matched actor, temperature, K, grader, and tasks, but Seq@5 used the same-task feedback setting with the highest score.The authors note that choosing that setting on the same tasks makes the Seq@5 values optimistic.Source: §C.2 · p. 17, Table 11 · p. 18
  • Across 26 pairs, Pass@5 and Seq@5 had Spearman correlation +0.68 and Pearson correlation +0.81, while their gains had correlations −0.08 and −0.04, respectively.The gain correlations had 95% intervals [−0.47, +0.31] and [−0.38, +0.26]; 11 pairs had no Pass@5 gain.Source: Table 11 · p. 18, §C.2 · p. 18

D.1 SUCCESS BY ATTEMPT

The appendix reports DKR success by attempt under actor-specific feedback setups, then describes the revision annotation and the selected trajectory examples. The annotation uses held-out tasks and revisions, while the examples illustrate patterns rather than their frequency.

D.1 SUCCESS BY ATTEMPT

  • Table 12 reports DKR Seq@K for K = 1 to 5 on 30 tasks per actor, but its rows do not compare actors under the same feedback.Opus 4.7 and GPT-5.2 receive directive feedback from themselves as critics; other listed actors receive Socratic feedback from GPT-5.2.Source: §D.1 · pp. 18–19, Table 12 · p. 19
  • In two alternate DKR feedback runs, Qwen3.6 Max’s EGS was 0.57 and 0.50, compared with 0.20 in Table 4.Final success also changes across feedback setups; EGS and LGS describe an actor under a given setup, not the actor alone.Source: §D.1 · p. 19

D.2 ANNOTATION OF REVISIONS

  • The revision annotation reuses four settings that hold feedback, critic, grader, temperature, and history fixed within each setting.Tasks were randomly split; the held-out set contains 102 distinct tasks, and coding annotations sample 20 tasks, excluding defective-task revisions and revisions with no valid request from Figure 5b.Source: §D.2 · p. 19
  • Annotators compare each failed attempt and its feedback with the next attempt, label requested changes, and classify whether the revision satisfies them.Invalid requests are dropped; requests already met at attempt t count as satisfied if they remain met at t+1.Source: §D.2 · pp. 19–20
  • Two annotators agreed on the revision category for 80% of 165 double-annotated revisions (κ = 0.73), with lowest agreement in DKR (κ = 0.53).These are LLM labels, not human annotations; a third blind Claude Opus pass resolved disagreements, leaving conclusions unchanged.Source: §D.2 · p. 20

E.1 HOW WE CHOSE AND READ THE EXAMPLES

  • The 31 trajectories read across four benchmarks illustrate revision patterns, not how often those patterns occur.The examples were selected from noticed cases, runs of the same tasks under other feedback, and one coding pair; the annotation separately covers 102 other tasks.Source: §E.1 · p. 20

E.2 REVISIONS THAT DO NOT ACT ON THE FEEDBACK: A TESTED FUNCTION THAT IS NEVER DEFINED

The appendix illustrates how revisions can ignore feedback or fix a flagged requirement while breaking another. It also contrasts those trajectories with runs that pass, and reports criterion-repair results for DKR.

E.2 REVISIONS THAT DO NOT ACT ON THE FEEDBACK: A TESTED FUNCTION THAT IS NEVER DEFINED

  • On the defective coding task, Qwen3 Coder Flash repeatedly resubmits unchanged code despite feedback naming the tested function, while Sonnet 4.6 renames its function and passes.The prompt omits the tested function’s name, and Qwen’s attempt 4 recognizes the mismatch in its explanation but keeps the old name.Source: §E.2 · pp. 20–21

E.3 REVISIONS THAT BREAK REQUIREMENTS THAT PASSED BEFORE: PODCAST REVIEW

  • In the podcast-review run, revisions fix some flagged requirements but leave others failing, and the trajectory never passes.The failed-requirement sets change from {4, 8, 11, 16} to {8, 11, 16}, {4, 16}, {4}, and finally {11}.Source: §E.3 · p. 21

CONTRAST

  • In the contrast runs, Qwen3.6 Max and Qwen3.6 27B pass at attempt 3, while DeepSeek V4 Pro passes on its first attempt.GPT-5.2’s failed-requirement count falls from 4 to 1 but the failing requirements change; Opus 4.7 fixes R16 before its remaining failure moves to R8 and then R9.Source: CONTRAST · p. 21

E.4 LIST OF TRAJECTORIES

  • Table 14 lists every trajectory used, including first-attempt successes and contrast runs, with run-specific settings and feedback arrangements.Unless noted, non-coding runs use seed 42, T = 0.7, and K = 5; the coding pair uses seed 1, T = 0.7, full history, and test-detail feedback.Source: §E.4 · pp. 21–22, Table 14 · p. 22

F.1 CRITERION REPAIR IN DKR

  • Binary feedback repaired 2.1 percentage points fewer criteria than independent sampling, with a 95% interval of [−11.5, +5.8].The analysis chooses each task’s attempt that passes the most criteria, using the latest attempt to break ties; regressions outside that chosen attempt are not counted.Source: §F.1 · p. 22, Table 15 · p. 23

F.2 BINARY FEEDBACK IN CODING

The appendix supplies coding scores used in the paper’s binary-feedback analysis and describes how summarized-history effects are calculated from paired trajectories.

F.2 BINARY FEEDBACK IN CODING

  • Table 16 reports coding scores with K = 5, averaging three runs at T = 0.7 over 45 MBPP Pro or 40 HumanEval Pro tasks.The supplied passages describe the table’s coverage but do not give its score values.Source: Table 16 · p. 23, §F.2 · p. 23

F.3 SUMMARIZED HISTORY

  • The summarized-history analysis pairs 2,520 same-task trajectories from 57 run pairs and reports average success-rate change by benchmark.Each pair matches benchmark, actor, feedback, budget, grader, prompt variant, seed, and temperature; coding rows include all 50 tasks, including defective tasks.Source: §F.3 · p. 23, Table 17 · p. 23
Loading 2610.37859v1…