Source-linked AI summary
The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
TL;DR
Existing benchmarks measure whether long-horizon agents finish tasks, but not the quality of decisions that shape their runs. This paper introduces Taste-Bench, where models choose between directions at trajectory forks, and finds that longer-horizon forks are harder while teacher distillation improves decisions on unseen tasks and end-to-end success on held-out tasks.
Problem
Existing long-horizon benchmarks report task completion but do not measure decision quality, even though agents’ choices can influence outcomes much later.
Source: §1 · p. 1Method
Taste-Bench uses 502 questions mined from engineering and machine-learning trajectories, freezing each decision fork’s prefix and hiding later work while models choose a direction.
Source: §1 · p. 2Results
Forks become harder as their horizon increases; GPT-5.6 Sol achieves 59.7% Average accuracy, and teacher distillation improves unseen-task judgments and held-out end-to-end success.
Source: §1 · p. 2, §4.2 · p. 5Takeaways & Limitations
Taste-Bench provides a practical basis for measuring and training long-horizon agent judgment.
Source: §7 · p. 10
Abstract
from arXiv · showhide
LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent. While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork. We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly. We further find that forks whose deciding evidence appears later in the trajectory are much harder for every model, and that a larger reasoning budget does not improve the accuracy. Finally, we show that taste can be trained. We distill the judgment of a teacher that has seen the outcome into a student model, and the student makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.
1 Introduction
The paper defines agent taste as choosing well at long-horizon decision points, where mistakes may look reasonable until their costs emerge later. It introduces Taste-Bench, which uses hindsight from agent trajectories to assess those choices without requiring expert annotations.
Introduction
- Existing long-horizon benchmarks [9] [12] [13] measure task completion rather than decision quality, while taste is an agent’s ability to choose well when later outcomes reveal which direction was better.Decision quality is difficult to judge at the fork because good and bad choices can look equally reasonable, and expert annotation is expensive and hard to scale.Source: §1 · pp. 1–2
- The authors measure taste by freezing a trajectory at a decision fork, hiding later work, and asking a model to choose between directions whose outcomes provide hindsight labels.Forks come from divergences between independent attempts at the same task or from an agent’s self-correction within a single run.Source: §1 · p. 2
- Taste-Bench contains 502 questions from software engineering and machine-learning research trajectories, and human review shows close agreement with labels based on recorded outcomes.The benchmark mines supervision from trajectories agents have already produced, allowing it to scale with those trajectories.Source: §1 · p. 2
- Taste-Bench questions become harder as the time horizon of the fork increases.Source: §1 · p. 2
- Distilling a teacher’s reasoning about the correct direction improves a student’s judgments on unseen tasks and yields end-to-end gains on held-out agent tasks.Source: §1 · p. 2
2 Measuring Taste via Long-Horizon Judgment
Taste is long-horizon judgment: choosing the direction whose advantage appears in later work, when that outcome is not yet visible. The section measures it by mining decision forks from agent trajectories and testing whether a model selects the direction supported by later outcomes.
2 Measuring Taste via Long-Horizon Judgment
- Taste is long-horizon judgment: choosing a direction whose advantage appears in later work, rather than at decision time.For example, a clean implementation may pass the same tests as a hasty one, with its advantage appearing as later changes become easier.Source: §2 · p. 2
- Repeated attempts at the same task provide decision forks: equivalent prefixes diverge into judgments whose branches have different outcomes.Sampling randomness assigns judgments to branches, helping separate judgment quality from execution quality and environment; forks are retained only when outcomes clearly follow from the judgments. Trajectories supply hindsight because they record both judgments and subsequent work.Source: §2 · pp. 2–3
- A taste question hides everything after the fork and asks the model to choose between two candidate directions; taste is the fraction of questions answered correctly.Each branch produces evidence mapped to an outcome score, and the candidate with the better realized outcome labels the fork, estimating which candidate has the higher expected outcome given the shared prefix.Source: §2 · p. 3
3 Taste-Bench
Taste-Bench contains 502 decision-fork questions mined from agent trajectories in engineering and research tasks, using parallel-attempt and within-trajectory detour constructions. Questions test whether a model can choose the better direction from the task prefix without seeing what happens next.
3.1 Forks from parallel trajectories
- Parallel forks pair attempts that diverge at the same task point, using their differing outcomes to label which direction is supported.The shared pre-fork trajectory becomes the prefix, and neutral descriptions of the two branches become the candidates; this construction captures wrong directions the agent never notices.Source: §3.1 · p. 3
3.2 Forks from detour trajectories
- Detour forks contrast an abandoned direction with a later recovery, placing the question before the agent takes the abandoned direction.A generator identifies the direction, its observed failure, and the completing recovery, then rejects cases where the better direction is only nameable after the failure or either option is implausible at the fork.Source: §3.2 · p. 4
3.3 Mining, filtering, and composition
- 502 questions remain from 4,657 candidate forks after filtering, with 390 engineering and 112 research questions.The trajectory pools include 2,677 engineering rollouts on 517 SWE-bench Pro tasks and 1,132 research runs on 47 AI R&D tasks from RE-Bench and HCAST [13] [14], released through MALT. In a 100-question human review, 170 of 172 explicit A/B judgments agreed with mined labels (98.8%); reviewers agreed with each other on 98.6% of 74 questions where both chose A or B (Cohen’s κ = 0.973).Source: §3.3 · pp. 4–5
3.4 Evaluation protocol
- Accuracy counts a question correct only when both candidate orders are answered correctly, with mean accuracy over the two orders also reported.Each question is evaluated once in a deterministic seeded order and once in the exact reverse order to address position bias [16].Source: §3.4 · p. 5
4 Experimental Results
The section evaluates 14 models on 502 Taste-Bench questions and finds that the best reaches 59.7% Average accuracy. Accuracy also falls with later deciding evidence, while larger reasoning budgets do not improve it.
4.1 Setup
- Fourteen models answer 502 Taste-Bench questions under a shared interface and token budget; Average is the 1:1 mean of research and engineering accuracy.Because both candidate orders must be correct, random guessing scores 25%, while always preferring the same position scores 0%.Source: §4.1 · p. 5
4.2 Accuracy of current models
- 59.7% Average accuracy makes GPT-5.6 Sol the top model, narrowly ahead of GPT-5.5 at 59.5%.The remaining models score below them, with additional variation between the research and engineering subsets.Source: §4.2 · p. 5, Figure 3 · p. 6
- Detour forks are harder than parallel forks in both domains, and that gap exceeds the difference between domains.Source: §4.2 · p. 6
4.3 Effect of the time horizon
- Mean accuracy across 14 models falls from 62.3% for in-prefix forks to 21.0% for more-work forks, near random guessing.The four time-horizon levels are in prefix, inferable, next step, and more work.Source: §4.3 · pp. 6–7, Figure 4 · p. 6
4.4 Effect of the reasoning budget
- From lowest to highest reasoning effort, GPT-5.6 Sol changes by −0.2 points and GPT-5.6 Luna by +2.2 points, without accuracy improvement.Accuracy overlaps across reasoning settings at every time horizon; the models reason longest on more-work forks, where accuracy is lowest.Source: Figure 4 · p. 6, §4.4 · p. 7
4.5 Comparison with end-to-end benchmarks
- Taste-Bench Average and SWE-bench Verified correlate at r = +0.63, while the engineering-subset correlation is r = +0.37.The four highest SWE-bench Verified models are within 4.0 points of one another on that benchmark but 10.7 points apart in Taste-Bench Average.Source: §4.5 · pp. 7–8, Figure 5 · p. 7
5 Generalizing Taste through Distillation
The section distills outcome-informed judgments about decision forks into a student model and evaluates its transfer to unseen questions and tasks. Student advice improves task success on held-out SWE-bench Pro tasks, while the student also transfers judgment to task-disjoint engineering questions.
5.1 Distillation recipe
- The distillation recipe trains on teacher-generated reasoning rather than binary labels, aligning those tokens and the final choice under a student context that contains only the benchmark question.The teacher and student are the same frozen Qwen3.6-27B base model; training uses LoRA adapters and an SDPO-style token-level forward KL on teacher-sampled continuations.Source: §5.1 · p. 8, §5 · p. 8
5.3 End-to-end evaluation
- On 41 held-out SWE-bench Pro tasks, student advice raised executor success to 33.7% (+19.1 percentage points over no advice); correct advice raised it from 14.6% to 39.0% (+24.4 points).At each fork, advice identified the candidate to avoid and the candidate to take; a fixed Qwen3.6-27B executor completed the task independently of the student.Source: §5.3 · p. 9, Figure 7
5.2 Transfer to unseen tasks
- On task-disjoint held-out engineering questions, student accuracy was 47.9% versus 30.0% for the base model, while mean two-order accuracy rose from 42.7% to 62.4%.Each student trained on one of two folds of the 390-question pool and was evaluated on the other, so it did not see evaluation tasks during training.Source: §5.1 · p. 8, §5.2 · p. 9
6 Related Work
Taste-Bench evaluates decisions among forks inside agent trajectories, rather than only task completion or judgments of standalone ideas. It uses recorded outcomes to identify better directions and evaluates choices before those outcomes are shown.
Long-horizon agent evaluation
- Prior long-horizon benchmarks assess task completion, while Taste-Bench assesses whether an agent selects the better direction within a task.Source: §6 · p. 10
Judging research directions before the outcome
- Prior direction-judgment work compares standalone ideas, ML solutions, or AI-safety proposals [33], and judgments can flip after execution [36]; Taste-Bench evaluates situated trajectory forks.Source: §6 · p. 10
Process evaluation and step-level judgment
Distillation and internalizing context
- The recipe transfers in-context judgment into student weights by demonstrating a supported candidate and sampling targets from the teacher, following SDPO [32].Context distillation also includes STaR with a correct answer [49], LEAP with privileged state [50], and SDPO with environment feedback [32]; advisor models steer executors with advice [52].Source: §6 · p. 10
7 Conclusion
Taste is an agent’s ability to choose a better direction before outcomes are visible; the paper measures it with Taste-Bench and shows that teacher judgment can be distilled to improve decisions and end-to-end success.
7 Conclusion
- Taste-Bench measures agents’ ability to choose better directions before outcomes are visible using 502 questions labeled by recorded outcomes.The benchmark is built from existing trajectories.Source: §7 · p. 10
- Distilling a teacher’s reasoning transfers judgment to questions from unseen tasks.The teacher is given the supported candidate.Source: §7 · p. 10
- Injecting the student’s judgment as advice improves end-to-end success.Source: §7 · p. 10
- The authors present Taste-Bench as a practical basis for measuring and training the judgment of long-horizon agents.Source: §7 · p. 10
Reproducibility Statement
The paper reports fixed benchmark and distillation settings, complete runs, and figures generated from recorded results. It releases the benchmark, evaluation code, protocol, and recorded model results.
Reproducibility Statement
- Reported results use a fixed benchmark version and complete question coverage, with documented construction, prompts, trajectory handling, scoring, and fixed distillation settings.The appendix specifies the benchmark procedures; distillation fixes the base-model version, random seeds, fold splits, and inference settings, and figures are generated directly from recorded results.Source: Reproducibility Statement · p. 10
- The benchmark, evaluation code, protocol, and recorded results for every evaluated model are released publicly.The benchmark is hosted on Hugging Face and the code, protocol, and model results are available on GitHub.Source: Reproducibility Statement · pp. 10–11
Ethics Statement
The authors identify dual-use risk in measuring long-horizon judgment and describe safeguards for reviewing and reporting benchmark results.
Ethics Statement
- The benchmark uses technical agent trajectories, removes credential strings before model submission, reports aggregate results without domain-specific harmful instructions, and collects judgments from two human reviewers.The authors note that measuring long-horizon judgment carries dual-use risk.Source: Ethics Statement · p. 11
AI Use Statement … D Full Results
This appendix documents the benchmark’s source data, fork statistics, construction prompts, filtering, and use of generative AI. It also presents examples of removed questions and released decision forks, and describes how models receive, answer, and are scored on the benchmark.
AI Use Statement
- The authors report using generative AI for data generation, research and writing tasks, implementation, and figure creation, while taking responsibility for the final content and identifying the work’s contributions as original.The disclosed uses also include translation, language editing, code editing, and literature searches.Source: AI Use Statement · p. 11
A.1 Source trajectories
- Engineering trajectories come from the authors’ SWE-bench Pro runs, and research trajectories come from METR’s MALT release.The engineering pool has 2,677 graded rollouts on 517 tasks across 11 repositories; the research pool has 1,132 scored runs on 47 RE-Bench and HCAST tasks.Source: §A.1 · p. 15
A.2 Statistics of the mined forks
- Parallel forks pair passing and failing attempts, while detour forks place the choice before an abandoned approach and require ordered evidence of failure and recovery.For detours, the generator must cite the poor commitment, observed failure, recovery commitment, and evidence that recovery succeeded; mechanical checks discard candidates that fail these requirements.Source: §A.2 · p. 15
A.3 Prompts of the generator and the judges
- Generator prompts require outcome-grounded forks: parallel candidates need causal differences, while detours require an ordered commitment, observed wall, recovery, and success.A four-model judge panel uses shuffled candidate order; one filter sees only the choices, and another sees the task, prefix, continuations, and outcomes.Source: §A.3 · pp. 16–19
A.4 Counts at each stage
- Of the 4,657 mined candidate forks, 1,809 passed the generator rubric, 729 survived the trivial filter, and 502 questions remained in the release.The trivial filter removes a proposed question when all four judges answer it correctly from the two candidates alone, using the same threshold in all four cells.Source: §A.4 · p. 19, Figure 8 · p. 20
- The removed-question examples show two filter failures: one choice was obvious from candidate wording, while another label depended on an unstated time budget and later timeout.In the first case, hardcoding three answers made the other candidate plainly preferable; in the second, evidence at the fork favored the ensemble, and only its later full-scale timeout determined the label.Source: §A.4 · pp. 20–21
B Example Questions
- The delegated-auth example supports selecting five endpoint fields: that candidate passes hidden tests, while assigning the entire discovery block fails deep equality.The full block also contains a state marker and an error field, which the hidden test does not expect.Source: B Example Questions · p. 22
- loss_validation was 7.3821 for supported candidate A’s fine-tuning branch, versus 10.1482 for the best embedding-rescaling variant.The agent fine-tuned the corrupted embedding matrix for 200 iterations with the rest of the model frozen; the sweep’s best variant used standard deviation 0.032.Source: B Example Questions · p. 22
- In the Ansible example, moving the placeholder onto the production call path yields 28 passed tests, after extending helper outputs caused 3 failed and 25 passed.The existing tests call the summary and documentation helpers directly and expect their default output shape; supported candidate A leaves those defaults unchanged.Source: B Example Questions · p. 23
- In the expression-discovery example, add(inverse(X4), cosine(X3)) achieved maximum absolute error 5.13e-10 and met tolerance on all 1000 rows.The supported candidate was A, testing allowed unary operators on X3; the regression-guided branch’s closest expressible variant had maximum absolute error 5.4016000766e-01.Source: B Example Questions · p. 23
C Evaluation Protocol Details
- Models receive the same prompt and 65,536-token output budget, choose between candidates with exactly one designated as better, and are scored across both candidate orders.Each question appears twice in seeded and reversed order; cell accuracy requires both answers correct, mean accuracy counts correct presentations, and domain accuracies pool engineering or research cells. Responses are parsed from a final ANSWER: X line (or a single letter); Figure 3 intervals use 4,000 within-domain bootstrap resamples.Source: C Evaluation Protocol Details · pp. 23–24
D.1 Results per model and per cell … G.2 Folds
Accuracy differs across Taste-Bench’s construction–domain cells, and model rankings can change between Taste-Bench and SWE-bench Verified. The passages also describe time-horizon annotation, reasoning-budget results, label checks, distillation, and task-disjoint evaluation folds.
D.1 Results per model and per cell
- 58.1% on parallel engineering, 50.9% on parallel research, 50.8% on detour research, and 35.9% on detour engineering show that construction gaps exceed domain gaps.Parallel engineering is the easiest cell, while detour engineering is the hardest; the paper therefore reports all four cells separately.Source: Table 4 · p. 25, §D.1 · p. 25
D.2 Comparison with SWE-bench Verified
- DeepSeek V4 Flash ranks 5th on SWE-bench Verified and 10th on Taste-Bench, while GPT-5.5 ranks 8th and 2nd, respectively.The comparison excludes three models with more than 9% unparsable responses; scores use a single public harness, reasoning effort is not matched for every model, and correlation intervals are wide.Source: §D.2 · p. 25
D.3 Time horizon annotation
- A GPT-5.5 judge assigns each question a time-horizon score, with five scores grouped into four levels because only 13 questions receive the highest score.The four levels contain 158, 219, 56, and 69 questions; the scale ranges from decisive evidence already in the prefix to substantial future work.Source: §D.3 · p. 26
D.4 Reasoning budget
- Reasoning-budget increases show no significant overall difference or interaction with time horizon for GPT-5.6 Sol or Luna, though Luna’s research accuracy rises from 43.8% to 54.5%.The more-work level receives the most reasoning tokens but has the lowest accuracy, between 13.0% and 23.2% across the six conditions.Source: §D.4 · pp. 26–27, Table 7 · p. 27
E Agreement of the Judges
- 51.7% pairwise agreement between judges on released parallel-engineering questions contrasts with unanimous labeled-candidate selection in the undecidable filter.The latter filter retains a question only when all judges agree.Source: Table 8 · p. 28
F.1 Sampling and annotation
- 98.8% of 172 retained judgments supported the mined label, and reviewers agreed on 73 of 74 questions with Cohen’s κ = 0.973.Questions were sampled across construction–domain cells and time-horizon levels, plus examples removed by the two filters.Source: §F.1 · p. 28, Table 9 · p. 28
G.1 Recipe
- Distillation uses the same frozen Qwen3.6-27B base model as teacher and student, with a task-specific demonstration in the teacher context and both candidate orders as student views.The teacher generates reasoning traces and the student is trained with forward-KL losses on reasoning and answer positions; calibration then adjusts the final choice using the student’s own traces. The recipe structurally prevents copying demonstration text, and training uses 156 and 142 questions across the two folds.Source: §G.1 · pp. 28–30
G.2 Folds
- 390 engineering questions are split into two task-disjoint folds, with each student trained on one fold and evaluated on the other.The folds share no question, task, source trajectory, or prefix; pooled out-of-fold evaluations are used in Section 5.2.Source: §G.2 · p. 30, Table 11 · p. 30
G.3 Transfer results
On held-out Taste-Bench questions, the student more often answered both presentations correctly than the base model, and its log-odds advice selected the supported candidate more often. The advice result uses a single combined decision per question rather than requiring correct answers to both presentations.
G.3 Transfer results
- The student answered both presentations correctly on 187 of 390 held-out questions, versus 117 for the base model.A question counted as correct only when both orders were answered correctly; unparseable presentations counted as wrong. Relative to the base model, the student was correct on 104 questions it missed and wrong on 34 questions it got right. Under the same scoring, the student ranked below GPT-5.6 Sol (56.9%) and GPT-5.6 Terra (50.0%), equal to GLM-5.2 (47.9%), and above Claude Opus 5 (46.7%); the base model was just below GPT-5.4 Nano (32.1%).Source: §G.3 · pp. 30–31
- Using summed log-odds over both presentations to make one decision per question, the student selected the supported candidate on 287 of 390 questions, versus 207 for the base model.These rates exceed the both-presentations-correct accuracy because this rule requires only the combined decision to be correct.Source: §G.3 · p. 31
H End-to-End Experiment Details
Student-generated advice selected the supported candidate on 77 of 98 forks across 41 held-out SWE-bench Pro tasks, and the advised executor achieved a 33.7% success rate. Advice was generated by students trained without the task and evaluated through the official SWE-bench Pro patch evaluation.
H End-to-End Experiment Details
- Each advice note gives the fork situation, the candidate to avoid, and the candidate to take, while warning that advice is not verification.Notes are placed before the original problem statement; the executor is instructed to work normally and verify its own patch.Source: H End-to-End Experiment Details · pp. 31–32
- The executor was SWE-agent 1.1.0 with Qwen3.6-27B, limited to 75 model calls and 4,800 seconds per run.Thinking was disabled, temperature was 0.7, top-p 0.8, top-k 20, presence penalty 1.5, and each command was limited to 600 seconds.Source: H End-to-End Experiment Details · p. 31
- 33.7%: the executor achieved this SWE-bench Pro success rate with student advice.Success means the official SWE-bench Pro evaluation accepts the patch.Source: H End-to-End Experiment Details · p. 32
- The student selected the supported candidate on 77 of 98 forks across 41 held-out tasks.For every task, advice came from the student trained on the fold that did not contain that task; the tasks span 11 repositories.Source: H End-to-End Experiment Details · pp. 31–32