Source-linked AI summary
AhaBench: Do Agents Learn from Prior Experience? A Benchmark for Long-Horizon Continual Learning
Zerui Cheng, Jiawei Xu, Huacan Chai, Jiayang Sun, Pramod Viswanath, Maxm Pan
TL;DR
Long-horizon evaluations often miss whether a fixed agent learns from experience after visible support is removed, changed, or delayed. AhaBench addresses this with three experience-and-evaluation benchmarks and finds that initial competence, later performance, and improvement are not interchangeable. Claude Opus 4.6 leads aggregate Post-Experience Score at 64.3 and aggregate Learning Lift at +25.8 on the common eight-model panel.
Problem
Standard evaluations often reset agents or score only one trajectory, leaving fixed-weight improvement after experience under a changed condition insufficiently measured.
Method
AhaBench combines no-hint puzzle exploration, held-out mathematical transfer, and incident-focused vending evaluation with Initial Score, Post-Experience Score, and Learning Lift.
Results
Claude Opus 4.6 leads aggregate Post-Experience Score at 64.3 and aggregate Learning Lift at +25.8 on the common eight-model panel.
Takeaways & Limitations
The scorecard shows that visible-support use, high post-experience performance, and improvement during a run are distinct forms of experience use.
Takeaways & Limitations
Numeric claims are derived from benchmark summaries, CSVs, plots, traces, and analysis scripts, with model names representing dated snapshot labels where available.
Abstract
from arXiv · showhide
Modern language agents are expected to operate over long horizons: they ask follow-up questions, reuse worked examples, handle tool feedback, and adapt to delayed consequences. Most evaluations still reset the agent after a prompt or score only the final state of one trajectory. AhaBench asks a more operational question: when a fixed model receives useful experience, does its later behavior improve under a related evaluation condition where the obvious support has been removed, changed, or delayed? The suite contains three components. Aha-Puzzle tests no-hint exploration after solved hidden-state puzzles; Aha-Euler turns Project-Euler-style mathematical ideas into generated taught/held-out tasks with exact validators; and Aha-Vending, an open-source implementation inspired by Vending-Bench, tests whether a simulated vending agent remains profitable while handling delayed feedback and operational incidents. AhaBench reports a three-part scorecard: Initial Score measures starting competence, Post-Experience Score measures the later empirical outcome, and Learning Lift is their difference. This decomposition is the main empirical message: models that use visible support well, models that reach high post-experience scores, and models that improve most during a run are not always the same. On the common eight-model panel, Claude Opus 4.6 leads aggregate Post-Experience Score at 64.3 and aggregate Learning Lift at +25.8, with Gemini 3.1 Pro close behind at 63.4. The component results explain the split: puzzle traces raise supported scores but often fail to become no-hint exploration behavior; Aha-Euler full teaching reaches 78.6-100.0% while answer-only transfer ranges from 0.0 to 73.9%; and Aha-Vending separates profitable incident handling from bankruptcy and no-order failure. We release benchmark tasks, rubrics, validators, simulator code, and interfaces for evaluating new agents.
1 Introduction
AhaBench evaluates whether fixed-weight agents improve after experience when later tests remove, change, or delay the original support. Its three-part suite and scorecard distinguish initial competence, post-experience performance, and learning during a run.
- Benchmark and question: Standard evaluations do not separate strong initial ability, temporary visible-hint use, and durable improvement after support is removed.AhaBench addresses this gap without fine-tuning or gradient updates.
- Benchmark and question: Each component provides experience and then tests a related condition that cannot be solved by copying the source support.The benchmark removes puzzle traces, changes mathematical inputs, or evaluates later business states caused by earlier decisions.
- Scorecard: Learning Lift is ∆= P −B, complementing Initial Score B and Post-Experience Score P in the common scorecard.These measures distinguish where a model started from how it performed later and how much its score changed.
- Results: 64.3 is Claude Opus 4.6's aggregate Post-Experience Score, while its aggregate Learning Lift is +25.8.Gemini 3.1 Pro follows with a Post-Experience Score of 63.4, illustrating that ranking by later performance and improvement are related but distinct.
- Benchmark and question: AhaBench tests fixed language agents across interactive puzzles, generated mathematical reasoning, and simulated vending operation.The suite is designed to measure experience use across three distinct domains.
- Contribution: The benchmark reports fine-grained analyses and reproducible artifacts, including generators, simulator code, rubrics, interfaces, baselines, and benchmark data.These materials support evaluation of new agents and analysis of different forms of experience use.
2 Long-Horizon Continual Learning
AhaBench formalizes experience use as a policy change measured before and after relevant interaction under a changed evaluation condition. It uses multiple diagnostics because visible support, transferable procedures, and later operational outcomes capture different forms of transfer.
- Formalization: The measured object is the later policy of the same fixed-parameter agent after prior curriculum, feedback, or environment trajectory D<t.The framework compares behavior before and after relevant experience rather than parameter updates.
- Formalization: The experience-use gain compares Initial Score Sinit with Post-Experience Score Spost after the evaluation condition changes.The paper uses this basic transfer estimate as a central measure of later behavioral change.
- Diagnostics: The support-reliance gap is Ssup −Spost, contrasting performance with visible support against performance after support is reduced.AHA-PUZZLE applies this to trace-supported averages versus the final no-hint challenge.
- Diagnostics: The procedure-reliance gap is Sfull −Spartial, separating answer-plus-code teaching from answer-only teaching in AHA-EULER.Support-gain retention additionally measures how much visible-support improvement survives when support is reduced.
- Scorecard: The main scorecard reports Initial Score B, Post-Experience Score P, and Learning Lift ∆= P −B alongside richer component diagnostics.The plural measurements preserve differences among transfer, support reliance, and operational trajectory behavior.
- Evaluation shift: AHA-PUZZLE withholds trace prefixes, AHA-EULER withholds explanation and code, and AHA-VENDING evaluates consequences over later simulated days.These evaluation shifts make copying the original support insufficient.
3 Benchmark Suite
The benchmark suite turns three standalone agent tasks into experience-and-evaluation tests: no-hint puzzle exploration, held-out mathematical transfer, and delayed-feedback operational control. Its designs preserve trajectories, exact validation, and failure modes rather than relying on final answers alone.
- Suite design: The three benchmark families evaluate exploration under hidden state, mathematical reasoning through related tasks, and operational planning under delayed consequences.These settings expose different ways agents may use prior experience.
- AHA-PUZZLE: AHA-PUZZLE uses human-verified traces and a final no-hint challenge to test whether solved episodes change exploration policy.Agents ask questions, receive host responses, reflect, and produce judged explanations across a balanced long-horizon schedule.
- AHA-EULER: AHA-EULER pairs a taught Qteach instance with a held-out Qtest input to measure transfer of a computational idea.Full teaching reveals answer, explanation, and code, while partial teaching tests answer-only method learning.
- AHA-EULER: The graph-composition abstraction decomposes mathematical tasks into dependent subcomputations that can be revealed locally and recombined on Qtest.Exact-match validation remains strict and auditable while Python supports method inference and execution.
- AHA-VENDING: AHA-VENDING exposes delayed operational feedback through sales, stockouts, supplier messages, weather, incidents, profit curves, survival, and termination reasons.The incident setting reveals failures that a single final answer or profit value could hide.
- Interpretation: The suite separates trace use from exploration transfer, code-supported adaptation from answer-only learning, and short-term profit from durable operation.It reports uncertainty at benchmark-specific levels, including endpoint intervals, exact-match intervals, trajectories, and termination modes.
4 Experimental Setup
The experimental setup compares dated model snapshots under a common protocol while preserving benchmark-specific scoring evidence. Aggregation balances puzzle languages, uses exact-match mathematical accuracy, and reports seeded incident-setting vending profit.
- Protocol: All numeric claims are derived from benchmark summaries, CSVs, plots, traces, and analysis scripts for dated model snapshots under the same protocol.The intended comparison is between learning behaviors rather than timeless model identities.
- Models and aggregation: AHA-PUZZLE and AHA-EULER use the common eight-model panel, while AHA-VENDING also includes additional models as anchors.Puzzle means balance the four languages, and endpoint bootstrap intervals are computed over scored trajectories.
- Models and aggregation: AHA-EULER reports exact-match accuracy on deterministic generated tasks after applying public scoring rules.This preserves a strict, reproducible outcome measure for mathematical transfer.
- Models and aggregation: AHA-VENDING reports average net profit over seeded incident-setting runs.The broader evaluation preserves trajectories and termination modes as evidence beyond the aggregate profit.
- Evidence preservation: Table 2 records whether each benchmark preserves human-checked rubric rules, exact validators, trajectories, executable code, and released objects.These evidence types support auditing and reproduction of benchmark outcomes.
5 Main Leaderboard and Metrics
The scorecard reports Initial Score, later Post-Experience Score, and Learning Lift to distinguish starting competence, later performance, and improvement. Its leaderboard and component views expose disagreement across experience-use axes.
- Initial Score B, Post-Experience Score P, and Learning Lift Δ=P−B measure early competence, later performance, and change on the same 0–100 scale.
- The official leaderboard prioritizes aggregate Post-Experience Score, with Lift reported beside it to show how much performance changed during the run.
- Component-level inspection identifies whether a model’s strength lies in operational control, hidden-state exploration, or reconstructing computation without visible procedures.
- Claude leads aggregate Post-Experience Score and aggregate Lift, while GPT-5.4 leads AHA-EULER Post-Experience Score and Qwen leads AHA-PUZZLE Lift.
6 Results
Results compare each experience source with a harder, related condition that removes support, withholds procedures, or delays consequences. Across components, supported performance often exceeds durable transfer, while incident handling separates robust from brittle policies.
- AHA-PUZZLE: Supported context is easier than support-free transfer: every reported AHA-PUZZLE model scores higher with trace support than at the final no-hint endpoint.
- AHA-PUZZLE: Qwen shows the clearest AHA-PUZZLE transfer gain, while high supported scores for GPT-5.4, Claude, Gemini, Kimi, and Hy3 Preview do not automatically yield no-hint exploration.
- AHA-EULER: Full AHA-EULER teaching produces high accuracy for nearly every model, whereas answer-only teaching sharply separates models that reconstruct procedures from those requiring explicit procedure.
- AHA-EULER: Successful GPT-5.4 partial-teaching traces use fresh programs, tool-feedback repair, and reparameterization rather than copying exposed reference code.
- AHA-VENDING: Claude and Gemini remain profitable across AHA-VENDING incident seeds, while Doubao and DeepSeek reach the bankruptcy floor and Hy3 Preview often terminates early without successful orders.
7 Cross-Benchmark Analysis
The three benchmarks measure separable forms of experience use rather than one universal capability. Descriptive overlap analysis supports treating exploration, mathematical transfer, and operational control as distinct axes.
- AHA-PUZZLE tests how models ask under hidden state, AHA-EULER tests computational reconstruction without exposed code, and AHA-VENDING tests durable operational policy.
- The benchmark’s visual comparison emphasizes operational robustness: profitable policies remain above zero under incidents, while brittle policies approach the bankruptcy floor.
- The eight-model overlap analysis is descriptive evidence that puzzle exploration, Euler-style transfer, and vending operation are not interchangeable skills.
8 Related Work
AhaBench builds on benchmark traditions, continual-learning research, and component-specific evaluations while changing the unit of measurement. It evaluates experience-to-evaluation pairs rather than isolated items or single trajectories.
- Benchmark traditions make scale, public protocols, reproducibility, leaderboards, and multi-metric comparison central to evaluation.
- AhaBench retains clear leaderboards and reproducible artifacts but changes the measurement unit from an isolated item to an experience-evaluation pair.
- Unlike parameter-update continual learning and standard in-context learning, AhaBench studies fixed-parameter agents under related but non-identical post-experience conditions.
- Its components connect to established math, puzzle, long-horizon-agent, and Vending-Bench lineages while combining them into one experience-use framework.
9 Conclusion
AHABENCH evaluates whether fixed-parameter agents convert experience into later behavior when direct support is removed, changed, or delayed. Its diagnostics distinguish exploration, mathematical transfer, and operational robustness rather than collapsing them into one score.
- Conclusion: AHABENCH makes experience-to-evaluation relations the measurement unit, testing whether later behavior improves when visible support is no longer directly copyable.Puzzle removes trace prefixes, Euler changes inputs while preserving computational ideas, and Vending evaluates consequences of earlier decisions.
- Conclusion: The benchmark reports support conditions, Post-Experience outcomes, retained gain, trajectory shape, and failure mode together.This structure is intended to reveal which form of experience use failed, not merely whether a final score is high.
- Conclusion: The suite is positioned as a measurement pattern for continual learning, test-time adaptation, and in-context learning with fixed model parameters.Unlike classical continual learning, AHABENCH studies deployment-time history without fine-tuning or gradient updates.
- Conclusion: AHA-PUZZLE targets interactive exploration, AHA-EULER targets mathematical reasoning under reduced teaching, and AHA-VENDING targets operational control under delayed feedback.The components connect to established puzzle, mathematical-programming, and vending-agent evaluation lines while retaining distinct experience conditions.
B Additional Benchmark Details
The additional benchmark details define balanced Puzzle sampling, executable Euler validation, and scripted Vending baselines. They emphasize trajectory-level measurement, auditable scoring, and diagnostics beyond a single endpoint.
- Puzzle: AHA-PUZZLE aggregates language-level means across selected models, with the final no-hint column representing the last curriculum slot.The released aggregation balances languages, while bootstrap procedures separately address endpoint and puzzle-identity variation.
- Puzzle: Puzzle scoring uses explicit explanation-chain rubrics, paraphrase-aware validation, and human checks of construction and sampled grading outputs.A puzzle is solved only when all required explanation elements are substantially covered.
- Puzzle: The endpoint bootstrap resamples paired model–language–repeat–puzzle trajectories, whereas the cluster bootstrap resamples puzzle identities.The cluster procedure treats puzzle identity as the effective independent unit.
- AHA-EULER: AHA-EULER separates Qteach from held-out Qtest and reports pass rates for single-step and compositional tasks under aligned and phase-level views.Generated task identifiers link prompts, traces, and validators, while exact-match evaluation supports auditable comparisons.
- AHA-VENDING: The appendix also reports hardened-versus-original baseline probes and scripted Aha-Vending profiles, including mean simulated days before completion or watchdog termination.These checks expose infrastructure-error sensitivity and provide fixed policy reference points for the simulator.
C Additional Aha-Vending Results
Additional results explain how the benchmark exposes temporal and operational behavior that final profit alone misses. Puzzle, Euler, and Vending use distinct interfaces for exploration, transfer, and control.
- AHA-PUZZLE: AHA-PUZZLE separates tester question selection, minimal host answers, and rubric-based final judging to isolate exploration policy.The model may ask up to 20 questions, stop, and provide an explanation without the host volunteering hidden links.
- AHA-PUZZLE: The Puzzle curriculum progressively removes trace support, distinguishing cold competence, trace-supported competence, and no-hint transfer.The final challenge requires using an exploration strategy without direct support.
- AHA-EULER: AHA-EULER instantiates related inputs from generator-backed problem families, separating full teaching with code from partial teaching with only question-plus-answer information.Exact-match validation tests whether the model can infer, implement, debug, and return the required computation.
- AHA-VENDING: AHA-VENDING exposes stateful actions and delayed business feedback across inventory, pricing, suppliers, cash flow, incidents, and bankruptcy.Mistakes compound through stockouts, procurement failures, inventory decay, and unreliable supplier messages.
E Metric Definitions and Reporting
AHABENCH defines comparable Initial, Post-Experience, and Learning Lift measures while preserving trajectory-level diagnostics. Its reporting framework treats long-horizon cost, failure modes, and capability disagreement as first-class evidence.
- Metrics: The scorecard compares early baseline performance with later post-experience performance and reports Learning Lift as their difference.Puzzle, Euler, and Vending use corresponding baseline and post-experience quantities, with Vending profit normalized for the incident setting.
- Diagnostics: Support-gain retention and late-minus-early profit efficiency explain how much visible support or operational progress survives under reduced support or over time.These quantities remain diagnostic rather than replacing direct partial-teaching accuracy or the main scorecard.
- Reporting: Long-horizon evaluation cost scales with accumulated turns and context, ∑t H_tC_t, rather than simply the number of curriculum stages.For this reason, AhaBench treats an item as a trajectory and audits support removal, held-out separation, curves, and samples.
- Capability surface: Puzzle, Euler, and Vending isolate epistemic exploration, algorithmic recomposition, and operational feedback control as distinct experience interfaces.These interfaces respectively test informative questioning, reusable computation from related examples, and durable policy under delayed rewards and shocks.
- Capability surface: A single leaderboard can hide disagreement between high final performance, transfer gains, and long-horizon robustness, so AHABENCH reports the capability surface with three score axes.Failure taxonomies and trajectory analyses distinguish errors that aggregate accuracy or final profit alone can obscure.
- Failure analysis: Aha-Vending trajectory examples show recovery after repeated slot-fit failures, later passivity after early profit, and losses under coupled shared-market demand.These patterns demonstrate why operational outcomes require curves, survival, termination reasons, and action histories.