PaperlayerView exact revision on arXiv

Source-linked AI summary

EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving

Shihan Dou, Ming Zhang, Chenhao Huang, Jiayi Chen, Feng Chen, Shichun Liu, Yan Liu, Chenxiao Liu, Cheng Zhong, Zongzhang Zhang, Tao Gui, Chao Xin, Chengzhi Wei, Lin Yan, Yonghui Wu, Qi Zhang, Xuanjing Huang

arXiv:2506.02672v3cs.CLcs.AI

TL;DR

Most LLM benchmarks test independent examples, leaving learning from prior task experience and learning speed underexplored. EvaLearn evaluates models on sequential problems with five metrics, and its nine-model study finds task-dependent learning patterns, benefits from feedback, and no clear across-task learning advantage for models with stronger static performance.

  • Problem

    Most benchmarks use independent parallel tests that measure static ability, leaving models’ capacity to learn within a task and the speed of that learning largely unexplored.

    Source: §1 · pp. 1–2

  • Method

    EvaLearn evaluates nine frontier LLMs by having them solve challenging problems sequentially and uses five metrics to quantify learning capability and efficiency.

    Source: §1 · p. 2

  • Results

    Learning patterns varied across tasks, feedback learning enhanced capability and efficiency, and stronger static performance gave no clear learning advantage across all tasks.

    Source: §1 · p. 2, §4 · p. 10

  • Takeaways & Limitations

    Learning capability and efficiency should be considered jointly because they provide a fuller assessment of model learning potential and are not strongly correlated with static capability.

    Source: §1 · p. 2

    The paper directs readers to Appendix A for limitations and future work, but the supplied limitation passages specify no concrete boundary.

    Source: §2 · p. 19

Abstract

from arXiv · show

We introduce EvaLearn, a pioneering benchmark designed to evaluate large language models (LLMs) on their learning capability and efficiency in challenging tasks, a critical, yet underexplored aspect of model potential. EvaLearn contains 648 challenging problems across six task types, grouped into 182 sequences, each sequence dedicated to one task type. Diverging from most existing benchmarks that evaluate models in parallel, EvaLearn requires models to solve problems sequentially, allowing them to leverage the experience gained from previous solutions. EvaLearn provides five comprehensive automated metrics to evaluate models and quantify their learning capability and efficiency. We extensively benchmark nine frontier models and observe varied performance profiles: some models, such as Claude-3.7-sonnet, start with moderate initial performance but exhibit strong learning ability, while some models struggle to benefit from experience and may even show negative transfer. Moreover, we investigate model performance under two learning settings and find that instance-level rubrics and teacher-model feedback further facilitate model learning. Importantly, we observe that current LLMs with stronger static abilities do not show a clear advantage in learning capability across all tasks, highlighting that EvaLearn evaluates a new dimension of model performance. We hope EvaLearn provides a novel evaluation perspective for assessing LLM potential and understanding the gap between models and human capabilities, promoting the development of deeper and more dynamic evaluation approaches. All datasets, the automatic evaluation framework, and the results studied in this paper are available at the GitHub repository.

1 Introduction

Most LLM benchmarks measure static performance with independent parallel tests, leaving learning from prior task experience and the speed of that learning largely unmeasured. EvaLearn introduces sequential evaluation and five metrics to assess these dynamics across challenging tasks.

2 EvaLearn

  • EvaLearn evaluates learning capability and efficiency through sequential problem solving and five metrics designed to remain independent of the learning method used.This method-independent design is intended to support different learning strategies.Source: §1 · p. 2

1 Introduction

  • Across nine frontier LLMs, learning varied by task; thinking-based models generally learned more effectively, feedback learning often outperformed demonstrations, and static ability did not predict learning consistently.The authors recommend considering learning capability and efficiency jointly.Source: §1 · p. 2

2.1 Datasets

  • The benchmark contains 648 problems in 182 seven-problem sequences across six task types, with each problem accompanied by a question, canonical answer, and human-written rubric.Problems within a sequence share a task category and are designed to challenge learning from prior experience.Source: §1 · p. 2, §2.1 · p. 3

2.2 Automatic Evaluation

  • Because many answers are difficult to verify with fixed rules or may have multiple valid forms, EvaLearn combines instance-level rubrics with an LLM judge.A human check found evaluation accuracy above 95% for all tasks.Source: §2.2 · pp. 3–4

2.3 Evaluation Metrics

  • The five metrics cover overall and position-wise accuracy, fitted learning slope, first-correct timing, offset to first learned correctness, consecutive correct runs, and post-warmup accuracy.Together they are presented as a nuanced measure of learning behavior in sequential problem solving.Source: §2.3 · pp. 4–5

3 Benchmarking LLMs with EvaLearn

EvaLearn benchmarks sequential learning across nine frontier LLMs, comparing learning settings and performance across tasks. Results show that learning capability and efficiency vary by model and task, without a consistent cross-task advantage for models with stronger static abilities.

3.1 Setup

  • Experiments compare zero-shot and few-shot parallel solving with sequential demonstration and feedback learning across nine frontier LLMs.Feedback includes prior solutions and judge-provided feedback assessed with instance-level rubrics.Source: §3 · p. 5, §3.1 · pp. 5–6

3.2 RQ I: Can LLMs learn a task by engaging with a sequence of problems?

  • Five models improve and four decline under feedback versus zero-shot; OpenAI-o3-mini has the highest overall improvement (+10.5%).Table 2 compares accuracy in the two settings, with parentheses showing their difference.Source: Table 2 · p. 6, §3.2 · p. 6
  • Feedback gains include GPT-4o’s 18.0% gain in mathematical reasoning and Claude-3.7-Sonnet-Thinking’s 13.5% gain in classification, while 7 of 9 models decline on summarization.The authors suggest clear solution steps may help feedback identify errors in reasoning and classification; summarization may rely more on pretrained knowledge and instruction following.Source: §3.2 · pp. 6–7
  • Most models achieve higher accuracy later in sequences, but learning stability varies by task; in summarization, high consecutive-correct streaks can coexist with declining post-warmup accuracy.Logical reasoning is harder for many models to sustain long streaks; thinking-based variants generally maintain more consecutive correct solutions than their non-thinking counterparts.Source: §3.2 · p. 7

3.3 RQ II: How efficient are LLMs at learning from a sequence of problems?

  • Non-thinking models tend to improve more rapidly, while thinking-based models show steadier gains; sequential reasoning remains challenging, with widespread declines.Claude-3.7-Sonnet has the highest overall learning efficiency (k = 2.08), while Claude-3.7-Sonnet-Thinking is the only model with meaningful positive learning on sequential reasoning (k = 1.89).Source: §3.3 · pp. 8–9

3.4 RQ III: Do different learning methods lead to differences in performance?

  • Sequential demonstration generally outperforms few-shot parallel solving, and feedback learning achieves higher average accuracy and learning efficiency than demonstration for most models.Claude-3.7-Sonnet-Thinking improves across five tasks over few-shot; DeepSeek-V3 and Doubao-1.5-Thinking-Pro learn from judge feedback across five tasks.Source: §3.4 · p. 9
  • No model consistently achieves stable learning improvements across all tasks, and state-of-the-art models show no clear cross-task advantage in learning capability or efficiency.Learning capability is not strongly correlated with static performance.Source: §3.4 · p. 9
  • Pfirst identifies how quickly models first succeed, while Poffset shows first learned successes typically occur three to four positions after the first zero-shot miss.For example, o3-mini and Doubao-1.5-Thinking-Pro have average Pfirst values of 1.57 and 1.80; models generally learn earlier in classification and extraction than logical reasoning.Source: §3.4 · pp. 9–10

4 Conclusion

EvaLearn offers a sequential benchmark for assessing model learning and efficiency, and its evaluation reveals varied learning profiles across frontier models.

4 Conclusion

  • EvaLearn sequentially evaluates models’ learning capability and efficiency within specific tasks.The conclusion presents it as a novel benchmark.Source: §4 · p. 10
  • Frontier models differ in performance across tasks, and their ability to benefit from teacher-model feedback also varies.Some models leverage feedback on previous solutions, while others struggle to benefit.Source: §4 · p. 10
  • The authors present EvaLearn as a new perspective on assessing LLM potential and a step toward dynamic evaluation.Source: §4 · p. 10

1. Claims

The paper clarifies its claims, discusses limitations and future work, reports no theoretical assumptions or proofs, and describes its experimental results and reproducibility materials.

1. Claims

  • The authors state that they clarified the claims in the abstract and introduction.Source: §1 · p. 19

2. Limitations

  • The paper discusses limitations and future work in Section 5.Source: §2 · p. 19

3. Theory assumptions and proofs

  • The authors characterize EvaLearn as a benchmark without theoretical assumptions or proofs.Source: §3 · p. 20

4. Experimental result reproducibility

  • The authors say experimental results and setups are reported in Section 3 and the appendix, with datasets, the evaluation framework, and results open-sourced in supplemental materials.Source: §4 · p. 20

5. Open access to data and code

The paper reports open access to its data and evaluation materials, notes that models require no training, and describes API-based evaluation and costs.

5. Open access to data and code

  • The authors say datasets, the automatic evaluation framework, and results are open-sourced in supplemental materials with a full explanation.Source: §5 · p. 21

6. Experimental setting/details

  • The benchmark requires no model training, and experimental setups are included in Section 3.Source: §6 · p. 21

7. Experiment statistical significance

  • All models were evaluated through their respective APIs, with implementations provided in Appendix D.Source: §7 · p. 21

8. Experiments compute resources

  • The authors report API costs in Appendix D and state that the benchmark requires no model training.Source: §6 · p. 21, §8 · p. 22

9. Code of ethics

The authors report ethics compliance, discuss societal impacts, and describe asset documentation, annotation labor, human-subject status, and manuscript-only LLM use.

9. Code of ethics

  • The authors state that their research conforms to the NeurIPS Code of Ethics and that societal impacts are discussed in Appendix A.Source: §9 · p. 22
  • The authors state that EvaLearn poses no such risks and that its data contain no biased information.Source: §9 · p. 23
  • Existing datasets are credited, with version information and licenses in Appendix B and asset licenses in supplemental materials.Source: §9 · p. 23
  • The authors report detailed asset descriptions and usage information in supplemental materials.Source: §9 · p. 24
  • Company staff with relevant professional expertise performed all annotation and received standard salaries.Source: §9 · p. 24
  • The authors state that the paper involves no crowdsourcing or research with human subjects.Source: §9 · p. 24
  • LLMs were used only to polish the manuscript and improve readability, not to affect the core methodology or originality.Source: §9 · p. 25

A Limitations and Future Work

EvaLearn identifies limits in current task coverage, model coverage, and learning-method analysis, while describing its relationship to prior evaluation paradigms and its data-annotation process.

A Limitations and Future Work

  • EvaLearn excludes highly challenging open-ended problems when systematic instance-level rubrics are difficult to construct, leaving those tasks outside its current coverage.The authors plan to develop evaluation strategies for open-ended responses.Source: A Limitations and Future Work · p. 26
  • The study benchmarks frontier models, but how open-source models compare in task-learning ability remains poorly understood.The authors plan to extend evaluation to more open-source models; they report current frontier models substantially outperform existing open-source models.Source: A Limitations and Future Work · p. 26
  • EvaLearn analyzes learning capability and efficiency, but the factors driving model differences warrant deeper investigation.The authors report that these measures are not strongly correlated with static abilities.Source: A Limitations and Future Work · p. 26
  • The study examines two learning methods, leaving broader strategies such as retrieval-augmented generation for future analysis.The authors plan to use EvaLearn to systematically analyze additional methods in task-learning scenarios.Source: A Limitations and Future Work · p. 26

B Related Work

  • Unlike parallel benchmarks focused on static abilities and interactive evaluations with i.i.d. tasks, EvaLearn tests sequential problem solving within one task.Its metrics are decoupled from the learning method, supporting evaluation across approaches.Source: B Related Work · pp. 26–27

C Broader Impact

  • The authors present EvaLearn as a dynamic assessment of LLM learning ability and efficiency and report no perceived negative societal impacts.They state that the data contain no biased information.Source: C Broader Impact · p. 27

D License For Artifacts and Data Consent

E Data Annotation Process

  • The annotation process uses human and LLM contributions to expand and filter challenging problems, construct and validate rubrics, and form randomized same-category sequences.Seed problems are retained when three sampled models all fail; rubrics are written for retained problems and revised if validation fails.Source: E Data Annotation Process · pp. 27–28

F List of Evaluation Models

This block lists evaluated models and illustrates sequential learning through two case studies: feedback helps Claude-3.7-Sonnet solve a maze, and DeepSeek-R1 refers to prior puzzle solutions.

F List of Evaluation Models

  • The evaluated models include Claude-3.7-Sonnet and its Thinking variant, DeepSeek-V3 and R1, and OpenAI-o3-mini.The list also describes Claude-3.7-Sonnet-Thinking as using extended thinking and OpenAI-o3-mini as a cost-efficient reasoning model.Source: F List of Evaluation Models · pp. 28–29

Case Study 1: Comparison with Parallel Solving (Zero-shot) and Sequence Solving (Feedback Learning)

  • In the maze case, Claude-3.7-Sonnet misses the exit at (7, 4) in zero-shot solving but reaches (7, 12) with feedback learning.With feedback, it defines directional conventions and tracks orientation after turns.Source: Case Study 1: Comparison with Parallel… · p. 29

Case Study 2: Evidence of learning from prior experience

  • DeepSeek-R1’s 4×4 puzzle reasoning refers to a previous solution and applies a learned pattern in an 11-step answer within the 12-move limit.The paper characterizes this case as extracting patterns from prior experience and applying them to related problems.Source: Case Study 2: Evidence of learning… · p. 31

G.2 Results of Overall sequence accuracy (Acc)

The figures compare overall accuracy across four solving methods, spanning parallel and sequential approaches.

G.2 Results of Overall sequence accuracy (Acc)

  • Overall accuracy is compared across four problem-solving methods.Figure 6 presents results for zero-shot, few-shot, demonstration learning, and feedback learning.Source: Figure 6 · p. 36, §G.2 · p. 36
  • Zero-shot and few-shot are the two parallel methods in the comparison.The figure caption distinguishes these from the sequential methods.Source: Figure 6 · p. 36
  • Demonstration learning and feedback learning are the two sequential methods compared with the parallel approaches.Source: Figure 6 · p. 36, §G.2 · p. 36

G.3 Results of Slope of fitted accuracy curve k

The appendix presents metrics and figures for learning speed, first success, and sustained correctness across sequences, separately for feedback and demonstration learning.

G.3 Results of Slope of fitted accuracy curve k

  • The fitted position-wise accuracy slope k summarizes learning speed across sequences, and Δk compares feedback with demonstration learning.The slope is obtained by fitting a straight line to position-wise accuracy using least-squares regression.Source: §G.3 · p. 37, Figure 7 · p. 37, Figure 8 · p. 38

G.4 Results of Average Position of First Correct Solution (Pfirst)

  • Pfirst measures the average position at which a model first succeeds, indicating how quickly it achieves an initial success in a sequence.Figures 10 and 11 show results for feedback and demonstration learning, respectively.Source: §G.4 · p. 40, Figure 10 · p. 40, Figure 11 · p. 41

G.5 Results of Average Number of Consecutive Correct Solutions (Nconsec)

  • Nconsec measures the average number of consecutive correct solutions, reflecting how consistently a model uses experience within a sequence.Figures 12 and 13 report results for feedback and demonstration learning, respectively.Source: §G.5 · p. 42, Figure 12 · p. 42, Figure 13 · p. 43

G.6 Results of Post-Warmup Accuracy (Accpw−K)

The appendix reports post-warmup and position-wise accuracy for both learning settings and includes the LLM-as-a-judge evaluation prompt.

G.6 Results of Post-Warmup Accuracy (Accpw−K)

  • Post-warmup accuracy measures performance after an initial warmup phase in which the model accumulates experience.Figures 14 and 15 show feedback and demonstration learning results, respectively.Source: §G.6 · p. 44, Figure 14 · p. 44, Figure 15 · p. 45

G.7 Results of Position-wise Accuracy Curve

H.1 Evaluation Prompt for LLM-as-a-judge

  • The LLM-as-a-judge prompt uses a strict two-level score: an answer receives 1 only if it meets every standard-answer requirement, otherwise 0.The prompt instructs the grader to compare each requirement and review its analysis before assigning the score.Source: §H.1 · p. 48, Figure 18 · p. 49

H.2 System Prompts for Different Solving Methods

The prompts distinguish parallel zero-shot and few-shot solving from sequential demonstration and feedback learning. The sequential templates show examples or prior questions with standard answers.

H.2 System Prompts for Different Solving Methods

  • The four prompt types pair zero-shot and few-shot with parallel solving, and demonstration and feedback learning with sequential solving.The system-prompt figures label these pairings explicitly.Source: Figure 19 · p. 49, Figure 20 · p. 49, Figure 21 · p. 50
  • The demonstration template frames the model as a student and introduces an example question with its standard answer in the history.The example question is represented by an {example_question_1} placeholder.Source: §H.2 · p. 49
  • The feedback template uses the same student framing and introduces a prior question with its standard answer in the history.Its placeholder is {previous_question_1}.Source: §H.2 · pp. 49–50

I Instances

The examples show related sliding-puzzle problems in one sequence, varying grid size, initial state, and symbol representation. Their move rules and step limits illustrate the logical-reasoning task.

I Instances

  • Instance 1 is a 4 × 4 sliding puzzle in which the blank space, marked '>', moves vertically or horizontally to restore the target configuration.The puzzle problems in the sequence center on sliding puzzles and logical reasoning.Source: I Instances · pp. 50–51
  • The two instances belong to one sequence and differ mainly in grid size, initial state, and symbol representation, testing whether solving experience carries across problems.EvaLearn uses sequential solving to evaluate whether models learn from experience and how efficiently they do so.Source: I Instances · p. 50, Figure 24 · p. 54
  • Instance 1 asks for a shortest valid move list and sets a maximum of 12 moves.Diagonal moves are disallowed.Source: I Instances · p. 51
  • Instance 2 uses a 6 × 6 grid, with 'B' as the movable space, and allows at most 18 moves to reach the goal.Its canonical answer contains 15 moves.Source: I Instances · pp. 52–54

Related papers

Loading 2506.02672v3…