PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamBench: Towards Benchmarking Continuous Improvement of Language Agents

Cheng-Kuang Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Hung-yi Lee

arXiv:2406.08747v2cs.CL

TL;DR

Existing benchmarks assess LLMs’ innate capabilities but not their improvement over time. StreamBench evaluates agents on input-feedback sequences and proposes streaming baselines; experiments report consistent gains for streaming methods and robust, cost-effective performance from shared memory across agents.

  • Problem

    Existing benchmarks focus on LLMs’ innate capabilities, leaving their ability to improve over time unevaluated despite the goal of gradual post-deployment improvement.

    Source: §1 · p. 1

  • Method

    StreamBench evaluates LLM agents over input-feedback sequences and proposes baselines that use streaming feedback to improve performance.

    Source: §1 · p. 2

  • Results

    Streaming methods outperform non-streaming methods overall, while correctness feedback on self-generated outputs improves performance consistently and MAM-StreamICL offers robust performance at the average cost of a single agent.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The experiments indicate that collecting correctness feedback and sharing memory across agents are promising streaming strategies within StreamBench.

    Source: §7 · p. 9

    StreamBench covers text tasks but not all task types or domains, and its simulated correctness feedback may not capture the more diverse, noisy, context-dependent feedback of real-world applications.

    Source: §8 · p. 9

Abstract

from arXiv · show

Recent works have shown that large language model (LLM) agents are able to improve themselves from experience, which is an important ability for continuous enhancement post-deployment. However, existing benchmarks primarily evaluate their innate capabilities and do not assess their ability to improve over time. To address this gap, we introduce StreamBench, a pioneering benchmark designed to evaluate the continuous improvement of LLM agents over an input-feedback sequence. StreamBench simulates an online learning environment where LLMs receive a continuous flow of feedback stream and iteratively enhance their performance. In addition, we propose several simple yet effective baselines for improving LLMs on StreamBench, and provide a comprehensive analysis to identify critical components that contribute to successful streaming strategies. Our work serves as a stepping stone towards developing effective online learning strategies for LLMs, paving the way for more adaptive AI systems in streaming scenarios. Source code: https://github.com/stream-bench/stream-bench. Benchmark website: https://stream-bench.github.io.

1 Introduction

Existing benchmarks assess LLMs’ out-of-the-box knowledge and reasoning, while recent work shows that agents can improve from experience. StreamBench evaluates that improvement online over input-feedback sequences and introduces baselines and analysis of streaming strategies.

1 Introduction

  • MMLU [3], GSM8K [4], and BIG-Bench-Hard [5] assess out-of-the-box knowledge or reasoning rather than improvement over time.The paper defines these abilities as LLMs’ innate capabilities.Source: §1 · p. 1
  • Prior agents improve through stored user feedback [6], repeated self-reflection [7], or cross-task experience [8].These examples motivate evaluating improvement beyond innate capabilities.Source: §1 · p. 1
  • StreamBench evaluates whether LLM agents improve over input-feedback sequences, addressing a gap in benchmarks focused on innate capabilities.The benchmark simulates a stream of users’ natural-language requirements and feedback across a wide range of tasks.Source: §1 · pp. 1–2
  • The authors propose simple streaming baselines, including a multi-agent method that outperforms other baselines at the average cost of a single agent.They also analyze advantages and potential pitfalls to identify insights into effective streaming strategies.Source: §1 · p. 2

2 Formulation

The formulation models an agent interacting with an external environment over a stream of inputs, receiving feedback after each prediction. At each step, agent components can be updated to improve accuracy across the sequence.

2 Formulation

  • An agent is an LLM with parameter θ and optional components such as external memory M and retriever r(·), used with a prompting template p(·).Its output is computed from the input and retrieved memory using the prompt and model.Source: §2 · p. 2
  • The environment g(·) supplies feedback, with possible sources including human users, code execution environments, and API responses.The feedback type depends on the downstream task and what feedback is collected.Source: §2 · p. 2
  • At each time step, the agent predicts on an input, receives environment feedback, and may update its prompt, retriever, memory, or parameters.The framework aims to maximize accuracy over the entire input sequence.Source: §2 · pp. 2–3
  • The baselines focus on updating prompts and memory; retriever-weight and model-parameter updates are left for future work because network updates are computationally expensive.Other possible strategies include prompt refinement, memory expansion, retriever updates, and parameter-efficient fine-tuning.Source: §2 · p. 3

3 StreamBench

StreamBench evaluates agents in a streaming setup using binary correctness feedback and final-step aggregate metrics. Its datasets span text-to-SQL, programming, tool use, medical diagnosis, and question answering.

3.1 General setup

  • Selected datasets are shuffled with fixed seeds for reproducible streaming order, and evaluation also uses five random seeds to test robustness.The authors also discuss distribution shifts in Appendix C.2.Source: §3.1 · p. 3
  • Binary feedback serves as a correctness proxy, offering a practical alternative because ground truth can be costly to obtain at every step.The authors leave ground-truth and natural-language feedback designs for future work.Source: §3.1 · p. 3
  • Agents are evaluated by their aggregate metric at the final time step, T.The metric function h is dataset-specific.Source: §3.1 · p. 3

3.2 Datasets

  • Text-to-SQL coverage uses Spider, CoSQL, and BIRD, which represent increasing difficulty; the selected tasks share costly per-step ground truth.CoSQL and BIRD development sets serve as test sets because their official test sets are unavailable.Source: §3.2 · p. 4
  • Programming uses DS-1000 questions requiring tested Python solutions, while tool-use evaluation uses ToolBench examples with stable, low-latency APIs.The selected ToolBench subset was collected in prior work [22].Source: §3.2 · p. 4
  • Medical diagnosis uses DDXPlus across 49 diagnoses, while HotpotQA tests reasoning over supporting and distracting documents using 1,500 sampled questions.HotpotQA contains 7,410 questions in total.Source: §3.2 · p. 4

4 Experiments

The experiments compare per-instance methods with methods that use past input-feedback streams, then evaluate whether streaming improves downstream-task performance. Streaming methods outperform non-streaming methods overall, while MAM-StreamICL has the most notable and consistent boost across datasets.

4.1 Baselines

  • Non-streaming methods improve performance independently per test instance, whereas streaming methods use information from past instances to improve future performance.The comparison includes two adapted streaming methods and two newly introduced approaches.Source: §4.1 · p. 4

4.1.1 Non-streaming methods

  • Non-streaming baselines include zero-shot, few-shot examples, chain-of-thought prompting, and Self-Refine’s iterative per-instance self-feedback.Self-Refine generates an initial response and refines it through multiple iterations.Source: §4.1.1 · p. 5

4.1.2 Streaming methods

  • GrowPrompt stores recent input-output-feedback triples in a sliding window, while MemPrompt retrieves past triples from external memory for inference.Both methods verbalize feedback to indicate whether the output was correct.Source: §4.1.2 · p. 5
  • Self-StreamICL stores only correct self-generated input-output pairs in memory, which accumulate as in-context examples without preconstructed few-shot data.It omits verbalized correctness feedback.Source: §4.1.2 · p. 5
  • MAM-StreamICL shares memory across agents and switches among them in round-robin order; only one agent answers per time step, keeping average cost at a single agent’s level.The shared memory receives correct outputs, and the scheduling lets each agent contribute in a balanced manner.Source: §4.1.2 · pp. 5–6

4.2 Implementation details

  • Experiments use GPT-3.5-Turbo, Gemini 1.0 Pro, and Claude 3 Haiku endpoints; the memory-based methods use a vector database with BAAI/bge-base-en-v1.5 embeddings.The three endpoints initialize MAM-StreamICL’s agents.Source: §4.2 · p. 6

4.3 Main results

  • Streaming methods outperform non-streaming methods across datasets to varying degrees, and MAM-StreamICL shows the most notable, consistent performance boost.Table 2 reports averaged performance for three LLM agents; MAM-StreamICL is the exception, run once on the streaming sequence.Source: Table 2 · p. 6, §4.3 · p. 6

5 Discussion

The discussion identifies correct self-output and shared memory as effective components of streaming strategies, then examines robustness across sequences and benefits for stronger models. It also notes that token usage is available to help benchmark users estimate cost.

5.1.1 Collecting correct self-output

  • Using only correct self-output consistently improves over zero-shot, while using incorrect self-output can degrade performance, sometimes below zero-shot.The ablations compare using all retrieved triples with using only correct or only incorrect triples; simply telling agents their output was incorrect does not help them learn from mistakes.Source: §5.1.1 · p. 7

5.1.2 Sharing memory across multiple agents

  • Different agents excel at different diagnosis categories, and sharing their past experiences is associated with enhanced overall performance at a cost similar to one agent.Figure 3 shows diagnosis confusion matrices for three LLMs and MAM-StreamICL; the discussion says agents take turns on incoming instances and can benefit from shared memory.Source: §5.1.2 · p. 7, Figure 3 · p. 8

5.2 Robustness to different streaming sequences

  • Self-StreamICL and MAM-StreamICL remain the top performers across shuffled sequences, with the ranking mostly consistent across datasets.The study averages Claude-3-Haiku results over five sequences; results for GPT-3.5-Turbo and Gemini-1.0-Pro are in Appendix C. A fixed sequence is selected for fair comparisons, while all five are released for thorough evaluation.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 Would stronger LLMs still benefit from streaming?

  • Self-StreamICL also produces significant performance improvements with the tested stronger models, GPT-4o-2024-08-06 and Gemini-1.5-Flash-001.Because of cost, the study runs only the methods shown in Table 3.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Cost analysis

  • The paper lists token usage for all baselines in Appendix E so benchmark users can estimate cost.No cost values are given in the supplied passage.Source: §5.4 · p. 8

6 Related work

The related work situates StreamBench among online learning and LLM feedback-improvement research. It highlights the lack of standard testbeds for online adaptation and consistent evaluation across diverse tasks.

6.1 Online learning

  • Online learning traditionally updates model weights, while newer LLM-agent strategies can update prompts, memory, or retrievers; the authors identify a lack of standard testbeds for this online setting.StreamBench is presented as a benchmark intended to support development of more dynamic adaptation methods for LLM agents.Source: §6.1 · p. 9

6.2 Improvement from feedback with LLMs

  • Prior feedback-improvement methods target individual instances or learn across time or tasks, but use different datasets and lack standardized evaluation.StreamBench provides a consistent empirical testbed across diverse tasks for evaluating LLM-agent improvement.Source: §6.2 · p. 9

7 Conclusion

The paper presents StreamBench as an instance of an evaluation setting for LLM-agent improvement on downstream tasks. Its experiments highlight correct self-generated outputs and shared memory as promising streaming strategies, while it points to several directions for future work.

7 Conclusion

  • The experiments find that collecting correct self-generated outputs improves performance consistently, whereas informing agents of incorrect outputs sometimes degrades it.They also find MAM-StreamICL achieves robust performance while maintaining the average cost of a single agent.Source: §7 · p. 9
  • Future work could explore active learning for requesting feedback, multi-agent collaboration as multi-arm bandits, and feedback signals beyond correctness.The paper gives natural-language user feedback as an example of an alternative signal.Source: §7 · p. 9

8 Limitations

The current StreamBench version covers several task types but not every possible task or domain, and is limited to text. Its simulated correctness feedback may not capture the diversity, noise, or context dependence of real-world feedback.

8 Limitations

  • StreamBench covers programming, text-to-SQL, medical diagnosis, question-answering, and tool use, but not all task types or domains; it also excludes image and audio modalities.The stated modality scope is text only.Source: §8 · p. 9
  • Simulated correctness feedback may differ from real-world feedback, which can be more diverse, noisy, and context-dependent than the benchmark captures.Source: §8 · p. 9

Checklist

The checklist points readers to the paper’s reproducibility, resource, asset, and limitations disclosures. Hyperparameters use matched context sizes for streaming baselines, while encoder results suggest model-specific choices can matter.

Checklist

  • The checklist reports that code, data, and reproduction instructions are available in the supplementary materials, with training details in Appendices A and F.It also directs readers to Section 5.4 for compute and resource information.Source: Checklist · p. 13
  • The checklist directs readers to the experiment results for error bars and to Section 8 for limitations, while marking theoretical proofs and societal-impact discussion not applicable.Source: Checklist · p. 13

A Hyperparameters

  • Streaming and few-shot baselines use the same k for fair comparison: k = 16 on five datasets and k = 4 on DS-1000 and HotpotQA to fit gpt-3.5-turbo-0125’s context.Source: A Hyperparameters · p. 14
  • Within the bge encoder family, the 109M-parameter model generally performs better, though 22.7M-parameter models can also perform strongly and encoder suitability may vary by LLM.Source: A Hyperparameters · p. 14, Table 4 · p. 14

B Main results of each LLM endpoint

The appendix presents results for three LLM endpoints, averages across five shuffled sequences, and tests Self-StreamICL under BIRD distribution shifts. The text reports that Self-StreamICL remains ahead of non-streaming baselines in the shifted sequence.

B Main results of each LLM endpoint

C.1 Performance on sequences shuffled by five different random seeds

C.2 Performance on the sequence with distributional shifts

  • Self-StreamICL still outperforms non-streaming baselines on BIRD when database ordering introduces 10 distributional shifts, and its performance does not differ drastically under those shifts.Source: Table 8 · p. 16, §C.2 · p. 16

D Detailed ablation study results

The appendix supplies model-specific correctness-ablation figures, token-cost tables, dataset construction and metrics, and task prompt templates. The available captions and prose specify scope and setup but do not provide numerical ablation or cost outcomes.

D Detailed ablation study results

E Token cost breakdown for each LLM endpoint

F.2 Details for each dataset

  • Spider uses its original test set and refers to Spider 1.0; CoSQL and BIRD use development sets because their official test sets are unavailable, and CoSQL is flattened into 1,007 question-SQL pairs.Source: §F.2.1 · p. 18, §F.2.2 · p. 19, §F.2.3 · p. 19

F.2.1 Spider

  • For Spider, CoSQL, and BIRD, Text-to-SQL performance uses Execution Accuracy, the proportion of instances whose predicted and gold SQL execution results match.Source: §F.2.1 · p. 18

F.2.4 DS-1000; F.2.5 ToolBench

  • DS-1000 uses its test set and pass@1; ToolBench uses 50 curated APIs with 15 test instances each and requires a correct API name and arguments.Source: §F.2.4 · p. 19, §F.2.5 · p. 19

F.2.6 DDXPlus; F.2.7 HotpotQA

  • DDXPlus is reduced from over 100,000 cases to 1,764 samples balanced across diagnoses; HotpotQA uses 1,500 randomly sampled distractor-setting development instances.Source: §F.2.6 · p. 19, §F.2.7 · p. 19
  • DDXPlus is evaluated with accuracy, while HotpotQA uses exact match as its primary metric and also calculates F1.Source: §F.2.6 · p. 19, §F.2.7 · p. 20

F.3 Prompt templates; F.3.1 Text-to-SQL; F.3.2 Medical diagnosis

  • Prompt templates aim to minimize prompt engineering across tasks; Text-to-SQL and medical diagnosis include zero-shot, CoT, Self-Refine feedback/refinement, and past-information prompts.Past-instance templates differ by baseline, and the Self-Refine pipeline alternates feedback and refinement after its initial zero-shot output.Source: §F.3 · p. 21, §F.3.1 · pp. 21–22, §F.3.2 · p. 23

F.4.1 Algorithm of ablation studies with MAM-StreamICL

The ablation compares multi-agent inference paired with a single chosen agent’s memory against the full MAM-StreamICL design. The reported results identify both multi-agent memory and multi-agent inference as beneficial for performance.

F.4.1 Algorithm of ablation studies with MAM-StreamICL

  • The counterfactual retains round-robin multi-agent inference but uses one selected agent for memory; the results report benefits from both multi-agent memory and inference.Source: §F.4.1 · p. 25
Loading 2406.08747v2…