Source-linked AI summary
Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving
Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, Ge Zhang, Jiaheng Liu, Xingyao Wang, Sirui Hong, Chenglin Wu, Hao Cheng, Chi Wang, Wangchunshu Zhou
TL;DR
Agent frameworks and their memory systems typically cannot share accumulated problem-solving experience across architectures, leaving agents to repeat solutions and mistakes. AGENT KB provides a framework-agnostic structured memory with hybrid planning and feedback retrieval plus a disagreement gate, and evaluations report consistent gains across reasoning and software-engineering benchmarks. The approach works without retraining, while automatically distilled experiences perform comparably to curated ones.
Problem
Agent frameworks operate in isolation, while existing memory systems generally remain individual-agent or framework-specific and do not support cross-architecture knowledge transfer.
Method
AGENT KB abstracts heterogeneous agent trajectories into structured experiences, exposes them through lightweight APIs, retrieves them for planning and feedback refinement, and gates incoherent updates.
Results
AGENT KB consistently improves diverse agent–model combinations across GAIA, HLE, GPQA, and SWE-bench, including 18.7pp for smolagents on GAIA pass@3 and 4.0pp for OpenHands on SWE-bench Lite pass@1.
Takeaways & Limitations
A shared, evolving memory backbone offers a practical step toward collective agent intelligence across heterogeneous agent frameworks.
Takeaways & Limitations
Supporting complex tasks requires better experience structuring and retrieval, even though larger knowledge bases reliably improve performance.
Abstract
from arXiv · showhide
AI agent frameworks operate in isolation, forcing agents to rediscover solutions and repeat mistakes across different systems. Despite valuable problem-solving experiences accumulated by frameworks like smolagents, OpenHands, and OWL, this knowledge remains trapped within individual systems, preventing the emergence of collective intelligence. Current memory systems focus on individual agents or framework-specific demonstrations, failing to enable cross-architecture knowledge transfer. We introduce AGENT KB, a universal memory infrastructure enabling seamless experience sharing across heterogeneous agent frameworks without retraining. AGENT KB aggregates trajectories into a structured knowledge base and serves lightweight APIs. At inference time, hybrid retrieval operates through two stages: planning seeds agents with cross-domain workflows, while feedback applies targeted diagnostic fixes. A disagreement gate ensures retrieved knowledge enhances rather than disrupts reasoning, addressing knowledge interference in cross-framework transfer. We validate AGENT KB across major frameworks on GAIA, Humanity's Last Exam, GPQA, and SWE-bench. Results show substantial improvements across diverse model families: compared to baseline pass@1, smolagents with AGENT KB achieve up to 18.7pp gains at pass@3 (55.2% -> 73.9%), while OpenHands improves 4.0pp on SWE-bench pass@1 (24.3% -> 28.3%). Similar improvements are observed across all base model families. Ablations confirm that hybrid retrieval and feedback stages are essential, with automatically generated experiences matching manual curation. This establishes the foundation for collective agent intelligence through shared memory infrastructures.
1 INTRODUCTION
AGENT KB addresses fragmented agent knowledge by enabling cross-framework experience sharing without retraining, using structured experiences, staged retrieval, and disagreement-gated integration. Across diverse frameworks, tasks, and model families, it reports consistent performance gains.
- Agent frameworks and memory systems largely operate within isolated architectures, forcing repeated problem solving and repeated mistakes.
- AGENT KB abstracts heterogeneous trajectories into structured experiences and exposes them through lightweight APIs for framework-compatible sharing without retraining.
- Its disagreement gate selectively integrates coherent external updates to address knowledge interference during cross-framework transfer.
- 18.7pp is the largest reported GAIA smolagents gain at pass@3, rising from 55.2% to 73.9%.
- 4.0pp is the reported SWE-bench Lite OpenHands gain at pass@1, increasing from 24.3% to 28.3%.
2 RELATED WORK
Prior work advances agent memory, trajectory retrieval, workflow reuse, and structured planning, but the cited approaches remain limited in information management, transfer, or framework scope.
- Modern memory systems support complex reasoning through latent, graph-based, and hierarchical representations, yet still struggle with large-scale information management and knowledge transfer.
- The related approaches described remain organized around individual systems, narrow task families, or limited framework settings rather than universal cross-framework sharing.
- Trajectory-based systems retrieve exemplar traces, mined sub-workflows, templates, or structured repositories to guide planning and tool use.
3 METHODOLOGY
AGENT KB converts agent traces into shared structured experiences, evolves the memory through quality-aware maintenance, and injects retrieved knowledge during planning and feedback while gating disruptive refinements.
- Overview: AGENT KB captures execution traces, abstracts them into reusable experiences, and retrieves them to guide planning and execution refinement.
- Self-Evolving Agent KB: Heterogeneous trajectories and human seeds are indexed, while memory evolution adds, resolves conflicts, deduplicates, and evicts experiences as quality and utility change.
- Experience Representation: The experience schema represents task embeddings, goal constraints, action–reasoning pairs, and cross-framework metadata using E = ⟨π,γ,S,C⟩.
- Retrieval Pipeline: Hybrid retrieval combines BM25 lexical shortlisting with embedding-based semantic ranking before deduplication and refinement.
- Planning Stage: Planning adapts retrieved trajectories through entity mapping, tool substitution, and step reordering to produce an executable native-environment plan.
- Feedback Stage: Feedback retrieves fixes from execution traces and applies only refinements whose plan similarity passes the disagreement gate threshold β = 0.8.
4 EXPERIMENT
AGENT KB is evaluated across reasoning and software-engineering benchmarks, agent frameworks, model families, retrieval configurations, and knowledge-base sizes. Results show broad performance gains, with hybrid retrieval, feedback, refinement, and structured knowledge contributing to transfer while overhead remains modest.
- Main results: GAIA results show consistent gains across heterogeneous agent stacks and model families, including GPT-4.1 smolagents rising from 55.2% to 73.9% pass@3.The largest reported smolagents lift is +18.7 points overall, while OWL with GPT-4o improves from 43.6% to 63.6%.
- Main results: SWE-bench Lite results improve across iteration budgets and backbones, with Claude-3.7 gaining 21.0 points at 50 iterations.Claude-3.7 rises from 30.0% to 51.0% at 50 iterations and from 41.3% to 53.3% at 100 iterations.
- Main results: AGENT KB also improves scientific question answering, with OpenHands reaching 14.1% at pass@3 on HLE and GPT-4.1 rising from 62.6% to 72.7% on GPQA.These improvements require neither additional fine-tuning nor tool customization.
- Ablation studies: Ablations identify refinement, planning, feedback, hybrid retrieval, and knowledge-base scale as important components, while complex tasks remain constrained by abstraction quality and retrieval structure.Removing Refine causes the largest drop (−6.06); hybrid retrieval peaks at k = 3 with 83.0% on GAIA Level 1, and larger stores do not improve advanced reasoning when abstraction quality is limiting.
- Ablation studies: Automatically refined experiences match manual curation on GAIA and outperform it on Level 3, while latency, memory, and monetary overhead remain modest.Automatic experiences achieve 75.15% versus 76.97% on GAIA and 57.69% versus 53.85% on Level 3; retrieval adds less than 0.4% of GAIA evaluation cost.
5 CONCLUSION
AGENT KB abstracts heterogeneous agent traces into reusable experiences and combines hybrid retrieval with disagreement-gated refinement. Across four benchmarks, it consistently improves performance, while automatically generated experiences perform comparably to curated ones and better on harder tasks.
- AGENT KB abstracts heterogeneous agent traces into reusable experiences as a cross-framework memory layer.
- Hybrid retrieval and disagreement-gated refinement address representation heterogeneity, context mismatch, and knowledge interference.
- AGENT KB produces consistent improvements across GAIA, HLE, GPQA, and SWE-bench.
- Automatically generated experiences perform comparably to curated experiences and surpass them on harder tasks.
- Future work targets richer modalities and longer-horizon reasoning.
ETHICS STATEMENT
The paper reports no direct ethical concerns from its publicly available, licensed datasets. It nevertheless highlights privacy, bias in retrieved knowledge, and misuse risks in high-stakes deployments, alongside its experience-construction procedures.
- The study uses publicly available datasets and follows their respective licenses.
- The experience knowledge base spans code reasoning, web navigation, multi-hop retrieval, and human-level evaluation tasks.
- Table 5 summarizes source datasets, original task counts, and resulting experience-entry counts.
- The construction includes task-specific experiences from BrowseComp, MultiHopRAG, HLE, WebWalkerQA, RepoClassBench, SWE-Gym-Raw, and RepoEval.
- Human annotators inspect failed logs and abstract recurring issues into correction templates for experience generation.
- Handcrafted workflows are authored from successful and failed GAIA logs, standardized into tool-neutral instructional prompts, and evaluated by few-shot testing.
C ABLATION DETAILS OF Reason-Retrieve-Refine MODULES
The ablation framework tests the planning and feedback retrieval stages, reasoning, retrieval, refinement, and raw-workflow alternatives. These components are removed systematically to measure their contributions to performance and reasoning quality.
- The deployment loop contains two retrieval phases with distinct objectives, evaluated through systematic ablations.
- Planning and feedback: Planning forms the initial executable workflow by cycling through reasoning, retrieval, and refinement.
- Planning and feedback: Feedback reuses the cycle on execution traces, retrieving precedents and refining plans under the disagreement gate.
- Ablated modules: The ablations remove planning, feedback, reasoning, retrieval, or refinement modules.
- Ablated modules: The raw-workflow variant uses the full retrieval pipeline without explicit modular control through the Reason and Refine phases.
- The experiments assess each module’s contribution to accuracy, robustness, and coherence in complex reasoning tasks.
D INFERENCE COST BREAKDOWN
AGENT KB adds small inference-time costs relative to full evaluation budgets. Its retrieval loop costs less than 0.4% of GAIA evaluation and under one cent per SWE-bench Lite issue with GPT-4.1.
- GAIA: In GAIA, the retrieval loop adds $0.27 to an $86.0 USD full evaluation, less than 0.4% per run.
- SWE-bench Lite: On SWE-bench Lite, AGENT KB hinting costs less than $0.004 USD per issue with prompts shorter than 7,000 tokens.
- Tables 6 and 7 report token and monetary budgets for GAIA and SWE-bench Lite, respectively.
- The offline ingestion step is a one-time expense amortized over future runs.
- The examples section presents execution demonstrations across domains and task types.
E.1 EXECUTION EXAMPLE
The execution examples show AGENT KB correcting an initially incomplete reasoning path through feedback and supplying targeted guidance for software-engineering tasks.
- EC-number example: The initial retrieval found an EC number for only one chemical and incorrectly concluded that the answer was solely “3.1.3.1.”This produced two sequential erroneous returns.
- EC-number example: Feedback clarified that only enzymes have EC numbers, preventing searches for non-enzyme chemicals and avoiding a mistaken no-answer response.The refined directive narrowed the task trajectory and expected outcome.
- EC-number example: The corrected reasoning identified alkaline phosphatase and horseradish peroxidase, returning their alphabetized EC numbers as 3.1.3.1;1.11.1.7.The model prediction and final code output agree on this ordered answer.
- Software-engineering examples: SWE-bench examples provide specific guidance for fixing API-documentation escaping and symbolic-mathematics evaluation bugs.The retrieved advice emphasizes context-sensitive string handling and preserving validation for invalid mathematical inputs.
E.2.2 RAW LOG TO EXPERIENCE GENERATION
This subsection describes converting raw agent execution logs and complex task analyses into structured experiences containing reusable strategies, failure modes, and domain-specific guidance.
- Raw-log transformation: AGENT KB extracts reusable experiences from raw logs, including how agents adapt when unexpected file formats disrupt a bioinformatics workflow.The subsection presents this transformation as part of the system’s learning capability.
- Key insights: The examples identify domain adaptation, error recovery, and precision management as capabilities supported by stored experiences.They span software engineering, bioinformatics, and biographical research.
- Raw-log transformation: The extracted bioinformatics guidance preserves exact numerical and formatting requirements, including three-decimal Ångström outputs without units.The experience specifies using the first two ATOM coordinates and retaining the computed value’s three-decimal precision.
- Structured experience generation: Generated JSON structures separate planning strategies and experiences for general and specialized agents, enabling systematic knowledge transfer and strategy refinement.The structures capture successful strategies and lessons learned from task execution.
- Structured experience generation: The generation procedure simulates agent planning, constructs realistic error scenarios, and produces actionable, generalizable guidelines without directly revealing the supplied answer.For embedded Search Agents, it also specifies query formulation, source prioritization, information extraction, and validation.
G.2 EXPERIMENTAL SETUP
The evaluation covers four agent frameworks powered by varied LLM configurations and measures performance using ground-truth task-completion metrics.
- Agent backbones: The study evaluates smolagents, OWL, SWE-Agent, and OpenHands across distinct LLM configurations, with AGENT KB interfacing through standardized APIs.The tested configurations include GPT-4o, GPT-4.1, Claude-3.7, Qwen, DeepSeek-R1, and o3-mini variants.
- Evaluation methodology: Performance assessment uses exact-match accuracy for GAIA and test-passage rates for SWE-bench rather than LLM-generated evaluation scores.The metrics are based solely on ground-truth task completion.
- Transparency: The manuscript reports that researchers authored the technical contributions, experimental design, results interpretation, and scientific claims, while language models assisted only with editorial tasks.The disclosed editorial assistance covered grammar, typo detection, and prose clarity.
G.4 METHODOLOGICAL CONSIDERATIONS
AGENT KB is designed to remain independent of individual agent architectures, but its use incurs measurable inference costs during experience construction and retrieval.
- Computational overhead: $3.0–$4.5 per task in estimated inference overhead accompanies the reported 4.0–18.7 percentage-point performance gains.The cost analysis is provided in Appendix D.
- Architectural independence: AGENT KB transfers knowledge across frameworks through semantic embeddings and standardized action vocabularies without requiring changes to agent architectures.This design supports portability across model families and API interfaces.