PaperlayerView exact revision on arXiv

Source-linked AI summary

KAPSO: A Knowledge-grounded framework for Autonomous Program Synthesis and Optimization

Alireza Nadafian, Alireza Mohammadshahi, Majid Yazdani

arXiv:2601.21526v2cs.AIcs.CLcs.SE

TL;DR

Reliable, optimized software requires repeated experimentation, evaluation, and reuse of expert engineering workflows rather than one-shot synthesis. KAPSO makes synthesis an operator in an evaluator-grounded optimization loop and evaluates this modular framework on MLE-Bench and ALE-Bench.

  • Problem

    Turning domain experts’ intent into reliable, runnable, and optimized software requires iterative experimentation and high-leverage engineering workflows.

  • Method

    KAPSO iteratively generates improvement hypotheses, synthesizes and edits code, executes artifacts, evaluates outcomes, and integrates git-native experimentation, structured knowledge retrieval, and cognitive memory.

  • Results

    KAPSO is instantiated and evaluated on the complementary MLE-Bench and ALE-Bench benchmarks for end-to-end performance.

  • Takeaways & Limitations

    The modular design supports applying the same evaluator-grounded optimization loop across domains by swapping evaluators and knowledge sources.

  • Takeaways & Limitations

    The relatively low number of ALE-Bench competitions may introduce noise, so larger task sets and multiple seeds are needed for more robust agent comparisons.

Abstract

from arXiv · show

We introduce KAPSO, a modular framework for autonomous program synthesis and optimization. Given a natural language goal and an evaluation method, KAPSO iteratively performs ideation, code synthesis and editing, execution, evaluation, and learning to improve a runnable artifact toward measurable objectives. Rather than treating synthesis as the endpoint, KAPSO uses synthesis as an operator within a long-horizon optimization loop, where progress is defined by evaluator outcomes. KAPSO targets long-horizon failures common in coding agents, including lost experimental state, brittle debugging, and weak reuse of domain expertise, by integrating three tightly coupled components. First, a git-native experimentation engine isolates each attempt as a branch, producing reproducible artifacts and preserving provenance across iterations. Second, a knowledge system ingests heterogeneous sources, including repositories, internal playbooks, and curated external resources such as documentation, scientific papers, and web search results, and organizes them into a structured representation that supports retrieval over workflows, implementations, and environment constraints. Third, a cognitive memory layer coordinates retrieval and maintains an episodic store of reusable lessons distilled from experiment traces (run logs, diffs, and evaluator feedback), reducing repeated error modes and accelerating convergence. We evaluated KAPSO on MLE-Bench (Kaggle-style ML competitions) and ALE-Bench (AtCoder heuristic optimization), and report end-to-end performance. Code Available at: https://github.com/Leeroo-AI/kapso

1 Introduction

KAPSO frames program synthesis as part of an evaluator-grounded optimization loop for improving runnable artifacts through repeated experimentation. It combines reproducible experiment tracking, structured knowledge retrieval, cognitive memory, and modular evaluation across benchmarked domains.

  • The framework addresses long-horizon coding-agent failures involving lost state, repeated integration errors, and weak reuse of engineering expertise.
  • KAPSO iteratively proposes improvements, edits code, executes artifacts, evaluates outcomes, and learns from measured feedback.
  • KAPSO supports pluggable evaluators and knowledge sources, and is evaluated on MLE-Bench and ALE-Bench with a released reproducible knowledge package.
  • A git-native experimentation engine isolates each attempt as a reproducible branch with explicit provenance.
  • A knowledge pipeline converts repositories, playbooks, documentation, papers, and web material into a typed, workflow-oriented knowledge base for optimization.
  • A cognitive memory layer retrieves structured knowledge and stores reusable lessons from run logs, diffs, and evaluator feedback to reduce repeated failures.

2 Framework Overview

KAPSO exposes operations for evolving, deploying, learning, and researching software artifacts around an explicit evaluator contract. Its orchestration loop combines retrieved knowledge, episodic insights, experiment history, isolated execution, and measured outcomes.

  • The evolve operation proposes improvements, applies code edits, executes artifacts, and uses evaluator measurements to guide later iterations.
  • Deploy adapts a selected solution to a target runtime and returns a unified software handle.
  • Learn curates repositories, playbooks, and benchmark artifacts into human-facing and retrieval-facing knowledge representations.
  • Research discovers papers, documentation, web-derived notes, and repositories that can seed later learning or provide context for evolution.
  • An explicit evaluator boundary defines execution, required outputs, and quality measurement, including automated, stochastic, LLM-judged, or preference-based evaluation.
  • The orchestrator alternates between constructing context from task state, retrieval, memory, and history, and running isolated experiments for measured outcomes.

3 Formalization (Notation and Algorithms)

KAPSO formalizes evaluator-grounded optimization as repeated search over executable artifacts under explicit goals, budgets, and evaluator contracts. The framework supports quantitative or preference-based selection, stochastic evaluation, feasibility handling, and history-dependent stopping.

  • 3.1 Evolve instance and evaluator contract: KAPSO formalizes each run with a natural-language goal, budget specification, and evaluator contract governing execution, measurement, comparison, and stopping.The formalization is intended to make KAPSO comparable with program synthesis, agentic search, and black-box optimization.
  • 3.1 Evolve instance and evaluator contract: The evaluator executes artifacts under a configuration, applies scalar utility or preference-based selection, and may aggregate stochastic rollouts.Selection can use automated metrics, test suites, rule-based comparators, LLM judges, or human-in-the-loop policies.
  • 3.1 Evolve instance and evaluator contract: An executable artifact is a repository state with sufficient entrypoint and configuration to run under the evaluator.A measurement record includes status, quantitative measurements, qualitative feedback, and auxiliary artifacts such as logs or traces.
  • 3.2 Objective, feasibility, and stochastic evaluation: KAPSO improves artifacts according to the evaluator’s selection rule, targeting expected utility when execution is stochastic.The objective distinguishes E[U(R)] from U(E[R]) when utility is nonlinear or records contain multiple components.
  • 3.2 Objective, feasibility, and stochastic evaluation: KAPSO estimates stochastic performance from K evaluator rollouts and aggregates their measurement records and utility values.The evaluator does not directly observe the conceptual optimum; aggregation produces an estimated utility for selection.
  • 3.2 Objective, feasibility, and stochastic evaluation: Errors define infeasible artifacts, while selection rules rank feasible artifacts above infeasible ones using sentinels or explicit preferences.When no scalar utility exists, KAPSO compares aggregated records with a preference relation that can incorporate multiple metrics and qualitative feedback.

3.3 Experiments, provenance, and history

KAPSO preserves an explicit history of executed experiments, representing each attempt as an isolated branch with its specification, artifact state, budget progress, rollout count, and evaluation outputs.

  • 3.3 Experiments, provenance, and history: Each experiment is an isolated execution on an experiment branch that records its motivating specification, produced artifact, and measured outcomes.The history is designed to retain experiment state across iterations.
  • 3.3 Experiments, provenance, and history: An experiment record stores branch identifier, specification, budget progress, rollout count, repository state, and evaluator records.The artifact state is defined as the repository state checked out from the experiment branch.

3.4 Knowledge grounding via seed repositories and retrieval

KAPSO grounds optimization in a seed repository and typed knowledge graph, retrieving reusable knowledge conditioned on the goal, starting codebase, and failure signals. Repository confidence determines whether initialization uses a retrieved repository or scaffold, while error recovery augments context after failures.

  • 3.4 Knowledge grounding via seed repositories and retrieval: KAPSO combines an optional seed repository with a typed knowledge graph containing principles, implementations, heuristics, and environment constraints.These assets provide initial runnable code and reusable context for optimization.
  • 3.4 Knowledge grounding via seed repositories and retrieval: A repository corpus stores candidate codebases with metadata such as tags, language, dependencies, and embedding representations for retrieval.Repository retrieval returns a candidate seed repository and confidence score.
  • 3.4 Knowledge grounding via seed repositories and retrieval: If repository confidence ρ ≥ τ, KAPSO initializes from the retrieved repository; otherwise it uses an empty or template scaffold.The resulting initial state becomes the parent for the first experiment branch.
  • 3.4 Knowledge grounding via seed repositories and retrieval: The typed knowledge graph represents wiki pages as typed nodes connected by relations such as IMPLEMENTED_BY, USES_HEURISTIC, and REQUIRES_ENV.Node types include Principle, Implementation, Environment, and Heuristic.
  • 3.4 Knowledge grounding via seed repositories and retrieval: Given a goal, optional seed repository, and failure signal, KAPSO retrieves a bounded knowledge packet through goal- and repository-conditioned retrieval.The packet can include grouped pages, confidence and provenance metadata, a repository reference, and recovery attachments.
  • 3.4 Knowledge grounding via seed repositories and retrieval: Conditioning retrieval on a seed repository surfaces compatible dependencies, entrypoints, idioms, and other repository-specific constraints.This connects retrieved knowledge to the starting codebase rather than treating the goal as the only retrieval signal.
  • 3.4 Knowledge grounding via seed repositories and retrieval: After failed experiments or repeated contract violations, error-recovery augmentation adds failure-conditioned heuristics, diagnostics, and alternative implementations.Without a failure signal, KAPSO uses the base knowledge packet instead.

3.5 Core solve loop (Orchestrator)

At the orchestrator level, KAPSO repeatedly builds context, runs a search strategy, records new experiments, and returns the best artifact under the evaluator’s selection rule. The loop is explicitly a search over executable artifacts driven by repeated experiments and evaluator-defined assessment.

  • 3.5 Core solve loop (Orchestrator): The solve loop repeatedly builds context and executes experiments until a stop condition fires.The implementation maps this orchestration to OrchestratorAgent.solve(), Kapso.evolve, and a selected search strategy.
  • 3.5 Core solve loop (Orchestrator): The orchestrator receives an evaluator contract, search strategy, context manager, and budgets as its core inputs.The search strategy may be linear or tree-based.
  • 3.5 Core solve loop (Orchestrator): KAPSO initializes an empty experiment history and iterates while the outer iteration count remains below N.Each iteration advances the experiment index after running the search strategy.
  • 3.5 Core solve loop (Orchestrator): The context manager supplies budget-aware problem context, knowledge, episodic memory, and prior experiment history to each iteration.The resulting context is passed to the search strategy together with the current budget progress.
  • 3.5 Core solve loop (Orchestrator): The search strategy runs on the current context and budget, and the returned experiments are added to the next history state.This preserves newly generated experiments for subsequent iterations.
  • 3.5 Core solve loop (Orchestrator): The loop returns the best artifact in the accumulated history according to the evaluator’s selection rule.Thus, the formal process searches over executable artifacts rather than ending at code synthesis.

3.6 Implement-and-debug loop (SearchStrategy)

KAPSO’s implement-and-debug loop runs each candidate in an isolated branch, using bounded debugging to repair execution failures and evaluator-detected contract violations. Tree search then prunes, expands, and selects solution specifications for black-box evaluation.

  • Implement-and-debug loop: The bounded debug loop targets execution errors and evaluator-detected contract violations rather than objective optimization.Examples include incorrect output formats and missing required files.
  • Implement-and-debug loop: Each branch-level attempt creates an isolated experiment, implements a solution from the specification and context, runs it, and finalizes the session.The inputs include a solution specification, rendered context, debug budget, branch name, and parent branch.
  • Tree search: At each outer iteration, tree search may prune leaf nodes, expand selected nodes with new child specifications, and select top-k leaves for execution.Expansion balances exploration and exploitation and uses an LLM ensemble to generate child specifications.
  • Tree search: The search combines a learned proposal distribution over solution specifications with black-box execution and selection under the evaluator’s ordering or utility criterion.Each tree node stores a solution specification and may store the corresponding experiment outcome.

3.8 Cognitive memory (cascaded retrieval, episodic learning, decisions)

KAPSO’s cognitive memory layer extracts reusable lessons from experiment traces, retrieves them alongside current knowledge, and chooses whether to retry, pivot, or complete. The same rendered context grounds both coding and decision-making while preserving provenance.

  • Episodic learning: KAPSO maintains episodic memory of lessons extracted from run logs, diffs, and evaluator feedback, then retrieves relevant lessons after each experiment.The retrieved lessons help determine whether to continue the current workflow or pivot to alternative knowledge.
  • Controller state: The controller state combines the goal, current knowledge, latest experiment record, retrieved episodic insights, and meta-statistics such as consecutive failures.This state is used for iteration-level control.
  • Cascaded retrieval: Cascaded retrieval uses WSR with PFR fallback and adds ERA when the latest signal indicates failure or contract violation.Episodic updates store generalized lessons from errors and qualitative feedback.
  • Decision policy: The controller’s action policy chooses among Retry, Pivot, and Complete for each iteration.The implementation exposes these as the policy’s iteration-level action space.
  • Failure recovery: After errors or contract violations, KAPSO records an extracted issue and augments knowledge with recovery heuristics and provenance-backed alternatives.The action decision uses the goal, knowledge, experiment, and retrieved episodic context.
  • Shared grounding: The coding agent and decision maker receive the same rendered context, while ERA records provenance through query_used and source_pages.This couples proposal generation and controller decisions to a shared evidence context.

4 System Implementation

KAPSO’s implementation fixes framework semantics while keeping execution, storage, indexing, and adapters modular. Its system combines reproducible git-based experimentation, structured knowledge retrieval, and episodic memory grounded in experiment provenance.

  • System architecture: The implementation keeps artifact, evaluator contract, history, seed selection, retrieval, and controller semantics fixed while allowing modular concrete backends and adapters.This design separates framework behavior from implementation choices for execution, storage, and indexing.
  • Experimentation engine: The experimentation engine isolates attempts, makes outcomes reproducible, and enables successful experiments to become parents for later work.Each experiment is represented as a branch inside an ExperimentWorkspace.
  • Experiment sessions: Each session starts from a parent branch, applies coding-agent edits, executes the evaluator, and commits artifact state and run outputs to the new branch.Branches can be checked out and re-executed to reproduce the evaluator interaction and artifacts.
  • Experiment sessions: Local git push publishes completed branches into the ExperimentWorkspace so they are immediately available as parents for child experiments, rather than guaranteeing publication to GitHub.Typical deployments use a file:// repository as the origin.
  • Knowledge system: The knowledge pipeline produces runnable seed repositories and structured reusable knowledge covering entrypoints, dependencies, environment requirements, and implementation patterns.Repository ingestion extracts principles, implementations, heuristics, and environment constraints.
  • Knowledge system: Typed pages and links organize principles, implementations, environments, and heuristics for bounded, repository-conditioned retrieval.The reference system uses pluggable graph and vector indices, including Neo4j and Weaviate.
  • Knowledge retrieval: Retrieval first selects an optional seed repository, then retrieves typed knowledge and adds failure-conditioned heuristics or alternative implementations after failures or repeated contract violations.The released package includes reproducible knowledge and repository snapshots with provenance metadata.
  • Cognitive memory: Episodic memory stores reusable, provenance-linked lessons from experiment traces and renders relevant memories into later proposal and debugging contexts.Lessons include triggers, generalized guidance, recommended actions, and links to branches, logs, diffs, and evaluator feedback.

5 Evaluation

KAPSO is evaluated on MLE-Bench and ALE-Bench, covering machine-learning competitions and heuristic optimization contests with benchmark-specific execution and scoring protocols. Leeroo outperforms the compared baselines on reported MLE-Bench medium and hard tasks and achieves the highest aggregated ALE-Bench performance, although ALE-Bench results may be noisy because few competitions are included.

  • MLE-Bench: MLE-Bench evaluates Python programs that train on competition data and produce a correctly formatted final_submission.csv.The protocol debugs first, validates the submission format, runs full mode, and grades the output on the competition split.
  • MLE-Bench: MLE-Bench runs for up to 24 hours or $200, stopping early when the grading library records any medal; medal rate counts competitions with medals across difficulty categories.Private score is returned by the benchmark grader.
  • MLE-Bench: 44.74% and 40.00% are Leeroo’s reported accuracies on Medium and Hard MLE-Bench tasks, versus 21.05% and 22.22% for R&D-Agent.Both systems achieve 68.18% on Low-difficulty tasks.
  • ALE-Bench: ALE-Bench evaluates C++23 solutions that optimize contest-defined objectives under strict runtime limits, with Docker execution and repeated public scoring for accepted solutions.The highest-public-score experiment receives private evaluation, final performance, rank percentile, and cost measurements.
  • ALE-Bench: 1909.4 final performance and 6.1% rank percentile are reported for Leeroo on ALE-Bench, versus 1879.3 and 6.8% for the original ALE-Agent.Leeroo’s reported total cost is $914.8, compared with $1003.3 for ALE-Agent.
  • ALE-Bench: The ALE-Bench comparison reports higher Leeroo scores than ALE on most AHC competitions, including 2022 versus 1457 on ahc016 and 2040 versus 1965 on ahc026.The passage also reports comparable rank percentiles across these competitions.
  • Limitations: ALE-Bench conclusions are limited by the relatively low number of competitions, which may introduce noise because agent performance varies across tasks.The authors propose larger competition sets and multiple seeds for more robust comparisons.

6 Conclusion

KAPSO combines evaluator-grounded experimentation, structured knowledge retrieval, and episodic learning into a modular framework for building software. Its design emphasizes auditable reuse through reproducible artifacts and an execution protocol for MLE-Bench and ALE-Bench.

  • KAPSO builds software through evaluator-grounded experiments, structured knowledge, and learning from episodic experience.
  • Its core components are a git-native experimentation engine, scalable knowledge acquisition and retrieval, and a workflow-aware cognitive memory layer.The retrieval system uses cascaded retrieval with WSR, PFR fallback, and ERA augmentation.
  • The cognitive memory layer stores and reuses lessons from experiment traces, while the framework maintains a clear evaluator contract and reproducible artifacts.
  • KAPSO provides an execution protocol for MLE-Bench and ALE-Bench to support reproducible evaluation.
Loading 2601.21526v2…