Source-linked AI summary
AutoFyn Technical Report: Non-Parametric Expert Iteration for Long-Horizon Agents
Adib Hasan, Daniel Schaffield, Akashnil Dutta, Tarik Adnan Moon
TL;DR
Long-horizon agents need to manage accumulating context and unreliable self-evaluation. AutoFyn addresses this with fresh sessions, persistent distilled state, specialized search, and external verification, achieving strong results across mathematics, data science, and cybersecurity. Its non-parametric update changes the context that conditions a frozen model, but verifier errors and harness differences limit what improvement can be attributed to the loop alone.
Problem
Long-horizon agents can suffer from uneven context use and accumulated errors when they evaluate their own work without external feedback.
Method
AutoFyn adapts a frozen model through fresh-round contexts, orchestrated candidate search, externally verified rewards, and distilled persistent state rather than model-weight updates.
Results
Across three domains, every IMO-tested model with room to improve scored higher under AutoFyn than its provider’s coding agent, while AutoFyn built the top-ranked Spider 2.0 dbt agent and produced 16 maintainer-confirmed advisories.
Takeaways & Limitations
AutoFyn provides an auditable long-horizon agent infrastructure that carries forward verified state while keeping the underlying model frozen.
Takeaways & Limitations
AutoFyn has no guarantee that verifier-guided updates improve performance, and the reported IMO totals compare deployments differing in episode length, tools, retrieval, and multi-agent structure.
Abstract
from arXiv · showhide
We introduce AutoFyn, an agent harness inspired by the Expert Iteration algorithm, adapting a frozen model across many rounds by updating persistent state from verified reward signals rather than model weights. Each round begins from a fresh model session, and durable information is reintroduced only through explicit interfaces such as persistent memory files, reports, and repository state. Within a round, an orchestrator explores, plans and builds many alternative approaches with specialized agents, while a task-grounded verifier verifies the work and supplies an objective reward for measuring progress. This reward is distilled back into the persistent state, which updates the effective policy for the next round. In this technical report, we formalize this loop and describe its persistent state and verification interfaces. We then demonstrate its use in three domains, namely olympiad mathematics, data science, and cybersecurity. On the six fresh problems of the 2026 International Mathematical Olympiad, every model with room to improve scores higher under AutoFyn than in its provider's own coding agent. AutoFyn also built the top-ranked agent on the Spider 2.0 dbt benchmark, and has produced $16$ maintainer-confirmed vulnerability advisories in Next.js, MetaMask, pnpm, Warp, LiteLLM, Langflow, and Open WebUI.
1 Introduction
AutoFyn addresses long-horizon agent failures by resetting context across rounds and carrying forward only externally verified, distilled state. The report introduces this non-parametric expert-iteration loop and evaluates it across mathematics, data science, and cybersecurity.
- Motivation: Long-horizon agents face degraded context use and errors that can accumulate when models evaluate their own work.These failures recur as operational horizons span many model calls and tool interactions.
- Approach: AutoFyn starts each round in a fresh context and carries forward bounded state recorded through durable interfaces.The orchestrator records progress to disk, while the next round is initialized only from that record.
- Approach: External verification determines whether a round counts as progress, preventing the agent’s own assessment from closing the loop.Only what is confirmed outside the agent is carried forward as progress.
- Results: AutoFyn reports results in olympiad mathematics, data science, and cybersecurity, including audited IMO evaluation and maintainer-confirmed advisories.The report also identifies the top-ranked agent on Spider 2.0 dbt.
- Contribution: AutoFyn formulates this process as non-parametric expert iteration, with search proposing candidates, verification supplying the criterion, and distillation updating persistent state.The model remains frozen while the effective policy changes through persistent context.
2 Related Work
AutoFyn combines established ideas on expert iteration, context management, and verification with explicit persistent state and hard round boundaries. Its distinctive contribution is composing these mechanisms with provenance-bearing state and externally grounded acceptance.
- Expert Iteration: Classical expert iteration alternates stronger search with an update, whereas AutoFyn uses verifier-grounded review and revision to adapt non-parametric context state.It updates persistent context rather than model parameters or an explicit search tree.
- Long Context Agents: AutoFyn extends fresh-context and memory techniques by composing hard round boundaries, provenance-bearing state, and grounded acceptance.Unlike a shell loop relying on human judgment, its orchestrator retains an artifact only when the external score improves.
- Verification: Verification pipelines provide a related verifier-driven shape, but AutoFyn emphasizes explicit persistent state and typed verification records.The harness remains part of what determines the meaning of a check.
3 Method
The method models AutoFyn as a round-based loop with frozen role models, persistent state, external verification, candidate search, selection, retention, and distillation. Each round reconstructs a fresh context from durable state and produces verified records that guide subsequent updates.
- Problem Setting: Role assignments remain frozen throughout an episode, while persistent state M_t is written to disk and reconstructed in a fresh context each round.Different roles may use different fixed model tiers.
- Problem Setting: The verifier maps each candidate artifact to a typed record containing status or score, evidence, assurance class, and verifier identity.Later rounds can act on why a candidate failed rather than only on a binary outcome.
- Round Loop: A round constructs context, searches over candidate trajectories, selects among candidates using their records, retains an incumbent, and distills the round into persistent state.The episode reports the retained incumbent under the environment’s acceptance rule.
3.2 Core Design Principles
AutoFyn keeps rounds independent at the conversation level while enabling progress through explicit, provenance-bearing state. It uses role-specialized branching to search alternatives and requires environment-grounded evidence rather than self-assessment for acceptance.
- Fresh Sessions over Explicit State: Every round begins a fresh model session, with information returning only through persistent files, repositories, artifacts, reports, and activity records.These interfaces preserve provenance for evaluations and failed approaches.
- Search over Candidates: The orchestrator dispatches specialized agents to explore alternatives, produce competing plans, build candidates, and revise rejected work.Branching breadth is controlled by the round’s budget rather than an explicit search tree.
- Grounded Acceptance: Environment evidence, not model self-assessment, determines whether a round is accepted as progress.Self-assessment may guide within-round search but cannot certify success.
- Loop Control: An external process starts each fresh orchestrator session, reconstructs the round, records reports and repository state, and decides whether another round begins.Git save and push operations are performed by the harness rather than the orchestrator.
3.4 Non-Parametric Expert Iteration
AutoFyn updates its effective policy without changing model parameters by reconstructing each round from persistent state and applying a fresh-session loop of search, verification, selection, retention, and distillation.
- Formalization: The formal loop maps goals, environments, budgets, frozen role assignments, orchestration, verification, selection, retention, and distillation into one round-based system.Table 1 presents the structural correspondence underlying this formulation.
- Contextual policy: Changing persistent state changes the contextual policy across rounds even though the role assignment Θ remains frozen.This is a contextual update rather than parameter training.
- Round loop: Each round starts from a fresh session with durable state only, then orchestrates search, evaluates candidates, updates the incumbent, distills state, archives outputs, and discards the session.The loop initializes context with no inherited transcript and returns the retained incumbent when continuation ends.
3.5 Comparison with Classical Expert Iteration
AutoFyn instantiates expert iteration through verifier-driven review and revision, but updates non-parametric context state rather than model weights and lacks classical improvement guarantees.
- Structural correspondence: AutoFyn’s expert procedure reviews and revises candidates against the environment until one is accepted, while distillation folds accepted findings into persistent state.The verifier grounds the loop, and the expert procedure spends more compute across candidates and revision rounds.
- Differences and limits: Unlike classical game-playing expert iteration, AutoFyn optimizes no parameters, constructs no explicit expert policy, and cannot assume every update improves performance.Its verifier may accept a wrong candidate or reject a right one, so improvement must be established by later verified performance.
3.6 Verification
AutoFyn uses environment-grounded checks whose conclusions are narrow and reproducible, with domain-specific verification determining what counts as progress.
- Verification scope: Verification is reproducible from the artifact and environment, but each check establishes only the property covered by its cases, evaluator, or searched refutation space.The record names the checked property while leaving other properties open.
- Data science: Data-science verification scores submissions against held-out expected outputs and returns a per-case breakdown that guides the next round.The breakdown identifies failed cases rather than relying on the bare score.
- Security-audit traces: Cumulative exploit curves rise only when new exploits fire against target builds and flatten during rounds without accepted additions.The pnpm run is truncated at round 8 before a final chain closes at round 129.
- Security auditing: Security verification executes candidate exploits against the target build to demonstrate reachable paths, not the vulnerability’s full scope or severity.Regression tests provide diagnostic pass-or-fail signals throughout.
- Olympiad mathematics: Olympiad verification searches for counterexamples to claimed bounds and symbolic discrepancies in algebraic computations, while an unsuccessful search establishes nothing.The checks refute targeted steps but cannot establish that a proof’s logic is sound.
3.7 Selection, Memory, and Incumbent Retention
AutoFyn retains the best accepted incumbent under comparable grounded scoring while organizing persistent memory by role and distinguishing verified outcomes from speculation.
- Incumbent retention: Under deterministic comparable scoring, incumbent retention guarantees q(I_t+1) ≥ q(I_t) when an accepted candidate improves the incumbent.If no improvement occurs, the incumbent and recorded goal remain unchanged.
- Incumbent retention: The retention rule bounds the incumbent but not the contextual policy because state distillation is lossy.Thus incumbent quality has a deterministic guarantee that later policy behavior does not.
- Security-audit traces: Figure 2 reports 56 confirmed exploits for Hermes Agent over 19 rounds, while pnpm reaches thirteen by round 8 and closes a final chain at round 129.Flat stretches indicate rounds that produced no accepted improvement and left the incumbent in place.
- Persistent memory: Persistent state separates goals, evaluation history, distilled role-specific rules, working status, and the incumbent artifact with evidence.Role-owned memory files let builders and reviewers accumulate distinct knowledge without polluting one another.
- Persistent memory: Verified outcomes remain distinguishable from hypotheses, failures, and next steps because working status stores these categories separately.Rules retain the round and reason they were learned and are retired when their referenced code disappears.
3.8 What the Architecture Enforces
AutoFyn mechanically controls how runs terminate and how progress is carried between rounds. It distinguishes time-unlocked stopping from time-locked execution with optional consolidation.
- Enforcement: The harness separates mechanically enforced behavior from prompt-requested behavior, with archived runs confirming the latter.Table 2 categorizes representative behaviors accordingly.
- Termination: Unlocked runs may stop when the goal is accepted, whereas time-locked runs continue until the budget expires or the user intervenes.A configured grace round may follow for consolidation.
- Termination: Episode duration measures the full run rather than the time required to reach the final candidate, whose retained incumbent is the reported output.This distinction matters when interpreting reported timings.
4 Applications
AutoFyn is evaluated in olympiad mathematics, data science, and security auditing using increasingly direct task verifiers. Across these applications, it improves or achieves leading outcomes, while the olympiad comparison does not isolate the loop from other deployment differences.
- Evaluation setup: The three applications use refutation-based proof acceptance, benchmark evaluation, and live exploit reproduction as their respective verification regimes.The environments are ordered by verifier strength.
- Olympiad Mathematics: The IMO 2026 evaluation used six newly released problems, five models, three harnesses, completion-based 0-or-7 grading, and independent audits.The problems were released after every model’s training cutoff.
- Olympiad Mathematics: Every model with room to improve scored higher under AutoFyn than its provider’s coding agent, with gains of 2.0 points for GPT-5.6 Sol and 11.3 for GLM 5.2.GLM 5.2 rose from 20.7 to 32.0 and crossed a medal band; Claude Fable 5 had no room to improve.
- Olympiad Mathematics: The olympiad totals compare deployments rather than AutoFyn alone because the harnesses differ in episode length, tools, retrieval, and multi-agent structure.The authors do not isolate what produces the difference.
- Data Science: SignalPilot Agent, autonomously constructed and optimized by AutoFyn for Spider 2.0 dbt, holds the top position on the public leaderboard.The benchmark evaluates realistic data-transformation workflows against held-out expected outputs.
- Security Auditing: Security auditing found over 150 individual vulnerabilities across thirteen projects and produced 43 advisories, including 16 maintainer-confirmed advisories.The remaining 27 advisories were still open.
5 Conclusion
AutoFyn bounds each round’s context to distilled state and accepts progress only through an external verifier. The same loop was applied across three domains with an auditable record of every run.
- Conclusion: AutoFyn adapts a frozen model by changing the context-producing state rather than the model’s weights.The report interprets this as expert iteration over a contextual policy.
- Conclusion: Each round carries forward a bounded distilled state and counts as progress only when an external verifier supplies a confirming signal.This is the report’s central architectural characterization.
- Conclusion: The same loop was run in three domains differing by available verifiers, with AutoFyn made available as infrastructure and every run auditable.The conclusion describes the resulting outcomes as impressive without further qualification.