PaperlayerView exact revision on arXiv

Source-linked AI summary

Phronix: The Capability-Centric Learning Paradigm for Agents

Hariharan Devarajan, Neeraj Rajesh, Aniruddha Marathe, Nikoli Dryden, Kin Ng, Michela Taufer

arXiv:2610.31789v1cs.AI

TL;DR

Scientific-computing agents face recurring engineering tasks, but harnesses often rely on people to curate knowledge rather than learn from executions. Phronix builds and validates reusable capabilities from those executions. Across models, recurring-task first-attempt success reaches 93–95%, with warm starts using 3.2–3.9× fewer tokens per task than one baseline attempt.

  • Problem

    Existing harnesses depend on user-curated knowledge, so agents may fail to reproduce solutions, lose execution lessons, and require user intervention to learn.

    Source: §I · p. 1

  • Method

    Phronix builds reusable capabilities from agent executions, validates their outputs with tools, and corrects capabilities using the findings.

    Source: §I · p. 2

  • Results

    Across models, recurring-task first-attempt success reaches 93–95%; warm starts use 3.2–3.9× fewer tokens per task than one baseline attempt.

    Source: §I · p. 2, §VIII · p. 10

  • Takeaways & Limitations

    Phronix lets harnesses build, validate, refine, and reuse capabilities from execution without relying on user-curated memory.

    Source: §I · p. 2

Abstract

from arXiv · show

Scientific computing increasingly relies on large language model-based agents for performance engineering tasks such as instrumentation, analysis, and workflow porting. However, agent effectiveness depends heavily on the surrounding harness, which typically relies on manually written skills and curated memory. This limits autonomous learning, forcing agents to repeatedly rediscover solutions rather than retain and reuse knowledge from prior executions. We introduce Phronix, a capability-centric learning paradigm that enables agentic harnesses to acquire, validate, and refine reusable capabilities from their own executions. Phronix operates through a predictor-validator-corrector loop: MCP tools and associated skills propose solutions, tool executions validate them, and observed outcomes refine the stored capabilities. It distills every observed interaction into a capability store and researches a failing task rather than abandoning it. First, such learned capabilities solve a recurring task on the first attempt in 93-95% of cases across every model, significantly reducing solution non-determinism and improving reliability. Second, Phronix turns an agent's own execution into 7.6-30.8 durable capabilities per task without user instruction, and creation falls below two once the needed capability exists. Third, warm-start runs consume only 0.38-0.40 million tokens per task, which is three to four times less than a single baseline attempt, while a cold-start costs no more than the baseline. On the 1000 Genome Pegasus workflow, instrumenting a production application for I/O performance analysis, Phronix passes all 11 correctness checks against six for the baseline harness, and also lifts the task solving rates of two cheaper models from 29% and 27% of SWE-bench tasks solved to 91% and 87%, respectively, matching the performance of the most expensive evaluated model.

I. INTRODUCTION

Scientific-computing agents handle recurring engineering work, but existing harnesses depend on people to retain and reuse what an execution teaches. Phronix learns reusable capabilities from executions and validates and corrects them; warm starts reduce token costs and improve solve rates.

I. INTRODUCTION

  • Scientific-computing agents perform repetitive work across applications, inputs, and systems, while harnesses increasingly rely on human-written skills and rulebooks.The harness coordinates the model, tools, skills, and memory that shape an agent’s work.Source: §I · p. 1
  • Existing harnesses leave agents dependent on user-curated knowledge, making solutions non-deterministic, execution lessons easy to lose, and learning dependent on user intervention.These challenges recur in scientific-computing tasks such as instrumentation, analysis, and workflow porting.Source: §I · p. 1
  • Phronix makes capability building central to the harness: capabilities predict an artifact, tools validate it, and validation findings correct it for reuse.It also tracks interactions and researches failing tasks, feeding findings into the capability store.Source: §I · p. 2
  • 3.2–3.9× fewer tokens per task than one baseline attempt, while solve rates rise from 27–65% to 87–91% on SWE-bench tasks.These are reported warm-start results.Source: §I · p. 2

II. BACKGROUND

An agent’s capabilities and continuity depend on the harness that coordinates its model, tools, skills, and memory. Existing knowledge-retention approaches do not inherently turn successful executions into validated, executable capabilities.

A. Agentic System Architecture

  • A model is stateless and stochastic, so the loop around it supplies reliability and memory by making each result the next input.The model proposes the next step while surrounding components carry it out.Source: A. Agentic System Architecture · p. 2
  • The model proposes steps, tools execute actions, skills guide task procedures, and memory carries preferences and past decisions between sessions.The harness selects what enters context, presents tools, executes calls, and returns results to the model.Source: A. Agentic System Architecture · p. 2

B. Current Learning Capabilities

  • Memory, skills, retrieval, and fine-tuning support knowledge reuse, but do not inherently capture successful executions as validated, executable capabilities.Harnesses therefore often depend on user-authored or curated knowledge, leaving agents to rediscover solutions.Source: B. Current Learning Capabilities · pp. 2–3

III. MOTIVATION

A 1000 Genomes workflow run illustrates how a solved task can leave no reusable knowledge: its commands were session-specific, its lessons were not recorded, and a repeat run would rediscover them. Phronix treats durable, retrievable, verifiable knowledge as a harness capability.

III. MOTIVATION

  • The baseline 1000 Genomes run completed a working trace through 39 shell commands, but its improvised routes and session-specific paths made the solution hard to replay.Seven commands embedded a timestamped directory, while others depended on temporary paths, session identifiers, or prior module state.Source: §III · p. 3
  • 49 of 50 calls were shell-tool calls, so the discarded transcript left no skill or MCP tool despite recording reusable build and dependency lessons.The tracer needed an extra link flag, failed under the default compiler, and the workflow’s dependencies were unnecessary for this input.Source: §III · p. 3
  • Without an executable capability, a second run would repeat 50 calls and process 2.84M input tokens; the first run also made 6 incorrect annotation decisions out of 11.The write loop was annotated, but the outer compression function lacked a span.Source: §III · p. 3

IV. PHRONIX

  • Phronix aims to make task knowledge durable, retrievable, and verifiable, putting deterministic steps in code and judgment-dependent decisions in prose.It acquires knowledge implicitly and admits predictions only after deterministic validation and any required correction.Source: §IV · p. 3

A. High-Level Design

Phronix routes tasks through a Builder, Tracker, and Researcher around a shared capability store; validators check predicted outputs and corrections improve reusable capabilities. Its Builder separates deterministic execution from judgment and retrieves only relevant knowledge.

A. High-Level Design

  • The Builder loads stored capabilities, the Tracker records every interaction, and the Researcher investigates failures and returns tasks with progress-based attempt budgets.The Tracker distills task observations into skills and MCP tools for later runs.Source: A. High-Level Design · p. 3
  • Tool-only validators provide repeatable verdicts, while structural checks and authored verifiers assess capability validity and task-specific claims.A missing dependency can produce an inconclusive verifier result; repeated failures or lack of improvement stop further attempts.Source: A. High-Level Design · pp. 3–4
  • The shared store has framework, user, and project layers, and its knowledge graph lets components retrieve needed objects without loading the whole store.The project layer can travel with a repository, while the graph indexes capabilities, skill blocks, and research records.Source: A. High-Level Design · p. 4

B. Capability Builder: Deterministic and Intelligent

  • The Builder encodes a repeatable tracer link-flag installation as an MCP tool, but leaves the judgment of which functions to annotate in a skill.New capability modules are compiled, checked for a register entry point, stripped of machine-specific paths, and fingerprinted to reject duplicates.Source: B. Capability Builder: Deterministic… · p. 4
  • Retrieval returns relevant skill blocks rather than entire files, limiting context; above 64 capabilities, a dispatcher pair keeps per-capability schema cost at O(1).At roughly 1,150 learned tools, schemas reached about 95K tokens; the dispatcher adds one level of indirection.Source: B. Capability Builder: Deterministic… · p. 4

C. Tracker: Automatic Tracking and Knowledge Creation

The Tracker captures interactions without interrupting task execution and distills them into reusable knowledge. It promotes task findings to durable storage as soon as outcomes are known.

C. Tracker: Automatic Tracking and Knowledge Creation

  • Automatic tracking addresses the need for unconditional, generalizable reporting that does not delay the task being learned from.Observations are filed by operation kind, and distillation runs in the background.Source: C. Tracker: Automatic Tracking and… · pp. 4–5
  • Observations record attempted operations and outcomes, then group them by operation type rather than by workflow so later tasks can retrieve relevant lessons.Reports are normalized and grouped by token overlap without a model invocation; a socket publishes them without blocking the caller.Source: C. Tracker: Automatic Tracking and… · p. 5
  • Background distillation runs on a bounded one-minute cycle, keeping learning off the task’s critical path and avoiding overhead that grows with the store.The per-cycle work is linear in new observations and quadratic only within a bounded candidate set.Source: C. Tracker: Automatic Tracking and… · p. 5
  • Promotion makes findings durable after each task’s outcome is known, merging updates atomically rather than deleting or mirroring existing files.Task-level promotion preserves graded outcomes even if a run is later killed.Source: C. Tracker: Automatic Tracking and… · p. 5

D. Researcher: Continual-Learning-Based Research

The Researcher treats missing capabilities as a reason to investigate, not simply retry, and uses measured progress to govern effort. The implementation shares capabilities and learning across harnesses and runs.

D. Researcher: Continual-Learning-Based Research

  • Research begins when a deterministic coverage check finds fewer than two matching catalogue blocks, rather than relying on the model’s confidence or retrying a known failure.Research runs once before the first attempt, and graph queries locate prior research on the same subject.Source: D. Researcher:… · p. 5
  • A fixed figure of merit and held-out measurements determine whether an approach is marked working or falsified.Approaches also retain their source.Source: D. Researcher:… · p. 6
  • Progress is measured by the furthest plan step reached; the budget starts at three attempts, can rise to five, and ends after two identical failures or a repeated patch.Attempts that never ran are void, and the resulting loop adds a constant number of agent invocations per task.Source: D. Researcher:… · p. 6

E. Implementation

  • The implementation publishes 84 MCP tools across 16 services and resolves capabilities through bundled, user, and project store layers, with higher layers shadowing lower ones.Writes default to the project layer, and a one-minute watcher registers new capabilities without a restart.Source: E. Implementation · p. 6
  • One store and neutral agent definitions serve multiple harnesses, while project commits, a promotion root, and upstream contributions share learned capabilities.Upstream contributions are screened by a privacy scan and exclude memory.Source: E. Implementation · p. 6

V. EVALUATION

The evaluation combines SWE-bench Lite repository tasks with an HPC workflow use case, comparing baseline, cold-start, and warm-start runs under a controlled protocol. It measures task success, capability creation and reuse, attempts, and token use.

V. EVALUATION

  • The evaluation establishes an overall solve-rate baseline, evaluates each of three concepts, benchmarks end-to-end token cost for task solving, and applies Phronix to an HPC I/O performance-characterization pipeline.Source: §V · p. 6

1) Testbed:

  • SWE-bench experiments run on CLUSTER nodes with 48-core AMD Rome processors, 256 GB memory, and Lustre storage, using eight nodes per run and isolated rootless containers per attempt.Each container is built from the task’s official SWE-bench image.Source: 1) Testbed · p. 6

2) Harness and Models:

  • Comparisons use Claude Code at low reasoning effort with gpt-5.4, gpt-5.5, and gpt-5.6-sol; the HPC use case uses gpt-5.5.The paper selects gpt-5.5 for the use case because its populated-store task cost was lowest among the three models.Source: 2) Harness and Models · p. 6

3) Workloads:

  • The workload includes 30 deterministically selected SWE-bench Lite tasks spanning 12 repositories, plus the 1000 Genomes Project workflow run with Pegasus and DFTracer.Every model receives the same 30 tasks in the same order.Source: 3) Workloads · p. 6

4) Run Protocol:

  • The protocol compares baseline, empty-store cold-start, and promoted-store warm-start arms on shared tasks, containers, and models, with up to five repeats and paired-run exclusions.Baseline is charged five attempts per task; Phronix starts with three and may earn up to two more based on progress, stopping after two identical failures.Source: 4) Run Protocol · pp. 6–7
  • Table I reports solved tasks over attempted tasks, pooled across runs, for 30 SWE-bench Lite tasks in five categories.This identifies the table’s scope without specifying cell values or an outcome.Source: Table I · p. 7

5) Metrics:

Phronix raises solve rates across all three evaluated models, while learned tool-and-skill capabilities make previously solved tasks highly reproducible on warm starts. The evaluation defines success by official test outcomes and tracks capability and resource use.

5) Metrics:

  • A task is solved only when every official FAIL TO PASS test passes and no PASS TO PASS test regresses; the reported solve rate is the fraction of attempted tasks solved.The evaluation also counts skills and tools created and reused, tracker entries, attempts, and tokens including prompt-cache reads.Source: 5) Metrics · p. 7

B. Overall Solve Rate

  • 91%, 87%, and 89% of tasks are solved by warm-start runs for gpt-5.4, gpt-5.5, and gpt-5.6-sol, respectively, versus baseline rates of 29%, 27%, and 65%.Cold-start rates are 67%, 74%, and 88%; warm-start runs bring all three models to roughly the same solve rate.Source: B. Overall Solve Rate · p. 7

1) Deterministic but Intelligent Capabilities:

  • 94.8%, 93.5%, and 93.3% of previously solved tasks are solved on the first warm-start attempt for gpt-5.4, gpt-5.5, and gpt-5.6-sol.Four of five categories reach 95–100%; Scientific is the exception at 62–75%, while Data & ML reaches 100% in every run.Source: 1) Deterministic but Intelligent… · p. 7, Figure 5 · p. 7
  • Each model creates both MCP tools and skills in every category; gpt-5.4 builds 7–9 capabilities per task versus 19–34 for the other two, and skills outnumber tools two to four times.Tools encode what can be coded, while skills carry contextual judgment and corner cases.Source: 1) Deterministic but Intelligent… · p. 7

2) Automatic Tracking and Knowledge Creation:

Phronix automatically turns execution records into reusable capabilities, and its progress-based research loop spends fewer attempts on solved tasks while preserving outputs from unsolved ones.

2) Automatic Tracking and Knowledge Creation:

  • Figure 6 tracks model and run arm on the x axis, capability counts split into skills and tools on the left axis, and observations per task on the right.Observation volume varies with timestamp attribution, so the figure’s tracking line indicates continued recording rather than a precise volume.Source: Figure 6 · p. 7, 2) Automatic Tracking and Knowledge… · p. 8
  • 7.6–30.8 capabilities per task are distilled without instruction in cold-start runs, while warm-start runs create fewer than two as tracking continues.The creation difference reflects whether needed capabilities already exist, not whether the tracker stops recording.Source: Figure 6 · p. 7, 2) Automatic Tracking and Knowledge… · p. 8

3) Continual-Learning-Based Research:

  • Figure 7 compares attempts by task outcome against the baseline budget and plots research outputs per task, showing progress-priced stopping and continued knowledge production.Whiskers show standard deviation.Source: 3) Continual-Learning-Based Research · p. 8, Figure 7 · p. 8
  • 1.05–1.91 attempts solve a task in cold-start research, versus 1.05–1.08 in warm-start runs, against the baseline’s fixed five-attempt budget.Failed tasks stop after 2.67–3.23 attempts; research still produces 2.93–6.34 outputs per cold-start task and 4.16–4.35 per warm-start task.Source: 3) Continual-Learning-Based Research · p. 8, Figure 7 · p. 8

D. End-to-End Benchmarking

End-to-end tests find that warm-start capability use costs substantially fewer tokens than a baseline attempt, while Phronix completes the 1000 Genomes instrumentation with more correct decisions and checks.

D. End-to-End Benchmarking

  • 0.38M, 0.40M, and 0.39M tokens per task are consumed by warm-start runs, 3.9, 3.2, and 3.7 times less than one baseline attempt, respectively.Cold-start runs cost 1.39M, 1.24M, and 0.85M tokens per task, at or below a single baseline attempt.Source: D. End-to-End Benchmarking · pp. 8–9
  • Figure 8 compares tokens per task across models and arms; 88–92% are prompt-cache reads, so a currency comparison would further favor Phronix.The test counts all task tokens, including prompt-cache reads, to compare arms attempting different numbers of times.Source: D. End-to-End Benchmarking · pp. 8–9

E. HPC Use Case: The 1000 Genomes Workflow

  • 11 correctness checks pass with Phronix on the 1000 Genomes workflow, compared with six for the baseline, which completes the smoke test but leaves three checks failed outright.Table II reports only five of the baseline’s 11 decisions as sound; Figure 9 shows six baseline setup calls rebuilding the environment and swapping the toolchain.Source: E. HPC Use Case: The 1000 Genomes… · p. 9, Table II · p. 9, Figure 9 · p. 9
  • Phronix uses 41 MCP tool calls and 15 skill invocations, while every baseline call is a model-composed shell command.Skills retain contextual judgment where it cannot be frozen into static code, and shell executes steps they prescribe.Source: E. HPC Use Case: The 1000 Genomes… · p. 9

VI. DISCUSSION

The discussion identifies limits on cross-family transfer and benchmark determinism; related work spans agent composition and memory, while Phronix leaves executable capabilities rather than records to reconstruct from.

VI. DISCUSSION

  • Cross-family capability reuse remains unmeasured, and skills may need rewriting because they encode how their target model prefers to be instructed.The authors identify cross-family reuse as a next step; the research loop can recover a failed translation by rewriting the skill.Source: §VI · pp. 9–10
  • Astropy tasks do not replay on the first warm-start attempt, with the authors attributing this boundary to faults in the SWE-bench environment image.The two Astropy tasks account for the reuse gap; the cited conftest issue and Sphinx Jinja drift are environment faults.Source: §VI · pp. 9–10

A. Multi-Agent Frameworks and Agent Generation

B. Agent Memory and Reflective Adaptation

  • Memory approaches retain experiences or trajectories for later context, while Phronix stores executable tools paired with skills so recurring tasks need not be reconstructed from descriptions.The discussion contrasts semantic and episodic memory, FORGE, and ExpeL with executable capability.Source: B. Agent Memory and Reflective… · p. 10

C. Tool Selection and Tool-Use Efficiency

Phronix complements tool-selection methods by creating capabilities, then stores executable tools and skills for reuse; the conclusion reports gains in recurring-task reliability, token cost, and workflow correctness.

C. Tool Selection and Tool-Use Efficiency

  • Phronix addresses where a tool set comes from, complementing selection methods that optimize choices among tools already covering a task.A growing store of tools is a setting where cheaper selection can also help.Source: C. Tool Selection and Tool-Use… · p. 10

D. Self-Evolving Capability Construction

  • Phronix learns from complete harness executions, separating deterministic behavior into executable tools and contextual judgment into skills stored across tasks.The discussion contrasts this with tool-making frameworks such as LATM and CREATOR, and retrieval-based IOAgent.Source: D. Self-Evolving Capability Construction · p. 10

VIII. CONCLUSIONS

  • 93–95% of recurring tasks are solved on the first attempt across models; Phronix also produces 7.6–30.8 capabilities per task and uses 0.38–0.40M tokens per warm-start task.On the 1000 Genome workflow it passes all 11 checks against six for the baseline, and lifts gpt-5.4 and gpt-5.5 SWE-bench solve rates from 29% and 27% to 91% and 87%.Source: §VIII · pp. 10–11

Related papers

Loading 2610.31789v1…