PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing LLM Agents with Skill Programs

Hongjun Liu, Yifei Ming, Shafiq Joty, Chen Zhao

arXiv:2605.17734v1cs.AI

TL;DR

LLM agents often rely on reusable skills that remain advisory rather than explicitly controlling decisions in the agent loop. HASP turns skills into executable Program Functions that intervene during inference, support post-training, and evolve through validation, improving performance across web-search, math reasoning, and coding tasks, with web-search average accuracy reaching 51.0% under inference-time intervention.

  • Problem

    Existing textual skills lack reliable mechanisms for when to activate and how to explicitly alter agent behavior during the policy loop.

  • Method

    HASP represents reusable skills as executable Program Functions that detect state-action conditions and intervene on the agent’s next decision.

  • Results

    HASP improves performance across web-search reasoning, mathematical reasoning, and coding tasks; inference-time PF intervention reaches 51.0% average accuracy on web-search reasoning.

  • Takeaways & Limitations

    PFs provide a shared interface for intervening during inference, supplying post-training signals, and evolving validated external skills.

  • Takeaways & Limitations

    Evaluation focuses on benchmark-style search, math, and coding tasks with relatively clear verification signals, leaving transfer to open-ended or weakly verifiable environments unresolved.

Abstract

from arXiv · show

Equipping LLM agents with reusable skills derived from past experience has become a popular and successful approach for tackling complex and long-horizon tasks. However, such lessons are often encoded as textual guidance that remains largely advisory, lacking explicit mechanisms for when and how to intervene in the agent loop. To bridge the gap, we introduce HASP(Harnessing LLM Agents with Skill Programs), a new framework that upgrades skills into executable Program Functions (PFs). Rather than offering passive advice, PFs act as executable guardrails that activate on failure-prone states and modify the next action or inject corrective context. HASP is highly modular: it can be applied at inference time for direct agent-loop intervention, during post-training to provide structured supervision, or for self-improvement by evolving validated, teacher-reviewed PFs. Empirically, HASP drives substantial gains compared to both training-free and training-based methods on web-search, math reasoning, and coding tasks. For example, on web-search reasoning, inference-time PFs alone improve the average performance by 25% compared to (multi-loop) ReAct Agent, while post-training and controlled evolution achieve a 30.4% gain over Search-R1. To provide deeper insights into HASP, our mechanism analysis reveals how PFs trigger and intervene, how skills are internalized, and the requirement for stable skill library evolution.

1 Introduction

HASP converts passive textual skills into executable Program Functions that intervene in an LLM agent’s policy loop when needed. The modular harness supports inference-time intervention, post-training, and self-improvement, with strong results across web-search, mathematical reasoning, and coding tasks.

  • Method: HASP reframes skills as executable Program Functions that inspect the current state and candidate action, then modify the action or inject context when intervention is needed.PFs replace passive guidelines with state–action intervention functions that can be triggered on demand.
  • Method: The external HASP harness retrieves relevant PFs, evaluates activation predicates, executes valid interventions, and feeds revised actions or injected context back into the agent loop.At each step, the base policy proposes an action before the harness applies any validated intervention.
  • Modularity: HASP supports inference-time action revision without model updates, structured post-training supervision, and validated evolution of its active skill library.PFs enter the active library only after syntax, interface, and mock-execution validation.
  • Results: 51.0% average accuracy is achieved by inference-time PF intervention on web-search reasoning, rising to 56.2% with an auxiliary teacher for PF selection.The evaluation covers web-search reasoning, mathematical reasoning, and coding tasks.

2 Related Work

Prior work improves LLM agents through post-training for search, reasoning, tool use, and coding, while another line reuses experience through skills, memories, routines, and self-improving evolution. These approaches extend agentic workflows but primarily emphasize training or experience reuse.

  • Post-training for agent reasoning and tool use: Post-training methods target search, reasoning, tool-use, and coding agents through interaction with search or tool environments and policy optimization.Examples include Search-R1, ReSearch, ZeroSearch, StepSearch, VerlTool, SimpleRL-reason, Open-Reasoner-Zero, General-Reasoner, ToRL, AceCoder, and GRPO-based code training.
  • Skill-augmented and self-improving agents: Skill-augmented agents interleave thoughts and actions, use external tools, and extend these loops through richer agentic workflows.This line of work treats skill or experience reuse as a strategy for improving agent behavior.
  • Skill-augmented and self-improving agents: Reflexion, ExpeL, and Voyager reuse past experience by storing verbal lessons, memories, or routines.Recent systems such as MemSkill and SkillRL investigate memory or skill evolution for self-improvement.

3 Harnessing LLM Agents with Skill Programs

HASP turns reusable agent skills into executable Program Functions that intervene during reasoning by detecting failure-prone states and repairing subsequent decisions. The same intervention records support post-training and controlled skill-library evolution.

  • Inference-time intervention: HASP wraps a base agent policy with an external harness that retrieves relevant PFs and applies them during task-solving rollouts.The harness operates at inference time over the agent’s current state and proposed action.
  • Program Functions and Skill Library: Each PF contains should_activate and intervene components that determine when to fire and how to repair the next decision.This makes skills executable rather than natural-language reminders that merely state general principles.
  • PF-guided intervention in the agent loop: PFs intervene either by modifying the next action or by injecting corrective context into subsequent reasoning.Examples include rewriting over-constrained search queries, redirecting retrieval, and warning about similar entities.
  • Post-training supervision: PF activation records preserve the original action, corrected action, context, metadata, and feedback, enabling supervision over intermediate decisions.HASP scores these events using timing, mode, correctness, and outcome signals, then trains on PF-corrected actions and trajectories rather than final answers alone.
  • Self-improving PF evolution: Controlled skill-library growth proposes PFs from recurring failures and admits candidates only after executable validation and teacher review.Every candidate must specify both an activation condition and an intervention behavior to fit the rollout harness.

4 Experiment

Experiments evaluate HASP’s inference-time intervention, teacher-assisted dispatch, post-training internalization, and closed-loop skill-library evolution across web-search, mathematical reasoning, and coding tasks. Results show gains from executable PFs, PF-derived supervision, and filtered evolution, while analyses identify intervention mechanisms, concentrated failure patterns, and selective skill internalization.

  • Experimental scope: HASP evaluates PF decision improvement, PF-event internalization through post-training, and external skill-library updates through filtered evolution.The experiments cover inference-time intervention, post-training, and self-improvement through evolving validated PFs.
  • Inference-time intervention: PF-only intervention improves over the base multi-loop agent and Prompt-Only Skills by directly changing actions or injecting corrective context.The comparison shows executable intervention is more effective than skill text alone.
  • Post-training: 56.2% rises to 56.8%, 59.3%, and 62.5% for fixed-library SFT, RS, and OPD on web-search reasoning.These variants use PF-corrected traces scored by PF-derived criteria as post-training supervision.
  • Skill-library evolution: 60.3% on web-search reasoning and 45.4% on mathematical reasoning are reached by HASP-Evolve + RS, improving over fixed-library RS.Closed-loop evolution summarizes residual failures into candidate PFs and filters them before updating the external library.
  • Mechanism analysis: 65.1% of web-search PF events revise actions, while 34.9% inject context, and activations concentrate in decompose_complex_question, insufficient_exploration, and answer_completeness.The three most frequently activated skills record 322, 138, and 100 total triggers, respectively.
  • Skill internalization: Behavior-correcting PFs internalize most strongly: multi_hop_reasoning_failure and retrieval_failure become silent on 100% of previously triggered cases.PFs tied to the question itself generally remain active after training.

5 Conclusion · Appendix

HASP represents skills as reusable Program Functions that intervene directly in agent decisions, provide post-training supervision, and evolve through validated residual failures. Across web-search, mathematical reasoning, and coding, it improves inference-time behavior, supports selective internalization, and benefits from filtered skill evolution.

  • 5 Conclusion: HASP represents skills as reusable state-action intervention functions called Program Functions.PFs convert skills into executable modules rather than passive guidance.
  • 5 Conclusion: PFs directly intervene on an agent’s next decision during inference.The framework uses a shared interface for agent-loop intervention.
  • 5 Conclusion: PFs provide structured signals that can teach agents during post-training.The same skill representation supports post-training supervision.
  • 5 Conclusion: The shared PF interface unifies inference, post-training, and skill-library evolution.PFs can act, teach, and grow within one modular framework.
  • 5 Conclusion: PFs evolve from recurring residual failures under validation and teacher review.Validated skill evolution is treated as external-memory growth.
  • 5 Conclusion: Across web-search reasoning, mathematical reasoning, and coding tasks, HASP improves inference-time behavior.The reported task coverage spans three domains.
  • 5 Conclusion: HASP supports selective skill internalization without requiring full reinforcement learning.This is presented as a benefit of the framework’s training interface.
  • 5 Conclusion: HASP benefits from filtered skill evolution as skills are expanded through validated experience.The conclusion characterizes PFs as a complementary path for improving agents.

A Limitations … B.5 Prompt-Equivalent Skills

HASP combines executable, typed skill interventions with phase-level reminders and handlers across web-search, math, and code, while limiting teacher dependence and distinguishing genuine interventions from prompt-equivalent reminders. Prompt-equivalent skills are common in web-search and code but absent from math, and their added effect is small and mixed across datasets.

  • A Limitations: HASP’s stronger variants use external teachers for PF selection, review, or on-policy distillation, increasing cost and potentially introducing teacher-specific bias.PFs mainly help models use and absorb existing strategies rather than discover fundamentally new ones.
  • B.1 PF Interface and Intervention Types: Each PF is a typed Python object with activation and intervention methods, supporting MODIFY_ACTION, INJECT_CONTEXT, or auditable NOOP outcomes.PFs are deterministic by default, run every step, and are subject to per-PF and per-skill firing caps that prevent oscillation.
  • B.2 Phase Instructions and Handlers: Phase instructions attach reminders to matching reasoning phases, while FINAL-time handlers can override a FINAL action back to SEARCH.The handler family is enabled only in web-search, and such overrides are capped at one per episode.
  • B.3 An Example PF: Retrieval Failure: The retrieval_failure PF recovers from failed searches by reformulating queries, using synonyms, or decomposing complex questions into sub-queries.Its phase reminders also require reading a relevant document before proceeding to FINAL.
  • B.4 Skill Libraries Across Domains: The shared loader, registry, and dispatcher support distinct web-search, math, and code skill libraries whose activation predicates and markdown content vary by domain.Web-search combines PF and prompt-only skills, math organizes PFs around correctness and verification, and code uses static analysis with context hints or NOOP audits.
  • B.5 Prompt-Equivalent Skills: A PF is prompt-equivalent when it only injects a fixed literal context and never modifies actions, making it functionally equivalent to a phase-instruction reminder.This criterion is defined by intervention behavior and the absence of runtime substitution from step context.
  • B.5 Prompt-Equivalent Skills: 10 of 26 web-search skills and 2 of 12 code skills are prompt-equivalent, whereas no math skill satisfies the criterion.The marginal effect of adding these skills is small and mixed in sign across six datasets, so prompt reminders alone do not consistently explain HASP’s gains.

B.6 Multi-Layer Skill Selection at Inference … D Additional Experimental Details

HASP uses layered inference-time filtering and a controlled self-improvement pipeline to select, validate, review, version, and gradually expand executable skills. The pipeline combines deterministic failure detection, teacher review, executable checks, and strict library controls to keep skill evolution useful and auditable.

  • B.6 Multi-Layer Skill Selection at Inference: Five concentric filters determine whether a PF reaches the prompt or fires, using master switches, sequential selectors, runtime predicates, vote aggregation, and budgets.Difficulty gating asks the teacher for a 1–5 score and can bypass skills below threshold; heuristic fallback uses question length, multi-hop markers, and constraint phrases.
  • B.7 Skill Retrieval and PF Selection Prompt: Teacher-ranked retrieval complements lexical scoring by favoring fewer targeted PFs while retaining three mandatory PFs that cannot be dropped.This second filtering stage reduces broad or off-topic skills affecting the trajectory.
  • C Self-Improving Pipeline Details: Each self-improving epoch revisits residual failures through eight phases that produce trajectories, analyses, candidate skills, validation reports, reviews, gradient streams, and training-data files.The process returns an updated library together with supervised- and preference-training data.
  • C.1 Per-Epoch Phase Sequence: The per-epoch sequence runs PF-aware rollout, failure clustering, capped proposal, executable validation, teacher review, library update, signal scoring, and training-data construction.Validation uses approximately 250 seed and 250 validation samples across five active web datasets, with cached splits reused unless the seed budget is −1.
  • C.2 Phase B — Failure Signal Detection: Failure detection combines twelve heuristic rules with teacher abstractions, independently deduplicates patterns, and prioritizes clusters with novelty ≥0.3 as new categories.The teacher generations use temperature=0.3 and max_tokens=400, with concurrency bounded by a thread pool.
  • C.3 Phase C — Skill Proposal: For each surviving cluster, the student proposes a SKILL.md specification and ProgramFunction subclass, with deterministic should_activate logic and a cap of five candidates per epoch.The proposer forbids LLM calls inside should_activate and constrains the Intervention interface.
  • C.4 Phase D — Executable Validation / C.5 Phase E — Teacher Five-Dimensional Review: Each candidate undergoes syntax, interface, mock-execution, and return-type checks, covering 3 mock contexts × {SEARCH, READ, FINAL} = 9 invocations.Teacher review scores concept, trigger, intervention, executability, and validation utility; Qexec < 0.3 hard-rejects, while Qskill ≥0.60 accepts and ≥0.42 revises.
  • C.6 Phase F — Library Update and Versioning / C.7 Failure Clustering and Candidate PF Proposal / C.8 Why Strict Filtering Matters: Accepted skills are versioned, audited, and capped at 50 active entries, while deduplication and rate-limited growth prevent noisy or redundant libraries from degrading retrieval precision and policy quality.Only the highest version per base id fires; older versions remain available for trajectory replay.

D.1 Datasets and Train/Test Splits … F.1 Stage-by-Stage Contribution

The paper specifies domain-specific data splits, six post-training configurations, shared rollout and optimization settings, and multiple skill-evolution and reward mechanisms. A stage-by-stage analysis attributes gains to inference-time PF intervention, post-training internalization, and closed-loop evolution.

  • D.1 Datasets and Train/Test Splits: Web-search files use 200 test and 800 training questions, while math evaluates AIME24, AMC23, and GameOf24 on their full datasets.The web-search validation pool uses 50 seed and 50 validation samples per dataset from the training tail, whereas math disables the difficulty gate.
  • D.1 Datasets and Train/Test Splits: Coding splits reserve 50 unique HumanEval+ and MBPP+ problems and 100 BigCodeBench entries for testing, with remaining problems forming training pools.EvalPlus expansion yields 100 test entries for the first two datasets, while BigCodeBench retains 1,040 training entries.
  • D.2 Detailed Definition of E1–E6: The six experiments form a 2 × 3 grid combining V1 open-loop or V2 closed-loop topology with SFT, rejection sampling, or on-policy distillation.V1 fixes the skill library after data collection; V2 evolves the library and refreshes data, with GPT-4o teaching E3 and E6.
  • D.3 Shared Training Configuration: All experiments use Qwen2.5-7B-Instruct with LoRA rank 16, α = 32, bf16 precision, gradient checkpointing, and approximately 65K effective tokens per step.SFT runs for 10 epochs, while rejection-sampling and closed-loop runs use at least 8 epochs per phase; rollouts use group size 2, temperature 0.9, top-p 0.95, five reasoning steps, and three search calls.
  • D.5 Closed-Loop Evolution Modes: Closed-loop evolution offers full and lite modes: E4 uses the complete pipeline, whereas E5 and E6 use cheaper lite evolution with 20 failed trajectories and at most three proposals.Lite evolution performs compile-checking and deduplication without teacher calls, clustering, review, or gradient computation, at roughly 5% of full wall-clock cost.
  • D.6 Data Construction Pipeline: Training data proceeds from PF-aware trajectories through four-signal step scoring and layered processing into experiment-specific YAML, with V2 rebuilding data between phases.V1 collects trajectories once from a frozen library, while V2 reruns collection each cycle using the current checkpoint and evolved library.
  • D.7 Post-Training Recipes: SFT converts PF corrections into weighted targets, rejection sampling retains trajectories that are successful and intervention-clean, and on-policy distillation distills corrected self-generated behavior.All three recipes share one signal interface, keeping recipe choice orthogonal to signal ablation; coarse signals weight timing, modality, correctness, and outcome at 0.15, 0.10, 0.25, and 0.50.
  • F.1 Stage-by-Stage Contribution: The results analyze stage contributions, coding transfer, and skill-library dynamics, showing that inference-time PFs provide most web-search gains before post-training internalizes repairs.Table 14 selectively enables stages while holding defaults fixed, isolating inference-time intervention, post-training internalization, and closed-loop evolution.

F.2 Detailed Coding Results … G.1 Inference-Time Component Ablation

The appendix shows that executable PF interventions are especially useful for difficult coding cases, complement capability internalization, and target web-search failure modes such as entity confusion and premature finalization. It also characterizes skill-library evolution and separates the contributions of PFs, teacher access, textual skills, and baseline ReAct execution.

  • F.2 Detailed Coding Results: Inference-time PFs help most on edge-case-heavy coding benchmarks, where static checks target missing length guards, off-by-one indexing, and stdin parsing.The harder splits are BigCodeBench/Hard and MBPP/Plus, while HumanEval/Base already reaches 81.7% with Qwen2.5-7B-Instruct; phase instructions are described as the dominant lever.
  • F.2 Detailed Coding Results: PF-derived supervision primarily elicits reliable execution of existing coding strategies, whereas GRPO and KodCode-RL expand the policy frontier through reward-shaped exploration.The two directions are described as largely orthogonal.
  • F.3 Per-Dataset Robustness: Web-search PFs help most on MuSiQue, where multi-hop chains create opportunities for entity confusion and premature FINAL targeted by retrieval_failure, wrong_entity_confusion, and insufficient_exploration.The supplied passage also places 2Wiki in the middle and notes that AgentFlow remains stronger there, but the excerpt is truncated.
  • F.4 Evolution Dynamics Across Rounds: Self-improvement tracks candidate-proposal success, teacher-reviewed skill acceptance, and cumulative library size across rounds.The passage reports that acceptance stays high early and that the library remains below max_library_size = 50 by the final round.
  • F.4 Evolution Dynamics Across Rounds: Web-search and math show qualitatively similar evolution curves, while coding accepts fewer skills per round because many PFs reduce to audit NOOPs and proposals are conservative.Later-round accepted skills tend to have higher Q_concept but lower Q_trigger, consistent with later proposals being more concept-focused.
  • G Signal and Filtering Ablations: Table 5 defines an inference-time component ablation, a signal ablation on closed-loop run E5, and a filtering ablation using the same backbone, rollout configuration, and evaluation pool.The shared setup uses Qwen2.5-7B-Instruct and HotpotQA / 2Wiki / MuSiQue; Table 15 separately identifies four core supervision signals for skill-conditioned intervention.
  • G.1 Inference-Time Component Ablation: 56.2% average is reported for Full, where PFs and the teacher jointly intervene at decision time, compared with 31.2% for RA-Agent (multi-loop), where both are disabled.RA-Agent uses the same multi-step ReAct loop without state-to-intervention machinery and is identified as the lower bound.
  • G.1 Inference-Time Component Ablation: The component ablation separately tests text-only skills, PFs without teacher access, and teacher access without PF-mediated state-to-intervention control.PF-only mode falls back to deterministic code-only paths for teacher-dependent skills, while teacher-only mode retains multi-loop ReAct and outer-loop format post-processing or retries.

G.2 Signal Ablation

Signal ablations show that all four supervision signals contribute distinct gains in closed-loop rejection sampling. Modality has the largest impact, while timing, correctness, and outcome each prevent different intervention failures.

  • Experimental setup: The ablation varies only which subset of {T, M, C, O} contributes to per-step weights, using E5 closed-loop rejection sampling with full Exec + Teach filtering.The default all-signal configuration uses weights (λt, λm, λq, λo) = (0.15, 0.10, 0.25, 0.50) and reaches 60.3%.
  • Signal contributions: −7.8 points without timing shows that rewarding intervention on risky steps and penalizing intervention on safe steps prevents mistimed or missed interventions.The remaining signals are renormalized after timing is removed.
  • Signal contributions: −15.5 points without modality is the largest drop, showing that distinguishing pre-action modification from post-observation context injection is crucial.Without modality credit, SEARCH-query rewriting and reading-reminder injection are treated as interchangeable.
  • Signal contributions: −12.1 points without correctness indicates that local syntactic validity, semantic appropriateness, and domain consistency filter invalid interventions.Otherwise, locally invalid actions such as malformed SEARCH arguments may be retained when the trajectory eventually succeeds.
  • Signal contributions: −12.8 points without outcome shows that downstream EM, cost, and side-effect feedback prevents internalizing locally plausible but ultimately ineffective trajectories.Removing outcome credit disconnects intervention quality from downstream results.
  • Overall finding: Together, the four drops support irreducible supervision across when, how, whether well-formed, and whether effective, with each contributing 7–15 points.The ablations collectively support the claim that PF-derived supervision cannot be reduced to a single scalar.

G.3 Filtering Ablation … Coding case: LiveCodeBench-easy — “Equally”

The paper shows that validated PF evolution materially outperforms ungated or partially gated evolution, while case studies demonstrate PFs correcting search, mathematical counting, and coding failures through targeted interventions. Across domains, PFs expose concrete failure modes and alter agent context or actions to produce verified solutions.

  • G.3 Filtering Ablation: 60.3% is achieved with full filtering, requiring both executable validation and teacher review before candidate skills enter the library.Candidates must pass four executable checks and reach Qskill ≥0.60 on five-dimensional teacher review.
  • G.3 Filtering Ablation: −24.0 points from full filtering occurs with no filtering, supporting the memory-pollution hypothesis that unvalidated skills degrade retrieval and policy decisions.The resulting noise includes skills that fail to import, fire too aggressively, or conflict with existing PFs.
  • G.3 Filtering Ablation: 60.3 > 59.3 > 48.8 > 47.2 > 36.3, showing that full filtering beats no evolution, exec-only, teacher-only, and no filtering.Both gates are necessary because removing either allows evolution noise to outweigh the benefit of new skills.
  • G.4 Case Study: The case studies render complete ReAct trajectories, including thoughts, actions, observations, and PF trigger and intervention events for baseline failures and successful PF-augmented recoveries.The trajectories use Qwen2.5-7B-Instruct with the rollout configuration of §D and are taken verbatim from evaluation logs.
  • Web-search case: HotpotQA — multi-hop entity resolution: In the HotpotQA case, PFs replace baseline guessing, repeated search, and missing reads with query decomposition, forced reading, and answer-completeness verification.The PF-augmented agent correctly identifies Helen Walton’s husband as Sam Walton after reading supporting evidence.
  • Math case: AMC23 — distinct-roots polynomial counting: In the AMC23 case, PFs collapse symmetric root permutations into distinct (a,b) pairs and verify all five candidates, changing the answer from 9 to 5.The verification hook substitutes candidates into the polynomial and confirms three distinct integer roots.
  • Coding case: LiveCodeBench-easy — “Equally”: Coding PFs expand the branch structure from {odd, zero, two-of-three} to {all-equal, odd, sum≤3, sum∈{4, 6}, two-of-three, else}.The intervention changes context rather than rewriting code, and the agent emits the revised solution after the static check fires.
  • Coding case: LiveCodeBench-easy — “Equally”: In the coding case, the baseline fails the all-equal input, while code_edge_cases injects a static-analysis hint that adds an all-equal branch first.The baseline passes sample 1 but fails sample 2, whereas the revised structure adds the only substantive change: the all-equal branch.

H Training Dynamics … H.3 Loop-Topology Comparisons

The training-dynamics analysis standardizes six runs, documents their logging and canonical-trace construction, and shows that optimization remains stable across recipes. Loop-topology comparisons further indicate that closed-loop refreshes alter the gradient signal, producing visible iteration-boundary changes rather than simply replaying identical data.

  • H Training Dynamics: All six runs use the same Qwen2.5-7B-Instruct LoRA fine-tuning configuration, while recipe-specific learning rate, epoch, and rollout settings vary.The shared configuration includes rank 16, α = 32, bf16, and gradient checkpointing.
  • H.1 Data-Collection Protocol: Training curves come from Weights & Biases logs emitted every 10 steps, with each record containing seven plotted scalars.The scalars are loss, grad_norm, learning_rate, entropy, num_tokens, mean_token_accuracy, and epoch.
  • H.1 Data-Collection Protocol: Canonical traces use the longest available V1 run or concatenate iter_0 and iter_1 for closed-loop V2 settings to capture the first evolution boundary.Examples include E1 V1 SFT at 600 optimizer steps and E4 V2 SFT at 820 + 180 steps.
  • H.2 Optimization Diagnostics: All six runs remain in the same gradient-norm magnitude band, ∼100 to 101, without iter-boundary spikes, ruling out unstable optimization as the source of V2 loss bumps.The learning-rate panel reports 1 × 10−5 peak for SFT and 5 × 10−6 for RS and OPD.
  • H.3 Loop-Topology Comparisons: Replotting against global step confirms recipe ordering is robust to the X-axis choice, while V2 spans fewer steps but more wall-clock seconds.The companion figures separate the six runs along orthogonal axes.
  • H.3 Loop-Topology Comparisons: V1 SFT is smooth, whereas V2 SFT shows a clean iteration-boundary break and a flatter post-evolution slope because its refreshed target pool has higher-accuracy targets.This pattern directly indicates that closed-loop refresh changes the gradient signal rather than merely rerunning the same data.

I Additional Analysis of Skill Evolution … J.1 Hardware Configuration

The analysis examines how self-improving skill libraries evolve across domains, including review quality, admission and filtering outcomes, recurring skill families, and resource requirements. It also specifies the hardware and reproducibility setup used for the experiments.

  • I Additional Analysis of Skill Evolution: Self-improving PF evolution is evaluated through per-skill admission events, review scores, library snapshots, and web-search uplift across evolution epochs.The analysis reads these statistics from closed-loop run artifacts and aggregates web-search gains by skill family and epoch.
  • I.1 Per-Family Review Scores: Executability scores highest at 0.90–1.00, while trigger scores lowest at 0.65–0.87 across the three domains.The executable validation gate filters out skills that fail to import or return the wrong type, whereas writing precise activation predicates remains difficult.
  • I.2 Per-Domain Filter Outcomes and Admission Counts: The runs span 5 epochs for web-search and 3 epochs each for math and code, with at most five proposed candidates per epoch and a 50-skill active-library cap.Under audit-only acceptance, compiling candidates are appended to the library with review scores stored alongside them.
  • I.2 Per-Domain Filter Outcomes and Admission Counts: Code admits approximately 18 candidates per epoch, compared with 4–6 for web-search and math, while versioned slots keep the active library within the 50-skill cap.The faster code growth reflects more aggressive failed-trajectory resampling and multiple proposer variants per residual cluster.
  • I.4 Skill-Family Evolution Pattern: Recurring evolved families include premature_final, no_read_before_final, reasoning_hallucination, wrong_entity_focus, and format_mismatch across web-search and math, with incomplete_implementation and incomplete_logic added in code.These families overlap with the Phase B failure-detector taxonomy and indicate convergence in the proposer’s selected skill ids.
  • J.1 Hardware Configuration: All experiments were run on an SLURM cluster constrained to NVIDIA L40S GPUs, with RSS-only memory accounting and GPU reuse between shared rollout and proposal stages.Per-job allocations and SLURM wall-clock ceilings are summarized in Table 20, while measured completion times are reported separately.

J.2 Wall-Clock and Aggregate Compute … K Cost and Latency Breakdown

The paper reports wall-clock, teacher-API, reproducibility, software-release, and cost details for HASP. Teacher invocation costs are quantified for web search, with math and coding estimated lower because their rollouts collapse to one step.

  • J.2 Wall-Clock and Aggregate Compute: J.2 Wall-Clock and Aggregate Compute aggregates six post-training experiments, one bootstrap rollout, three self-improving runs, and eight skill-evaluation ablation rows.Wall-clock figures use the longest single Weights & Biases run as the canonical curve and exclude the amortized bootstrap rollout and reused post-training evaluation.
  • J.3 Teacher API Budget: J.3 Teacher API Budget pins gpt-4o-2024-11-20 through the OpenAI Chat Completions API for reproducible teacher behavior.Future changes to the default gpt-4o alias would affect reruns unless the snapshot suffix is preserved.
  • J.4 Random Seeds and Variance: J.4 Random Seeds and Variance uses seed = 42 for training and a separate fixed seed for validation-pool shuffling.Rollout sampling remains non-deterministic with temperature 0.9 and top-p 0.95, while re-rollout variance is bounded primarily by group size g = 2.
  • J.4 Random Seeds and Variance: J.4 Random Seeds and Variance reports single-seed results because rerunning E1–E6 across multiple seeds is prohibitive.The passage states that trajectory-generation variance is driven mainly by group size rather than seed choice.
  • J.5 Software, Code, and Checkpoint Release: J.5 Software, Code, and Checkpoint Release implements the codebase in Python 3.11 using transformers, trl, peft, and vllm.Pre-trained LoRA adapters and evolved skill libraries are slated for camera-ready release, while teacher API logs are not released.
  • J.6 Reproducibility Checklist Pointers: J.6 Reproducibility Checklist Pointers map datasets, hyperparameters, rollout configuration, prompts, evaluation protocols, and qualitative comparisons to specified appendix sections and boxes.The pointers include §D.1, Table 13, §D, Boxes C.1–C.4, and §G.4.
  • K Cost and Latency Breakdown: K Cost and Latency Breakdown quantifies per-episode token budgets and approximate dollar costs for each Table 5 setting.The framework uses up to 17 distinct teacher invocation sites per episode; web search has the longest rollouts, while math and code costs are roughly 30–50% lower.

K.1 Per-Setting Cost Aggregates

Under the full HASP setting, each question costs approximately $0.045 on average, with eight FINAL-time LLM-assisted handlers comprising the main token expenditure. Mid-rollout PFs contribute about 10%, while pre-rollout components cost roughly 2k tokens once per episode.

  • Per-Setting Cost Aggregates: $0.045 per question is the approximate average cost under the full HASP setting.Token costs use the OpenAI gpt-4o-2024-11-20 list price at experiment time: input $2.50/M and output $10.00/M.
  • Per-Setting Cost Aggregates: 50% of teacher tokens per episode come from the eight FINAL-time LLM-assisted handlers in the full setting.Mid-rollout PFs are rate-limited, while pre-rollout components such as the Difficulty Gate and PFSelector cost about 2k tokens once per episode.
  • Per-Setting Cost Aggregates: 10% is the contribution of the rate-limited mid-rollout PFs: retrieval_failure, format_extraction_error, reasoning_error, and answer_confidence_guard.Pre-rollout costs become relatively smaller on longer multi-step rollouts because they are incurred once per episode.
Loading 2605.17734v1…