PaperlayerView exact revision on arXiv

Source-linked AI summary

Hypothesis-Driven Skill Optimization for LLM Agents

Fangxin Shang, Yehui Yang

arXiv:2606.22330v1cs.AIcs.SE

TL;DR

Persistent skill updates can improve frozen action agents but may preserve spurious or executor-incompatible rules when derived from sparse or noisy trajectories. HDSO uses frozen curator and executor endpoints to propose falsifiable skills, validate them through paired executions, and admit only supported candidates. On ALFWorld, it improves executor-only baselines while providing evidence about robustness, transfer, and cross-model alignment.

  • Problem

    Persistent skill updates risk encoding spurious shortcuts or rules that target executors cannot reliably follow when distilled from sparse or noisy trajectories.

    Source: §1 · p. 1

  • Method

    HDSO uses a frozen curator and executor in a hypothesis lifecycle with explicit validation, paired control/treatment executions, behavior review, and progressive skill disclosure.

    Source: §1 · p. 1, §3.1 · p. 4, §3.2 · p. 4

  • Results

    +6.9 Avg. SR points for Qwen3-8B and +4.0 points for Qwen3.6-27B improve over executor-only baselines on ALFWorld.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    HDSO provides an auditable alternative to unvalidated prompt or memory accumulation for frozen action agents, with transfer and failure records exposing cross-model alignment boundaries.

    Source: §5 · p. 10, §6 · p. 11

    Empirical evidence is limited to ALFWorld, and the same skill interface should not be assumed to transfer unchanged to single-turn question answering or pure reasoning tasks.

    Source: §5 · p. 10

Abstract

from arXiv · show

External skills can improve action-oriented LLM agents without changing model weights, but persistent skill updates are risky when they are distilled from sparse or noisy trajectories. A plausible reflection may encode a useful procedure, a spurious shortcut, or a rule that the target executor cannot reliably follow. We propose Hypothesis-Driven Skill Optimization (HDSO), a train-free framework in which both the skill curator and the agent executor are frozen inference endpoints. The curator observes executor traces, proposes a falsifiable hypothesis with an explicit validation plan, instantiates the hypothesis as a candidate skill package, validates the package through paired control/treatment executions, reviews behavior differences, and consolidates only supported candidates into an approved repository. The executor consumes approved skills through progressive disclosure, preserving the executor-only path when no skill is selected. On ALFWorld, HDSO improves executor-only baselines by +6.9 Avg. SR points for Qwen3-8B and +4.0 points for Qwen3.6-27B. Under 20% randomly flipped success/failure feedback during skill discovery and validation, HDSO preserves a +7.1-point gain for Qwen3-8B. Transfer and heterogeneous-pair diagnostics further show that validated repositories can be useful beyond the run that produced them, but cross-model curation succeeds only when curator diagnosis, executor capability, and validation evidence align. HDSO provides an auditable skill lifecycle for frozen action agents rather than an unconstrained memory accumulation procedure.

1. Introduction

HDSO treats persistent skill updates as falsifiable hypotheses rather than automatically accumulated reflections, using paired validation before repository admission. It provides a train-free, auditable lifecycle for improving frozen action agents while preserving the executor-only path.

1. Introduction

  • External skills target recurring procedural failures such as invalid actions, unmet preconditions, unproductive search, and missing state transformations without changing model weights.This is especially useful when serving models are fixed by cost, compliance, latency, or vendor constraints.Source: §1 · p. 1
  • The lifecycle avoids blindly appending reflections because trajectories can reflect domain gaps, exploration weakness, formatting errors, invocation mistakes, executor limits, or unreliable feedback.Uncontrolled accumulation can create over-general or contradictory rules.Source: §1 · p. 1
  • HDSO frames persistent skill updates as hypotheses that specify expected behavioral changes, scope, motivation, risks, and falsifying observations.Candidates are tested prospectively against matched control and treatment executions before promotion.Source: §1 · p. 1
  • HDSO combines a frozen curator and frozen executor across proposal, paired validation, behavior review, consolidation, and rejection.Approved skills enter the repository, while rejected hypotheses remain auditable evidence.Source: §1 · p. 2, Figure 1 · p. 2
  • HDSO supports asynchronous train-free operation with auditable approved skills, rejected hypotheses, validation pairs, and failure attributions.The approach requires neither fine-tuning nor reward-model training.Source: §1 · p. 2

2. Related Work

Prior work uses external skills, learned or untrained curators, and reflection-based memories to improve agents. HDSO distinguishes itself by making prospective falsification and evidence-based admission the repository update rule.

Self-evolving agents and external skills

  • HDSO differs from external-skill and self-evolving systems by accepting a skill only when its falsifiable hypothesis survives prospective validation.This emphasizes admission control rather than merely storing reusable artifacts.Source: §2 · p. 3

Automatic skill optimization

  • SkillOS is the closest comparison, but HDSO differs in requiring an observe–hypothesize–validate–review–consolidate lifecycle with validation artifacts and rejected-hypothesis records.The comparison concerns the update rule, not simply whether the curator is trained.Source: §2 · p. 3

Reflection and memory

  • Reflection, experiential learning, context evolution, and reasoning-memory methods convert trajectories or evidence into lessons, memories, or updated operating context.HDSO does not claim to dominate all such methods on raw score.Source: §2 · p. 3
  • HDSO instead targets trust management by attaching scope, expected effect, validation evidence, and rollback rationale to each approved skill.This provides a more explicit basis for managing persistent agent knowledge.Source: §2 · p. 3

3. Method

HDSO updates an approved skill repository while keeping curator and executor models frozen, preserving the executor-only path when skills are absent or unused. It proposes structured skill hypotheses, tests them with paired validation, reviews behavior changes, and consolidates only supported candidates.

3.1. Problem Setup

  • The curator and executor remain frozen while the system updates only the approved repository using observable trajectories, outcomes, and validation summaries.The curator does not access executor private reasoning.Source: §3.1 · p. 3, §3.3 · p. 4
  • HDSO formulates skill updates as hypotheses with explicit scope, falsification conditions, and evidence, retaining rejected candidates for future refinement.Its invariants separate observation, proposal, validation, review, consolidation, and rejection.Source: §3.1 · p. 4, §3.3 · pp. 4, 6

3.2. Executor with Skill Conditioning

  • The executor retrieves a small set of compact skill cards and discloses a full skill only when relevant, avoiding repository-wide prompt expansion.When no skill is available or requested, action generation remains the same as the no-skill ReAct executor.Source: §3.2 · p. 4

3.3. Curator Optimization Loop

  • Candidate skills contain behavioral claims, supporting evidence, scope, expected changes, risks, rollback conditions, invocation metadata, and optional payloads.This structure makes each proposed intervention explicit before validation.Source: §3.3 · pp. 4–5
  • The curator reviews treatment-specific behavior and promotes candidates only when effect, guardrail, invalid-action, scope-regression, and review criteria support the proposed direction.Accepted candidates enter the repository, while rejected or revisable candidates remain in the hypothesis bank and evidence ledger.Source: Figure 2 · p. 5, §3.3 · pp. 5–6
  • Validation proceeds from small screening to medium screening and independent confirmation, comparing control and treatment executions on the same task indices.The promotion gate records paired outcomes, success, step, invalid-action, guardrail, scope, uptake, and statistical diagnostics.Source: §3.3 · p. 5

4. Experiments

HDSO evaluates train-free skill optimization on ALFWorld through controlled setup, benchmark comparisons, frozen-transfer delivery, and noisy-feedback tests. Across these settings, validated skills improve frozen executors, transfer beyond their source run, and remain useful under moderately unreliable optimization feedback.

4.1. Setup

  • HDSO uses ALFWorld train-split traces for discovery and prospective paired control/treatment validation, with held-out evaluation, fixed seeds, and staged screening and confirmation.Promotion requires success-rate improvement, net wins, and limited guardrail regressions; both curator and executor remain frozen.Source: §4.1 · pp. 6–7

4.2. ALFWorld Main Results

  • +6.9 Avg. SR for Qwen3-8B and +4.0 for Qwen3.6-27B over executor-only baselines, with fewer Avg. Steps in both homogeneous settings.A heterogeneous Qwen3-8B curator also raises the Qwen3.5-9B executor from 44.3 to 52.1 Avg. SR (+7.8).Source: §4.2 · p. 7
  • Gains concentrate in recurring procedural-failure families such as heating, cooling, and look-at tasks, especially pick_heat_then_place, while clean and multi-object rows remain unchanged.The task-type pattern supports scoped skills with guardrails rather than broad reminders.Source: §4.2 · p. 8, Table 2 · p. 8

4.3. Skill Delivery

  • Progressive disclosure transfers an approved repository from the Qwen3.6-27B run to Qwen3-8B, improving Avg. SR from 40.5 to 46.2 (+5.7), whereas full-context delivery yields only modest gains.Compact skill cards let the executor request relevant skill bodies instead of receiving the entire repository unconditionally.Source: Table 3 · p. 8, §4.3 · p. 9

4.4. Noisy Feedback

  • With 20% success/failure labels randomly flipped during optimization, Qwen3-8B reaches 47.6 Avg. SR versus 47.4 with clean feedback and 40.5 for the executor-only baseline.The noisy run promotes a narrow cleaning skill, and requests occur in 28 of 420 episodes, concentrated on its target task family rather than unrelated tasks.Source: §4.4 · pp. 9–10
  • The noisy-feedback result is robustness evidence only for this model pair and corruption level, not evidence that corrupted feedback improves skill learning.HDSO avoids directly absorbing every noisy reflection by retaining scoped validation, behavior review, and rejection-sensitive consolidation.Source: §4.4 · p. 10

5. Discussion

HDSO shows that validated skills can transfer beyond their producing run, but cross-model curation depends on matching curator diagnosis, executor capability, and validation evidence. Its empirical scope is limited to ALFWorld and does not establish unchanged applicability to single-turn or pure reasoning tasks.

5. Discussion

  • Validated repositories can contain reusable external knowledge, whereas autonomous cross-model curation is harder because proposed skills must match the target executor’s capabilities.The discussion distinguishes reuse of an already validated repository from generating skills that another model can reliably execute.Source: §5 · p. 10
  • Cross-model curation can fail when a stronger curator proposes procedures beyond a weaker executor’s stable capability or when accepted skills do not cover dominant failures.The 27B→8B and 27B→9B cases show reversals, vanished effects, or narrow local validity during validation and full evaluation.Source: §5 · p. 10
  • HDSO records lifecycle breakpoints across proposal quality, executor uptake, action reliability, validation robustness, coverage mismatch, and review conservatism.These records make heterogeneous failures diagnostically useful rather than treating them only as unsuccessful outcomes.Source: §5 · p. 10
  • The empirical evidence is limited to ALFWorld, and the same skill interface should not be assumed to transfer unchanged to single-turn question answering or pure reasoning tasks.Those settings require different adapter designs because they lack the same multi-step environment-decision trajectories and validation evidence.Source: §5 · p. 10

6. Conclusion

HDSO provides a train-free, hypothesis-driven lifecycle for improving frozen action-oriented LLM agents through validated external skills. On ALFWorld, it improves executor-only baselines and supports transfer diagnostics while leaving broader question-answering and pure reasoning settings to future adapter designs.

6. Conclusion

  • HDSO casts persistent skill updates as a train-free hypothesis lifecycle for action-oriented LLM agents.A frozen curator proposes falsifiable hypotheses, validates candidates through paired execution and behavior review, and promotes only supported skills.Source: §6 · p. 11
  • On ALFWorld, HDSO improves executor-only baselines, remains effective under noisy feedback, and exposes when cross-model curation aligns with the target executor.The resulting transfer and failure records support an auditable alternative to unvalidated prompt or memory accumulation.Source: §6 · p. 11
  • Broader question-answering and pure reasoning settings remain outside the demonstrated scope and require future adapter designs.Source: §6 · p. 11

A. Curator Prompt and Output Schemas

The curator prompt operationalizes HDSO as evidence-backed, narrowly scoped hypothesis generation and review for a frozen executor. Its schemas require structured proposals, paired-validation inputs, explicit promotion criteria, and review diagnoses.

A. Curator Prompt and Output Schemas

  • The curator and executor remain frozen inference endpoints, with runtime evidence supplied through task-specific prompts rather than changes to the static system prompt.The proposal call receives trace evidence, repository state, outcome statistics, approved skills, and existing hypotheses.Source: A. Curator Prompt and Output Schemas · pp. 13–14
  • The curator must separate execution lapses from recurring skill-addressable failures and ground each hypothesis in contrasting successful and failed traces.A failure alone diagnoses a symptom but cannot establish that the proposed policy is better.Source: A. Curator Prompt and Output Schemas · p. 13
  • Prospective paired validation is mandatory, and review rejects candidates without attributable treatment wins or with guardrail, invalid-action, or harmful-overreach regressions.The review separately distinguishes wrong direction, poor implementation, and insufficient validation power.Source: A. Curator Prompt and Output Schemas · p. 13
  • Candidate skills must remain narrowly scoped, avoid benchmark-oracle knowledge, and encode executable policies, progress checks, or output-contract repairs rather than generic advice.Rejected hypotheses become negative evidence, and revised skills must be validated again as new candidates.Source: A. Curator Prompt and Output Schemas · p. 13
  • Each candidate must specify applicability conditions, expected improvements, guardrails, risks, falsification, promotion, and rollback conditions.The proposal schema returns these fields together with evidence, attribution, confidence, and a candidate skill package.Source: A. Curator Prompt and Output Schemas · p. 14
  • The review input contains compact hypotheses, paired-validation summaries, behavior pairs, skill uptake, and metrics summaries, while noisy-feedback promotion uses only the observed success/failure channel.Hidden oracle metrics are reserved for post-hoc diagnosis rather than curator decisions.Source: A. Curator Prompt and Output Schemas · pp. 14–15

B. Representative Approved Skills

The representative repository contains narrowly triggered skills that replace inefficient action sequences with validated direct procedures. Each example combines explicit state checks, action priorities, reset conditions, and safety guards.

B. Representative Approved Skills

  • Both skills use explicit task scope, invocation triggers, state checks, action priorities, reset conditions, and safety guards to operationalize specific failure mechanisms.The repository preserves executable bodies and activation conditions rather than broad recommendations.Source: B. Representative Approved Skills · pp. 15–16
  • A heating skill directly selects “heat [target_obj] with microwave” when the agent holds the object at the microwave, bypassing redundant move, open, close, and examine loops.It preserves baseline behavior outside that inventory-and-location condition and does not apply when the microwave is locked or broken.Source: B. Representative Approved Skills · pp. 15–16
  • An acquisition skill prioritizes taking the target object before using a light source when a look_at_obj task mentions illumination and the object is not already held.The rule suppresses use or examination until acquisition succeeds, then resets after activation or task completion.Source: B. Representative Approved Skills · pp. 16–17
Loading 2606.22330v1…