PaperlayerView exact revision on arXiv

Source-linked AI summary

FAPO: Fully Automated Prompt Optimization of Multi-Step LLM Pipelines

Paul Kassianik, Baturay Saglam, Huaibo Zhao, Blaine Nelson, Supriti Vijay, Aman Priyanshu, Amin Karbasi

arXiv:2606.19605v2cs.SEcs.AI

TL;DR

Multi-step LLM pipelines can fail across interacting retrieval, reasoning, formatting, and control-flow steps, limiting prompt-only optimization. FAPO inspects intermediate steps and iteratively applies evidence-scoped prompt or structural changes, outperforming GEPA in 15 of 18 model-benchmark comparisons with a mean gain of +14.1 pp.

  • Problem

    Multi-step LLM pipelines can fail across interacting retrieval, reasoning, formatting, and control-flow steps, so single-prompt tuning is insufficient.

  • Method

    FAPO repeatedly evaluates pipelines, inspects intermediate steps, diagnoses bottlenecks, and validates scoped prompt-first changes before structural edits.

  • Results

    FAPO outperformed GEPA in 15 of 18 model-benchmark comparisons, with a mean accuracy gain of +14.1 pp.

  • Takeaways & Limitations

    These results support pipeline-aware, evidence-grounded optimization for general-purpose and security-focused LLM tasks.

  • Takeaways & Limitations

    The GEPA comparison is a reproduced benchmark comparison rather than an exact fairness match because the optimizers and allowable search spaces differ.

Abstract

from arXiv · show

Multi-step LLM pipelines fail through interactions among retrieval, reasoning, and formatting steps, so prompt-only optimization can miss bottlenecks in the chain. We present Fully Automated Prompt Optimization (FAPO), a framework that lets Claude Code optimize an LLM pipeline inside a standardized codebase. FAPO evaluates a pipeline, inspects intermediate steps, diagnoses failures, proposes scoped changes, and validates variants repeatedly to optimize against a score function. It first tries prompt edits and, only when prompt optimization appears insufficient, changes chain structure within the permitted scope when attribution identifies a structural bottleneck. Across six benchmarks and three task models, FAPO beats the baseline GEPA in 15 of 18 model-benchmark comparisons. In 11 model-benchmark comparisons, FAPO wins with non-overlapping mean $\pm$ trial-standard-deviation ranges, and the mean FAPO-GEPA gain is +14.1 pp. In the six HoVer and IFBench comparisons where prompt-first search escalated to structural changes, FAPO wins all six with a mean gain of +33.8 pp. FAPO also improves performance on security tasks: on CTIBench-RCM, a security CVE-to-CWE task, prompt-only FAPO lifts test accuracy by +4.0 pp on GPT-5, +7.1 pp on Foundation-Sec-8B-Instruct, and +2.0 pp on Foundation-Sec-8B-Reasoning. These results position FAPO as a state-of-the-art pipeline optimization technique for both general-purpose and security-focused tasks.

1 Introduction

FAPO addresses the limits of prompt-only optimization for complex, inspectable LLM pipelines by using Claude Code to diagnose failures and iteratively improve prompts or permitted pipeline structure. Across six benchmarks and three task models, it outperforms GEPA in most comparisons while providing a reproducible optimization workspace.

  • Motivation: FAPO targets failures that propagate across multi-step pipelines, where single-turn prompt tuning cannot address every bottleneck.These pipelines combine LLM calls with code-based steps, and failures can occur at any stage.
  • Results: 15 of 18 model–benchmark comparisons favor FAPO over GEPA, with a mean FAPO–GEPA gain of +14.1 percentage points.The evaluation covers six benchmarks and GPT-4.1-mini, GPT-5.4-mini, and Gemma 3-12B.
  • Method: FAPO uses Claude Code to inspect intermediate failures, propose scoped variants, evaluate them, and validate changes within isolated workspaces.It represents pipelines as stateful LangGraph graphs and searches for higher-scoring task-specific pipelines.
  • Method: FAPO begins with prompt edits and escalates to structural changes only when failure attribution indicates that prompting cannot resolve the dominant bottleneck.This prompt-first procedure is a central design principle of the technique.
  • Reproducibility: FAPO records final outputs, intermediate step outputs, configurations, and variant history to support reproducible pipeline optimization.The workspace procedure is designed to make optimization evidence and changes inspectable.

2 System Overview

FAPO optimizes LLM pipelines as inspectable, bounded workflows: it records intermediate evidence, attributes failures, proposes scoped changes, reviews variants, and retains improvements. Its shared runtime and isolated tenant workspaces support reproducible evaluation across task-specific pipelines.

  • Evidence-based inspection: FAPO records each pipeline step’s inputs, outputs, and logs to localize failures to prompts, evidence sources, or chain structure.Recorded intermediate steps reveal whether errors arise during retrieval, reasoning, formatting, or the final response step.
  • Workspace architecture: Each task workspace defines instructions, examples, scoring rules, current prompts, datasets, and permitted changes for isolated optimization.Tenant workspaces preserve task-local chain code, variants, and evaluation materials, while the shared runtime handles execution and scoring.
  • Optimization loop: FAPO evaluates the current workflow, groups mistakes by likely cause, proposes one scoped variant, reviews it, and reruns evaluation.Failure categories include missing evidence, unsupported abstention, verbosity, malformed output, and weak final instructions.
  • Scoped intervention: Prompt edits are attempted first; parameter or chain changes occur only when recorded failures show prompts are insufficient and the scope contract permits them.This implements FAPO’s preference for the smallest useful change while keeping optimization bounded by task-specific rules.
  • Reproducibility and isolation: Variants and scores are recorded within isolated tenant directories, enabling visibility into prior attempts while preventing cross-task assumptions from affecting optimization.The shared tester evaluates many tasks, whereas each tenant can define its own pipeline, scoring, and deployment methods.

3 Claude-Driven Optimization

FAPO uses Claude Code as an orchestration layer that evaluates pipelines, attributes failures, proposes scoped variants, and reviews changes under tenant-defined constraints. Its prompt-first loop escalates to chain parameters or structure only when attribution identifies a bottleneck that prompt edits are unlikely to fix.

  • Orchestration: Claude Code orchestrates optimization separately from the evaluated task model by editing the workspace, running evaluations, dispatching subagents, and recording variants.The mechanism supports pipelines using closed or open-source task models.
  • Orchestration: Three agents drive the loop: optimization, step-attribution, and variant-reviewer.Attribution classifies failures as prompt-addressable or structural, while review checks scope compliance, placeholder integrity, data leakage, and scorer compatibility.
  • Optimization loop: The optimizer defines a scope contract allowing prompt text, chain parameters, or chain structure, then evaluates variants, attributes failures, proposes scoped changes, reviews them, and compares candidates.The loop uses training outputs with intermediate-step evidence and aggregate validation scores to guide continuation or escalation.
  • Optimization loop: Prompt changes are tried first, with escalation to chain parameters or structure only when prompt optimization appears insufficient, the contract permits escalation, and attribution identifies a structural bottleneck.This preserves a prompt-first policy even when chain-level variants are allowed.
  • Guardrails: Four guardrails limit overfitting: split access controls, scope constraints, iteration memory, and variant immutability.These restrict information access, independently enforce allowed changes, record variants and scores, and create a new file for every accepted or rejected attempt.

4 Evaluation

FAPO was evaluated against GEPA across six benchmarks and three task models, typically outperforming GEPA while escalating from prompt edits to structural changes on HoVer and IFBench. Prompt-only optimization also improved CTIBench-RCM accuracy across all three evaluated models.

  • Overall comparison: FAPO typically outperformed GEPA across six benchmarks and three task models, with AIME the only benchmark where GEPA led across all three model comparisons.FAPO won 9 of 12 prompt-only comparisons, including six with non-overlapping mean ± trial-standard-deviation ranges.
  • Structural optimization: +24.78 to +48.56 pp gains on HoVer followed attribution of insufficient retrieval coverage and expansion from three-hop retrieval to four–five hops with multi-query BM25 search and entity-aware rescue.HoVer was one of the benchmarks where FAPO escalated to pipeline optimization after starting at prompt level.
  • Structural optimization: +19.84 to +38.95 pp gains on IFBench followed attribution of format failures and addition of deterministic post-processing nodes enforcing instruction constraints.IFBench centers on verifiable constraint satisfaction, making format and constraint enforcement central failure modes.
  • Experimental design and limitations: FAPO’s broader search scope permitted chain-parameter and structural changes after prompt optimization, but escalation increased run-to-run variation through path-dependent trajectories.GEPA remained a fixed-program prompt optimizer, whereas FAPO began at prompt level and could modify the pipeline under the scope contract.

5 Related Work

Prior work optimizes prompts, demonstrations, modules, chain structure, training code, or adversarial prompt searches. FAPO extends agent-driven closed-loop search toward stable constructive improvement of deployable multi-step pipelines using aggregate validation performance and attribution.

  • Pipeline and prompt optimization: Pipeline optimization spans prompt text, module composition, and chain topology, including GEPA, DSPy, MIPRO, and APE.These methods represent established approaches to optimizing multi-step LLM systems at different granularities.
  • Automated research agents: Autoresearch uses an LLM agent to edit training code and hyperparameters under fixed experiments and one scalar training metric, whereas FAPO targets discrete pipeline optimization.Both use agent-driven closed-loop experimentation, but they optimize different objects and evaluation settings.
  • From jailbreaking to prompt optimization: Automated jailbreaking searches discrete prompt actions with verifier or score feedback, including tree search, capability-scaling studies, and adversarial reasoning over reasoning strings.Its technical lineage includes universal adversarial triggers, AutoPrompt, and GCG’s token-level prompt generation for aligned chat models.
  • FAPO’s constructive continuation: FAPO is a constructive continuation of this search pattern, optimizing aggregate validation performance for one deployable pipeline variant rather than rare adversarial successes.It preserves the evaluate–analyze–propose–iterate loop while emphasizing mean behavior across examples, task constraints, and multi-step pipeline attribution.

6 Conclusion

FAPO addresses failures caused by interactions across multi-step LLM pipeline components rather than isolated prompts. It provides a reproducible loop for evaluating pipelines, diagnosing intermediate bottlenecks, proposing scoped changes, and validating variants.

  • Multi-step LLM pipelines fail through interactions among retrieval, reasoning, formatting, and control flow.
  • FAPO is a Claude Code-based framework that turns pipeline failures into a reproducible optimization loop.
  • The loop evaluates the pipeline, inspects intermediate steps, diagnoses bottlenecks, proposes scoped changes, and validates resulting variants.
  • FAPO starts with prompt edits and escalates to structural changes only when attribution indicates that prompt tuning is insufficient.

A System Implementation Details

This appendix presents technical implementation details that are summarized at a higher level in Section 2.

  • The appendix provides technical details of the system implementation.
  • Section 2 summarizes the implementation at a higher level.
  • The appendix expands on the higher-level summary in Section 2.

A.1 Runtime and Task Workspaces

FAPO separates a reusable evaluation runtime from tenant-local pipeline definitions behind an eval-config boundary. The same runner and scorer contract supports interchangeable task models and both single-node and multi-node LangGraph chains.

  • Runtime and workspace separation: FAPO’s reusable runtime provides configuration, dataset loading, prompt rendering, provider adapters, chain loading, scoring, artifact writing, progress tracking, storage, and failure attribution.Tenant-local workspaces contain the task chain, prompt variants, scorer implementation, and dataset conversion scripts.
  • Configuration boundary: The eval config specifies the tenant, provider, dataset, scoring profile, output directory, concurrency, run identifier, and chain module, factory, and local settings.Chain-local settings can carry prompt paths and task parameters such as retrieval depth.
  • Evaluation control flow: The runner validates configuration, loads JSONL cases and the tenant scorer, constructs a provider adapter, builds the tenant chain, then streams, scores, tracks, and records each case.Supported provider adapters include OpenAI, SageMaker, and Baseten-compatible providers.
  • Model and tenant abstraction: A small ProviderClient.generate(messages) interface keeps task models interchangeable while task-specific logic remains outside the core package.The runner and scorer contract is shared across all tenants.
  • Chain configurations: Single-node AIME and CTIBench-RCM tenants use one-node LangGraph chains, whereas HotpotQA uses six nodes for two-hop retrieval, summarization, query generation, and final answering.HotpotQA’s sequence includes BM25 retrieval, two summaries, follow-up-query generation, a second retrieval hop, and final answer generation.

A.2 Chains and Pipeline-Aware Scoring

FAPO evaluates standardized executable LangGraph chains that preserve a shared state protocol and expose named intermediate outputs. Its runner streams execution for diagnostics and resilience, while pipeline-aware scorers can evaluate intermediate steps and validate composite results.

  • Chain protocol: FAPO targets compiled LangGraph StateGraph chains built through a fixed build_chain(provider, config) factory while preserving the required ChainState fields.Chains may use typed or dictionary state and may add fields, while the runner initializes the protocol for every case.
  • Node execution: make_llm_node renders prompts from case context and prior outputs, calls the provider, optionally parses results, and writes named state outputs.Prior outputs are exposed under steps.<name>.output, with output_text and step_outputs updated for inspection.
  • Node execution: Custom nodes follow the same state-update contract, enabling multi-hop retrieval and generation prompts to reference outputs from earlier steps.HotpotQA retrieval uses case context or previous outputs, queries an in-process BM25 index, and writes formatted passages into step_outputs.
  • Evaluation runtime: The runner streams chain execution, records per-node timings, and continues evaluation after exceptions by recording diagnostics and scoring failed cases with empty outputs.It also warns and scores an empty final answer when a chain never sets output_text.
  • Pipeline-aware scoring: Pipeline-aware scorers can score intermediate step outputs, while the default scores the final output and the runtime requires a finite composite_score in [0, 100] plus a numeric score_breakdown.HotpotQA explicitly scores the answer step when present.

A.3 Run Artifacts and Failure Attribution · A.4 Tenant Isolation

FAPO records self-contained run artifacts and uses lightweight deterministic heuristics, optionally enriched with trace evidence, to attribute failures. Tenant-local layouts, configurations, operational contracts, and immutable variants enforce isolation and scoped optimization.

  • A.3 Run Artifacts and Failure Attribution: Each evaluation writes a self-contained directory with resolved run settings, per-case results, and atomically updated progress status.Artifacts include run_config.json, results.jsonl, and progress.json.
  • A.3 Run Artifacts and Failure Attribution: attribute_failures filters low-scoring cases and deterministically assigns likely chain-step failures using named heuristics, including retrieval overlap and hit, partial, or miss classifications.Retrieval-like steps are recognized by names containing retrieval-related terms.
  • A.3 Run Artifacts and Failure Attribution: FAPO can attach compact trajectory evidence from Inspect logs or synthesized step outputs and timings without changing the scoring contract.Trajectories may include turns, tool calls, errors, token counts, wall-clock time, expected answers, and final outputs.
  • A.4 Tenant Isolation: Each tenant uses a self-contained directory layout separating protected inputs, derived datasets, code, tests, chains, prompts, configurations, outputs, and reports.The layout also includes storage and documentation areas.
  • A.4 Tenant Isolation: Evaluation configurations keep dataset, chain, prompt, scorer, and output paths tenant-local.The evaluation config is described as the first enforcement point.
  • A.4 Tenant Isolation: Customer artifacts use canonical tenant storage configuration, while local evaluation configs and outputs remain ephemeral and operational documents are checked in.Customer-data synchronization is performed through python -m hephaestus.cli customer-data.
  • A.4 Tenant Isolation: Before editing, Claude reads the tenant playbook and emits a scope contract; prompt and structural variants are immutable clones with metadata and configuration-derived prompt paths.Structural variants live under chains/variants/ and describe their parent chain and hypothesis.

B Optimized Prompt Variants · C CTIBench-RCM Full Variant Progression

The optimized prompts are model-specific: GPT-5 benefits from expanded NVD confusion-pair rules, whereas concise prompts work best for the Instruct and Reasoning models. On HotpotQA, optimization adds brevity, must-answer, and exact-format rules to address near-miss and abstention failures.

  • B Optimized Prompt Variants: Optimized prompts differ by model even for the same task, with CTIBench-RCM prompts shown for each model and HotpotQA prompts shown before and after optimization.This section compares model-specific prompt variants rather than presenting one universally optimal template.
  • B.1 CTIBench-RCM: Baseline (variant-001, all models): CTIBench-RCM uses a cybersecurity-expert system prompt across all baseline models.The baseline asks the model to map each CVE to a CWE, briefly justify the choice, and place only the CWE ID on the final line.
  • B.2 CTIBench-RCM: GPT-5 Best (variant-029, 76.1% test): GPT-5’s expanded rules cover distinctions including buffer overflows, command injection, hardcoded credentials, malformed-input DoS, weak cryptography, authorization, integer overflow, and NULL dereference.The prompt also warns against using CWE-20 as a catch-all and directs attention to root cause rather than impact or attack vector.
  • B.3 CTIBench-RCM: Foundation-Sec-8B-Instruct Best (variant-037, 71.0% test): For Foundation-Sec-8B-Instruct, added rules hurt format extraction, and the best prompt is 2 lines—2× shorter than baseline.Its wording identifies the model as a CWE-classification expert, requests the root-cause CWE under NVD conventions, and places the ID on the last line.
  • B.4 CTIBench-RCM: Foundation-Sec-8B-Reasoning Best (variant-072, 73.0% test): The Foundation-Sec-8B-Reasoning best prompt is almost identical to the Instruct prompt, with “standard NVD abstraction level” contributing +2.9 pp.The prompt maps CVEs to the most appropriate CWE under NVD conventions and outputs the CWE ID on the last line.
  • B.5 HotpotQA: Answer Generation Before and After: On HotpotQA, variant-003 raises validation exact match from 39.22% to 70.3%.The optimized prompt adds brevity rules, a must-answer rule, and format guidance targeting near-miss and abstention failures.
  • B.5 HotpotQA: Answer Generation Before and After: The HotpotQA optimized prompt requires the shortest possible answer, forbids abstention, and enforces exact entity, number, date, or yes/no output formats.It also directs best-effort inference from partial summaries and exact copying of names.
Loading 2606.19605v2…