PaperlayerView exact revision on arXiv

Source-linked AI summary

APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answering

Jie Ding, Rui Sun, Xinyuan Zhang, Zeyu Zhang, Xin Liu

arXiv:2609.02253v1cs.AIcs.CL

TL;DR

Deep research agents need better ways to learn from prior interaction experience because existing approaches either retrieve verbose task-specific traces or produce skills disconnected from policy adaptation. APEX addresses this with hierarchical memories and skills coupled through Executor, Distiller, and Planner modules, trained with alternating GRPO and used for skill-guided test-time adaptation. Across seven benchmarks, APEX achieves state-of-the-art performance, including a 14.7-point gain over GPT-5.4 and a 3.0-point gain over the strongest memory-augmented baseline.

  • Problem

    Existing experience-utilization methods either burden decision-making with verbose task-specific trajectories or leave distilled procedural skills decoupled from downstream policy adaptation.

  • Method

    APEX organizes interaction history into instance-level memories and category-level skills, couples Executor, Distiller, and Planner in a closed loop, and applies alternating GRPO with skill-guided TTRL.

  • Results

    64.4% average performance across seven benchmarks surpasses GPT-5.4 by 14.7 points and the strongest memory-augmented baseline by 3.0 points.

  • Takeaways & Limitations

    Distilled skills function as procedural priors for ground-truth-free online Planner adaptation, while hierarchical utilization reduces injected memory tokens.

  • Takeaways & Limitations

    APEX is currently built on small open-source models, has not been tested for scaling to more powerful backbones, and uses sequential rather than end-to-end joint module training.

Abstract

from arXiv · show

Deep research agents augment large language models with external tools to answer complex, long-horizon questions through multi-turn reasoning. Learning from prior experience is crucial for continual improvement, yet existing methods either retrieve verbose task-specific traces that burden decision-making, or distill procedural skills that remain decoupled from downstream policy adaptation. We propose APEx, a hierarchical experience utilization framework that organizes interaction history into instance-level trajectory memories and category-level procedural skills, and couples them through a closed-loop architecture of Executor, Distiller, and Planner. The three modules are optimized via a three-stage alternating GRPO training paradigm, enabling reward-guided skill distillation rather than fixed-prompt generation. At test time, distilled skills serve as procedural priors for online Planner adaptation through skill-guided test-time reinforcement learning, allowing ground-truth-free self-improvement with skill-alignment regularization to prevent policy drift. Experiments on 7 benchmarks demonstrate that APEx achieves state-of-the-art performance, surpassing GPT-5.4 by 14.7 points and the strongest memory-augmented baseline by 3.0 points.

1 Introduction

APEX addresses continual improvement in deep research agents by organizing experience hierarchically and closing the loop between execution, skill abstraction, planning, and test-time adaptation. It replaces verbose or decoupled experience reuse with reward-guided procedural learning and achieves strong benchmark results.

  • Motivation: Deep research agents need effective experience utilization to improve on complex, long-horizon, open-ended questions without repeating past mistakes.These tasks involve multi-step tool orchestration, iterative retrieval, and multi-source synthesis.
  • Limitations of prior work: Existing episodic memories often retrieve lengthy, task-specific, noisy trajectories, while reflective summaries are generated by fixed prompts and remain external to policy adaptation.These limitations make transferable strategy inference occur at decision time rather than through adaptive priors.
  • APEX framework: APEX organizes experience into instance-level memories and category-level skills, preserving concrete traces while abstracting reusable procedural knowledge for planning.The hierarchy couples concrete task-solving evidence with generalizable guidance.
  • APEX framework: APEX couples an Executor, Distiller, and Planner in an experience-to-action loop that collects trajectories, distills skills, and reuses them for strategic planning.The design aligns execution, abstraction, and planning for continual improvement.
  • Training and adaptation: APEX uses three-stage alternating GRPO training so skill distillation is reward-guided rather than fixed-prompt generation.The paradigm is designed to stabilize cross-module credit assignment.
  • Results: 64.4% average performance surpasses GPT-5.4 by 14.7 points and the strongest memory-augmented baseline by 3.0 points across seven benchmarks.The skill-only variant also reduces injected memory tokens, supporting the context efficiency of hierarchical experience utilization.

2 Methodology Overview

APEX combines hierarchical memory with a closed-loop workflow in which retrieved experiences guide planning, execution produces new trajectories, and those trajectories update memories and skills. Its modules are optimized with GRPO, while TTRL updates the Planner online using retrieved procedural guidance.

  • Memory: APEX maintains instance-level memories for concrete execution trajectories and category-level skills for reusable procedural knowledge.The two levels provide complementary forms of long-term experience.
  • Workflow: For a query, APEX routes it to a category–modality bucket and retrieves relevant memories and the corresponding skill.The retrieved experiences are organized into structured context for planning.
  • Workflow: Conditioned on the query and structured experience context, the Planner generates a research plan that the Executor follows through multi-turn tool interactions.Execution produces a trajectory and final answer.
  • Experience update: After execution, APEX stores a summarized trajectory as a new instance-level memory and updates the corresponding category-level skill when sufficient experience accumulates.This creates a continuing experience-to-action loop.
  • Optimization: Distiller, Planner, and Executor are optimized with GRPO using group-normalized relative rewards and token-level probability ratios.GRPO computes advantages relative to other responses sampled for the same input.
  • TTRL: At test time, the Planner retrieves skill and memory guidance, generates a plan, and is updated online by GRPO using a final reward from a multi-judge LLM-as-Judge mechanism.This implements skill-guided TTRL without standard majority-voting rewards.

3 Methodology

APEX builds a hierarchical experience memory that links concrete trajectories to reusable category-level skills, then jointly trains execution, distillation, and planning through alternating reinforcement learning. At test time, skill-guided adaptation uses judge-based rewards and skill alignment to update the Planner while anchoring it to verified procedures.

  • Hierarchical Experience Memory: Instance-level memories preserve task-solving traces, outcomes, and reuse win rates, organized by task category and input modality.Each memory is represented as mi = (qi, ξi, yi, wi), where ξi summarizes the workflow, yi records correctness, and wi is the success-to-usage ratio.
  • Hierarchical Experience Memory: Category-level skills distill memories within each category–modality bucket into recommended steps, failure modes, descriptions, and skill-level win rates.The Distiller can synthesize, refine, create, or skip updates as new supporting memories accumulate.
  • Three-Stage Alternating RL Training: APEX alternates reinforcement learning across Executor, Distiller, and Planner modules, freezing the other modules during each training stage.The Executor is trained first, the Distiller second, and the Planner third.
  • Three-Stage Alternating RL Training: The Executor reward combines answer correctness, tool-use effectiveness, and format validity during multi-turn tool interaction.Correctness is judged from the execution, tool reward encourages valid use in each segment, and format reward checks structural compliance.
  • Three-Stage Alternating RL Training: The Planner uses retrieved skills and memories to generate a plan, execute it, assess the trace, and optionally re-plan when the initial execution fails.The decision reward encourages stopping after a correct first execution and triggering re-planning after failure.
  • Skill-Guided Test-Time Reinforcement Learning: Skill-guided test-time reinforcement learning adapts the Planner on unlabeled queries with multi-judge rewards and skill-alignment regularization.Judges assess reasoning consistency, evidence support, and answer validity; alignment rewards encourage recommended procedures and avoidance of known pitfalls.

4 Experiment

APEX is evaluated across seven benchmarks, memory-efficiency settings, generalization scenarios, training dynamics, and component ablations. Results show strong performance, reduced context costs, transfer across executor models, and complementary benefits from hierarchical memories and closed-loop adaptation.

  • Main Results: 64.4 average score across 7 benchmarks makes APEX the best-performing system, exceeding GPT-5.4 by 14.7 points and MIA by 3.0 points.APEX also surpasses its backbone Qwen2.5-VL-7B by 46.8 absolute points.
  • Context Efficiency: APEX improves average accuracy over MIA by 1.7 points while reducing injected memory tokens by 45.6% and total token consumption by 36.8%.These results are reported across FVQA-test and 2Wiki.
  • Generalization Analysis: 11.0% average relative improvement on HotpotQA and 7.0% on LiveVQA over ReAct shows generalization across unseen closed-source executor models without additional training.The comparison uses different closed-source executors and preserves the trained APEX pipeline.
  • Experience Utilization: Removing concrete memories, skills, or both lowers average accuracy by 1.8, 3.1, and 10.1 points, respectively, showing complementary roles for instance-level and category-level experience.Skills provide general templates, whereas instance memories provide concrete problem-solving examples.
  • Post-training Optimization: Removing GRPO from the Distiller, Planner, and Executor reduces average accuracy by 2.3, 1.8, and 8.4 points, respectively.The largest degradation without Executor optimization indicates its particular importance for reliable multi-step tool interaction.
  • Closed-loop Adaptation: Removing test-time reinforcement learning or iterative skill updates reduces average accuracy by 4.2 and 2.5 points, respectively, supporting closed-loop adaptation.Improved trajectories are distilled into better skills during test-time feedback.

5 Related Work

Related work moves from trajectory retrieval and textual reflection toward adaptive experience management. APEX distinguishes itself by coupling category-level procedural skill abstraction with downstream Planner adaptation through confidence-weighted alignment.

  • Deep Research Agents: Deep research agents combine dynamic reasoning, adaptive tool-mediated search, and compact experience retrieval for complex multi-turn information-seeking questions.Prior studies improve long-horizon search through curated trajectories, supervised fine-tuning, and agentic reinforcement learning.
  • Memory-Augmented Agents: Earlier memory systems retrieve external documents, trajectories, or summaries, while newer approaches emphasize adaptive experience management.Trajectory retrieval can remain lengthy, task-specific, and noisy, requiring decision-time inference of transferable strategies.
  • APEX: APEX distills trajectories within category–modality groups into reusable procedural skills and uses confidence-weighted alignment to regularize online Planner updates.This couples procedural abstraction with downstream parametric adaptation in a closed experience-to-action loop.

6 Conclusion

APEX is a hierarchical experience utilization framework that organizes interactions into memories and skills, couples execution, distillation, and planning, and supports skill-guided test-time adaptation. Across 7 benchmarks, it achieves state-of-the-art performance while reducing policy drift through skill-alignment regularization.

  • APEX organizes accumulated interactions into instance-level memories and category-level skills for reusable procedural knowledge across tasks.
  • GRPO-based alternating optimization couples the Executor, Distiller, and Planner in a closed loop spanning execution, skill abstraction, and adaptive planning.
  • At test time, distilled skills act as procedural priors for skill-guided TTRL, enabling ground-truth-free online adaptation while reducing drift through skill-alignment regularization.
  • APEX achieves state-of-the-art performance across 7 benchmarks.

Limitations

The current APEX framework has limitations in model scale, training organization, and test-time supervision. The authors identify broader backbone scaling, joint training, and minimal supervision as directions for improvement.

  • APEX has been evaluated with small open-source backbone models, leaving its scaling to more powerful models unexplored.
  • The three-stage alternating GRPO paradigm sequentially optimizes the Executor, Distiller, and Planner rather than exploiting possible end-to-end joint-training synergies.
  • Skill-guided TTRL currently operates online without ground-truth supervision, although minimal supervision could improve distilled-skill quality and accelerate adaptation.

Ethical Considerations

APEX relies on models and open-web retrieval that can expose users to inaccurate, biased, harmful, private, or offensive content. The authors recommend content filtering and specify usage and licensing safeguards.

  • The underlying language and vision-language models may produce inaccurate, biased, or harmful outputs because of pre-training-corpus biases.
  • Open-web retrieval at inference time may expose APEX to personally identifiable information or offensive material, motivating appropriate content filtering by deployers.
  • The authors state that released code or weights will include guidelines prohibiting malicious uses such as misinformation generation and unauthorized surveillance.
  • Proprietary baseline models are accessed through provider APIs and remain subject to their providers’ terms of use.

A Memory and Skill Retrieval

APEX retrieves memories through semantic similarity and historical reuse, gates skills by confidence, and combines category-level skills with instance-level memories for planning. This supports a closed adaptation loop in which improved trajectories update the Planner and future skills.

  • APEX scores candidate memories using query and caption or contextual similarities, with question-only similarity when captions are unavailable.
  • The final retrieval score combines normalized semantic similarity with historical reuse win rate, and successful and failed memories are retrieved separately.
  • Category-level skills combine win rate with evidence sufficiency, and skills are injected into the Planner only when confidence exceeds a minimum threshold.
  • The Planner combines category-level skills and instance-level memories through structured prompt concatenation to use abstract strategies with case-based guidance.
  • For highly reliable skills, a gate can skip test-time updates, preventing degradation from noisy rewards and avoiding unnecessary computation.
  • Skill-guided TTRL forms an adaptation loop in which skills improve plans, plans improve trajectories, and trajectories update the Planner and future skills.

C Training Setups.

Training uses three-stage alternating GRPO across the Executor, Distiller, and Planner, with dedicated models, datasets, rewards, and hyperparameters for each module.

  • Infrastructure: Training runs on one 8-GPU A100 node with FSDP offloading, asynchronous rollouts, and a Qwen3-32B LLM Judge for reward computation.The infrastructure supports 7B/8B model training within GPU memory constraints.
  • Executor Training: The Executor is initialized from Qwen2.5-VL-7B and trained on FVQA-train for multi-turn tool interaction.Its GRPO reward uses λ1 = 0.7, λ2 = 0.2, and λ3 = 0.1.
  • Distiller Training: The Distiller uses Qwen3-8B to transform instance-level memories into structured category-level skill documents in a single turn.Its reward combines skill quality and evolution criteria, with µ1 = 0.5, µ2 = 0.1, and µ3 = 0.4.
  • Planner Training: The Planner uses Qwen3-8B and is trained on FVQA-train without images plus MATPO data through a plan–execute–evaluate–replan loop.Its reward weights are α1 = 0.7, α2 = 0.2, α3 = 0.05, and α4 = 0.05.

D Test Setups.

At test time, APEx adapts the Planner with skill-guided TTRL on unlabeled queries, using configured search tools and skill-alignment regularization.

  • Test-Time Adaptation: Skill-guided TTRL adapts the Planner on unlabeled test queries after training, using a no-ground-truth reward constructed from multiple LLM judges.Category-level skills act as regularization priors during this adaptation.
  • Search Configuration: Test-time search uses wiki25 for several text benchmarks, Serper for other benchmarks, and Serper for all multimodal image-to-image searches.The listed wiki25 benchmarks are HotpotQA, 2WikiMultiHopQA, and SimpleVQA.
  • Reward Formulation: The final TTRL reward combines a no-ground-truth reward with a skill-alignment reward.The no-ground-truth component is Rnogt = 0.9 · A + 0.1 · Rfmt.
  • Skill Alignment: Skills with higher win rate and more supporting evidence exert stronger regularization, while low-confidence or newly created skills have minimal influence.The configuration sets λbase = 0.1 and Nthreshold = 10.

E Dataset Description

The evaluation covers seven multimodal and text-only benchmarks, compares direct, search-augmented, and memory-based agents, and documents the prompts governing the multi-agent workflow.

  • Evaluation Datasets: The experiments evaluate seven benchmark datasets spanning both multimodal and text-only scenarios.Table 8 summarizes the evaluation datasets.
  • Training Analysis: Figure 5 reports each module’s mean reward and corresponding mean response length across training batches.The visualization distinguishes raw training records from batch-averaged smoothed trajectories.
  • Baselines: Baselines are grouped into direct-answer, search-agent, and memory-based search-agent categories.Direct-answer models use internal knowledge, search agents follow multi-turn ReAct tool calling, and memory baselines vary memory incorporation under matched training settings.
  • Prompt Coverage: Dedicated prompts govern the Executor, Planner, Distiller, and TTRL and skill-quality judges within the multi-agent pipeline.The prompt appendix includes text-only and image variants where applicable.
  • Executor Prompt: The Executor prompt makes each turn choose between calling one tool and providing a final answer, then requires observation-driven return to analysis after tool use.Tool calls and final answers use dedicated tags, and final responses must contain only the synthesized answer.
  • Planner Prompt: The Planner prompt conditions action plans on relevant skills, correct and incorrect memories, the question, and available text or image search tools.It requests one concise, generalizable, step-by-step work plan with atomic actions.
Loading 2609.02253v1…