PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI

Jitao Sang, Jinlin Xiao, Jiarun Han, Jilin Chen, Xiaoyi Chen, Shuyu Wei, Yongjie Sun, Yuhang Wang

arXiv:2510.16720v2cs.AI

TL;DR

Agentic AI aims to move beyond generative models’ reactive outputs toward systems that reason and act, but training agentic behaviors from complete annotated trajectories is costly. This survey reviews how reinforcement learning supports the shift from externally scripted pipelines to model-native capabilities, and concludes that RL-driven internalization is a central direction in agentic AI.

  • Problem

    Generative AI remains largely reactive, while agentic tasks require goal-directed behavior and complete supervised trajectories can be prohibitively expensive to construct.

    Source: §1 · p. 3

  • Method

    The survey synthesizes reinforcement learning as a route to internalizing agentic capabilities and reviews shifts across planning, tool use, memory, and applications.

    Source: §1.3 · p. 6, §3.1 · p. 14, §4.1 · p. 21

  • Results

    The survey identifies a shift from supervised fine-tuning to RL and from process reward toward outcome reward in the model-native paradigm.

    Source: §3.4 · p. 19

  • Takeaways & Limitations

    The survey concludes that RL can help models acquire core agentic capabilities through exploration without costly step-by-step supervision.

    Source: §1.3 · p. 6

    Open-web noise may let outcome-driven RL amplify hallucinations, and subjective research qualities make reward definition difficult.

    Source: Deep Research Agent · p. 5

Abstract

from arXiv · show

The rapid evolution of agentic AI marks a new phase in artificial intelligence, where Large Language Models (LLMs) no longer merely respond but act, reason, and adapt. This survey traces the paradigm shift in building agentic AI: from Pipeline-based systems, where planning, tool use, and memory are orchestrated by external logic, to the emerging Model-native paradigm, where these capabilities are internalized within the model's parameters. We first position Reinforcement Learning (RL) as the algorithmic engine enabling this paradigm shift. By reframing learning from imitating static data to outcome-driven exploration, RL underpins a unified solution of LLM + RL + Task across language, vision and embodied domains. Building on this, the survey systematically reviews how each capability -- Planning, Tool use, and Memory -- has evolved from externally scripted modules to end-to-end learned behaviors. Furthermore, it examines how this paradigm shift has reshaped major agent applications, specifically the Deep Research agent emphasizing long-horizon reasoning and the GUI agent emphasizing embodied interaction. We conclude by discussing the continued internalization of agentic capabilities like Multi-agent collaboration and Reflection, alongside the evolving roles of the system and model layers in future agentic AI. Together, these developments outline a coherent trajectory toward model-native agentic AI as an integrated learning and interaction framework, marking the transition from constructing systems that apply intelligence to developing models that grow intelligence through experience.

1 Introduction

The survey frames agentic AI as a response to generative AI’s reactive limits and traces a shift from externally orchestrated pipelines toward model-native capabilities, with reinforcement learning as a central driver.

1 Introduction

  • Generative AI largely responds to prompts, while agentic AI emphasizes self-directed behavior, complex reasoning, and interaction with environments.The survey identifies planning, tool use, and memory as core agentic capabilities.Source: §1 · p. 3

1.1 Paradigms

  • Pipeline-based agents use external, handcrafted logic, whereas model-native agents progressively internalize planning, tool use, and memory through end-to-end training.Pipeline systems can be rigid and brittle when circumstances change; the model-native policy maps state to action within the LLM.Source: §1.1 · pp. 4–5

1.2 Applications

  • Model-native Deep Research agents internalize research strategy, supporting long-horizon consistency, deeper exploration, and adaptability, while open-web noise and subjective rewards remain challenges.The survey also describes model-native GUI agents that internalize perception, planning, grounding, and action, but face fine-grained errors and changing interface states.Source: Deep Research Agent · p. 5, GUI Agent · p. 6

1.3 Algorithms

  • Agent tasks make complete supervised trajectories costly or infeasible, while multi-turn interactions and sparse rewards challenge methods reliant on dense, step-level supervision.The survey presents RL as learning through exploration and feedback rather than imitation alone.Source: §1.3 · p. 7
  • GRPO uses relative rewards within sampled response groups, and DAPO combines decoupled clipping with dynamic sampling for long-horizon agent training.The survey identifies LLM + RL + Task as a unified training solution and describes these advances as drivers of the model-native paradigm.Source: §1.3 · pp. 7–8

1.4 Survey Structure

  • The survey reviews RL foundations, capability shifts in planning, tool use, and memory, application evolution in Deep Research and GUI agents, and emerging collaboration, reflection, and layer roles.Its organization assigns these topics to Sections 2 through 7.Source: §1.4 · p. 8

2 Algorithm: RL for LLM

The section presents reinforcement learning as a way to internalize agentic behaviors that prompt-based pipelines only elicit, and explains how pretrained knowledge and task environments make this approach broadly applicable. It frames RL-driven reasoning and interaction data as part of a feedback loop that converts compute into intelligence.

2.1 Necessity: The Shortage of Procedural Data

  • CoT prompting injects reasoning structure through examples, but its out-of-distribution prompts can make reasoning brittle and limit generalization.Source: §2.1 · pp. 8–9
  • SFT requires query, reasoning, and answer triples, but human annotation can be prohibitively expensive or impossible, while synthetic data is limited by base-model capability.Source: §2.1 · p. 9
  • RL replaces fixed-trajectory imitation with policy exploration and outcome-based rewards, enabling learning without full procedural supervision or explicit ground truth.Source: §2.1 · p. 9

2.2 Feasibility: Classical RL vs. RL for LLM

  • Pretrained knowledge gives RL a structured policy prior, improving sample efficiency and enabling meaningful trajectory discovery early in training.Source: §2.2 · p. 10
  • Textual or multimodal contexts, generated text or tool actions, and semantic rewards provide a shared interface through which an LLM can bridge reasoning, decision-making, tool use, and memory management.Source: §2.2 · pp. 10–11
  • The LLM + RL + Task framing treats the task as the learning environment, with work spanning data synthesis, reward design, and environment and benchmark construction.Source: §2.2 · p. 11

2.3 A Data Synthesis Perspective

  • Post-training RL synthesizes procedural reasoning data internally and interaction data through tool use and environmental feedback.Source: §2.3 · pp. 12–13
  • RL-driven data synthesis links pretraining, post-training, and inference in a feedback loop that the paper presents as continually converting compute into intelligence.Source: §2.3 · p. 13

3 Core Capability: Planning

Planning has shifted from external symbolic planners and prompt-engineered reasoning toward capabilities trained directly into model parameters. The survey traces this shift through symbolic and prompt-based pipelines, supervised data construction, and reinforcement-learning approaches.

3.1 Overview

  • Planning begins with an initial state and goal; LLMs use learned world knowledge to interpret instructions, decompose tasks, and adjust plans, while pipeline approaches rely on external planners or prompts.The model-native direction uses supervised fine-tuning and reinforcement learning to embed planning and reasoning in model parameters.Source: §3.1 · pp. 13–14

3.2 Pipeline-based Paradigm

  • Symbolic planners such as STRIPS and PDDL search explicit action models effectively in closed domains, but formalizing natural-language tasks is costly and sensitive to modeling errors.LLM-generated PDDL can reduce semantic extraction and modeling burdens, yet hybrid methods remain constrained in open, dynamic, or noisy environments.Source: §3.2.1 · pp. 14–15
  • Prompt-based planning ranges from linear Chain-of-Thought to branching Tree-of-Thought and search methods, trading flexibility for prompt sensitivity, instability, and high token costs.Branching methods support multi-path exploration but depend on evaluator quality and increase computation; linear reasoning can struggle with backtracking.Source: §3.2.2 · p. 15

3.3 Model-native Paradigm

  • Supervised planning internalizes reasoning from offline data, using synthetic multi-path samples or tree search, while distillation transfers a strong teacher’s reasoning chains to a student.Multi-path sampling selects complete trajectories; tree search offers finer step-level control, and distillation quality depends on the teacher’s reasoning ability.Source: §3.3.1 · p. 16
  • DeepSeek-R1 illustrates combining direct reinforcement learning with subsequent data distillation, supporting scalable, automated reasoning-data production with less manual annotation.The cited studies use the model as a teacher and apply simple verification and filtering to produce data.Source: §3.3.1 · p. 17
  • Process rewards provide step-level supervision but require costly, high-quality annotations and face ambiguous step correctness, noisy labels, and reward hacking.Outcome rewards instead assess final-answer correctness, avoiding intermediate-step evaluation in tasks with verifiable answers.Source: §3.3.2 · pp. 17–18

3.4 Summary and Discussion

  • The model-native shift moves planning into model parameters; it also marks broader transitions from SFT to RL and from process rewards to outcome rewards.Outcome rewards are described as sufficient for capable models in verifiable domains such as mathematics and code, often alongside rule-based rewards.Source: §3.4 · p. 19
  • The pipeline-to-model-native trajectory also appears in multimodal reasoning, moving from external tools and prompting toward end-to-end processing of multimodal information.The survey identifies future directions including implicit reasoning, self-supervised internalization, and transferable planning across domains.Source: §3.4 · pp. 19, 21

4 Core Capability: Tool Use

Tool use adds a tactical layer of planning: agents must decide when and in what sequence to invoke tools, then execute calls and use feedback. The survey traces a shift from externally orchestrated workflows toward learned policies, with training choices shaped by credit assignment and environment conditions.

4.1 Overview

  • Tool use combines action-level planning of tool-call timing and sequence with execution that generates valid commands and interacts with the environment.This tactical planning is distinct from decomposing a high-level task into subgoals.Source: §4.1 · p. 21

4.3 Model-native Paradigm

  • Model-native tool use internalizes tool decisions; modular training learns a planner while leaving invocation and environment interaction to a separate executor.This can isolate decision learning from execution noise, conserve training budget, and improve sample efficiency and training stability.Source: §4.1 · p. 22, §4.3.1 · pp. 23–24

4.2 Pipeline-based Paradigm

  • Pipeline-based methods range from fixed workflows, which are predictable and easy to debug but inflexible, to prompts that let models plan dynamically.Prompt methods separate planning from execution or interleave reasoning, actions, and observations; the latter adapts through feedback, while separation suits predictable paths.Source: §4.2.1 · p. 22, §4.2.2 · p. 23

4.3 Model-native Paradigm

  • End-to-end training jointly learns multi-step planning and action execution, with credit assignment ranging from final-outcome rewards across trajectories to finer step- or turn-level signals.Trajectory-level rewards are simple and scalable, but sparse rewards and diffuse credit can weaken policy improvement in long-horizon, multi-tool tasks.Source: §4.3.2 · pp. 24–25
  • Training environments trade stable feedback in simulations for noisier, less stable interaction in dynamic real-world settings.Methods address live-environment noise through filtering or preprocessing feedback and through training frameworks designed for online instability; real-world methods bring higher training and engineering complexity.Source: §4.3.2 · pp. 25–26
  • Under matched training conditions, ReTool performs better with the reasoning base DeepSeek-R1-Distill-Qwen-32B than with Qwen2.5-32B-Instruct.The comparison supports the survey's account that base-model reasoning capacity enhances learning of tool-integrated reasoning.Source: §4.3.2 · p. 26

4.4 Summary and Discussion

  • The survey identifies credit assignment and environmental noise as ongoing challenges, and points to modular learning and finer-grained rewards as ways to improve training stability.Hybrid architectures, robust training in open environments, and agents that create tools are also identified as future directions.Source: §4.4 · pp. 26, 28

5 Core Capability: Memory

This section examines memory as action-oriented evidence governance across storing, managing, retrieving, and using information, including short- and long-term memory. It traces a gradual shift from externally orchestrated workflows toward learned, model-native capabilities, while external stores continue to support persistence and governance.

5.1 Overview

  • Memory governs evidence across the task lifecycle by deciding what to store, how to organize it, when to retrieve it, and how to use it in reasoning and action.The overview distinguishes short-term memory for in-session information from long-term memory that persists across sessions.Source: §5.1 · pp. 28–29

5.2 Short-term Memory: Long Context

  • Long-context effectiveness depends on using information, not merely expanding the context window; noisy or poorly used contexts can hide crucial facts and dilute attention.External pipelines manage context through methods such as sliding windows, summarization, and retrieval-augmented generation; proximity-based truncation can discard distant evidence, while summaries can trade detail for compactness.Source: §5.2 · pp. 29–31
  • Long-sequence curricula help models internalize retrieval and reranking for end-to-end reasoning, while positional extrapolation and efficient attention support longer inputs.UltraLLaDA extends context stability to 128K tokens for retrieval and 32K tokens for complex reasoning when position encoding extrapolation is combined with targeted training; efficient attention methods address the quadratic cost of standard attention.Source: §5.2 · p. 32

5.3 Short-term Memory: Context Management

  • Hybrid context management replaces fixed rules with learned components or trains models to execute workflows, bridging external pipelines and fully internalized memory.Learned memory slots distill information into structured representations, and modular retrieval policies improve task performance while retaining architectural interpretability; workflow-internalizing approaches learn when to retrieve, rewrite, or evaluate information.Source: §5.3.2 · pp. 34–36
  • Fully model-native context management makes memory operations part of a unified policy, with agents actively curating working memory and coupling retrieval with generation.MemAct uses explicit editing operations trained with DCPO to balance memory curation with long-term task objectives; external repositories can remain factual foundations while the model controls their use.Source: §5.3.2 · p. 36

5.4 Long-term Memory

  • Long-term memory uses external repositories or model parameters as carriers; repositories support traceability, while parameter storage can lower latency but reduce interpretability and risk fact interference.Repository methods emphasize precise retrieval and structured, verifiable evidence; stable, frequently reused knowledge can be internalized through training, while targeted model editing suits sparse factual corrections rather than large-scale migration.Source: §5.4 · p. 36, §5.4.1 · p. 37, §5.4.2 · p. 37

5.5 Summary and Discussion

  • Short-term memory is shifting from pipeline-managed capacity toward learned context management that filters noise, locates evidence, and uses it in reasoning.Early systems such as MemoChat [131] used structured templates but accumulated cross-module errors and left the model passive; expanded context windows have shifted the challenge toward effective information use.Source: §5.5 · p. 38
  • Internalizing memory is gradual and depends on long-sequence data synthesis and curriculum design, with retrieval, compression, and verification proposed as explicit learning objectives [11] [74] [314].Models may learn retriever-like attention and router-like resource allocation [121], leaving vector stores as backend infrastructure for compliance, persistence, and offline preparation.Source: §5.5 · p. 39
  • Long-term memory is moving from fixed external repositories toward personalized evidence cards, while co-training, online write-back, and adapters increasingly link it to short-term memory.Earlier cascaded re-ranking [83] could not rewrite content online and could retrieve relevant but reasoning-irrelevant material [6]; methods include co-training retrieval and generation [74] [303], adapters, and systems such as Mem0 [26].Source: §5.5 · p. 39
  • Memory evaluation is expected to move beyond raw capacity toward benchmarks testing long-range dependency, interference robustness, and logical closure.Source: §5.5 · p. 39
  • Future memory systems are expected to unify short- and long-term memory as model-native skills and routinely co-train retrieval with generation.Personalized, dynamic long-term memory will require stronger governance, privacy controls, and user-facing correction for consistency and reliability.Source: §5.5 · p. 39

6 Applications

This section examines the shift in Deep Research and GUI agents from externally orchestrated pipelines toward model-native systems that learn agentic behaviors through reinforcement learning and end-to-end training, while noting persistent challenges for GUI agents.

6.1 Deep Research Agent

  • Pipeline systems use prompt templates and tool chains for one-shot retrieval and synthesis or iterative reasoning-action retrieval; they are controllable but manually designed and vulnerable to prompt sensitivity and long-horizon instability.Multi-turn methods interleave reasoning, retrieval, and observations, while context management and evidence refinement help maintain focus and reduce noise.Source: §6.1 · pp. 39–41
  • Model-native Deep Research internalizes search strategies through reinforcement learning, improving long-horizon coherence and adaptation while reducing goal drift, redundant retrieval, and unnecessary tool calls.The model can adjust retrieval depth and evidence integration to task complexity, supporting generalization across open-domain, multi-hop, and unstructured-evidence tasks.Source: §6.1 · p. 40
  • Model-native training internalizes planning and retrieval end-to-end: offline closed-corpus learning offers control and reproducibility but bounded knowledge, while online web learning adds freshness alongside engineering and optimization costs.ZeroSearch simulates a search engine to approximate web variability while retaining controlled training dynamics.Source: §6.1 · pp. 42–43
  • Deep Research agents converge across four dimensions: API- or browser-based information acquisition, definitive answering or generative synthesis, LLM-to-LRM-to-LMM base models, and single- to multi-agent architectures.The survey summarizes these dimensions in its overview of Deep Research methods.Source: §6.1.4 · p. 43, Table 6 · p. 44
  • Future directions include adaptive API or browser selection, interactive, continuous, and predictive research objectives, complexity-aware agent collaboration, and curiosity-driven or self-reward mechanisms.The proposed collaboration approach models workflows as mutable graphs optimized through evolutionary or adaptive algorithms.Source: §6.1.4 · p. 47

6.2 GUI Agent

  • GUI agents autonomously perceive, plan, and execute tasks in graphical interfaces, and their development also moves from external pipeline-based systems toward a model-native paradigm.Source: §6.2 · p. 47

Model-native Paradigm

  • Model-native work internalizes GUI capabilities through modular or end-to-end training; modular components and fusion support targeted learning, while error propagation motivates fuller integration.Modular training can improve interpretability and data efficiency, but the field continues toward end-to-end training to reduce inter-module errors.Source: §6.2 · pp. 47, 50, 51, 53

Pipeline-based Paradigm

  • Prompt-based LLM agents replaced fixed scripts with natural-language-driven, zero-shot task handling, but remained pipeline-orchestrated and faced perception, state-tracking, and inference-cost limits.Prompt-based methods enabled agents to handle complex tasks without manual scripting, while limitations motivated direct model training for greater stability and generalization.Source: §6.2 · pp. 47, 50

Modular Training

  • Modular work trains grounding, planning, and execution components; grounding improvements are linked to task success, while atomic and widget-level executors trade precision and generality against semantic alignment.Planning approaches include imitation from demonstrations and generative planning based on learned environment dynamics or world models.Source: §6.2 · pp. 51–52
  • Advanced modules add reflection for action or sequence evaluation and correction, alongside exploration that generates interaction data to improve sample efficiency, robustness, and generalization.Exploration progresses from broad trajectory synthesis to backward task synthesis and targeted, few-shot exploration.Source: §6.2 · p. 52

End-to-end Training: Offline

  • Offline end-to-end training evolves from imitation datasets and staged curricula to policy-level optimization, directly mapping interface perception through planning to execution in one architecture.GUI-R1 uses rule-based reinforcement fine-tuning with a unified action space and relative rewards to improve execution stability and generalization.Source: §6.2 · p. 53

End-to-end Training: Online

  • Online end-to-end training advances from pixels-to-actions optimization to self-evolving, multi-task agents that use environmental rewards and sustained interaction to improve across platforms.Later systems extend online optimization to multi-turn interaction and task transfer, supporting policy generalization and self-evolution over long-term interactions.Source: §6.2 · pp. 53–55

Summary and Discussion

  • The section frames GUI-agent evolution as two leaps: LLMs enabled natural-language plans, then reinforcement learning supported model-native systems for continuous improvement.The second leap internalizes the perceive-plan-act loop through data-driven training, culminating in systems such as UItron and GUI-Owl.Source: §6.2.4 · p. 55

6.2 GUI Agent

  • GUI agents remain bottlenecked by scarce high-quality labels, weak foundation-model GUI priors, and unrealistic environments; Ultron[293] calls for robust data engineering and interaction infrastructure.Source: §6.2.4 · p. 55
  • Data-efficiency work targets adaptation to new interfaces with few samples through domain knowledge injection, dense rewards, self-supervised annotation, and better exploration.Source: §6.2.4 · p. 55
  • GUI perception should improve fine-grained visual-language recognition and localization, and fuse screenshots, OCR, and structured metadata for grounding, as in MobileFlow[163].Source: §6.2.4 · p. 55
  • Evaluation should pair unified benchmarks such as A3[15] and MLA-Trust[269] with automated pipelines, then move beyond static tests to realistic dynamic environments.Source: §6.2.4 · pp. 55–56

7 Future Direction and Discussion

The discussion connects the shift from human-designed logic to data-driven learning with two future questions: which capabilities may become model-native, and how the system layer may evolve. It reviews emerging approaches to multi-agent collaboration and reflection, then considers a system layer shifting from capability compensation toward ecosystem support.

7.1 Emerging Model-native Agentic Capabilities

  • Fig. 10 maps future capabilities by implementation difficulty and current model-native progress, highlighting multi-agent collaboration and reflection as mid-term research frontiers.The roadmap spans quick internalization, such as output formatting, to long-term challenges such as safety and alignment.Source: §7.1 · p. 56, Figure 10 · p. 57
  • Model-native collaboration progresses from reinforcement-learning-trained agent roles to learned system topologies, while distillation can encode multi-role patterns in one model for orchestration-free inference.The survey describes two stages: improving individual agents, then learning system architecture.Source: §7.1.1 · pp. 58–59
  • Multi-agent reinforcement learning can internalize collaboration, but concurrent agents complicate credit assignment and make learning non-stationary; proposed directions include better attribution and more stable algorithms.Changing peer policies can make an action that was once optimal become suboptimal.Source: §7.1.1 · p. 59
  • Reflection evolved from prompt-driven self-check loops to trained external pipelines, which can improve their applicability but remain template-bound and face training-data cost and quality bottlenecks.Early loops can reduce errors, but depend on the base model's evaluation abilities and scale poorly with task complexity or interaction length.Source: §7.1.2 · p. 59
  • Model-native reflection trains self-correction into generation, with approaches using corrected trajectories, masked self-reflection, or context-sensitive verification; challenges include data design, reward modeling, interpretability, and compute trade-offs.The survey identifies self-supervised training as a possible way to reduce dependence on explicit error labels.Source: §7.1.2 · p. 60

7.2 System vs. Model: Evolving Roles in Agentic AI

  • The survey forecasts three stages: pipeline design (2023–2025), a model-native transition (2025–2027) with scaffolding still supporting deployment, and autonomous evolution from 2027 onward.The final stage anticipates standardized AgentOps and more autonomous capabilities.Source: §7.2 · p. 61
  • The system layer's role shifts from compensating for model deficits to supporting capabilities as they become model-native, including dialogue, planning, and tool use.The survey notes a divide between researchers anticipating internalized capabilities and industry practitioners' engineering-heavy view of production systems.Source: §7.2 · pp. 60–61
  • As memory and collaboration become model-native, the system layer is envisioned as ecosystem infrastructure for synchronization, compliance, privacy, scheduling, communication, identity, and execution observability.The survey describes this future as foundational AgentOps rather than capability-specific patches.Source: §7.2 · pp. 61–63, Figure 12 · p. 62

8 Conclusions

The conclusion frames agentic AI's shift from externally orchestrated pipelines to model-native systems as a redefinition of how intelligence is trained and deployed. It presents reinforcement learning and task experience as central to internalizing capabilities and growing intelligence through interaction.

8 Conclusions

  • Reinforcement learning bridges perception and action, turning static models into adaptive, goal-directed entities that learn from environmental interactions.The conclusion frames this as a shift from externally scaffolded reasoning, memory, and action to internalized capabilities.Source: §8 · p. 63
  • Planning, tool use, and memory are progressively absorbed into model policy through the unifying LLM + RL + Task framework and cycles of pretraining, post-training, and inference.The survey characterizes this framework as transforming compute into intelligence.Source: §8 · p. 63
  • The model-native transition is presented as a move from systems that use intelligence to models that grow intelligence through experience and interaction with their environments.The conclusion emphasizes learning, collaboration, and evolution through experience as features of the next era.Source: §8 · p. 63

Related papers

Loading 2510.16720v2…