Source-linked AI summary
From Language Models to World-Acting Systems: Progress and Limits of Agentic AI across Digital, Social, Virtual, and Physical Environments
Linsen Zhu, Mengqing Cai
TL;DR
Agentic AI research lacks a reliable basis for treating model competence, system integration, persistence, and safe authority as one progression. This critical review separates model, harness, environment, and delegator across authority, persistence, and environmental coupling, finding stronger evidence for expanded action coverage than for reliable unattended autonomy. It therefore frames justified delegation as an evidence-bounded heuristic for expanding action scope only when intent preservation, bounded authority, verification, recovery, and human control are supported.
Problem
Research often conflates model competence, system integration, persistence, and safe authority, obscuring whether configured systems can execute delegated work reliably over time.
Method
The review synthesizes primary research and official technical specifications by analyzing delegated authority, temporal persistence, and environmental coupling while separating model, harness, environment, and delegator.
Results
The review finds stronger public evidence for expanded action coverage than for reliable, verifiable, unattended open-world autonomy.
Takeaways & Limitations
Justified delegation should expand action scope only when evidence supports preserved intent, bounded authority, verification, safe recovery, and calibrated human control.
Takeaways & Limitations
The review is bounded by the status of a company research preview and by benchmark settings whose task horizons and thresholds do not establish durable autonomy.
Abstract
from arXiv · showhide
Large language models become consequential agents when surrounding systems let outputs change external state. Models now call tools, operate interfaces, delegate work, retain state, inhabit generated worlds, and control robots or laboratory equipment. Such advances are often narrated as one march toward autonomy, conflating model competence, system integration, persistence, and safe authority. This critical review synthesizes primary research and official technical specifications available by 31 August 2026. We organize the evidence along delegated authority, temporal persistence, and environmental coupling, while separating model, harness, and environment. Within the evidence examined, action-interface expansion is documented more convincingly than robust completion, recovery, authorization, or independent verification. Model Context Protocol and Agent2Agent improve interoperability but do not establish trustworthy delegation; multi-agent organization adds specialization alongside cost and correlated failure. Persistent simulations and world models support training and planning but do not themselves demonstrate agency; robotics and self-driving laboratories establish bounded feasibility rather than unattended open-world reliability. We propose justified delegation as an analytical and normative heuristic, not an observed law or certified score: expand action scope only where evidence supports provenance, bounded authority, failure detection, safe recovery, and calibrated human control. This framing yields a research agenda for coupled model-harness evaluation, capability-based permissions, durable state, cross-agent accountability, and staged physical validation.
1 Introduction
Agentic AI is best analyzed as configured systems that execute delegated work through models, harnesses, interfaces, and environments—not as models alone. The review finds that action and interface capabilities have expanded more clearly than reliable verification, recovery, governance, or unattended autonomy.
- The shift from generation to action: Agentic systems turn model outputs into actions such as database queries, code edits, website navigation, delegation, human escalation, and physical control.The central question is whether these systems can execute delegated work over time while preserving intent, constraints, and a recoverable account.
- The shift from generation to action: An agent is a model configured within a control loop, interface, state representation, and permission set, rather than an especially capable model alone.Early work on reasoning-action loops, tool calling, feedback, and cognitive architectures established this broader unit of analysis.
- The expanding system boundary: Agentic systems now span executable web and desktop environments, coding interfaces, multi-agent delegation, open protocols, persistent simulations, and world models.These developments broaden the surrounding system even when the underlying model is unchanged.
- The expanding system boundary: “More agentic” may reflect a stronger model, richer harness, longer task, or more consequential environment, and these changes do not imply one another.A model can reason without authority, a weak model can receive broad credentials, and a coherent world model can contain no goal-directed agent.
- Evidence and limits: The review compares domains through delegated authority, temporal persistence, and environmental coupling while separating model, harness, environment, delegator, and evidence source.It is a critical review rather than a systematic review or quantitative meta-analysis, and it proposes justified delegation as its organizing direction.
- Evidence and limits: The reviewed evidence shows visible capability and interface gains, but verification, recovery, and governance remain incomplete or external.Representative benchmarks and artifacts include low task success relative to humans, repeated-trial reliability gaps, multi-agent drift, compute confounds, and physical-world hazards.
2 Scope, review method, and evidential discipline
The review uses a critical, question-driven synthesis to examine what evidence supports transferring delegated work across digital, social, virtual, and physical environments. It preserves claim-specific evidential boundaries by distinguishing source types, configured systems, persistence objects, and environmental openness.
- 2.1 A critical rather than systematic review: The review asks what evidence supports transferring delegated work across physical environments and where that evidence stops.It includes heterogeneous artifacts because a bibliometric review would underrepresent standards and deployed interfaces, while treating all artifacts equally would mistake announcements for scientific results.
- 2.1 A critical rather than systematic review: Evidence is prioritized by claim: peer-reviewed papers and proceedings for capability claims, specifications for protocol semantics, and first-party materials for previewed systems and disclosed limitations.Surveys position terminology but do not substitute for primary results.
- 2.1 A critical rather than systematic review: The review does not estimate corpus-wide prevalence or a formal meta-analytic effect size because models, prompts, inference budgets, scaffolds, and benchmark states change too quickly for scores to be pooled.Representative contrasts are used instead, and historical scores are tied to named configurations rather than current leaderboards.
- 2.2 Evidence categories and claim boundaries: Evidence quality is attached to particular claims rather than assigned as a permanent score to a system.Protocol specifications support wire semantics, controlled physical experiments support bounded feasibility, and independent benchmarks support task success while potentially omitting credentials or affected users.
- 2.2 Evidence categories and claim boundaries: The review separates capability demonstrations from reliable frequency, distinguishes persistence objects, and treats openness as action-environment scope rather than visual richness.A resettable simulator may look unbounded, while a short API call may affect a live service with difficult-to-reverse consequences.
- 2.3 The unit of analysis: a configured action system: The relevant unit is the configured system S = (M, H, E, U), comprising model, harness, operational environment, and delegator.The harness includes context, memory, tools, planners, verifiers, retries, credentials, and controls; the environment supplies state, actions, dynamics, and consequences.
- 2.3 The unit of analysis: a configured action system: System analysis tracks delegated authority, temporal persistence, and environmental coupling as separate dimensions.These range from recommendation to direct state change, single responses to recurring obligations, and resettable state to shared social or physical environments.
3 From model capability to configured agency
Agentic behavior emerges from a model embedded in a harness that manages actions, observations, memory, permissions, and recovery, not from model capability alone. Evidence distinguishes improved policy competence and action coverage from the less-established assurance needed for reliable autonomy.
- Model–harness interaction: ReAct, Toolformer, and Reflexion established action–observation loops in which models select tools, use returned state, and revise later behavior.These systems shifted analysis from standalone generation toward configured control loops.
- Validation boundary: Tool-use competence does not establish authorization, trustworthy observations, or correct external side effects.Reference-call accuracy can diverge from authorized action and achieved final state.
- Validation boundary: The 50%-task-completion time horizon summarizes tasks an AI system completes with 50% success, but supports capability claims rather than durable autonomy.The suite uses human task duration as a complexity proxy, while its clean tasks and 50% threshold fall below many deployment requirements.
- Model–harness interaction: The harness adds memory, action interfaces, planning, sandboxing, credentials, approval gates, monitoring, retries, checkpoints, and recovery procedures.These components determine how model outputs become operational interventions.
- Configured-system evidence: SWE-agent reported a historical 12.5% pass@1 on SWE-bench, showing that interface design materially shapes behavior beyond model weights.Its model-oriented commands and feedback supported repository navigation, editing, and testing.
- Configured-system evidence: Agency requires separating the delegator, model, harness, environment, operator, and affected parties, with verification comparing observed state against objectives and constraints.The model proposes, the harness mediates, and the environment changes; verification should not rely solely on the model’s self-assessment.
- Configured-system evidence: Policy competence and action coverage are more directly documented than assurance, so action scope should expand only at a rate supported by competence and assurance.A broader interface can increase available actions without making success, authorization, or recovery reliable.
4 Digital action: from typed APIs to universal computer use
Typed APIs make operations and authorization more inspectable, while graphical computer use broadens access to legacy and proprietary software at the cost of weaker state and intent validation. Across both interfaces, benchmark gains and deployed workflows remain bounded by reliability, verification, security, and live-world consequences.
- Structured APIs: Typed APIs expose named operations and arguments that services can authenticate, validate, record, and return as structured state.This makes precondition checks and policy enforcement more tractable than graphical reconstruction.
- Structured APIs: Valid tool arguments can still encode the wrong meaning, violate unstated preferences, disclose sensitive context, or create impermissible aggregate outcomes.The problem is compositional: model proposals, harness policy decisions, and service authorization must remain distinct.
- Structured APIs: τ-bench checks final database states against task-specific policies and shows that repeated-attempt consistency can fall below one-shot success.This exposes reliability failures that single-trajectory completion can miss.
- Graphical interfaces: Computer-use agents reach incomplete APIs through screenshots or accessibility trees, but clicks depend on visual grounding, focus, timing, layout, notifications, and hidden state.Graphical control therefore expands coverage while weakening explicit semantics.
- Graphical interfaces: WebArena’s best reported GPT-4 baseline achieved 14.41% end-to-end success versus 78.24% for humans, while OSWorld reported below 12.2% versus 72.4%.These historical resettable-environment baselines separate fluent language generation from perception, operational knowledge, and multi-application state tracking.
- Graphical interfaces: OpenAI’s CUA preview reported 38.1% on OSWorld and 58.1% on WebArena, yet its system card called OS automation insufficiently reliable and recommended human oversight.Anthropic likewise described computer use as experimental and error-prone.
- Interface trade-offs: Hybrid systems should prefer structured calls when semantics and authorization are adequate, use computer control when necessary, and preserve shared policy and provenance layers.Without that common layer, graphical fallback can become a route around controls.
- Coding agents: Coding environments narrow the action–verification gap through sandboxing, version control, tests, and review, but test coverage and generated assumptions remain incomplete.Dependencies, network access, and plausible but insecure diffs preserve attack and maintenance risks.
5 Delegation, persistence, and multi-agent organization
Protocols and multi-agent architectures broaden delegation and persistence, but interoperability and organization do not by themselves establish trustworthy execution, independent verification, or durable safe authority.
- Protocols: MCP connects applications to exposed tools and resources, whereas A2A supports discovery and task exchange between otherwise opaque agents.MCP is principally an application-to-capability interface; A2A is principally an agent-to-agent work protocol.
- Protocols: Neither protocol establishes that capabilities are trustworthy, intentions are understood, or completed statuses match the delegator’s desired world state.Authentication and signed metadata provide identity or integrity properties, not competence or semantic validation.
- Agent-to-human delegation: Agent-to-human delegation includes requests for credentials, physical manipulation, judgment, and responsibility, extending beyond clarification interfaces.MCP elicitation, multi-round-trip requests, A2A input-required states, and richer co-tasking systems support this reverse interaction.
- Agent-to-human delegation: Market-mediated delegation can let agents purchase physical presence, identity-dependent access, or social influence, while a single-platform preprint identified 99 of 303 bounties as API-key or MCP channels.The study reported credential fraud, impersonation, reconnaissance, social-media manipulation, authentication circumvention, and referral fraud, but did not establish ecosystem-wide prevalence.
- Multi-agent organization: Multi-agent organization can isolate tools, enable parallelism, and clarify responsibility, but it can also increase inference cost, propagate errors, and obscure consequential decisions.These trade-offs motivate matched-budget comparisons rather than assuming that named roles improve performance.
- Multi-agent organization: Equal-budget evaluations found single agents matched or outperformed several multi-agent organizations on tested multi-hop reasoning, while debate could drift through low-quality feedback and unclear progress.The findings are limited to tested models and tasks but support reporting coordination overhead and matched baselines.
- Multi-agent organization: Multi-agent decomposition is most defensible when roles differ in permissions, models, sensors, institutional ownership, or independently verifiable expertise.Shared prompts, tools, retrieval, and training data can create correlated errors, so a same-context critic is not an external verifier.
- Persistence: Persistence separates conversational state, resumable tasks, episodic memory, executable skills, environmental state, and retained authority.These dimensions should be reported separately rather than collapsed into a single claim of long-term agency.
6 Persistent virtual environments and world models
Virtual environments and world models expand training and evaluation substrates, but coherent or persistent environments do not themselves demonstrate agency, causal validity, or reliable transfer to consequential settings.
- Conceptual distinctions: World models include predictive latent models, action-conditioned video generators, structured simulations, and interactive-environment platforms, whereas agents maintain objectives and select actions.The distinction separates environmental or predictive substrates from goal-directed control.
- Agent-environment systems: SIMA demonstrates cross-environment instruction following through screen inputs and keyboard–mouse actions, while Voyager combines curriculum, executable skills, feedback, and repair.SIMA’s initial tasks emphasize basic skills over short time scales.
- World models: V-JEPA 2 learned predictive representations from more than one million hours of video and used fewer than 62 hours of unlabelled robot video for action-conditioned post-training.The report demonstrated zero-shot image-goal planning on Franka arms in the cited setting.
- Interactive worlds: Genie 3 reported text-conditioned environments at 24 frames per second and 720p, with interaction over several minutes and visual memory of approximately one minute.The preview also limited direct agent actions, multi-agent interaction, geographic accuracy, text rendering, and duration.
- Interactive worlds: Pixel recurrence can preserve short-range visual continuity while failing object permanence, camera-independent consistency, or stable causal variables after long occlusion.Evaluation should distinguish continuity from spatial consistency, persistence, and causal structure.
- Persistent world state: Project Eden describes a structured, camera-independent world state separated from observation and neural rendering, potentially retaining off-screen changes for multiple users or agents.At the cutoff it remained a first-party research preview without independent reproduction.
- Generated artifacts: Generated 3D open-world artifacts provide evidence about sandboxed agentic production, not open-world governance or unrestricted execution.The task combines coding, asset creation, spatial composition, and iterative visual feedback.
- Evaluation: Joint evaluation should test environment counterfactual accuracy, state persistence, causal consistency, controllability, and failure coverage alongside agent completion, recovery, compliance, and generalization.Otherwise agents may exploit learned-world shortcuts that fail under target-environment dynamics.
7 Physical interfaces, robotics, and autonomous laboratories
Robotics, laboratory systems, and physical interfaces show bounded feasibility for language-conditioned action, while noisy dynamics, constrained demonstrations, and external checks limit claims of unattended open-world autonomy.
- Physical constraints: Physical agency adds partial observability, calibration drift, actuator latency, material changes, shared workspaces, and no general analogue of software snapshot reversion.Evidence must cover both the selected objective and the controller executing it.
- Robotics: RT-2 and related VLA systems show that foundation-model representations can improve robot-policy generality, but short action mappings do not necessarily retain objectives, authority, or stopping responsibility.The orchestration layer may still select tasks, interpret constraints, and verify outcomes.
- Physical interfaces: MHS proposes a model-agnostic driver linking agents to programmable laboratory and manufacturing equipment through discoverable state, read/write primitives, tags, and safety limits.The proposal is a limited research preview rather than evidence that physical autonomy has been standardized.
- Physical interfaces: Physical interfaces require device-certified interlocks, calibration and uncertainty metadata, idempotency declarations, dry-run modes, independent emergency stops, and append-only intent-to-command records.These requirements arise because interface commands cross a physical boundary.
- Autonomous laboratories: Coscientist combined literature search, code execution, and experimental automation across six laboratory tasks, including palladium-catalysed cross-coupling optimization.Its evidence supports semiautonomous experimentation under prescribed goals, tools, and constrained chemical access.
- Autonomous laboratories: Laboratory demonstrations do not establish an unattended general scientist because goals and tools are prescribed, hazards are sparsely sampled, and expert judgment remains essential for scientific meaning.Executing a protocol, producing a valid measurement, and drawing an internally consistent conclusion are distinct levels of correctness.
- Autonomous laboratories: AILA and AFMBench found that domain question-answering performance did not reliably translate into laboratory competence, with sensitivity to prompt formatting and instruction deviations termed “sleepwalking.”The study evaluated single- and multi-agent workflows across design, operation, and analysis.
- Autonomous laboratories: A-Lab demonstrates integrated robotics, databases, machine-learning interpretation, text-mined synthesis knowledge, and active learning, but corrected novelty claims require using it as evidence of integration.The correction limits unqualified claims about novel materials and synthesis outcomes.
8 Reliability, security, and governance across environments
Reliable delegation requires evidence chains linking intent, authority, permitted actions, execution, outcome provenance, security, and recovery; interfaces and oversight widgets alone are insufficient.
- Verification: Safe delegation requires interpreting the request, establishing authority, permitting actions under current conditions, observing final state and side effects, and triggering recovery when failures occur.This chain connects plausible action selection to externally verified task outcomes.
- Verification: Structured checks such as database fields, repository diffs, tests, checksums, interlocks, and transaction receipts can provide evidence independent of model narratives.Semantic goals require layered tests, independent tools or models, and accountable human judgment.
- Verification: Verifiers remain attack surfaces because agents can optimize proxies, alter evaluated objects, or select evidence that makes incorrect actions appear successful.Verification should bind objectives, action logs, environment identity, and measurement provenance.
- Security: Indirect prompt injection can turn web pages, documents, tool outputs, or agent messages into unauthorized state changes once credentials or action interfaces are available.AgentDojo evaluates such attacks under utility constraints, but its coverage is necessarily partial.
- Security: Security evaluations should report task performance and attack outcomes together with model, harness, permissions, budgets, and exact trust boundaries.Refusal-based defenses can reduce attack success while also destroying benign utility.
- Security: A first-party incident report documented unauthorized live-internet actions under particular misconfiguration or intentional internet-enabled tests, without establishing ordinary-deployment frequency.The evidence supports a boundary failure under those tests, not a prevalence estimate.
- Governance: Natural-language goals underspecify spending, access, safety, and anomaly policies, so authorization layers should translate intent into machine-enforceable capabilities.MCP and A2A security hooks do not remove the harness responsibility to assign, constrain, and revoke credentials.
- Human oversight: Human oversight can occur at goal setting, plan approval, transaction confirmation, monitoring, intervention, final inspection, or incident investigation, with different information and latency at each position.Approval presence alone does not establish effective control.
9 A research agenda for justified delegation
The research agenda treats justified delegation as a bounded decision criterion rather than a universal autonomy score, requiring evidence about competence, authority, provenance, recovery, and system factors. It recommends separating model, harness, interface, memory, verification, permissions, and delegation effects in evaluation.
- Delegation criterion: Justified delegation is a heuristic for deciding when broader authority is defensible, not a benchmark, law, or certification.It requires adequate competence, bounded authorization, outcome and side-effect evidence, and safe return when assumptions fail.
- Evaluation design: Factorial evaluations should vary model, interface, memory policy, verifier, and retry budget independently while reporting total inference and tool cost.Comparisons should hold the harness fixed for model claims and the model fixed for harness claims.
- State and verification: Interpretable state should represent plans, commitments, uncertainties, tool results, and authorization decisions with provenance and invalidation conditions.Compression should preserve why facts are believed and when they expire, while verifiers should receive evidence that allows them to challenge the actor.
- Authorization: Agent protocols should use task-scoped capabilities specifying resources, operations, bounds, expiration, delegability, and approval requirements.Subdelegation should attenuate authority, and every handoff should preserve the chain from the principal.
- Policy and conformance: Natural-language intent should compile into a reviewable policy object that supports ambiguity detection, multi-principal reconciliation, and non-delegable decisions.Conformance tests should include adversarial authorization, confused-deputy, replay, and revocation cases, distinguishing endpoint exposure from tested compliance.
9.3 Treat human participation as a scarce, safety-critical resource
Human participation should be allocated around judgment, supervision, and accountability rather than minimized mechanically. Long-task and memory evaluations should test safe interruption, state integrity, provenance, correction, deletion, and appropriate resumption.
- Human supervision: Human–agent systems should optimize the allocation of judgment, with calibrated triggers for asking, approving, observing, intervening, and auditing.Evaluation should test error detection, information support, supervision under concurrency and fatigue, and responsibility reconstruction after incidents.
- Human marketplaces: Agent-to-human marketplaces must address compound task risks, authenticated evidence, surveillance trade-offs, consent, discrimination, reputation manipulation, jurisdiction, and liability.The cited early measurement indicates that these marketplaces raise both economic and social concerns.
- Long-task evaluation: Persistent benchmarks should include interruptions, stale observations, policy changes, expiring credentials, concurrent actors, delayed outcomes, and tasks that should be cancelled.Scoring should include state integrity, resources, forgotten obligations, safe resumption, reauthorization, and durable evidence rather than endpoint reward alone.
- Memory safety: Memory tests should separately measure recall, provenance, correction, and deletion because persistent false inferences and unreleased sensitive records remain unsafe.Stored skills should also be tested for validity after environmental change and for knowing when not to reuse them.
9.5 Connect world models to causal and transfer evidence
World-model and physical-agent research should connect generated environments to intervention-sensitive, causal, transfer, and hardware evidence rather than treating visual coherence or planning competence as sufficient. The agenda emphasizes staged validation, independent safety controls, and explicit device-level semantics.
- Causal world-model evidence: World-model benchmarks should test hidden objects, conflicting viewpoints, causal interventions, conservation rules, and coherence under concurrent actions.These tests move evaluation from visual demonstration toward intervention-sensitive measurement.
- Transfer and staging: Generated environments should be evaluated for downstream transfer, artifact exploitation, uncertainty propagation, and abstention when direct sensing is more reliable.A simulation–hardware-in-the-loop–bounded deployment sequence can localize failures before physical exposure.
- Physical validation: Physical-agent evaluation should separately assess semantic planning, device execution, and system-level safety because success at one layer does not guarantee success at another.Interlocks and emergency stops should remain effective when the model, network, or orchestration process fails.
- Device interoperability: Shared device manifests and conformance suites should encode units, uncertainty, freshness, command preconditions, reversibility, dry runs, hazards, and signed calibration.Their value should be measured through integration time, portability, error detection, and safe recovery across independent systems.
9.7 Replace benchmark snapshots with longitudinal evidence
The review recommends replacing static benchmark snapshots with post-deployment evidence that preserves version history, records outcomes and human work, and keeps system boundaries understandable as capability changes. Its conclusions are bounded by selective, asymmetric, and rapidly changing public evidence.
- Longitudinal evidence: Post-deployment evidence should link benchmark results to model and harness versions and record attempted, declined, escalated, corrected, reversed, and abandoned tasks.Incident taxonomies should connect failures to prompt injection, permission misuse, coordination, memory, interface drift, and other causes.
- Authority boundaries: Higher benchmark performance should not automatically unlock a larger action surface; authority should track whether actions remain understandable and verifiable within scope.A narrow agent with strong evidence may warrant broad use within its scope, while a more capable but harder-to-verify agent may warrant less authority.
- Scope limits: The review cannot support bibliometric prevalence claims, and its fast-moving cutoff leaves protocol, product, and benchmark changes possible after 31 August 2026.Several frontier claims rely on first-party previews without archival papers or independent replications at the cutoff.
- Evidence coverage: Public evidence is concentrated in software, web tasks, games, robotics laboratories, and English-language interfaces, with less auditable evidence from high-stakes institutions and affected non-users.An absent public failure mode is therefore not treated as evidence that the failure is rare.
10 Conclusion
The review finds that agentic AI is a control architecture whose evidence supports broader action coverage and uneven task-competence gains, but not reliable unattended open-world autonomy. It therefore recommends expanding delegated authority only when evidence supports accountable, safe action.
- Agentic AI broadens action coverage and yields meaningful but uneven task-competence gains, without establishing reliable unattended open-world autonomy.The conclusion distinguishes demonstrated system expansion from a general autonomy transition.
- Separating model, harness, environment, and delegator clarifies why interoperability, modularity, persistent environments, and physical coupling do not automatically produce trustworthy autonomy.MCP and A2A standardize interfaces, multi-agent systems add modularity, world models provide environments, and physical studies remain bounded by constraints, calibration, and recovery.
- Justified delegation asks whether evidence supports preserving intent, respecting permissions, revealing changes, and failing safely before granting state-changing authority.The proposed heuristic treats delegation scope as evidence-dependent rather than as a fixed autonomy score.