Source-linked AI summary
Know When to Escalate: Critic-Guided Adaptive Execution for Cloud-Local Embodied Agents
Wei Zhang, Lirong Xiang, Charles Fleming
TL;DR
Embodied agents need fast closed-loop decisions, but local models may be error-prone and cloud models add substantial latency. CARE uses a critic to transfer control persistently between local and cloud executors while sharing handoff memory; across benchmarks, it improves task success over collaboration baselines while reducing latency.
Problem
Cloud–local embodied agents must balance responsive local execution against stronger but slower cloud reasoning, while fixed-role hierarchies and stateless routers do not adapt executor choice across a trajectory.
Source: §I · pp. 1–2Method
CARE uses a critic to escalate from local to cloud execution when local execution deteriorates, sustain cloud control through recovery, and share task context across handoffs.
Source: §I · p. 2Results
CARE improves task success by 6.3–18.3 percentage points over the strongest evaluated collaboration baseline and reduces median time-between-action latency by 28.8–82.6%.
Source: §I · p. 2Takeaways & Limitations
Across Habitat, EmbodiedBench, and VLABench, executor selection should adapt with the embodied trajectory rather than remain fixed throughout a task.
Source: §VI · p. 7CARE is evaluated at the decision layer, and its behavior under sensing noise, network variability, and hardware execution failures remains unverified.
Source: §V · p. 7
Abstract
from arXiv · showhide
Recent advances in foundation-model agents have enabled increasingly capable embodied systems for long-horizon planning and interaction. However, embodied agents require fast, closed-loop decisions as they continuously perceive and act on a changing environment. Deploying strong cloud agents leaves the platform idle for seconds between consecutive actions, while lightweight local models respond quickly but are more vulnerable to errors. Existing hierarchical architectures typically assign models fixed roles, whereas conventional model routers make stateless per-request decisions. We present CARE, a critic-guided adaptive execution framework that dynamically determines when to escalate control from a local executor to a stronger cloud model. A lightweight critic model monitors execution reliability and triggers escalation when local execution becomes unreliable; once escalated, control persists until the agent recovers, avoiding step-wise switching. To support such control transfer, a shared handoff memory maintains execution context and task-level progress across executors, enabling the model to intervene without restarting from scratch. Across diverse embodied benchmarks, CARE improves task success by 6.3-18.3 percentage points over the strongest evaluated collaboration baseline. CARE further approaches or exceeds Cloud-only success while reducing median time-between-action latency by 28.8-82.6%.
I. INTRODUCTION
Embodied agents must balance strong reasoning with responsive closed-loop interaction, but executor suitability can change as task difficulty evolves. CARE uses critic-guided, persistent control transfer and shared handoff memory; across three benchmarks, it improves task success over the strongest evaluated collaboration baseline while reducing latency.
I. INTRODUCTION
- Executor suitability can change during a task because a lightweight executor may struggle after ambiguous observations, failed actions, or deviations from the expected state.Overlooking action feedback can lead to repeated actions, lost progress, and compounded errors.Source: §I · p. 1
- Cloud–local systems trade responsive local execution against stronger cloud reasoning at seconds-per-decision latency, while fixed-role hierarchies and per-query routers do not model changing suitability along a trajectory.Replanning may revise actions without reconsidering which model should execute the subsequent trajectory.Source: §I · pp. 1–2
- The critic selectively escalates to cloud when local execution becomes unreliable, keeps cloud control through recovery, then returns control to local execution when reliability recovers.Shared handoff memory preserves observations, completed progress, and failure context across handoffs.Source: §I · p. 2
- CARE formulates cloud–local collaboration as stateful adaptive execution, dynamically escalating control and safely resuming local control.Its mechanism combines persistent cloud intervention with shared execution memory to maintain continuity across handoffs.Source: §I · p. 2
- CARE improves task success by 6.3–18.3 percentage points over the strongest evaluated collaboration baseline while reducing median time-between-action latency by 28.8–82.6%.The evaluation spans three long-horizon embodied benchmarks with diverse observation modalities and execution interfaces.Source: §I · p. 2
II. RELATED WORK
Prior embodied-agent work improves planning and organizes execution hierarchically, but typically fixes executor assignments in advance. CARE addresses the runtime question of which executor should hold authority as a plan unfolds.
A. Long-horizon embodied agents
- Planning and hierarchical execution improve instructions or pair foundation models with faster executors, but fix executor assignments in advance.Prior approaches include program or skill generation, intermediate representations, slow–fast reasoning, and semantic guidance.Source: A. Long-horizon embodied agents · p. 2
B. Embodied execution monitoring
- Unlike cascades and routers that decide for a self-contained input, embodied execution requires stateful authority changes because actions alter the state for later decisions.Failure-recovery methods diagnose failures or revise plans, whereas CARE’s critic decides which executor holds authority and when it changes.Source: B. Embodied execution monitoring · p. 2
III. METHOD
CARE treats cloud–local executor selection as a stateful decision coupled across an embodied trajectory. Its critic-guided transfer mechanism changes execution authority while shared context and memory support continuity.
III. METHOD
- The critic transfers control from local to cloud when needed and keeps it there until local execution is supported again, while shared handoff memory lets the incoming executor continue from current progress.This transfer changes which executor proposes actions without changing the perception–action loop.Source: §III · p. 2, Figure 2 · p. 3
A. Problem Formulation
- At each step, local and cloud executors share the task instruction, current observation, and memory accumulated along the trajectory; the active executor produces an action whose outcome informs later control.The two executors share an abstraction level and produce navigation or manipulation actions, while cloud reasoning generally has higher inference latency.Source: A. Problem Formulation · p. 3
- Executor selection is coupled across the trajectory because each action changes the subsequent observation and task state.The objective is to approach Cloud-only task success while reducing closed-loop response time, particularly the delay between consecutive actions.Source: A. Problem Formulation · p. 3
B. Critic-Guided Control Transfer
CARE uses a critic to assess local-execution reliability and guide control transfers between local and cloud executors. Asymmetric thresholds keep cloud control active through recovery until local reliability supports a return.
B. Critic-Guided Control Transfer
- The critic evaluates shared context and a local proposal, assigning a local or cloud control label based on whether local execution remains reliable.The relative likelihoods of the candidate labels are normalized into a reliability-aware confidence score.Source: B. Critic-Guided Control Transfer · p. 3
- CARE uses asymmetric escalation and return thresholds, so cloud control persists until local reliability recovers above the return threshold.The thresholds satisfy θesc < θret, introducing hysteresis to avoid frequent handoffs during multi-step recovery.Source: B. Critic-Guided Control Transfer · p. 3
- At each interaction step, the controller evaluates when scheduled, updates its control state, commits the selected executor’s action, and records the resulting transition in memory.When cloud control is active, the cloud action is awaited; otherwise the current executor acts.Source: B. Critic-Guided Control Transfer · p. 4
- Speculative cloud proposals are discarded if local control continues, but reused if escalation occurs, reducing control-transfer latency without changing the critic score or switching policy.The critic runs every k steps, with the interval set per benchmark.Source: B. Critic-Guided Control Transfer · p. 4
C. Shared Handoff Memory
CARE keeps execution context in shared handoff memory so an incoming executor can continue from the existing task state. The memory combines recent transitions with a persistent episode-level summary and is shared across executors and the critic.
C. Shared Handoff Memory
- Shared handoff memory addresses the continuity gap at control transfer by preserving completed subgoals, failed actions, and the path to the current state.Without this context, an incoming executor may repeat resolved work or re-attempt a known failure.Source: C. Shared Handoff Memory · p. 4
- Memory combines a bounded window of recent observation–action–outcome transitions with an episode summary of verified progress and persistent failures.The window preserves recent context; the summary retains important information beyond that window.Source: C. Shared Handoff Memory · p. 4
- After each action, the transition is added without another model call, while only observation- or feedback-verified state is promoted to the episode summary.Unexecuted proposals, reasoning traces, and privileged simulator state are excluded.Source: C. Shared Handoff Memory · p. 4
- The same memory is exposed to the local executor, cloud executor, and critic at handoff, allowing the incoming executor to resume without reconstructing prior progress.Actions during cloud control update the shared memory before control returns to the local executor.Source: C. Shared Handoff Memory · p. 4
A. Experimental Setup
The evaluation compares CARE and baselines across three embodied benchmarks with different action interfaces. It reports reliability and latency measures, and the table caption characterizes CARE’s reliability–latency trade-off among collaboration methods.
A. Experimental Setup
- Claude Sonnet 4.6 serves as cloud planner and executor, while one Qwen3.6-35B-A3B checkpoint serves as both local executor and critic.The models use temperature 0; escalation and return thresholds are selected per benchmark from a sweep.Source: A. Experimental Setup · p. 4
- CARE has the strongest reliability–latency trade-off among collaboration methods and is the only method to exceed 70% success on all three benchmarks.The table’s italic row reports CARE relative to Cloud-only in points for SR and CRR and percent for E2E and TBA.Source: Table I · p. 5
- The evaluation covers 120 Habitat rearrangement instances, 80 EmbodiedBench instances, and 80 VLABench instances across mobile manipulation, household interaction, and tabletop manipulation.The benchmarks use grounded PDDL-style actions, discrete action IDs, and structured macro skills, respectively.Source: A. Experimental Setup · p. 5
- Baselines include Local-only and Cloud-only executors, ReAct variants, and collaboration methods representing fixed hierarchy, model routing, and feedback-triggered replanning.All methods use the same instances and memory representation; transferred requests receive the memory intact.Source: A. Experimental Setup · pp. 4–5
- The study reports task success rate, end-to-end episode latency, time between actions, trajectory length, and cloud request ratio.The cloud request ratio counts cloud executor requests, including speculative requests, but excludes critic calls.Source: A. Experimental Setup · p. 5
B. Overall Performance
CARE improves task success over Local-only, approaches or exceeds Cloud-only success, and reduces response latency. It achieves this with selective cloud use, adapting escalation to how much of the trajectory local execution can handle.
B. Overall Performance
- CARE improves success by 16.3–35.0 percentage points over Local-only, exceeds Cloud-only success on Habitat and VLABench, and remains within 5.0 points on EmbodiedBench.Median time-between-action latency falls 28.8–82.6% relative to Cloud-only, with lower median end-to-end latency across all three benchmarks.Source: B. Overall Performance · p. 5
- CARE uses cloud models for only 24.4–34.4% of executor decisions while remaining on the empirical non-dominated frontier across all three benchmarks.Hi Robot, RouteLLM, and AdaPlanner use comparable or fewer cloud requests but achieve lower task success.Source: B. Overall Performance · p. 5, Figure 3 · p. 6
- On EmbodiedBench, where Local-only achieves 40.0% success, CARE escalates more often and recovers substantially more success while remaining faster than Cloud-only.CARE does not enforce a fixed cloud-use target; its responsiveness depends on how much of the trajectory can be handled locally.Source: B. Overall Performance · p. 5
C. Does CARE Know When to Escalate?
CARE is evaluated against random and periodic escalation schedules at matched cloud usage, including on Habitat instances where Local-only fails. Its advantage grows when the cloud budget is sufficient, while episode pathways show both selective escalation and sustained control transfers.
C. Does CARE Know When to Escalate?
- Over 30–50% cloud request ratio, CARE’s success rises from 88.4% to 93.0%, while state-independent schedules remain between 74.4% and 79.0%.At about 20% cloud request ratio, all three methods achieve similar success.Source: C. Does CARE Know When to Escalate? · p. 6
- CARE itself saturates at 93.0% around 50% CRR, showing diminishing returns from more aggressive thresholds as cloud request ratio increases.Source: C. Does CARE Know When to Escalate? · p. 6
- 43% of episodes finish without escalation, and 93% of those succeed; returned episodes also succeed at 93%.30% remain under cloud control until termination, and 81% of takeovers span multiple decisions.Source: C. Does CARE Know When to Escalate? · p. 6
- CARE’s episode-level control transfers are state-dependent rather than independent per-step model choices.The evaluation compares CARE with Random and Periodic schedules under matched cloud usage on Habitat instances where Local-only fails.Source: C. Does CARE Know When to Escalate? · p. 6
D. System Analysis and Ablation Studies
CARE lowers execution latency relative to Cloud-only, with modest critic and speculative-transfer overhead. Handoff memory preserves progress at control transfers, and the framework retains effectiveness across executor and critic substitutions.
D. System Analysis and Ablation Studies
- Across P5–P95 episodes, CARE reduces Habitat execution latency by 1.8–7.1× relative to Cloud-only.Critic evaluation adds 0.15–2.45 seconds, and speculative transfer overlaps cloud generation with critic evaluation; removing speculation costs 0.05 seconds at P50 and 2.32 seconds at P95.Source: Figure 5 · p. 7, D. System Analysis and Ablation Studies · p. 7
- Removing handoff memory reduces success from 100% to 25% and more than doubles trajectory length and cloud usage at matched Habitat execution states.Episodic memory alone yields 90% success, observation memory alone 75%, and their combination restores full success while reducing execution cost.Source: D. System Analysis and Ablation Studies · p. 7, Table II · p. 7
- Across executor and critic substitutions, CARE retains 83.3–88.3% success, although the default pairing achieves the best success, trajectory length, and end-to-end latency.Executor substitutions increase trajectory length and latency more than critic substitutions.Source: D. System Analysis and Ablation Studies · p. 7, Table III · p. 7
- An 8B critic retains 86.7% success with substantially lower degradation than executor replacements, supporting reduced critic capacity when deployment cost is prioritized.Source: D. System Analysis and Ablation Studies · p. 7
V. LIMITATIONS AND FUTURE WORK
CARE’s reported benchmark results combine higher task success and lower median time-between-action latency, while matched-budget comparisons attribute gains to selecting useful intervention states. Its decision-layer evaluation leaves physical-platform behavior under sensing noise, network variability, and hardware failures unverified.
V. LIMITATIONS AND FUTURE WORK
- CARE operates above the low-level controller, and its behavior under sensing noise, network variability, and hardware execution failures remains unverified.Reported latencies measure decision-layer responsiveness, not end-to-end physical robot response.Source: §V · p. 7
- Physical-platform evaluation is a next step, since real network conditions may affect cloud-assistance latency and availability.Source: §V · p. 7
- Future work should characterize how model scale, task difficulty, and action abstraction shape the boundary between local and cloud competence.Source: §V · p. 7
VI. CONCLUSION
- Task success improves by 6.3–18.3 percentage points over the strongest collaboration baselines, while median time-between-action latency falls by 28.8–82.6% relative to Cloud-only.These results span Habitat, EmbodiedBench, and VLABench.Source: §VI · p. 7
- Matched-budget comparisons indicate gains come from selecting useful intervention states rather than simply increasing cloud usage.Source: §VI · p. 7
- The conclusion is that executor selection should adapt with the embodied trajectory rather than remain fixed throughout a task.Source: §VI · p. 7