PaperlayerView exact revision on arXiv

Source-linked AI summary

CarryOn: A Dual-Loop Harness for Personalized and Reliable Robot Assistance

Zixuan Wang, Yuqi Liu, Yuxin Chen, Sengqiao Yang, Jinhui Ye, Pengguang Chen, Changsheng Lu, Shu Liu, Bei Yu, Jiaya Jia

arXiv:2610.21453v1cs.ROcs.CV

TL;DR

Robot assistance requires task decisions grounded in user needs and current scenes, alongside reliable execution. CarryOn adds revisable context, shielding, and reflective recovery around a frozen VLA; studies report lower constraint violations and higher recovery completion with refined skills.

  • Problem

    Robot assistance must connect context-sensitive task decisions with reliable execution as user needs, scenes, and physical conditions change.

    Source: §1 · p. 2

  • Method

    CarryOn augments a frozen VLA with persistent user and scene memories, observation-driven updates, geometric action filtering, and reusable recovery skills refined through reflection.

    Source: §1 · p. 2, §3.3.1 · p. 5

  • Results

    Shielding reduces violations for the safety-finetuned VLA from 52.6% to 6.1%; iterative recovery reaches 76.10% cumulative completion in simulation.

    Source: §4.2 · p. 9, §4.3 · p. 11

  • Takeaways & Limitations

    Persistent context and physical feedback support assistance beyond isolated task execution in the reported real-robot and simulation studies.

    Source: §5 · p. 11

Abstract

from arXiv · show

While generalist vision-language-action (VLA) models have enabled diverse open-world manipulation, real-world robotic assistance requires two critical capabilities they currently lack: personalization and reliable execution. First, executing an instruction like "make me a sandwich" requires grounding subtask decomposition in persistent user preferences rather than relying on a single generic solution. Second, reliable deployment demands explainable safety assurance that proactively prevents hazards and gracefully recovers from occasional physical failures. To address these gaps, we present CarryOn, a dual-loop harness designed to augment a frozen VLA policy. The planning loop maintains persistent user-level and scene-level context that is continually grounded against new observations and revised through interaction, enabling personalized task selection and execution grounding across conversations and changing scenes. The execution loop actively supervises policy rollouts, shielding actions from predicted hazardous contact. When execution fails, the system invokes reusable control-based recovery skills that restore task progress and are refined through reflection on earlier failures. On a real robot and in simulation, CarryOn dispatches preference-consistent subtasks across users and scenes, reduces the violation rate of an already safety-finetuned VLA from 52.6% to 6.1%, and increases recovery success from 13% to 80% when refined skills are reused.

1 Request & scene

The robot uses remembered context to interpret a broad food-serving request, then combines a conditional plan with shielding and recovery during execution.

1 Request & scene

  • CarryOn retrieves context from memory after the user asks only, “Serve me food.”The planner then dispatches subtasks to a frozen VLA policy.Source: Figure 1 · p. 1
  • The plan opens the pink cover and places the carrot in a bowl only if it is underneath.Source: Figure 1 · p. 1
  • During execution, shielding steers the gripper and carrot around an obstacle, while recovery re-grasps the carrot so the task can resume.Source: Figure 1 · p. 1

1 Introduction

Everyday robot assistance requires connecting user- and scene-sensitive task decisions with reliable physical execution. CarryOn places persistent, revisable context and protection and recovery around a frozen VLA.

1 Introduction

  • Everyday assistance requires deciding what meets a user’s needs in the current environment and executing reliably as physical conditions change.Instruction-conditioned manipulation alone does not bridge this gap.Source: §1 · p. 2
  • Personalized plans must adapt to changing interactions and scenes, while hazardous contact and post-failure task interruption require distinct execution responses.User preferences and personal objects provide context, but stored context also needs revision as circumstances change.Source: §1 · p. 2
  • CarryOn connects adaptive task decisions and reliable execution through persistent, revisable context and visual feedback around a frozen VLA.The planning loop selects what to do; the execution loop protects and repairs the active subtask.Source: §1 · p. 2
  • Dialogue revises user context and observations update scene context, allowing later plans to rely on updated evidence.The execution loop combines proactive geometric shielding with reactive recovery skills refined through offline reflection.Source: §1 · p. 2
  • Real-world studies demonstrate preference-oriented planning across users and scenes; the reported violation rate falls from 52.6% to 6.1%, and recovery success rises from 13% to 80% with refined skills.Source: §1 · p. 2

2 Related Work

Related work addresses personalization, policy execution, recovery, and safety through varied memories, planners, and runtime mechanisms. CarryOn combines persistent personalization with action filtering and physical repair around a retained VLA.

Personalization through persistent context

  • Prior personalization work uses placement preferences, reference-image memory, and memory across user interactions; CarryOn links persistent preferences to revisable scene knowledge.User memory constrains task selection, while scene memory guides later actions.Source: §2 · p. 3

Harnesses for robot policy execution

  • Robot harnesses and hierarchical systems connect language requests to execution, while CarryOn retains the VLA for manipulation and couples personalization with action filtering and physical repair.Both loops share the active task, preserving the user-conditioned goal through recovery and using observed outcomes to govern task progression.Source: §2 · p. 3

Recovery and adaptation from failure

  • Prior work studies semantic failure feedback, learned recovery, runtime critics, and reusable skills; CarryOn confines refinement to control-based recovery while keeping the VLA and geometric shield fixed.Offline reflection revises supported parameters and strategies for later attempts.Source: §2 · p. 3

Geometric protection during manipulation

  • Safety filters include reachability and barrier-function methods; CarryOn predicts spatial response and filters against minimum-distance constraints for the gripper or carried object.The shield checks before hazardous contact and complements recovery after failure without online learning.Source: §2 · p. 3

3 Method

CarryOn surrounds a frozen VLA with two linked loops: planning uses persistent user and scene context, while execution filters risky actions and recovers after failures. Reflection revises recovery skills across attempts.

3 Method

  • CarryOn updates planning context through dialogue and observation while retaining execution experience in a versioned recovery-skill library.The harness runs a pretrained VLA, checks actions for collision risk, and invokes control-based recovery when manipulation fails.Source: §3 · p. 3

3.1 Overview of CarryOn

  • The planner combines user and scene memories with a request and skill descriptions to select subtasks and endpoints; a visual judge advances on success or invokes recovery on failure.Observation results return to the planner for plan and memory updates.Source: §3.1 · p. 3

3.2.1 Planning with persistent, revisable context

  • User memory supplies retained preferences and object references, while scene memory records inventories and locations to guide endpoint selection and observation requests.Planning combines both contexts with the current instruction and available skills.Source: §3.2.1 · p. 4

3.2.2 Updating context from dialogue and observation

  • Dialogue updates lasting user preferences while temporary instructions remain in the conversation; observations revealing targets or stale locations require explicit scene-memory updates.A validator returns plans missing a required update_scene operation for revision.Source: §3.2.2 · p. 4

3.3 The Execution Loop: Proactive Protection and Reactive Recovery

  • Execution divides support into fast proactive shielding and slower, event-triggered recovery, with offline reflection refining recovery versions while the VLA remains fixed.Even a safety-finetuned checkpoint violates constraints in 52.6% of 900 episodes, and frozen StarVLA succeeds in 13.0% after a 12 cm object displacement.Source: §3.3 · p. 5

3.3.1 Proactive shielding within an episode

  • The shield builds task-specific obstacle sets from RGB-D observations and target and destination descriptions, predicts gripper and carried-object motion, and filters each action chunk.If the first action is blocked, it returns a short retreat instead of an executable prefix.Source: §3.3.1 · p. 5
  • A short-horizon response model predicts end-effector poses from action chunks and executed-motion history; actions stop before the first predicted distance reaches its threshold.If no action can be released, the shield retreats and reassesses using fresh observations.Source: §3.3.1 · pp. 5–6

3.3.2 Reactive recovery with reusable skills

  • After a judged failure, a reusable control skill re-localizes and grasps a dropped object, checks holding during lift and transport, then resumes the interrupted instruction if checks pass.Failed retrieval or a failed transport check marks the recovery attempt unsuccessful.Source: §3.3.2 · p. 6

4 Experiments

The experiments evaluate planning, shielding, and recovery in real-robot assistance and controlled simulation. They test personalized decisions across changing users and scenes, physical safety, recovery from disturbances, and integrated assistance.

4.1 Personalization across Interactions and Scenes

  • Memory-guided dispatch reaches 100% (24/24) for sandwich preferences and 100% (16/16) for activity-dependent packing; packing without habits reaches 80.0% (16/20).After sandwich preference updates, dispatch accuracy is 83.3% (20/24); profile isolation reaches 88.9% (16/18).Source: §4.1 · pp. 7–8
  • For covered ingredients, remembered locations yield 100% (12/12) dispatch accuracy, versus 80.0% (8/10) for search without memory and 75.0% (9/12) with stale memory.Fresh observations can correct stale scene context used for planning.Source: §4.1 · p. 8
  • Across 88-request text and image modes, memory improves endpoint selection and reduces latency; after relocation, corrections succeed in 76/88 text and 77/88 image trials.Follow-up dispatch is correct in 83/88 text and 85/88 image trials.Source: §4.1 · pp. 8–9, Table 2 · p. 8

4.2 Safety during Execution

  • Shielding reduces safety-finetuned π0.5 violations from 52.6% to 6.1%, while violation-free task success remains similar at 25.3% without shielding and 25.6% with it.Across other backbones, shielding also reduces violations: OpenVLA-OFT from 42.0% under VLSA to 17.2%, and VLAct from 44.8% under VLSA to 26.1%.Source: Table 3 · p. 9, §4.2 · p. 9

4.3 Recovery and Improvement from Failure Experience

  • Iterative reflection and retries raise perturbation-task completion from 29.0% at R0 to 70.5% at R1 and 80.0% at R2.R1 resolves 83/142 previous failures, and R2 resolves another 19/59.Source: Table 4 · p. 10, §4.3 · p. 10
  • On recorded natural failure states, recovery raises completion from 32.08% to 50.00%, then iterative refinement reaches 76.10%.In real-robot trials, completion rises from 3/20 with the raw policy to 9/20 with base recovery and 18/20 with a revised skill.Source: Table 1 · p. 7, Table 5 · p. 10, §4.3 · p. 11

4.4 Integrated Assistance on the Real Robot

  • In integrated food-serving runs, shielding avoids obstacle collisions and recovery restores a dropped carrot, while newly revealed meat leads the system to refuse serving under the vegetarian preference.The observed meat also replaces the stale carrot entry in scene memory.Source: Figure 6 · p. 10, §4.4 · p. 11

5 Conclusion

CarryOn pairs revisable user and scene memories with proactive shielding and reactive recovery around a frozen VLA. Real-robot and simulation studies evaluate these capabilities, while future work targets longer-term adaptation and balancing protection with task completion.

5 Conclusion

  • CarryOn combines persistent, revisable context for planning with proactive geometric shielding and reactive recovery around a frozen VLA.Offline reflection refines recovery skills while the underlying policy remains frozen.Source: §5 · p. 11
  • Real-robot and simulation studies evaluate preference-conditioned assistance, scene-memory revision, runtime protection, and recovery after disturbance.The conclusion characterizes these studies as showing that persistent context and physical feedback extend assistance beyond isolated task execution.Source: §5 · p. 11
  • Future work targets longer-term adaptation, transfer of recovery experience across tasks, and balancing protective intervention with task completion.Source: §5 · p. 11

A System Implementation and Runtime Coordination

The runtime coordinates planner outputs, endpoint subtasks, the VLA, the Judge, shielding, and recovery. Persistent memories are stored as Markdown blocks and rendered into planner context on each request.

A System Implementation and Runtime Coordination

  • A request builds context from the request, both memories, endpoints, and a skill library, then repairs invalid plans before dispatching subtasks.Subtasks without dependencies start immediately; dependent subtasks wait for earlier ones to succeed.Source: A System Implementation and Runtime… · p. 15
  • During physical subtasks, the harness shields VLA commands, resumes the original subtask after recovery, and replans when execution reveals new scene contents.The Judge determines success or failure; newly revealed contents must be recorded with update_scene in the next plan.Source: A System Implementation and Runtime… · p. 15

B.1 Storage and Planner Injection

  • Both memories use Markdown section blocks with key-value lines and are rendered into every planner context, with user facts grouped by user.The appendix also describes memory update paths and logged examples of revisions changing subsequent plans.Source: §B.1 · p. 15, B Persistent User and Scene Memory · p. 15

B.2 User Memory from Dialogue

User memory records durable preferences from dialogue and replaces superseded values; scene memory is revised when observations reveal changed contents. Reference images can also be stored as entity attributes for later requests.

B.2 User Memory from Dialogue

  • User memory can capture explicit preferences during a turn or use a background summarizer to store durable facts after a plan finishes.The durability filter excludes one-off wishes and volatile information, and a new value replaces the old one.Source: §B.2 · pp. 15–16
  • The same sandwich request yields different plans after a stored diet changes from vegetarian to non-vegetarian.The examples show a vegetarian sandwich plan changing to ham and lettuce after the profile update.Source: §B.2 · p. 16

B.3 Scene Memory from Observation

  • When an observation contradicts a remembered location, the validator requires an update_scene record before the next plan proceeds.In the example, the Rubik’s cube is found on the dining table rather than the garage bench, and a follow-up request dispatches directly to its revised location.Source: §B.3 · pp. 16–17

B.4 Remembered Reference Images

  • A reference image can be copied into memory as an entity attribute and made available to the planner through the user-memory block.The fact can be created by a summarizer processing an attached image or by an explicit remember-image command.Source: §B.4 · p. 17

C Proactive Shielding

The shield grounds scene geometry from instruction-named targets and destinations, then checks predicted gripper and carried-object contact against geometric rules. Blocked actions trigger a brief retreat from the nearest obstacle.

C Proactive Shielding

  • Text-prompted segmentation grounds target and destination names in image masks, with VLM point prompting available when text grounding finds no mask.Depth pixels are back-projected into the world frame, robot and mount pixels are removed, and the target’s source support is estimated from a local patch beneath it.Source: C Proactive Shielding · p. 17
  • The contact checks exempt permitted target, source-support, and destination contacts, while testing predicted gripper and carried-object geometry against remaining obstacles.A holding-state latch uses measured gripper width and the executed gripper command to determine when carried-object geometry is included.Source: C Proactive Shielding · p. 17
  • When the first candidate action is blocked, the controller retreats for three control steps using away-from-obstacle and upward components.Figure 7 illustrates action filtering and retreat; Table 7 lists the LIBERO-Safety shield parameters.Source: C Proactive Shielding · p. 17, Table 7 · p. 17

D Recovery Skills and Reflection

The appendix details geometric action filtering and retreat as part of CarryOn’s recovery and reflection mechanisms. The figure shows how predicted gripper geometry is checked against obstacles and how the robot retreats when its first action is blocked.

D Recovery Skills and Reflection

  • The appendix describes additional details of the recovery and reflection mechanism.Source: D Recovery Skills and Reflection · p. 17
  • Actions are released only before predicted gripper–obstacle distance first crosses the dashed safety threshold.The figure distinguishes predicted gripper geometry from current robot geometry.Source: Figure 7 · p. 18
  • If the first action is blocked, the robot retreats and rechecks before continuing.The retreat direction combines a component away from the nearest current end-effector obstacle point with an upward bias.Source: Figure 7 · p. 18

D.1 Base Recovery Skill

The base recovery skill seeks to restore the physical state needed for the interrupted subtask, then requests VLA resumption. It checks the grasp after lifting and again after transport to a destination stand-off pose.

D.1 Base Recovery Skill

  • The skill re-localizes the target, generates grasp candidates, and checks measured gripper width after lifting before transport.A passing check allows movement to a stand-off pose near the destination, followed by a second holding check.Source: §D.1 · p. 18
  • If both holding checks pass, the VLA resumes the original instruction from the destination stand-off pose.Source: §D.1 · p. 18
  • If retrieval or a required motion fails, the skill attempts a fallback pose and requests VLA resumption, while the Judge continues assessing completion.The fallback is the available candidate’s pre-grasp pose, or the recorded pre-failure pose when no candidate exists; resumption is requested even if fallback motion fails.Source: §D.1 · p. 18

D.2 Applying a Stored Recovery Version

Stored recovery versions retain the base procedure and any strategy or parameter revisions, which later attempts reuse and reflection can extend. Strategy changes target recovery operations or conditions, while parameter changes adjust numerical settings.

D.2 Applying a Stored Recovery Version

  • Each recovery version combines the base skill with stored strategy and parameter revisions, and later reflection builds on that version using remaining failures.Strategy revisions change operation sequence or conditions; parameter revisions adjust numerical settings such as grasp height.Source: §D.2 · p. 18
  • In simulation, VLM-assisted re-grounding is enabled only for hard-to-ground objects that lead to recovery failure.The VLM selects among numbered candidate masks using the target name, reference image, and fresh scene image.Source: §D.2 · p. 18
  • On the real robot, strategy revisions add retries with renewed perception so another grasp attempt can use an updated target location.Source: §D.2 · p. 18
  • After a Judge failure verdict, Algorithm 2 suspends the VLA, grounds the target, tries grasp candidates, checks holding, and requests resumption with the original instruction and remaining budget.If recovery fails, it attempts the candidate’s pre-grasp pose or recorded pre-failure pose before opening the gripper.Source: §D.2 · p. 19

D.3 A Complete Revision Chain

A BBQ sauce example traces reflection-and-retry rounds in the LIBERO perturbation study. The reflector uses failed attempts to revise strategy and parameter memory, and cumulative completion rises across the reported rounds.

D.3 A Complete Revision Chain

  • M1 records a visual grounding error and raises the contact height to 110.0 mm as a shallower-approach test.The selected failures had six grasp attempts at 66.0 mm, with descents jamming at 109.0–113.2 mm.Source: §D.3 · p. 19
  • R1 reaches the planned descent height instead of jamming, but the object often slips during the lift.Source: §D.3 · p. 19
  • M2 skips a strategy change because the crops match the reference bottle and updates the parameter to 105.0 mm as a grasp-stability hypothesis.Source: §D.3 · p. 19
  • 17 of 20 episodes are complete cumulatively by R2, after R0 completes 3/20 and R1 completes 3/17.The reflector reviews up to three failed recoveries between rounds and reruns failed episodes using the revised memory version.Source: §D.3 · p. 19

E Experimental Configuration

The experimental configuration passages describe the models and language-model module inputs and outputs, alongside prompts for personalized planning, action selection, scene grounding, and recovery. They also specify how agents assess subtask completion from visual evidence and keep recovery reflections grounded in observations rather than assumptions.

E Experimental Configuration

F.2 Agent Prompts

  • The planner must attribute durable facts to the active user, keep one-off wishes temporary, and avoid dispatching physical actions unless the user requests them.It must not apply another user’s stored facts when planning.Source: §F.2 · p. 21
  • For skill-library endpoints, the planner must copy a listed instruction verbatim and report uncertainty rather than force a visual object match.The prompt also requires the user-facing reply and dispatched subtasks to describe the same action.Source: §F.2 · pp. 21–22
  • Scene memory indicates where to observe, not what is currently present; the planner must inspect uncertain locations before manipulating targets found there.When observations contradict or add to memory, the prompt requires updating scene memory in the same plan as the next physical step.Source: §F.2 · pp. 22–24
  • Subtask judgments compare the baseline with the current scene, exclude work assigned to completed subtasks, and prioritize the planner’s explicit subtask objective over the raw instruction.When the apparent goal state was already present at baseline and no new change occurred, the prompt says to return running, then failed after the stuck timeout.Source: §F.2 · pp. 24–25
  • Placement success requires a newly achieved state with the correct object visibly inside the specified destination, not merely nearby or already there at baseline.The prompt requires identifying the placed object and checking that it matches the instruction; an uncertain identity or a different object rules out success.Source: §F.2 · p. 25
  • For observe subtasks, the agent reports newly revealed visual contents independently of its status verdict and flags the planner’s scene record for updating.If nothing was newly revealed, the report is null and scene_update_required is false.Source: §F.2 · p. 25
  • Progress is judged by task-relevant changes to the target object, and no progress beyond the timeout requires failure and correction unless the target is already placed.Arm motion alone does not count as progress; the already-placed guard prevents a failed verdict from undoing a completed placement.Source: §F.2 · pp. 25–26
  • Recovery reflection treats selected masks as unverified and whole-task failure as insufficient to diagnose identity, grasp, or grasp-height causes.The batch review uses actual attempts and outcomes, distinguishes planned poses from measured observations, and permits NO_UPDATE when evidence is insufficient or grasp height cannot reasonably address the observations.Source: §F.2 · p. 27

Related papers

Loading 2610.21453v1…