PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboPocket: Improve Robot Policies Instantly with Your Phone

Junjie Fang, Wendi Chen, Han Xue, Fangyuan Zhou, Tian Le, Yi Wang, Yuting Zhang, Jun Lv, Chuan Wen, Cewu Lu

arXiv:2603.05504v2cs.ROcs.AIcs.LG

TL;DR

RoboPocket addresses the inefficiency of open-loop handheld data collection and the hardware cost of interactive policy correction. It uses smartphone-based AR Visual Foresight with asynchronous online finetuning to guide targeted corrections without a physical robot. The system reports up to 2× gains in data or sample efficiency across centralized and distributed settings.

  • Problem

    Open-loop handheld collection does not reveal policy weaknesses, while interactive correction requires costly physical robot execution, limiting scalable coverage of critical states.

  • Method

    RoboPocket uses a smartphone as a computational co-pilot that visualizes predicted trajectories through AR and continuously finetunes policies with incoming corrective data.

  • Results

    Up to 2× data-efficiency gains were reported in centralized and distributed environments compared with open-loop baselines.

  • Takeaways & Limitations

    RoboPocket supports scalable policy adaptation by enabling non-experts to identify weaknesses and provide targeted corrections without physical robot supervision.

  • Takeaways & Limitations

    The reported efficiency comparison is framed against open-loop baselines in centralized and distributed environments.

Abstract

from arXiv · show

Scaling imitation learning is fundamentally constrained by the efficiency of data collection. While handheld interfaces have emerged as a scalable solution for in-the-wild data acquisition, they predominantly operate in an open-loop manner: operators blindly collect demonstrations without knowing the underlying policy's weaknesses, leading to inefficient coverage of critical state distributions. Conversely, interactive methods like DAgger effectively address covariate shift but rely on physical robot execution, which is costly and difficult to scale. To reconcile this trade-off, we introduce RoboPocket, a portable system that enables Robot-Free Instant Policy Iteration using single consumer smartphones. Its core innovation is a Remote Inference framework that visualizes the policy's predicted trajectory via Augmented Reality (AR) Visual Foresight. This immersive feedback allows collectors to proactively identify potential failures and focus data collection on the policy's weak regions without requiring a physical robot. Furthermore, we implement an asynchronous Online Finetuning pipeline that continuously updates the policy with incoming data, effectively closing the learning loop in minutes. Extensive experiments demonstrate that RoboPocket adheres to data scaling laws and doubles the data efficiency compared to offline scaling strategies, overcoming their long-standing efficiency bottleneck. Moreover, our instant iteration loop also boosts sample efficiency by up to 2$\times$ in distributed environments a small number of interactive corrections per person. Project page and videos: https://robo-pocket.github.io.

I. INTRODUCTION

RoboPocket targets the cognitive and hardware bottlenecks that limit scalable robot-learning data collection. It turns handheld recording into computationally guided collection and enables robot-free policy iteration through AR feedback and rapid online updates.

  • Handheld interfaces remove physical-robot dependence but largely remain open-loop, leaving operators without guidance about policy weaknesses or critical state coverage.
  • RoboPocket exports expert intuition into a smartphone-based co-pilot that unifies data collection, training, and testing roles.
  • Real-time visual feedback and validation let users verify SLAM stability and kinematic feasibility during collection, improving demonstration quality.
  • AR Visual Foresight projects policy trajectories onto the user’s view, enabling proactive failure identification and targeted corrective data collection without a physical robot.
  • Up to 2× improvement in data efficiency was demonstrated across diverse manipulation tasks, while distributed users achieved up to 2× sample-efficiency gains with as few as 12 interactive corrections per user.

B. Interactive Policy Learning and Correction

RoboPocket addresses the deployment paradox of interactive learning by decoupling policy correction from physical robots. Its handheld hardware is designed to expose policy intent while preserving visual and physical consistency with the target robot.

  • Interactive corrections address covariate shift but remain tethered to costly physical hardware, limiting safety and scalability.
  • AR Visual Foresight reveals the policy’s planned trajectory before execution, changing correction from reactive safeguarding to robot-free policy iteration.
  • 1) Real-Time Interaction Interface:: The iPhone Pro serves as an Edge-Compute Hub for real-time visual-inertial odometry, kinematic solving, and AR rendering, enabling remote policy inference.
  • Hardware isomorphism targets the Robotiq 2F-85 and reproduces underactuated gripper dynamics to reduce the embodiment gap.

3) Sensory Completeness:

RoboPocket augments the smartphone with sensors and verification mechanisms needed for reliable manipulation data. The system expands visual context, measures gripper width, and provides real-time physical and logical validation.

  • 3) Sensory Completeness:: Sensory Completeness addresses sensor information that standard smartphones cannot fully provide for robot learning.
  • A fisheye lens expands the smartphone’s visual context to capture both the surrounding environment and gripper-object interaction.
  • A magnetic encoder connected through an ESP32-based interface measures gripper width at 0.088° angular resolution and 30Hz.
  • Active Data Verification combines SLAM monitoring, on-device inverse-kinematics checks, and visual or haptic warnings to mark invalid frames during collection.
  • AR Trajectory Replay lets users inspect recorded motion immediately and verify SLAM fidelity and logical success such as stable grasping.

2) Multi-Device Spatiotemporal Synchronization:

RoboPocket frames policy improvement as targeted sampling of states where the current policy is deficient. Its server-client loop streams data, finetunes continuously, and returns updated predictions without physical robot execution.

  • A. Problem Formulation: The system seeks to collect the specific state distributions required by the robot rather than relying only on static demonstrations.
  • Compounding errors create OOD states, while physical execution makes on-policy correction costly and safety-sensitive.
  • Batch-oriented pipelines sever the immediate feedback needed for operators to understand current policy capabilities.
  • AR Visual Foresight and real-time parameter updates guide operators toward critical state-space regions where the policy is deficient.
  • The server-client architecture uploads trajectories, finetunes with weighted sampling, and synchronizes updated weights to the inference server.

1) Remote Inference:

RoboPocket streams observations to a remote inference server and projects predicted trajectories into the physical world through AR. Proactive intervention and asynchronous online finetuning create a rapid robot-free feedback loop for collecting targeted corrections.

  • Remote Inference:: Remote inference streams observations from the smartphone to a GPU server, achieving round-trip latency under 150ms over standard Wi-Fi.Persistent model states support fluid user interaction.
  • Remote Inference:: AR Visual Foresight projects the policy’s predicted trajectory into the real world so non-experts can interpret the policy’s intent.The system renders the predicted path as virtual cues aligned with the physical scene.
  • Proactive Intervention:: Proactive Intervention lets users trigger new inference queries and collect data specifically in weak regions of the policy’s state space.This changes intervention from reactive failure response to robot-free active learning.
  • Online Policy Iteration:: Asynchronous online finetuning continuously updates the policy by mixing 50% offline demonstrations with 50% newly collected online data.The weighted sampling strategy is designed to prevent catastrophic forgetting.
  • Online Policy Iteration:: The feedback loop operates in minutes: users see a failure, collect corrective data, and then observe the updated policy through AR.The near-instant update cycle is reported to improve collection efficiency and user engagement.

V. EXPERIMENTS

The experiments evaluate RoboPocket’s system fidelity, learning efficiency, and distributed scalability. Results show accurate tracking, faster collection with higher-quality trajectories, adherence to data scaling laws, and targeted corrections that address covariate shift.

  • Experiments: RoboPocket is evaluated across system fidelity, policy-learning data efficiency, and distributed in-the-wild scalability.The real-world evaluations use a Flexiv Rizon 4 arm with a Robotiq 2F-85 gripper and Diffusion Policy models.
  • System Capability Verification: 2.8mm average cumulative 3D Euclidean error and 0.4° rotation error outperform standard inertial-monocular SLAM results of 6.1mm and 3.5°.The single-device setup is compared against UMI’s reported SLAM performance.
  • System Capability Verification: 3m51s for data acquisition and 1m37s for transfer replace UMI’s 8m34s collection, 1m24s transfer, and 9m12s SLAM processing.RoboPocket uses online SLAM and avoids separate mapping collection and offline SLAM computation.
  • System Capability Verification: Zero position jumps and physically plausible acceleration limits contrast with UMI’s position jumps in 2 of 9 successful trials and acceleration spikes above 15m/s2.The comparison concerns trajectory quality after tracking and filtering.
  • System Capability Verification: A 1,600-demonstration Mouse Arrangement dataset across 64 environment-object pairs produces OOD success-rate scaling that mirrors the established power law.This result is presented as validation that RoboPocket supports large-scale robot-learning data collection.
  • Beyond Data Scaling Laws with Instant Iteration: Targeted failure-recovery data addresses Behavior Cloning’s covariate-shift plateau rather than relying only on additional expert demonstrations.The evaluation tests whether Robot-Free Instant Policy Iteration can break the diminishing returns of traditional data scaling.

1) Task Settings & Baselines:

RoboPocket is tested on four manipulation tasks with distinct long-horizon, rotation, deformable-object, and bimanual challenges, alongside fixed-data and correction-based baselines. Across these settings, instant policy iteration targets task-specific failure regions and can improve performance or stability with fewer corrections.

  • Task Settings & Baselines: The four manipulation tasks are Block Sorting, Seasoning Pouring, Towel Folding, and Snack Bagging.They represent long-horizon sequencing, aggressive wrist rotation, deformable-object perception, and bimanual precision challenges, respectively.
  • Task Settings & Baselines: The IL Only baseline trains on fixed pre-collected datasets of N = 100, 200, 300 demonstrations.This is compared with manual and RoboPocket-based policy-iteration baselines.
  • Task Settings & Baselines: IL + Manual PI uses an expert to analyze robot failure videos and collect 25 or 50 targeted correction demonstrations.The baseline requires physical robot execution for failure analysis.
  • Task Settings & Baselines: IL + Offline PI collects 25 or 50 AR-guided corrections using a fixed pretrained model.Unlike instant iteration, the policy is not updated during correction collection.
  • Results and Analysis: In Block Sorting, all policy-iteration strategies outperform pure data scaling, while RoboPocket matches IL + Manual PI without a physical robot.The task exposes wrong-order execution as a recurring pure-IL failure mode.
  • Results and Analysis: In Seasoning Pouring, IL + Instant PI reaches comparable results with 300 IL using fewer data, while its variance is 0.08 versus 0.30 for IL + Offline PI.Pure IL often fails after the first pour because large rotations induce OOD repositioning states.
  • Results and Analysis: In Towel Folding, IL + Manual PI drops from 0.73 to 0.50, whereas IL + Instant PI achieves stable gains at 0.88.The paper attributes the contrast to the difficulty of deformable-object perception and the risk of inaccurate corrective data.
  • Results and Analysis: In Snack Bagging, RoboPocket surpasses the 300 IL baseline at 0.56 versus 0.51 by targeting ambiguity regions.The task includes left-arm grasp failures and right-wrist camera occlusion.

D. Scalable and Generalizable Policy Iteration

RoboPocket supports scalable policy iteration across diverse environments by combining distributed data collection with robot-free interactive corrections. Its results show adherence to data scaling laws and up to 2× data-efficiency gains over open-loop or pure imitation-learning baselines.

  • Robot-Free Iteration: A smartphone-based closed-loop system enables non-expert users to identify policy weaknesses and provide targeted corrections without physical robot supervision.AR Visual Foresight supports proactive correction before deployment.
  • Performance Comparison: Up to 2× higher efficiency than pure imitation learning is achieved across four tasks, with performance comparable to expert manual intervention without a physical robot.The comparison includes block sorting, towel folding, and other manipulation tasks.
  • Distributed Generalization: Up to 2× policy-performance improvement is obtained across four environments with only 12 interactive corrections per user.Four collectors simultaneously perform Robot-Free Instant Policy Iteration in distinct rooms.
  • Limitations: The handheld rig is limited by its parallel-jaw design for high-dexterity in-hand manipulation and may induce fatigue during prolonged collection.The authors suggest lighter interfaces such as first-person AR glasses as future work.

APPENDIX

The appendix evaluates whether non-experts can use RoboPocket’s feedback and iteration tools effectively. Survey responses favor real-time feedback, Virtual Foresight, and Instant Policy Iteration, while feature-based analysis indicates non-expert state coverage comparable to experienced experimenters.

  • Study Protocol: The study compared five offline-iteration trajectories with five Instant Policy Iteration trajectories after a 10-minute warm-up.Participants had varying prior experience with imitation learning and UMI-based collection.
  • User Feedback: All users found real-time constraints and feedback beneficial for data collection, with more than half rating them “Very Helpful”.The survey evaluated the system’s core feedback features.
  • User Feedback: 7 out of 10 participants rated Virtual Foresight “Very Helpful” for discovering model failure cases.All users agreed that Virtual Foresight assisted failure discovery.
  • User Feedback: 8 out of 10 users considered Instant Policy Iteration highly beneficial for realizing model improvement.Participants also identified real-time uploading and online finetuning as making collection more effective.
  • State Coverage: PCA of DINOv2 state features shows non-expert data achieving state coverage comparable to experienced experimenters.Offline Policy Iteration data was used because each participant’s online iterations were limited.

B. Data Collection Details

RoboPocket’s data-scaling evaluation uses diverse environments, object pairs, demonstrations, and normalized task scores. The appendix defines scoring criteria for Mouse Arrangement, Block Sorting, Seasoning Pouring, Towel Folding, and Snack Bagging.

  • Scaling Protocol: Data diversity is more important for zero-shot generalization than merely increasing demonstrations per scene.Established scaling-law work relates policy performance to environment and object diversity through a power law.
  • Scaling Protocol: The Mouse Arrangement dataset spans 32 environments and 47 object pairs, with 25 demonstrations collected per environment-object pair.The environments include indoor and outdoor settings with diverse lighting and conditions.
  • Task Scoring: Mouse Arrangement uses a six-point score: three points for successful pickup and three for successful placement.Partial-credit criteria distinguish failed approaches, drops, unstable placement, and complete success.
  • Task Scoring: Block Sorting awards up to 9 points for sequentially placing red, green, and blue blocks into their corresponding boxes.Incorrect execution orders are scored using the Longest Increasing Subsequence match to the ground truth.
  • Task Scoring: Seasoning Pouring awards up to 12 points for ordered pickup, pouring, and stable replacement of three jars.Scores range from failed approaches to successful pouring and stable placement, with order handled using LIS matching.
  • Task Scoring: Towel Folding and bimanual Snack Bagging each score sequential task completion using three-point increments per fold or snack.Towel Folding has two folds and a six-point maximum; Snack Bagging has four snacks and a 12-point maximum.

D. Hardware Setup

RoboPocket’s hardware setup combines an isomorphic robot-gripper arrangement, smartphone streaming, portable power, and workstation-based data and inference services. The evaluation environments include varied lighting and textures.

  • Robot Setup: The robot setup uses a Flexiv Rizon 4 arm with a Robotiq 2F-85 adaptive gripper connected through a 90-degree adapter.The gripper uses the same TPU soft fingers as the RoboPocket collector to preserve physical consistency.
  • Training Environments: Scaling-law training environments cover a wide range of lighting conditions and textures.The environments are used to verify data scaling laws.
  • Robot Setup: Data Scaling Laws experiments use a DH-ROBOTICS AG-160-95 gripper and a corresponding isomorphic handheld collector.The hardware is matched between the robot and handheld collection systems.
  • Compute and Power: An iPhone streams camera feeds to a workstation that serves data, trains models, and supports the collection system.The workstation uses an Intel Core i9-12900K CPU, NVIDIA RTX 3090 GPU, and EcoFlow DELTA 3 MAX power station.
  • Compute and Power: Robot-Free Instant Policy Iteration uses a separate inference workstation with an Intel Core i9-13900K CPU and NVIDIA RTX 4090 GPU.The inference server supports remote policy prediction for the handheld workflow.

E. Policy Training Details

Policy training uses a CNN-based Diffusion Policy with a single-step observation horizon, while Robot-Free Instant Policy Iteration uses separately summarized training parameters.

  • The policy is trained with a CNN-based Diffusion Policy built on the Diffusion Policy and UMI code bases.
  • Tobs = 1 excludes explicit and implicit velocity information from observations collected during Robot-Free Instant Policy Iteration.This design avoids encoding instability in human motion speed, although it discards historical context.
  • Training hyperparameters and Robot-Free Instant Policy Iteration hyperparameters are summarized separately in Tables I and II.
Loading 2603.05504v2…