PaperlayerView exact revision on arXiv

Source-linked AI summary

TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Plans

William Shen, Nishanth Kumar, Sahit Chintalapudi, Ryan Lindeborg, Jie Wang, Christopher Watson, Edward Hu, Jing Cao, Dinesh Jayaraman, Leslie Pack Kaelbling, Tomás Lozano-Pérez

arXiv:2603.09971v2cs.RO

TL;DR

Reliable language-conditioned manipulation remains an open problem despite foundation models’ growing perception and reasoning capabilities. TiPToP combines pretrained models with GPU-accelerated TAMP, and its evaluations report competitive results on real hardware and first-place MolmoSpaces performance among non-ID methods.

  • Problem

    Foundation models provide open-vocabulary perception and reasoning, but converting these capabilities into reliable robot manipulation remains an open problem.

    Source: §I · p. 2

  • Method

    TiPToP composes pretrained foundation models with test-time TAMP in modular perception, planning, and execution modules, without robot training data.

    Source: §I · p. 2

  • Results

    Across 165 trials in 28 simulation and real-hardware scenes, TiPToP matches or outperforms π0.5-DROID, and ranks first overall on MolmoSpaces pick and pick-and-place tasks among methods not trained on in-distribution data.

    Source: §X · p. 9

  • Takeaways & Limitations

    The modular architecture supports component-level failure analysis, which identifies grasping as a dominant bottleneck in the evaluated trials.

    Source: §X · p. 9

    Open-loop execution is the most impactful limitation; grasping failures account for over half of observed failures, and many could be recovered by retrying the grasp.

    Source: §IX · p. 8

Abstract

from arXiv · show

We present TiPToP, a modular manipulation system that integrates pretrained foundation models with a GPU-accelerated Task and Motion Planner to solve tasks directly from RGB images and natural language. TiPToP composes perception, planning, and execution modules and requires no robot training data. It can be deployed on a standard DROID setup in under an hour and adapted to new embodiments with minimal effort. We evaluate TiPToP against $\pi_{0.5}\text{-DROID}$, a state-of-the-art VLA fine-tuned on 350 hours of demonstrations, across two real-world DROID setups (one operated by an external team) and simulation, where TiPToP attains a higher average success rate and faster average completion time. We also evaluate on the MolmoSpaces benchmark, where TiPToP ranks first overall on pick and pick-and-place tasks among methods not trained on in-distribution data. We further show that TiPToP's modularity enables us to trace failures to specific components, revealing where to target improvements. We release TiPToP open-source to serve as a reproducible baseline and to enable further research on modular manipulation systems. Project website and code: https://tiptop-robot.github.io

I. INTRODUCTION

TiPToP addresses the gap between foundation models’ perception and language capabilities and reliable manipulation by pairing pretrained models with test-time planning. Its modular, compositional, zero-shot design is evaluated across hardware and benchmarks, while related work distinguishes it from data-trained end-to-end policies and prior modular planners.

I. INTRODUCTION

  • TiPToP pairs pretrained foundation models with test-time TAMP and separates perception, planning, and execution for component-level improvement and failure tracing.New skills require symbolic predicates, an operator, and a parameterized controller; deployment on supported robots requires camera calibration and takes under an hour.Source: §I · p. 2
  • Across real hardware, TiPToP matches or exceeds π0.5-DROID’s success rate at comparable or shorter completion times; on MolmoSpaces, it outperforms non-ID VLA and WAM methods on 5 of 9 tasks.The independent MolmoSpaces evaluation covers all 9 pick and pick-and-place tasks across 9,000 episodes.Source: §I · p. 2

II. RELATED WORK

  • Compared with end-to-end VLA and WAM approaches requiring robot-data fine-tuning, TiPToP composes pretrained models without embodiment-specific training and allows components to be debugged independently.Prior TAMP approaches commonly require detailed object geometries; TiPToP uses GPU-parallelized cuTAMP and packages deployment for camera-, gripper-, URDF-, and trajectory-controller-equipped robots.Source: §II · p. 3

III. PROBLEM FORMULATION AND SYSTEM OVERVIEW

  • Language-conditioned manipulation maps images, joint configuration, and an instruction to actions; TiPToP instantiates this interface as a planner rather than a single trained network.The shared interface supports direct comparison between end-to-end learned policies and frozen foundation models combined with test-time planning.Source: §III · p. 3

A. TiPToP

TiPToP plans from a stereo image pair and language instruction, then executes a full trajectory open-loop. Its perception module combines 3D geometry and grasp proposals with semantic object identities and goals.

A. TiPToP

  • TiPToP observes once, plans a complete timed trajectory, and executes it open-loop; its modules construct an object-centric scene, search for a feasible plan, and track that plan.The planned trajectory includes joint configurations, velocities, and binary gripper actions.Source: A. TiPToP · p. 3

IV. PERCEPTION MODULE

  • The perception module takes the initial observation, joint configuration, and instruction, then merges scene geometry and grasps with object identities and symbolic goal propositions.Its 3D Vision and Semantic branches run in parallel before their outputs are combined.Source: §IV · p. 4

A. 3D Vision Branch

  • FoundationStereo produces cleaner depth maps than the ZED camera’s proprietary stereo matching, particularly on transparent, specular, and textureless surfaces.The depth map is unprojected into a world-frame point cloud using camera intrinsics and the camera-to-end-effector transform with forward kinematics.Source: A. 3D Vision Branch · p. 4
  • M2T2 predicts ranked 6-DoF grasps using the full scene point cloud, allowing the planner to select candidates according to task requirements and collision constraints.When objects lack predicted grasps, the planning module falls back to a heuristic 4-DoF sampler.Source: A. 3D Vision Branch · p. 4

B. Semantic Branch

The Semantic Branch uses a vision-language model to identify objects and translate language into symbolic goals, then segmentation links those identities to scene geometry. The combined representation supplies per-object meshes and assigned grasps to planning.

B. Semantic Branch

  • Gemini Robotics-ER jointly predicts object labels, 2D bounding boxes, and a symbolic goal expressed as predicates over detected objects.For “serve peanut butter crackers on each tray,” it identifies the two relevant packages and produces a goal placing one on each tray.Source: B. Semantic Branch · p. 4
  • SAM-2 generates pixel-level masks for detected objects, which are combined with the scene point cloud to extract per-object geometry and assign grasps.Source: B. Semantic Branch · p. 4

C. Combining Outputs

  • The default convex-hull mode builds watertight object meshes from segmented points, projecting them to each object’s lowest observed point before computing the hull.This per-object projection accommodates objects resting on other objects; the alternative RecGen mode produces full meshes and 6-DoF poses from RGB, depth, camera intrinsics, and masks.Source: C. Combining Outputs · p. 5
  • Convex hulls are cheap and usually provide conservative collision geometry, while shape completion gives higher-fidelity meshes for complex objects at greater compute cost.RecGen takes approximately 10 seconds per object on an RTX 3090; the authors find convex hulls sufficient for most evaluated tasks despite over-approximating concave objects.Source: C. Combining Outputs · p. 5

V. PLANNING MODULE

TiPToP’s planner searches symbolic action skeletons and optimizes continuous manipulation parameters before generating trajectories. Execution tracks plans open-loop, while the experiments examine task performance, total time, and failure modes.

V. PLANNING MODULE

  • The planner enumerates symbolic action skeletons, samples continuous parameters, and optimizes candidates for collision avoidance, stable placement, and kinematic feasibility.Longer skeletons can move an obstructing object before attempting a blocked grasp.Source: §V · p. 5
  • For each satisfying particle, cuRobo solves for collision-free, time-parameterized trajectories that form the manipulation plan.The plan specifies joint positions, joint velocities, and gripper commands.Source: §V · p. 5
  • cuTAMP adapts to imperfect scene reconstructions and oriented placement surfaces through collision-checking changes, more planning attempts, and extended placement costs.These modifications support reliable real-world deployment.Source: §V · p. 5

VI. EXECUTION MODULE

  • The custom impedance controller tracks plans open-loop; this suffices for static scenes with accurate tracking but fails when objects move or grasps slip.It keeps end-effector tracking error within 5 mm at high speeds, which the authors say is tight enough for their evaluation tasks.Source: §VI · p. 6

VII. EXPERIMENTS

  • The experiments assess performance across tabletop tasks without embodiment-specific training data, total planning-plus-execution time, and module-level failure modes.They include a controlled comparison against π0.5-DROID and evaluation on MolmoSpaces.Source: §VII · p. 6

A. Controlled Comparison with π0.5-DROID

TiPToP is compared with π0.5-DROID on real-robot and simulated tasks, then evaluated on MolmoSpaces and analyzed for failures. The results show strong performance on complex tasks and identify grasping and scene-modeling as key failure sources.

A. Controlled Comparison with π0.5-DROID

  • Across 165 trials, TiPToP beat π0.5-DROID overall in success rate (98/165 vs. 55/165) and task progress (74.6% vs. 52.4%).The 28-scene comparison covered simulation and two DROID setups, spanning simple pick-and-place through multi-step tasks.Source: Table I · p. 6, A. Controlled Comparison with π0.5-DROID · p. 6
  • TiPToP was faster than π0.5-DROID on five of six timed scenes and matched it on the sixth, often finishing single-step tasks in roughly half the time.The reported completion-time comparison averages time-to-success over successful trials; TiPToP’s planning time is included.Source: A. Controlled Comparison with π0.5-DROID · p. 6, Table II · p. 7

C. Failure Analysis

  • Of 55 traced failures, grasping accounted for 31, scene-completion errors for 13, VLM errors for 6, and cuTAMP failures for 5.Grasp failures arose from unsuccessful predicted or fallback grasps; mesh approximations caused scene errors, VLMs misdetected objects, and cuTAMP sometimes found no feasible plan within its time budget.Source: C. Failure Analysis · p. 7, Figure 5 · p. 7

D. Discussion

  • TiPToP’s strongest comparative performance grows with task complexity, with explicit VLM goal grounding helping distractor and semantic tasks and cuTAMP decomposing multi-step tasks.The discussion attributes these benefits to isolating task-relevant objects and resolving referring expressions, and to planning feasible collision-free action sequences.Source: D. Discussion · p. 7
  • TiPToP commits to a time-optimal trajectory while π0.5-DROID acts step by step; its time advantage narrows on multi-step tasks where execution dominates.The discussion reports higher success for TiPToP on multi-step tasks despite that narrower time advantage.Source: D. Discussion · p. 7
  • Open-loop execution prevents retries after missed or slipped grasps, while convex-hull meshes poorly represent concave shapes and provide coarse collision checking.Because failures map to specific components, the authors identify them as independently addressable.Source: D. Discussion · pp. 7–8

B. MolmoSpaces

  • TiPToP achieved 46.1% oracle success rate versus 41.3% for the next-best non-ID method, ranking first among non-ID methods on five of nine tasks.The evaluation covered nine pick and pick-and-place tasks and omitted unsupported open and close tasks; oracle success counts success anywhere within the episode horizon.Source: B. MolmoSpaces · p. 7, C. Failure Analysis · p. 7, Table III · p. 8

D. Discussion

  • The wiping demonstration extends TiPToP beyond pick-and-place to the instruction “erase the whiteboard and put everything into the bowl”.The authors describe this skill extension as straightforward.Source: Figure 6 · p. 8

VIII. EXTENDING TO NEW EMBODIMENTS AND SKILLS

TiPToP extends to new embodiments and skills through localized changes, while its discussion identifies perception and execution limitations and proposes integrating learned policies. The conclusion reports competitive performance and identifies grasping as a dominant bottleneck.

VIII. EXTENDING TO NEW EMBODIMENTS AND SKILLS

  • TiPToP was deployed on UR5e and Trossen WidowX AI arms; adapting the UR5e required robot and collision models, configuration, and camera and controller interfaces.The UR5e adaptation was completed within a few hours.Source: §VIII · p. 8
  • A whiteboard-wiping skill was added through three localized changes without modifying perception or execution, and the extension took under a day.The additions were predicates and a goal-grounding prompt, a planner operator, and a wiping controller.Source: §VIII · p. 8

IX. LIMITATIONS AND FUTURE DIRECTIONS

  • Open-loop execution is the most impactful limitation: grasping failures account for over half of observed failures, and many could be recovered by retrying.The proposed direct route to closing the loop is to rerun perception and planning after each pick-and-place step.Source: §IX · p. 8
  • Single-viewpoint perception requires every task-relevant object to be at least partly visible from one wrist-camera pose, while imperfect meshes can cause missed or unnecessary collisions.The paper proposes active camera movement or additional static cameras to reduce occlusion and improve shape estimates.Source: §IX · p. 8
  • TiPToP and π0.5-DROID have complementary failure profiles: TiPToP handles geometric reasoning, sequencing, and grounding, while π0.5-DROID offers closed-loop reactivity.The paper proposes integrating end-to-end policies as reactive skills or assigning them separate task substeps; planner integration requires specifying abstract preconditions and effects.Source: §IX · pp. 8–9
  • Belief-space cuTAMP is proposed to reason about uncertainty in object poses, grasp outcomes, and partially observable state.It could also support information-gathering actions, such as moving the camera to observe occluded regions before planning.Source: §IX · p. 9

X. CONCLUSION

  • Across 165 trials in 28 scenes, TiPToP matches or outperforms π0.5-DROID, especially on semantic grounding, distractor rejection, and multi-step sequencing.It ranks first on MolmoSpaces pick and pick-and-place tasks among methods not trained on in-distribution data; failure analysis identifies grasping as a dominant bottleneck.Source: §X · p. 9
  • The authors argue that modular systems built from off-the-shelf foundation models and planning algorithms can be strong manipulators, with components independently upgradeable.They suggest that combining planning with learned reactive skills could join structured reasoning with closed-loop visuomotor robustness.Source: §X · p. 9

APPENDIX

The appendix describes cuTAMP extensions for grasping, placement, collision handling, and task-search efficiency, alongside controller implementations for different robots.

A. cuTAMP Extensions

  • M2T2 grasp predictions initialize grasp particles, with collision filtering rejecting grasps that would collide with the target object.Placement surfaces also support oriented bounding boxes, with penalties near surface edges and samplers accounting for object extents.Source: A. cuTAMP Extensions · p. 14
  • Motion planning robustness was improved by increasing attempts over constraint-satisfying particles and optionally relaxing collision thresholds when path planning fails.The extensions address differences between cuTAMP's sphere collision representation and cuRobo's oriented-box checks.Source: A. cuTAMP Extensions · p. 14
  • Movable-object collision costs are evaluated only after an object's action activates, allowing initially colliding objects to avoid penalties until moved.The appendix also adds a Franka FR3 model and caches intermediate task-planner tree-search results to reduce redundant computation.Source: A. cuTAMP Extensions · p. 14

B. Controller Implementation Details

  • The DROID Franka controller tracks planned waypoints by computing joint torques from position and velocity errors, Coriolis and gravity compensation, and an inertia term.The inertia term uses desired acceleration estimated by filtered numerical differentiation.Source: B. Controller Implementation Details · p. 14
  • The FR3 and Panda controller gains were tuned for tracking, but high-speed execution can still produce up to 5 mm of end-effector position error.The appendix says the controller implementation would be open-sourced upon acceptance.Source: B. Controller Implementation Details · p. 14
  • For the UR5e, servoJ streams motion through a high-gain phase followed by settling to mitigate mechanical oscillation.The appendix also reports that SAM-2 generates eight pixel-level masks from Fig. 3c bounding boxes.Source: B. Controller Implementation Details · p. 14, Figure 7 · p. 14

C. Additional Experiment Details

The appendix details the evaluation scene descriptions, trial protocols, completion-time measurement, and a separate procedure for tracing failures across TiPToP trials.

C. Additional Experiment Details

  • Table IV lists each evaluation scene with its language instruction and task-progress metric.Source: C. Additional Experiment Details · p. 14
  • TiPToP plans once with a 30–60 second planning timeout, whereas π0.5-DROID runs continuously until manually terminated.π0.5-DROID stopping limits varied by simulation and real-world evaluator; the authors describe them as generous relative to typical completion times.Source: C. Additional Experiment Details · p. 14
  • Average completion time is reported only over successful trials; real-world timing used a remote iPad timer, automatically stopped for TiPToP and manually for π0.5-DROID.TiPToP used a cuRobo time-dilation factor of 0.6 in simulation and real-world experiments.Source: C. Additional Experiment Details · p. 14
  • The 173 failure-analysis trials covered 52 simple pick-and-place tasks and 121 distractor tasks spanning varied language goals and one- to five-object manipulation.Success was judged manually, and failures were traced using logging and visualization.Source: C. Additional Experiment Details · pp. 14–15

D. Deployment on UR5e

The UR5e deployment required robot, collision, and planner configuration plus camera and controller interfaces; the appendix also describes stereo-depth and control choices.

D. Deployment on UR5e

  • UR5e adaptation required a robot URDF, collision spheres, cuRobo configuration, cuTAMP loading changes, and interfaces for the camera and robot controller.With an existing robot controller, the changes took approximately 2–3 hours.Source: D. Deployment on UR5e · p. 15
  • RealSense infrared stereo produced noisier depth estimates than the DROID ZED Mini's RGB stereo, particularly on transparent, specular, and reflective objects.The appendix attributes this to active IR stereo's difficulty with surfaces that do not reliably reflect its projected pattern.Source: D. Deployment on UR5e · p. 15
  • The UR5e controller uses servoJ to interpolate sparse waypoints into a 125 Hz command stream, with high motion gain followed by lower-gain settling to mitigate oscillation.Source: D. Deployment on UR5e · p. 15

E. Whiteboard Wiping Skill

TiPToP adds whiteboard wiping through three localized changes without modifying its perception or execution infrastructure.

E. Whiteboard Wiping Skill

  • Three localized changes add whiteboard wiping without modifying perception or execution infrastructure.The changes span semantic goal grounding, task planning, and a wiping controller.Source: E. Whiteboard Wiping Skill · p. 15
  • The semantic branch adds IsEraser and IsCleaned predicates and extends goal grounding to express cleaning goals such as IsCleaned(whiteboard).Source: E. Whiteboard Wiping Skill · p. 15
  • A Wipe operator requires an eraser in hand and a surface target, marks that surface clean, and lets the planner sequence pick then wipe.Source: E. Whiteboard Wiping Skill · p. 15
  • The controller uses a VLM bounding-box query to localize the region, projects its corners into world coordinates, then covers it with back-and-forth strokes.The strokes use IK-based Cartesian control.Source: E. Whiteboard Wiping Skill · p. 15

F. MolmoSpaces Integration

TiPToP's MolmoSpaces integration supports pick and Pick & Place-NextTo predicates, using a wrist camera and depth data under stated view assumptions.

F. MolmoSpaces Integration

  • MolmoSpaces Pick and Pick & Place-NextTo tasks use Holding and Near predicates; support for Open/Close variants is left for future work.Source: F. MolmoSpaces Integration · p. 15
  • TiPToP uses only the wrist camera and assumes objects of interest are visible from its starting view, which it positions with an initial motion to a constant pose.Source: F. MolmoSpaces Integration · p. 15
  • TiPToP relies on depth data and camera pose; MolmoSpaces supplies ground-truth simulation depth, while real experiments derive depth from stereo RGB or infrared inputs.Source: F. MolmoSpaces Integration · p. 15
  • The evaluation table organizes scenes by identifier, language prompt, and task-progress metric; task categories include Simple, Distractor, Semantic, and Multi-step.Progress metrics can vary by evaluator and task, and some penalize distractor manipulation while others do not.Source: Table IV · p. 16, F. MolmoSpaces Integration · p. 16

G. VLM Prompting Details

The VLM prompt detects and labels relevant objects, then translates natural-language instructions into formal predicate goals.

G. VLM Prompting Details

  • Object detection returns image bounding boxes while excluding the robot, gripper, and table surface.Source: G. VLM Prompting Details · p. 18
  • The prompt limits detection to 25 relevant objects, names repeated objects by distinguishing traits or numbers, and requires unique, task-relevant labels.Source: G. VLM Prompting Details · p. 18
  • Task translation converts instructions into conjunctions of formal predicates, using detected object labels and referencing only objects detected in the image.Source: G. VLM Prompting Details · pp. 18–19
  • The trash example grounds goals only for the empty chips packet and soda can, excluding usable items such as a full bottle and candy bars.Source: G. VLM Prompting Details · pp. 18–19
Loading 2603.09971v2…