Source-linked AI summary
Leveraging Inter-object Affordances for Efficient Planning in Contact-rich Tasks
Pouya P. Niaz, Justus Piater, Alejandro Agostini
TL;DR
The paper addresses the limits of TAMP methods that simplify objects and defer physical reasoning to motion planning. It enriches U-TAMP with symbolic inter-object affordances, using VLM-based affordance detection, and reports higher success rates with faster planning in simulated kitchen organization tasks.
Problem
Traditional TAMP overlooks physical properties of realistic objects, limiting the constraints it can represent for contact-rich manipulation.
Method
The method enriches U-TAMP with PDDL-based object-centric abstractions of inter-object affordances for grasping, supporting, lifting, and sliding.
Results
UTAMP-2-GT achieves 100% success, UTAMP-2-VLM achieves 90%, and the other approaches remain below 50%.
Takeaways & Limitations
Rich symbolic affordance abstractions support faster planning and higher success rates for realistic contact-rich kitchen organization tasks.
Abstract
from arXiv · showhide
Traditional task-and-motion planning (TAMP) approaches primarily focus on defining sequences of actions along with the necessary geometric and kinematic constraints to execute long-horizon tasks. However, their applicability in real-world settings is limited, as they typically assume simplified object models that overlook key physical properties critical for the successful execution of contact-rich tasks. Moreover, they often use sub-symbolic reasoning during motion planning, which drastically increases planning time and decreases overall success rates. We propose a method that leverages a TAMP approach, defining object-centric abstractions of execution constraints, called Unified TAMP (U-TAMP), to execute robotic tasks involving interactions among objects with heterogeneous shapes, sizes, and materials. Using a Vision-Language Model (VLM), we generate abstractions of inter-object affordances for characterizing physical interaction constraints between objects in contact-rich tasks, such as grasp and support constraints. These constraints are used to enrich the U-TAMP planning domain to deal with objects with variable physical properties. We perform experiments in simulated kitchen table organization scenarios and compare our results with those of the original U-TAMP, as well as a state-of-the-art VLM-based planner that leverages common sense knowledge of objects' affordances for plan generation. Our approach achieves significantly higher planning success rates and improves planning times by one to two orders of magnitude compared to other methods.
I. INTRODUCTION
Traditional TAMP often relies on simplified object models that omit physical properties needed for realistic contact-rich manipulation. This work extends U-TAMP with symbolic inter-object affordances and evaluates the approach in simulated kitchenware organization.
- Simplified object models omit physical properties needed to represent manipulation and object-object interaction constraints.
- Inter-object affordances characterize what parts of objects can do with parts of other objects, including cutting, containment, and placement.
- The proposed method extends U-TAMP constraints to account for physical properties and affordances during task planning.
- Experiments use a kitchenware organization scenario with objects of varied shapes, sizes, and materials.
- The planning domain uses PDDL-based symbolic abstractions for grasping, supporting, and sliding to generate feasible long-horizon plans.
II. RELATED WORK
Related work addresses TAMP through simplified object models, language-model reasoning, and object-centric affordance or motion-constraint representations. These lines of work motivate incorporating affordances into symbolic planning.
- Traditional TAMP commonly defines constraints over simplified cuboid-like objects, limiting applicability to real settings.
- LLMs and VLMs have been used to extract affordances, generate goals, plan tasks, and determine constraints in TAMP.
- Several approaches estimate affordances or constraints before planning or within feedback loops involving execution and monitoring.
- Object-centric predicates can encode action affordances such as which object can be cut with a particular tool.
- U-TAMP builds on object-centric abstractions to represent motion constraints in action preconditions and effects.
A. Task and Motion Planning
TAMP first searches over symbolic actions that transform an initial state into a goal state, then uses motion planning to realize those actions. U-TAMP moves rich motion constraints into task-level search through object-centric abstractions.
- A. Task and Motion Planning: Task planning searches for symbolic actions that transform an initial symbolic state into a goal configuration.
- A. Task and Motion Planning: PDDL represents object states and relations with predicates, while planning operators specify action preconditions and effects.
- B. U-TAMP: U-TAMP includes abstractions of motion constraints in planning-operator preconditions and effects.
- B. U-TAMP: By incorporating constraints into heuristic task-plan search, U-TAMP minimizes the motion planner’s role and unifies task and motion planning.
- B. U-TAMP: U-TAMP defines finite, scenario-specific physically plausible interactions that map symbolic predicates directly to execution parameters.
IV. U-TAMP USING INTER-OBJECT AFFORDANCES
The method represents physical interaction constraints as inter-object affordances determined by object parts, shapes, sizes, and materials. These abstractions cover support, grasping, lifting, and sliding for contact-rich manipulation.
- Physical attributes create relative grasping and supporting constraints that original U-TAMP does not represent.
- The proposed affordance abstractions extend U-TAMP to grasping, placement, sliding, and lifting constraints for heterogeneous objects.
- Compound objects are represented through members and bounding-box parts, enabling object-centric interaction descriptions.
- Supportiveness and supportability distinguish parts that can stably support objects from parts suitable for placement.
- Inter-object supportiveness depends on whether one object part can stably support another object part.
- Inter-object graspability captures stable grasping of a compound object through a rigidly attached member such as a pan handle.
- The framework also defines lifting and sliding according to payload, grasping, placement, and surface-contact conditions.
- Grasping is categorized into wrap-grasping, edge-grasping, and handle-grasping using different members of compound objects.
A. Actions
The method defines three symbolic actions and extends U-TAMP constraints to represent grasping, picking, placing, lifting, and sliding interactions among object parts.
- Three actions are defined: pick-support grasps an object, place-support places it on a supporting surface, and slide-support moves it across that surface.
- StableGrasp requires stable grasping through a designated object member, with valid configurations represented by canGraspFrom and current states checked by isGraspFrom.
- The PDDL domain summarizes StableGrasp, StablePick, StablePlace, and StableSlide constraints using symbolic predicates over objects, members, parts, and virtual spaces.
- StablePlace requires clear supportive and supportable parts, while the support object must be fully within the robot’s reachable workspace.
- StableSlide applies when an object cannot be lifted, can slide on its current support, and has adjacent empty workspace that can contain it.
V. EXPERIMENTS
Experiments evaluate contact-rich kitchen-table organization with heterogeneous objects in a physically realistic simulation, using a robot to stack objects on a tray.
- The kitchenware task involves stacking real-life objects with varied shapes, sizes, and materials, using affordances for grasping, supporting, lifting, and sliding.
- The simulation uses CoppeliaSim with a Franka Emika Panda robot, a two-finger gripper, a kitchen table, and five objects arranged in varied initial configurations.
- Figure 3 presents the experimental pipeline used by the study’s main approaches.
- The goal is to organize the table by stacking all objects on top of one another on the tray while distinguishing containment from support.
A. Task Planners and Evaluation Metrics
The study compares PDDL-based planners that differ in whether they model inter-object affordances and physical attributes, while using PDDL descriptions and a common planning system.
- Task planning transforms a symbolic initial state into a goal configuration using operators with preconditions and effects.
- The extended U-TAMP domain represents object-centric inter-object affordances and constraints, with goals requiring all object-holding table slots except the tray to be clear.
- All PDDL-based approaches use the off-the-shelf Fast Downward planner.
- UTAMP-1 uses object-centric predicates for motion and kinematic constraints but excludes inter-object affordances and physical object attributes.
1) PDDL-based Task Planning:
The PDDL-based evaluation compares original U-TAMP, a hand-crafted affordance baseline, and a VLM pipeline that converts detected object attributes into affordances and PDDL problems.
- PDDL-based Task Planning: UTAMP-2 extends UTAMP-1 with supporting, grasping, lifting, and sliding affordances, workspace-engulfment preconditions, and slide-support for heavy objects.
- PDDL-based Task Planning: UTAMP-2-GT manually hand-crafts inter-object affordances as a ground-truth baseline with perfect affordance detection.
- PDDL-based Task Planning: UTAMP-2-VLM detects scene objects and physical attributes, then parses them into inter-object affordances and composes a PDDL problem.
- PDDL-based Task Planning: Plans are decoded into Cartesian end-effector poses for grasping, placement, and sliding before motion execution.
- PDDL-based Task Planning: Success requires physically valid preconditions and compatible accumulated effects, followed by issue-free execution of all planned actions.
- PDDL-based Task Planning: Experiments compare tasks with and without required sliding across scenes containing 2 ≤ n ≤ 5 manipulable objects, reporting success and planning time.
B. Results
Rich symbolic abstractions improve planning success and efficiency for realistic contact-rich manipulation, while performance degrades for less affordance-aware methods as scene complexity increases.
- Higher success rates and faster planning times: rich symbolic abstractions outperform language-model planning in the overall pick-and-place experiments.The comparison uses success rate and planning time, with planning time shown on a logarithmic scale.
- 100% success rate: UTAMP-2-GT performs best, followed by UTAMP-2-VLM at 90%, while other approaches remain below 50%.UTAMP-2-GT uses ground-truth affordances and constraints; UTAMP-2-VLM uses VLM-based affordance detection.
- 0.98 mean precision, recall, and F1: UTAMP-2-VLM detects affordances accurately, but a few incorrect supporting affordances can still make stacking infeasible.The resulting success rate is 90%, reflecting sensitivity to affordance-detection errors.
- 5 objects: VLM and UTAMP-1 seldom produce a successful plan or valid stacking order, while UTAMP-2 variants retain the highest success rate.Planning time rises with scene complexity, but UTAMP-2 variants have a flatter increase than the other methods.
- Sliding-task comparisons preserve the earlier pattern: VLM has the lowest success rate and longest planning time, while UTAMP-2-VLM trails UTAMP-2-GT.These configurations require sliding a partially unreachable, non-liftable wooden tray into the robot’s workspace before stacking.
- Symbolically valid UTAMP-2-VLM plans can still fail when missing supporting affordances leave no feasible stacking plan.The missing affordances arose mainly from inconsistent VLM detection of object attributes.
VI. DISCUSSIONS
The discussion contrasts symbolic planning’s constraint-validity benefits with language models’ affordance-detection strengths, while acknowledging simplifying assumptions in UTAMP-2.
- PDDL planners generate plans satisfying action-precondition constraints, making them efficient and symbolically valid.
- LLMs and VLMs are well-suited to affordance detection or symbol grounding because their common-sense reasoning can complement limited supervised or interaction-based learning.
- UTAMP-2 assumes rules such as small objects cannot support larger ones, round surfaces are unsuitable for placement, and large objects are too heavy to lift.It also places ideal support and grasp positions at a flat surface’s centroid and a member’s middle, respectively.
- These affordance and motion-planning assumptions cover many objects and scenarios but not all; future work targets more complex actions.
VII. CONCLUSIONS
The paper concludes that inter-object affordance abstractions enable efficient PDDL task planning for expressive contact-rich manipulation with realistic objects.
- The PDDL domain encodes grasping, supporting, lifting, and sliding constraints at task-planning time for realistic objects.This removes the need to reason about these constraints during motion planning while retaining fast, constraint-aware planning.
- VLM common-sense knowledge assists affordance detection, while alternative learning-based methods and hand-crafted ground-truth affordances are also supported.
- Empirical comparisons show that robust symbolic abstractions let AI planning quickly generate feasible and expressive task plans without sub-symbolic reasoning.