Source-linked AI summary
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, Li Fei-Fei
TL;DR
Robotic manipulation needs constraint representations that generalize across diverse tasks, avoid manual specification, and remain real-time optimizable. ReKep uses semantic 3D keypoints and model-generated Python cost functions within hierarchical optimization, demonstrating varied real-robot behaviors without task-specific data or environment models. Its scope is bounded by local rigidity assumptions and reduced robustness on long, temporally dependent tasks.
Problem
Robotic manipulation lacks a constraint formulation that is simultaneously versatile, automatically obtainable, and real-time optimizable.
Method
ReKep represents spatial relations as Python cost functions over semantically meaningful 3D keypoints and sequences these constraints across spatio-temporally dependent task stages.
Results
ReKep produces multi-stage, in-the-wild, bimanual, and reactive behaviors on two robot platforms without task-specific data or environment models.
Takeaways & Limitations
ReKep provides a visually grounded constraint representation that supports closed-loop hierarchical optimization and automated synthesis for varied manipulation behaviors.
Takeaways & Limitations
ReKep assumes locally rigid motion for grasped keypoints over the short optimization interval and lacks robustness on tasks with many temporally dependent stages.
Abstract
from arXiv · showhide
Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep is expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observations. We present system implementations on a wheeled single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models. Website at https://rekep-robot.github.io/.
1 Introduction
ReKep addresses the challenge of representing manipulation constraints that are versatile across tasks, automatically obtainable, and efficiently optimizable. It combines semantic 3D keypoints, model-generated Python constraints, and hierarchical optimization to produce real-time robot behaviors.
- Motivation: Manipulation tasks require spatial, temporal, and transitional constraints linking intermediate sub-goals to robot actions.Pouring tea requires grasping, upright transport, spout alignment, and correctly timed tilting.
- Challenges: Existing relative-pose representations omit geometric detail, require object models, and cannot handle deformable objects, while data-driven methods face combinatorial training-data demands.These limitations motivate representations adaptable to diverse real-world tasks.
- ReKep Representation: ReKep represents each constraint as a Python function mapping semantically meaningful 3D scene keypoints to a numerical cost.Keypoints can collectively specify lines, surfaces, and rotations when rigidity is enforced, and can relate robot arms, object parts, and other agents.
- Automated Specification: Large vision models propose fine-grained keypoints, while vision-language models write Python constraints from RGB-D observations and free-form language instructions.This pipeline automates ReKep specification instead of requiring manual task-by-task constraint writing.
- Optimization: Hierarchical optimization first solves SE(3) waypoint sub-goals and then computes dense receding-horizon actions, operating at approximately 10 Hz for the evaluated tasks.The optimization repeatedly reevaluates constraints using tracked keypoints.
- System Scope: The system is implemented on wheeled single-arm and stationary dual-arm platforms for multi-stage, in-the-wild, bimanual, and reactive manipulation without task-specific data or environment models.The stated contributions include both automated constraint generation and real-robot demonstrations across these behavior types.
2 Related Works
Prior work uses rigid-body, data-driven, and keypoint-based structural representations, alongside constrained optimization and foundation models for manipulation. ReKep combines foundation-model synthesis with continuous receding-horizon constrained optimization to address visual detail and open-world semantics.
- Structural Representations: Rigid-body poses efficiently model long-range object motions but often require environment geometry and dynamics to be specified beforehand.This motivates alternatives based on object-centric representations, particles, keypoints, or descriptors.
- Structural Representations: Data-driven structural representations include object-centric methods, particle-based dynamics, and keypoints or descriptors.The related work notes that keypoints have shown promise among these approaches.
- Constrained Optimization: Manipulation constraints support motion planning, contact-rich behavior, task-and-motion planning, and nonlinear programs over trajectories.Constraints may be manually written or learned as manifolds, feasibility models, or signed-distance fields.
- Constrained Optimization: ReKep formulates sequential manipulation as an integrated continuous mathematical program solved repeatedly in a receding-horizon fashion, with constraints synthesized by foundation models.This distinguishes its optimization formulation from prior constrained approaches described in the passage.
- Foundation Models: VLMs provide open-world reasoning but may lose visual detail, whereas self-supervised vision models provide fine-grained features without effective open-world semantic interpretation.ReKep uses DINOv2 for keypoint proposal and GPT-4o for visual reasoning in code.
3 Method
ReKep represents manipulation constraints as Python functions over task-specific 3D keypoints, then combines staged sub-goal and path constraints with hierarchical optimization. Large vision and vision-language models automate keypoint and constraint generation, while replanning and real-time solvers support reactive execution across diverse tasks.
- Relational Keypoint Constraints: ReKep maps task-specific 3D scene keypoints to numerical costs that encode spatial relations among robot arms, object parts, and other agents.Each constraint is implemented as a stateless Python function using potentially nonlinear and nonconvex NumPy operations; costs f(k) ≤0 indicate satisfaction.
- Task Formulation: Each manipulation task is decomposed into temporally ordered stages containing sub-goal constraints for stage endpoints and path constraints for transitions within stages.For pouring, the stages are grasp, align, and pour; path constraints maintain properties such as keeping the teapot upright during transport.
- Task Formulation: The optimization represents actions as a discrete-time sequence of end-effector poses in SE(3), with stage-transition timings and keypoint dynamics included in the formulation.The forward model h estimates keypoint changes from end-effector changes, while auxiliary costs address constraints such as collision avoidance and reachability.
- Decomposition and Algorithmic Instantiation: A hierarchical solver first optimizes the immediate next sub-goal and then a trajectory to reach it, using Dual Annealing followed by SLSQP and subsequent local optimization.The decomposition is designed for real-time solution, with the initial solve taking around 1 second and the system operating at approximately 10 Hz for the considered tasks.
- Decomposition and Algorithmic Instantiation: The controller checks prior-stage constraints during every loop and backtracks to an earlier stage when a previously achieved sub-goal is violated.This enables replanning across stages after disturbances such as a cup being removed from the gripper.
- Keypoint Proposal and ReKep Generation: Large vision models propose semantically meaningful keypoints from RGB-D observations, while vision-language models generate ReKep Python constraints from overlaid visual input and language instructions.The system also uses scene segmentation and tracking to support keypoint-based optimization; AnyGrasp supplies grasp choices when grasping is required.
4 Experiments
The experiments evaluate ReKep across diverse multi-stage, in-the-wild, bimanual, and reactive manipulation settings on two robot platforms. Results show that ReKep generates varied garment-folding strategies, supports coordination and disturbance response, and exposes interpretable failure sources.
- 4.1 ReKep for In-the-Wild and Bimanual Manipulation: The evaluation covers seven tasks spanning multi-stage, in-the-wild, bimanual, and reactive behaviors on wheeled single-arm and stationary dual-arm platforms.Three tasks are additionally tested under external disturbances.
- 4.1 ReKep for In-the-Wild and Bimanual Manipulation: ReKep handles temporal dependencies, commonsense requirements, bimanual and collaborative coordination, confined-space motion, and dense shoe packing.The system also replans within and across stages when tracked keypoints respond to external disturbances.
- 4.2 Generalization in Manipulation Strategies: The garment study evaluates eight categories requiring distinct folding methods, geometrical reasoning, commonsense reasoning, and bimanual coordination.Strategy Success tests feasible generated ReKep, while Execution Success measures system success for each clothing category over 10 trials.
- 4.2 Generalization in Manipulation Strategies: Garment categories produce drastically different folding strategies, including simultaneous sleeve folding and one-arm execution when two arms are unnecessary.These strategies are often aligned with human folding behavior.
- 4.3 System Error Breakdown: The point tracker causes the largest share of system errors, while keypoint proposal and VLM errors also contribute considerably.Optimization contributes less to failures because many problems admit multiple feasible solutions.
5 Conclusion & Limitations
The conclusion presents ReKep as a semantic-keypoint constraint representation automatically synthesized by vision models and solved in closed loop at real-time frequency. It reports broad real-robot demonstrations while identifying rigidity-based keypoint prediction and accurate tracking as limitations.
- Conclusion: ReKep specifies desired relations among robot arms, object parts, and other agents using constraints over semantic keypoints.The framework repeatedly solves these constraints through hierarchical optimization as a closed-loop policy.
- Conclusion: The system demonstrates multi-stage, in-the-wild, bimanual, and reactive behaviors on two robot platforms without task-specific data, additional training, or environment models.Constraint synthesis uses large vision and vision-language models.
- Limitations: The optimization framework assumes a rigidity-based forward model for keypoints, while closed-loop feedback reduces its accuracy requirement.Accurate point tracking remains necessary for closed-loop action optimization and is challenging.
A.1 Pseudo-code for Sequential Manipulation with Relational Keypoint Constraints
The sequential manipulation procedure advances through constraint-defined stages while repeatedly solving sub-goal and path problems. It executes receding action segments and can backtrack when current-stage constraints are violated.
- Sequential manipulation: The algorithm initializes stage and time indices, then loops until all sequential stages are completed.Each iteration checks constraint violations and end-effector proximity to the current goal.
- Backtracking: When a constraint in the current stage is violated, the procedure decrements the stage index and continues, enabling cross-stage backtracking.This permits replanning when a prior sub-goal no longer holds.
- Optimization and execution: For each active stage, the solver first obtains a sub-goal end-effector pose and then solves a path problem before executing the next action segment.The loop advances time after executing m actions.
A.2 Wheeled Single-Arm Platform
The wheeled single-arm platform combines a Franka arm with a manually movable base for deployment beyond laboratory settings. Its controller executes 6-DoF pose sequences at 20 Hz using workspace clipping, interpolation, and inverse kinematics.
- Platform: The platform uses a Franka arm mounted on a wheeled Vention-frame base that is mobile but not autonomously motorized.This setup supports investigation outside laboratory environments.
- Control: The pipeline produces 6-DoF end-effector pose sequences, while position control runs at a fixed frequency of 20 Hz.Target poses are clipped to a predefined workspace before execution.
- Control: Motion is generated by linearly interpolating from the current pose using 5mm position and 1 degree rotation steps, with inverse kinematics applied to each pose.These details describe the low-level execution of planned end-effector trajectories.
- Perception: Two RGB-D cameras mounted on opposite sides of the robot capture RGB images and point clouds at 20 Hz.The cameras face the center of the workspace.
A.3 Stationary Dual-Arm Platform
The stationary dual-arm platform uses two Franka arms controlled simultaneously through jointly optimized end-effector pose sequences. Three RGB-D cameras capture observations at 20 Hz.
- The platform consists of two Franka arms mounted in front of a tabletop workspace and controlled simultaneously at 20 Hz.The shared controller differs from the wheeled single-arm setup in its simultaneous dual-arm control.
- The pipeline jointly solves two 6-DoF end-effector pose sequences and sends them together to the low-level controller.The controller then calculates inverse kinematics for both arms.
- Three Orbbec Femto Bolt RGB-D cameras are mounted around the platform and capture RGB images and point clouds at 20 Hz.Two cameras are positioned on the left and right sides, and one is mounted at the back.
A.4 Evaluation Details
This section introduces the evaluation details for the experiments reported in Sections 4.1 and 4.2.
- The section presents evaluation details for the experiments reported in Sections 4.1 and 4.2.
A.4.1 Details for In-the-Wild and Bimanual Manipulation (Section 4.1)
The evaluation covers diverse in-the-wild and bimanual tasks with manually verified feasible initial configurations and operator-measured success. Tasks include pouring, sorting, stowing, taping, folding, packing, and collaboration.
- Evaluation protocol: Each task uses 10 different initial object configurations spanning the workspace, manually checked for robot kinematic feasibility.A human operator restores each scene and initiates every trial.
- Evaluation protocol: Success rates are measured by a human operator because automatic criteria are difficult for the diverse objects and environments.
- In-the-wild tasks: Pouring tea requires grasping a teapot, aligning it over a cup, and tilting it while keeping the teapot upright until pouring.
- In-the-wild tasks: Recycling requires reorienting one of three can types upright over a narrow recycle-bin opening before dropping it.
- In-the-wild tasks: Stowing a book requires placing it steadily into a 15 cm shelf opening without bumping the shelf or neighboring books.
- Collaborative tasks: Taping a box involves a robot grasping and aligning pre-unrolled tape while a human operator squeezes the box.The tape is pre-unrolled because unrolling typically exceeds the robot arm’s force limit.
- Bimanual tasks: Bimanual garment folding and shoe packing allow variable action stages, with success defined by final placement and collision-related criteria.Garment folding targets at most half the original surface area; shoe packing avoids stacking and bimanual self-collision.
- Collaborative tasks: Collaborative blanket folding requires grasping two corners opposite the human and aligning them with the human-held corners.The blanket is pre-folded because its size exceeds the workspace limit.
A.4.2 Details on Baseline Methods
The evaluation uses VoxPoser as the main baseline and adapts it to share the same vision-language model, camera input, and task context for fair comparison.
- Baseline selection: VoxPoser is selected as the main baseline because it also requires neither task-specific data nor predefined motion primitives.
- Baseline adaptation: For fairness, the adapted baseline uses GPT-4o and the same camera input as the proposed method.
- Baseline adaptation: The original VoxPoser prompt is augmented with this work’s prompt to provide sufficient task context.
A.4.3 Details for Generalization in Manipulation Strategies (Section 4.2)
The system evaluates ReKep across clothing-folding and other manipulation settings, combining visually generated strategies with hierarchical task-space optimization. It supports 6-DoF, bimanual, multi-stage, reactive behavior, while limitations remain in long-horizon robustness, kinematic feasibility, articulated-object reasoning, and bimanual coordination.
- Evaluation: The dual-arm system evaluates folding eight clothing categories using Strategy Success for generated keypoints and constraints, and Execution Success for robotic execution.These metrics separate strategy generation quality from execution given a successful strategy.
- Task generality: ReKep addresses 6-DoF single-arm and 12-DoF bimanual tasks by having vision models reason over 3D keypoints while numerical solvers determine rotations.This avoids requiring VLMs to explicitly predict full 3D rotation poses.
- Task generality: The formulation integrates high-level task planning and low-level actions in one continuous program, accounting for geometric dependencies across stages and backtracking after failures.For example, pouring can re-align the teapot or re-grasp it when conditions change.
- Execution: A high-frequency perception-action loop couples vision-language specification with constraint optimization, supporting reactive manipulation instead of only open-loop or slow execution.The broader system uses generated constraints with hierarchical optimization and operates at approximately 10 Hz for the considered tasks.
- Representation: Visual prompting lets code-generation express vectors, surfaces, volumes, and temporal dependencies through arithmetic relations among multiple keypoints.This extends the representation beyond the geometric information captured by a single point.
- Limitations: Existing VLMs are not robust for tasks with many stages and temporally dependent constraints, producing inconsistent success.This limitation is specific to complex, long-horizon strategy specification.
- Limitations: Task-space optimization can yield poses that are kinematically difficult to achieve because the solver does not explicitly model robot kinematics.Joint-space planning may resolve this issue but is less computationally efficient for the reported tasks.
- Keypoint proposal: DINOv2 provides sharper object-part distinctions than CLIP or ViT, while SAM’s objectness prior keeps keypoint proposals focused on scene objects rather than background.The comparison highlights fine-grained feature quality and object-focused proposal constraints as important for keypoint generation.