PaperlayerView exact revision on arXiv

Source-linked AI summary

KETO: Learning Keypoint Representations for Tool Manipulation

Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, Silvio Savarese

arXiv:1910.11977v2cs.RO

TL;DR

Robots need useful representations for manipulating novel objects as tools across different task goals. KETO predicts task-specific keypoints from tool point clouds and learns them through self-supervised robot interactions. It outperforms compared methods in task success rate across tool-manipulation scenarios.

  • Problem

    Prior tool representations can be difficult to generalize, while existing keypoint methods rely on costly manual annotations and category-level definitions.

    Source: §I · p. 1

  • Method

    KETO jointly predicts task-specific grasp, function, and effect keypoints from tool observations and uses them to guide actions, learning from self-supervised robot interactions.

    Source: §I · pp. 1–2, A. Keypoint Representations for Tool… · p. 3

  • Results

    KETO consistently outperforms compared methods in task success rate across task scenarios and evaluation criteria.

    Source: D. Quantitative Results · p. 5

  • Takeaways & Limitations

    The learned keypoints provide an interpretable structured tool summary and are demonstrated in creating novel tools from random object parts.

    Source: §VI · p. 6

Abstract

from arXiv · show

We aim to develop an algorithm for robots to manipulate novel objects as tools for completing different task goals. An efficient and informative representation would facilitate the effectiveness and generalization of such algorithms. For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation. For each task, a set of task-specific keypoints is jointly predicted from 3D point clouds of the tool object by a deep neural network. These keypoints offer a concise and informative description of the object to determine grasps and subsequent manipulation actions. The model is learned from self-supervised robot interactions in the task environment without the need for explicit human annotations. We evaluate our framework in three manipulation tasks with tool use. Our model consistently outperforms state-of-the-art methods in terms of task success rates. Qualitative results of keypoint prediction and tool generation are shown to visualize the learned representations.

I. INTRODUCTION

Robust tool manipulation calls for representations that support generalizable control while remaining compact and interpretable. KETO uses task-specific keypoints learned from robot interactions to represent tools, and outperforms baselines across three tool-use tasks.

I. INTRODUCTION

  • Black-box latent representations can lack compactness, interpretability, and compositionality, qualities sought for effective downstream manipulation.The paper motivates representations that retain neural-network representation power while being efficient for downstream tasks.Source: §I · p. 1
  • KETO represents each tool with task-specific keypoints intended to capture robot–tool–environment interactions and guide manipulation actions.The framework predicts grasp, function, and effect points for each task.Source: §I · p. 1
  • Self-supervised robot interactions train keypoint prediction without explicit human annotations for ground-truth keypoints.The method addresses the annotation cost and ambiguity of physically grounded keypoints, as well as prior approaches that predicted points individually or at category level.Source: §I · pp. 1–2
  • Across hammering, pushing, and reaching, KETO outperforms end-to-end neural policies and hardcoded-keypoint baselines in task success rate.The paper also visualizes predicted keypoints and tool generation.Source: §I · p. 2

II. RELATED WORK

KETO builds on object-keypoint manipulation and tool-use learning while distinguishing its representation through task-specific points and self-supervised training. It also contrasts with end-to-end latent representations and motion-primitive-based approaches.

A. Keypoint Representations

  • Robotic manipulation research uses keypoints to compactly represent visual information about objects and environments, [17],, [49],,, [34].Earlier keypoint approaches also describe object parts at category level or use low-level features for detection and matching.Source: A. Keypoint Representations · p. 2
  • Unlike kPAM [34], KETO uses task-specific keypoints, assumes no known object category at test time, and learns from self-supervised interactions rather than human annotations.The approaches share a trajectory-optimization formulation.Source: A. Keypoint Representations · p. 2

B. Understanding and Manipulation of Tools

  • Compared with end-to-end latent tool representations, KETO uses structured keypoints and, unlike [16], does not rely on parameterized motion primitives.The paper reports better generalization to unseen test objects than [16].Source: B. Understanding and Manipulation of… · p. 2

C. Self-supervised Robot Learning

The paper studies robot tool use as choosing a grasp and motion that maximize task success under a defined observation and task context. Its setup uses self-supervised learning in simulation and constrains the manipulation and sensing configuration.

C. Self-supervised Robot Learning

  • The framework trains each task from self-supervised robot interactions in a simulated environment, using trial and error to reduce reliance on explicit human annotations.The paper notes that simulation can be used when collecting real-world robot data is expensive and time-consuming.Source: C. Self-supervised Robot Learning · p. 2

III. PROBLEM STATEMENT

  • The robot observes a tool as a 3D point cloud and selects a grasp and subsequent motion to maximize the likelihood of task success under context c.Success is represented by a binary variable evaluated for an observation, context, grasp, and motion.Source: §III · p. 2
  • Tasks express goals as spatial relationships between tools and environment objects; tools are unseen at test time, while environment objects remain fixed for each task.Environment points summarize task context, and tools are randomly placed at the start of each episode.Source: §III · p. 2
  • The setup assumes a single-arm Sawyer with a parallel-jaw gripper, top-down depth sensing, external instance segmentation, top-down grasps, and planar tool motion.The sampled tool point cloud is separated from the task environment context before use.Source: §III · p. 2

IV. METHOD

KETO predicts tool keypoints from visual observations and uses them with predefined environment keypoints to optimize grasp and manipulation actions. Its points encode task-relevant relations among the robot, tool, and environment.

IV. METHOD

  • The framework predicts tool keypoints from visual observation, then optimizes grasp and manipulation trajectories; keypoint learning uses self-supervised trial and error.The method comprises keypoint representations, an action optimizer, and a keypoint generator.Source: §IV · p. 2, Figure 2 · p. 3

A. Keypoint Representations for Tool Manipulation

  • Environment keypoints are predefined to summarize the task workspace, while the model predicts tool keypoints to guide robot motion.Together, the two sets of points provide a compact task representation.Source: A. Keypoint Representations for Tool… · p. 3
  • The environment points encode a target contact location and a receiver point whose direction from the target specifies the expected force direction.The framework represents them as Kc = [xt, xr].Source: A. Keypoint Representations for Tool… · p. 3
  • Three tool points encode grasp location, functional contact location, and force direction; the grasp and function points lie on the tool, while the effect point may lie outside it.These are xg, xf, and xe, respectively.Source: A. Keypoint Representations for Tool… · p. 3

B. Grasp Selection and Motion Generation

KETO derives grasp and manipulation actions from intermediate tool keypoints. Motion generation optimizes a final tool pose so the functional point reaches the target and the tool’s force aligns with the desired direction.

B. Grasp Selection and Motion Generation

  • The grasp is selected as the robust grasp candidate closest to the predicted grasp point, while manipulation motion is parameterized by the tool’s final pose and effect point.The grasp is represented by position and orientation in the world frame.Source: B. Grasp Selection and Motion Generation · p. 3
  • The quadratic program selects a final tool pose by bringing the function point near the target while aligning the tool’s exerted force with the desired direction.Its solution gives the precise final pose, subject to inequality constraints defining a valid range.Source: B. Grasp Selection and Motion Generation · pp. 3–4
  • To improve optimization stability, the method replaces a division-based expression with a surrogate objective that maximizes vT z − ||xf −xg||2.The resulting objective is expressed as a quadratic form.Source: B. Grasp Selection and Motion Generation · pp. 3–4

C. Keypoint Generator

The keypoint generator proposes and scores task-relevant keypoint candidates from an object point cloud. It is trained from robot trials labeled by task success rather than explicit human annotations.

C. Keypoint Generator

  • The proposal network generates 256 keypoint candidates from the object point cloud and random seeds, and the evaluation network selects the candidate with the highest score.Both networks use PointNet as their backbone feature extractor.Source: C. Keypoint Generator · p. 4
  • The robot begins with a heuristic policy, switches to the neural model after training, and records point clouds, keypoints, and success labels after each trial.Success requires a robust grasp and achievement of the predefined task goal.Source: C. Keypoint Generator · p. 4
  • Training uses trial outcomes: successful keypoints supervise the proposal network, while the evaluation network learns to predict subsequent manipulation success or failure.The proposal network uses variational inference with an L1 reconstruction loss; the evaluator uses sigmoid cross-entropy.Source: C. Keypoint Generator · p. 4

V. EXPERIMENTS

The experiments are framed around evaluating KETO’s tool understanding and manipulation. They examine task performance, generalization to unseen objects, interpretability, and whether keypoints can support novel tool creation.

V. EXPERIMENTS

  • The experiments ask how KETO performs across manipulation tasks and how its keypoint representations affect generalization to unseen objects.The evaluation questions also include whether the representations offer intuitive, interpretable understanding.Source: §V · p. 4
  • The experiments examine whether predicted keypoints provide an intuitive and interpretable understanding of the tool.Source: §V · p. 4
  • The experiments ask whether keypoints can be used to create novel tools.Source: §V · p. 4

A. Experiment Setup

The setup evaluates tool manipulation in simulation across hammering, pushing, and reaching, using procedurally generated hammer and non-hammer objects. Baselines include an adapted end-to-end policy, template matching, and handcrafted keypoints.

A. Experiment Setup

  • The simulated tasks are hammering, pushing, and reaching, with success defined by task-specific nail, object-motion, or target-motion conditions.A top-down depth camera supplies the 3D point cloud observation.Source: A. Experiment Setup · p. 4
  • Training and testing use procedurally generated hammer and non-hammer tools, with 600 tools generated for each split.The object examples are categorized as hammers and non-hammers, with different instances drawn from the same distribution for training and testing.Source: A. Experiment Setup · p. 4, Figure 5 · p. 5

B. Baseline Methods

  • The End-to-End baseline adapts TOG-Net to point-cloud input and uses the same neural network encoder as KETO.Source: B. Baseline Methods · p. 4
  • The Template baseline transfers keypoints from the Chamfer-distance-nearest training object, while the Heuristic baseline uses handcrafted keypoint generation and may generalize less well to unseen objects.Template matching has significant computational burden on a large training set.Source: B. Baseline Methods · p. 5

C. Data Collection and Training

For each task, the model predicts tool keypoints from point-cloud observations and uses them to select actions in simulation. Successful outcomes provide positive keypoint examples, with a heuristic policy bootstrapping training.

C. Data Collection and Training

  • Each episode selects a training tool in a random tabletop pose, predicts keypoints, and feeds them to an action optimizer for execution in simulation.The keypoints are labeled positive when the task succeeds and negative otherwise.Source: C. Data Collection and Training · p. 5
  • A handcrafted heuristic policy bootstraps training by increasing positive examples in the first round.Approximately 100K keypoint and point-cloud tuples were collected for each task.Source: C. Data Collection and Training · p. 5
  • The authors trained a separate neural network for each task using Adam.Source: C. Data Collection and Training · p. 5

D. Quantitative Results

KETO outperforms the compared methods in task success rates and shows promising generalization to unseen tool distributions. Qualitative results illustrate task-specific keypoints, multi-stage tool use, and generation of tools from object parts.

D. Quantitative Results

E. Qualitative Results

D. Quantitative Results

  • KETO consistently outperforms end-to-end neural network, heuristic-keypoint, and template-matching baselines across the evaluated scenarios.Hammer tools generally achieve higher success rates than non-hammer tools.Source: D. Quantitative Results · p. 5, Figure 6 · p. 6

E. Qualitative Results

  • Tool generation composes randomly selected object parts by gradient-ascent optimization of their translations and rotations to raise the keypoint-conditioned confidence score.The resulting parts combine into a tool when their transformations converge.Source: E. Qualitative Results · pp. 5–6, Figure 9 · p. 6

VI. CONCLUSION

The conclusion presents a compact, interpretable keypoint representation learned through robot-environment interaction, with experiments outperforming competitive baselines. It also identifies simulation evaluation, single-step action generation, and current tool-generation methods as boundaries for future work.

VI. CONCLUSION

  • The proposed tool-manipulation keypoint representation is compact, effective, and interpretable.Source: §VI · p. 6
  • Robot interaction with the environment supplies self-supervision, removing the need for manual keypoint annotation.Source: §VI · p. 6
  • Experiments show that the keypoint intermediate representation outperforms competitive baselines, including end-to-end visuomotor learning.Source: §VI · p. 6
  • The keypoints serve as a structured tool summary and are also used to create novel tools from random object parts.Source: §VI · p. 6
  • Experiments were primarily conducted in physical simulation; future work includes hardware deployment, multi-step planning, and more sophisticated tool-generation models.Source: §VI · p. 6
Loading 1910.11977v2…