PaperlayerView exact revision on arXiv

Source-linked AI summary

OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction

Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, Cewu Lu

arXiv:2203.15709v1cs.CV

TL;DR

Machines need knowledge of both object affordances and human hand interactions to understand manipulation, but existing datasets lack comprehensive awareness of these components. OakInk addresses this gap with interrelated affordance and interaction knowledge bases, yielding 50,000 distinct interactions and supporting benchmarks and applications for pose and handover generation.

  • Problem

    Existing hand-object interaction datasets lack comprehensive awareness of object affordances and the hand interactions based on those affordances, while machines struggle to acquire human manipulation experience.

  • Method

    OakInk builds an object-centric Oak knowledge base describing object functionality and a human-centric Ink knowledge base containing affordance-conditioned interactions, with rich hand, object, contact, stress, and intent annotations.

  • Results

    OakInk contains 50,000 distinct hand-object interactions collected from 100 real-world objects and transferred to 1,700 virtual counterparts, and is benchmarked on pose estimation and grasp generation tasks.

  • Takeaways & Limitations

    OakInk provides a multimodal repository for visual and cognitive hand-object interaction research, including intent-based hand pose generation and human-to-human handover generation applications.

  • Takeaways & Limitations

    OakInk does not record dynamic interactions with movable articulated-object parts and does not transfer human-hand interaction knowledge to multi-finger robot arms.

Abstract

from arXiv · show

Learning how humans manipulate objects requires machines to acquire knowledge from two perspectives: one for understanding object affordances and the other for learning human's interactions based on the affordances. Even though these two knowledge bases are crucial, we find that current databases lack a comprehensive awareness of them. In this work, we propose a multi-modal and rich-annotated knowledge repository, OakInk, for visual and cognitive understanding of hand-object interactions. We start to collect 1,800 common household objects and annotate their affordances to construct the first knowledge base: Oak. Given the affordance, we record rich human interactions with 100 selected objects in Oak. Finally, we transfer the interactions on the 100 recorded objects to their virtual counterparts through a novel method: Tink. The recorded and transferred hand-object interactions constitute the second knowledge base: Ink. As a result, OakInk contains 50,000 distinct affordance-aware and intent-oriented hand-object interactions. We benchmark OakInk on pose estimation and grasp generation tasks. Moreover, we propose two practical applications of OakInk: intent-based interaction generation and handover generation. Our datasets and source code are publicly available at https://github.com/lixiny/OakInk.

1. Introduction

OakInk addresses missing affordance and interaction knowledge by combining object-centric and human-centric resources. It builds Oak and Ink, extends recorded interactions through Tink, and supports broad benchmarking and applications.

  • Motivation: Existing hand-object datasets lack comprehensive awareness of object affordances and the hand’s interactions with them.Real-world datasets contain few objects or interactions, while synthetic grasps do not reflect human interaction distributions or object affordances.
  • Knowledge bases: Oak describes object affordances, while Ink records diverse human hand interactions performed according to those affordances.The two knowledge bases represent object-centric and human-centric perspectives on manipulation.
  • Oak construction: 1,800 household objects are organized in an object knowledge graph using taxonomy and attribute abstractions.The graph supports extending the collection to new objects and clustering objects.
  • Ink construction: 100 representative objects were recorded with 12 human subjects using a multi-sensor motion-capture platform.The recordings contain 230,064 RGB-D frames and up to 5 intent-oriented interactions across 32 categories.
  • Repository and evaluation: OakInk combines affordances and human experiences with image-based and geometry-based datasets and rich interaction annotations.Its annotations include hand and object poses, scanned models, affordances, contact and stress patterns, and intent labels.
  • Repository and evaluation: The authors benchmark OakInk on three existing tasks and propose intent-based hand pose generation and human-to-human handover generation.These tasks target pose estimation, pose generation, and practical interaction synthesis.

2. Related Work

Prior hand-object interaction datasets use real-world or synthetic sources and collect annotations through several sensing and reconstruction strategies. Related tasks focus on contact and pose understanding, while OakInk derives and transfers contact information for virtual objects.

  • Dataset types: ObMan and YCBAfford use grasp simulators to synthesize or label static grasps in synthetic datasets.Real-world datasets are categorized by how their annotations are collected.
  • Contact annotation: Prior methods capture hand-object contact with force transducers, tactile sensors, thermal cameras, or accurate pose tracking.Contact information provides cues for reasoning about conjoint hand-object poses and optimizing grasps.
  • Contact annotation: OakInk derives contact regions and stress patterns through pose tracking, then maps contacts to virtual objects and optimizes poses.This connects recorded contact information with transferred interactions on virtual counterparts.
  • Pose-related tasks: Pose-related hand-object interaction research includes pose estimation and grasp generation from images or object shapes.Earlier methods focused on hand pose or object pose alone before conjoint hand-object estimation methods emerged.

3. Constructing the OakInk

OakInk is constructed from interrelated object-affordance and human-interaction knowledge bases, combining a 1,800-object knowledge graph with recorded and transferred hand-object interactions. The pipeline records multimodal demonstrations, annotates poses and contact patterns, and uses Tink to transfer interactions to virtual objects, producing 50,000 distinct interactions.

  • Repository design: OakInk comprises an object-centric Oak base for affordances and a human-centric Ink base for demonstrations of manipulating objects according to those affordances.The construction addresses affordance representation, human demonstration recording, and interaction transfer as three linked questions.
  • 3.1. Object-Centric Affordance Knowledge Base: 1,800 household objects are organized in Oak’s knowledge graph using two-level taxonomy and attributes to support object extension and clustering.The taxonomy includes manipulation tools and function tools at the top level, with 32 WordNet-based subcategories.
  • 3.2.3. Data Annotation: The annotations combine object 6 DoF poses, 3D MANO hand geometry from multiview keypoints, and per-hand-part contactness derived from pose proximity.Contactness peaks at 1 near an anchor and decreases to 0 beyond 25 mm, approximating stress spreading around physical contact.
  • 3.3. Interaction Knowledge Transfer: Tink transfers selected source interactions to virtual target objects through shape-aware refinement because direct pose copying can create disjointedness or intersections.The method represents shapes with signed distance functions and maps contact regions through intermediate landmarks before refining the transferred pose.
  • 3.4. Dataset Analysis: OakInk yields image-based and geometry-based datasets totaling 50,000 distinct hand-object interactions after recording real objects and transferring interactions to 1,700 virtual counterparts.OakInk-Image contains the recorded sequences, while OakInk-Core selects representative interactions and OakInk-Shape contains selected and transferred interactions.

4. Tasks and Benchmark Results

OakInk is benchmarked on three existing pose and grasp tasks and extended with intent-based interaction and handover generation. The experiments define task-specific data splits, baselines, metrics, and qualitative evaluations for these applications.

  • Benchmark scope: Five tasks are evaluated: HMR, HOPE, GraspGen, intent-based interaction generation, and human-to-human handover generation.The first three are existing tasks, while IntGen and HoverGen are proposed as novel tasks.
  • Existing benchmarks: HMR estimates 3D hand mesh geometry from a single image using OakInk-Image and evaluates I2L-MeshNet and HandTailor with MPJPE, AUC, and MPVPE.The default SP0 split uses train/val/test proportions of 70% / 5% / 25%.
  • Existing benchmarks: HOPE jointly estimates hand pose and object pose from a single image, evaluating Tekin et al. and Hasson et al. with MPJPE and MPCPE.Object pose is represented by the oriented eight corners of the 3D object bounding box in the hand wrist-relative system.
  • Existing benchmarks: GraspGen generates diverse hand poses conditioned on object shape; GrabNet is evaluated on OakInk-Shape using penetration, intersection, displacement, and perceptual-survey metrics.Oak objects are split into 80% training, 10% validation, and 10% testing.
  • Novel generation tasks: The novel generation tasks address intent-specific poses and handovers, extending GrabNet with intent conditioning or the giver’s hand as an additional condition.HoverGen aims for receiver hands that avoid colliding with or hindering the giver’s hand retraction path.

5. Discussion

OakInk is presented as a large-scale repository combining object affordances with human hand-object interaction data. Its current scope excludes dynamic articulated-object interactions and transfer to multi-finger robot arms.

  • Limitations: Current OakInk does not record dynamic interactions with movable parts of articulated objects such as scissors.The authors identify this as a limitation for future work.
  • Limitations: Current OakInk does not transfer interaction knowledge from human hands to multi-finger robot arms.The authors state that this transfer will be addressed in future work.
  • Discussion: OakInk combines the Oak and Ink knowledge bases to represent object affordances and human experiences for understanding hand-object interactions.The conclusion states that the repository was benchmarked on computer-vision and computer-graphics tasks.

Appendices

The Oak base organizes object affordance knowledge across 32 categories and 30 attribute phrases.

  • Oak base: Oak base documentation covers 32 object categories and 30 attribute phrases.These lists are provided in Table 7.

B. Data Annotation Details

The annotation pipeline fits 3D hand pose, shape, and wrist position from labeled image evidence using multiple geometric and anatomical constraints. Temporal filtering then smooths the solved poses across sequences.

  • Pose and geometry: The annotation process solves hand pose θ, shape β, and wrist position P_h,0, which drive a differentiable MANO hand model.The hand joints are represented by P_h ∈ R21×3 and mesh vertices by V_h ∈ R778×3.
  • Optimization terms: Geometry consistency penalizes hand vertices that lie inside the object’s signed distance field, discouraging hand-object interpenetration.The object pose is transformed from the MoCap system to the camera system before evaluating vertex distances.
  • Optimization terms: Silhouette consistency compares the rendered hand-object silhouette with an automatically extracted binary foreground mask.Foreground pixels are obtained using green-screen and depth-image matting, and the mismatch is penalized per pixel.
  • Optimization terms: Anatomical constraints restrict each joint’s rotation axes and angles to keep the MANO pose anatomically plausible.The formulation uses axial and angular components for joint rotations and treats MCP joints specially.
  • Temporal processing: Temporal smoothing applies a low-pass filter, such as a Kalman filter, to poses and wrist positions across each image sequence.This addresses sequence-level continuity beyond per-frame annotation precision.

C. More Dataset Analysis

OakInk’s interaction data shows structured hand-pose and contact patterns across objects. Similar poses cluster together, different grasping types separate, and frequent contact regions align with object affordances.

  • Hand Pose Distribution: Similar interacting hand poses on different objects map to adjacent locations in the t-SNE embedded 2D space.Poses transferred from the same OakInk-Core pose share a color in the visualization.
  • Hand Pose Distribution: Different grasping types are separated in the t-SNE embedded space.
  • Contact Distribution: Contact heatmaps reveal contact frequencies across interactions on six Oak base categories.The heatmaps are provided for example objects.
  • Contact Distribution: High-frequency contact regions are consistent with the described object affordances.These regions are represented as red “hot” areas in the heatmaps.

D. Implementation: IntGen and HoverGen

IntGen and HoverGen modify GrabNet’s design for intent-conditioned grasp generation and human-to-human handover generation. IntGen conditions on intent, while HoverGen conditions receiver-pose generation on object shape and the giver’s hand root pose.

  • Architecture: IntGen and HoverGen use architectures modified from the original GrabNet design.
  • IntGen: IntGen adds a real-valued vector representing the intents use, hold, and hand-out as network input.Different intents are mapped to different areas of the latent pose space Z ∈ R16 during training.
  • IntGen: IntGen trains for 1,000 epochs with batch size 32 and an initial learning rate of 1 × 10^-3.The learning rate decays by a factor of 0.5 every 200 epochs.
  • HoverGen: HoverGen provides the giver’s root rotation and position to CoarseNet and a hand-to-hand Chamfer distance to RefineNet.These inputs condition the receiver hand’s embedding space on object shape and the giver’s hand root pose.
  • HoverGen: At inference, HoverGen samples an embedding and decodes a receiver hand pose for a human-to-human handover.Its training adds an L1 loss on the predicted hand-to-hand Chamfer distance.

E. Perceptual Survey for Generation Tasks

The study evaluates generated hand-object interactions through perceptual surveys on Amazon Mechanical Turk. Participants judge naturalness-related statements using four views and a five-level Likert scale.

  • Survey Design: Three perceptual surveys evaluate pose predictions from GrabNet, IntGen, and HoverGen.
  • Survey Design: Each survey presents four views of every predicted hand-object interaction for audience evaluation.The example statement asks whether the hand interacts naturally with the object.
  • Survey Design: Audiences rate each statement on a 5-level Likert scale.“Strongly agree” corresponds to one end of the scale.

F. Additional Benchmark Results

Additional results evaluate OakInk-Image under subject- and object-disjoint splits and examine generation on unseen out-of-domain objects. Qualitative benchmark results are provided across pose estimation, grasp generation, and handover generation tasks.

  • Alternative Splits: OakInk-Image includes subject-split and object-split benchmarks in addition to the default view split SP0.
  • SP1: Subjects Split: SP1 uses 6/1/5 train/validation/test subjects, with test subjects excluded from training.
  • SP2: Objects Split: SP2 splits objects into 70%/5%/25% train/validation/test sets, with test objects excluded from training.
  • Out-of-Domain Objects: The Stanford bunny serves as an illustrative unseen out-of-domain object for testing GrabNet and HoverGen.OakInk distinguishes unseen in-domain objects from objects without similar training counterparts.
  • Qualitative Results: Qualitative results cover HMR, HOPE, GrabNet, IntGen, and HoverGen benchmarks.These results are shown in Figures 16–18.

G. Discussion on Personally Identifiable Data

The study collected hand-object interaction videos from 12 subjects through a third-party crowdsourcing company, with stated ethics safeguards. Subjects signed contracts covering portrait usage, data usage, and payment.

  • 12 human subjects were recruited through a third-party crowdsourcing company, and their actions were recorded as video sequences by the MulCam system.
  • The crowdsourcing company warranted that appropriate IRB approval, or an equivalent approval under local requirements, had been obtained.
  • Subjects signed contracts granting permission for portrait usage and acknowledging data usage and payment policies.
Loading 2203.15709v1…