PaperlayerView exact revision on arXiv

Source-linked AI summary

OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion

Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, Cewu Lu

arXiv:2403.19417v1cs.CV

TL;DR

Existing data-driven work has made progress on hand-object interaction, but human-level understanding of complex task completion remains a gap. OAKINK2 structures manipulation using affordances, Primitives, and their dependencies, and provides a bimanual dataset and CTC framework. The dataset contains 627 sequences, while CTC combines LLM-based Primitive planning with task-aware motion generation.

  • Problem

    Despite progress in hand-object pose estimation, interaction synthesis, and action imitation, methods still face a gap in human-level understanding of complex object manipulation and its ordered interactions.

    Source: §1 · p. 2

  • Method

    OAKINK2 organizes manipulation as object affordances, minimal Primitive interactions, and dependency-ordered Complex Tasks, and uses an LLM interpreter with task-aware motion generation for CTC.

    Source: §1 · p. 2

  • Results

    OAKINK2 contains 627 bimanual interaction sequences, including 363 Primitive and 264 Complex Task sequences, spanning 75 objects and 39 affordances.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    The dataset supplies demonstrations and task structure for bimanual object manipulation, and the paper applies them to decomposing Complex Tasks and generating Primitive-level hand motion.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present OAKINK2, a dataset of bimanual object manipulation tasks for complex daily activities. In pursuit of constructing the complex tasks into a structured representation, OAKINK2 introduces three level of abstraction to organize the manipulation tasks: Affordance, Primitive Task, and Complex Task. OAKINK2 features on an object-centric perspective for decoding the complex tasks, treating them as a sequence of object affordance fulfillment. The first level, Affordance, outlines the functionalities that objects in the scene can afford, the second level, Primitive Task, describes the minimal interaction units that humans interact with the object to achieve its affordance, and the third level, Complex Task, illustrates how Primitive Tasks are composed and interdependent. OAKINK2 dataset provides multi-view image streams and precise pose annotations for the human body, hands and various interacting objects. This extensive collection supports applications such as interaction reconstruction and motion synthesis. Based on the 3-level abstraction of OAKINK2, we explore a task-oriented framework for Complex Task Completion (CTC). CTC aims to generate a sequence of bimanual manipulation to achieve task objectives. Within the CTC framework, we employ Large Language Models (LLMs) to decompose the complex task objectives into sequences of Primitive Tasks and have developed a Motion Fulfillment Model that generates bimanual hand motion for each Primitive Task. OAKINK2 datasets and models are available at https://oakink.net/v2.

1. Introduction

OAKINK2 addresses the gap in human-level understanding of complex object manipulation by structuring tasks as object-affordance fulfillment. It contributes a bimanual dataset and a task-oriented framework for Complex Task Completion.

1. Introduction

  • OAKINK2 extends OakInk, which lacked demonstrations of affordance fulfillment and complex tasks involving multiple object affordances.The paper motivates this extension by noting that humans use appropriate object-interaction sequences to achieve desired outcomes.Source: §1 · p. 2
  • OAKINK2 structures manipulation as Affordances, minimal Primitive Tasks that fulfill them, and Complex Tasks that combine Primitives with dependencies.Complex Tasks can be represented as Primitive Dependency Graphs, whose nodes are Primitives and whose directed edges encode execution order.Source: §1 · p. 2
  • The dataset contains 627 real-world bimanual manipulation sequences, including 264 Complex Task sequences, with 4.01M frames from four views.It covers four scenarios, 75 objects, and nine invited subjects.Source: §1 · p. 2
  • CTC uses an LLM-based interpreter to decompose textual task objectives into Primitive execution plans and TaMF to generate hand motion for each Primitive.TaMF conditions hand-motion generation on the task-related object trajectory.Source: §1 · p. 2
  • The paper presents an object-centric, three-level abstraction, a real-world bimanual manipulation dataset, and a task-oriented framework for complex task and motion planning.The framework combines LLM-based task decomposition with diffusion-based motion generation for Primitive fulfillment.Source: §1 · p. 2

2. Related Works

Related work spans hand-object interaction datasets, hierarchical task decomposition, motion synthesis, and foundation models for manipulation. OAKINK2 combines affordance-based Primitive demonstrations with Complex Tasks whose Primitives have explicit dependencies.

Hand-Object Interaction Datasets

  • OAKINK2 captures demonstrations of minimal interactions fulfilling object affordances and of Complex Tasks where those affordances are fulfilled in dependency-constrained order.Prior datasets variously cover dynamic or bimanual interactions, articulated objects, affordance labels, or tool-action-object pairs.Source: §2 · p. 3

Decomposition of Manipulation Tasks

  • OAKINK2 introduces an object- and affordance-centric three-level abstraction for decomposing complex manipulation tasks into Primitives.Prior approaches use symbolic task planners, hierarchical neural programs, dexterous policies, or language-based action hierarchies.Source: §2 · p. 3

Motion Synthesis

  • OAKINK2 frames Task-aware Motion Fulfillment as synthesizing hand-motion trajectories from textual task descriptions and object motions.The related work describes prior motion synthesis using probabilistic backbones, including cVAE and denoising diffusion.Source: §2 · p. 3

Foundation Models in Manipulation Tasks

  • The paper explores LLM reasoning over affordance interdependencies and execution order, using OAKINK2's Primitive decompositions and image-textual descriptions.Related work also includes foundation models that output control signals and LLMs used for action selection or program generation.Source: §2 · p. 3

3. Construction of OAKINK2

OAKINK2 constructs an object-centric hierarchy of affordances, Primitive interactions, and Complex Tasks, then records task demonstrations with synchronized camera and motion-capture data. Its annotations include task structure and human, hand, and object motion.

3. Construction of OAKINK2

  • The construction process first acquires the three abstraction levels, then collects and annotates demonstrations of Primitive and Complex Tasks.Subjects complete both task types on a capture platform after the abstraction acquisition stage.Source: §3 · p. 3, §3.2 · p. 4

3.1. Complex Task Acquisition

  • Annotators propose Complex Tasks in four scenarios and select the object clusters needed for those tasks.The scenarios are kitchen table, study room table, demo chem lab, and bathroom table.Source: §3.1 · p. 3

3.1.1 Object Affordance Analysis

  • Each affordance pairs an object-part segmentation with a descriptive phrase tuple that expresses the part's function.For example, a bottle cap can be associated with the function “secure.”Source: §3.1.1 · p. 3

3.1.2 Primitive Task Design

  • A Primitive is a minimal interaction that fully completes an affordance without redundant interaction, with starting and terminal conditions and an intervening hand-object process.The knife-blade example describes cutting through an object so its separated parts can detach.Source: §3.1.2 · pp. 3–4

3.1.3 Complex Task Decomposition

  • Complex Tasks are decomposed into short-term, single-goal Primitives whose PDG records dependencies, from independent L0 Primitives to those completing the task.An expert instantiates the scene and target, a subject describes the completion order using the Primitive pool, and the expert organizes it into a PDG.Source: §3.1.3 · p. 4

3.2. Data Collection and Annotation

  • After abstraction acquisition, subjects perform Primitive and Complex Tasks on a platform combining multi-camera recording with optical motion capture.The capture and annotation system records the manipulation process and tracks poses.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 Capture Setup

  • The capture setup uses 12 Optitrack infrared cameras for MoCap and four RGB cameras, with three allocentric views and one egocentric view, synchronized at 30 fps.The transformation between the camera systems is calibrated.Source: §3.2.1 · p. 4, Figure 3 · p. 4

3.2.2 Data Annotation

  • Human pose and surface annotations use a two-stage SMPL-X fitting process: first fit shape and marker correspondence, then fit per-frame poses and reconstruct the body mesh.Other body representations, such as MANO, are derived from the reconstructed result; experts also provide Primitive-step commentary on the current state and forthcoming action, refined using GPT-4.Source: §3.2.2 · p. 5

4. The OAKINK2 Dataset

OAKINK2 provides multi-view RGB videos and 3D motion and task-specification annotations. Its dataset statistics describe broad task and object coverage, while its comparison identifies several features relative to existing datasets.

4.1. Data and Annotation List

  • Annotations include synchronized multi-view RGB videos, 3D pose and shape for the upper body, hands, and objects, and task specifications including affordances, Primitives, Complex Tasks, PDGs, commentary, and completion sequences.The paper reports cross-dataset validation for 3D hand keypoints and physical-property checks for grasping poses.Source: §4.1 · p. 5

4.2. Dataset Statistics

  • OAKINK2 contains 627 bimanual interaction sequences, including 363 Primitive and 264 Complex Task sequences, across 75 objects, 39 affordances, and 60 Primitive types.The dataset includes four scenarios, 38 long-horizon complex manipulation goals instantiated as 150 Complex Tasks, and 4.01M image frames; the paper highlights affordance grounding, long-horizon goals, hand-and-object 3D pose and shape, and Primitive-based task decomposition relative to other datasets.Source: §4.2 · p. 5

5. Selected Applications

The selected applications cover hand-mesh reconstruction, task-aware hand-motion generation, and a staged framework for completing complex tasks. The latter combines Primitive planning, object-trajectory retrieval, and hand-motion generation.

5.1. Hand Mesh Reconstruction

  • Hand-mesh reconstruction is benchmarked in single-view and multi-view settings using joint- and vertex-position errors, root-relative metrics, and keypoint AUC.The sequence-level split is approximately 70% train, 5% validation, and 25% test; Table 2 presents the evaluation results in millimeters.Source: §5.1 · p. 6, Table 2 · p. 6

5.2. Task-aware Motion Fulfillment (TaMF)

  • TaMF generates hand-motion sequences from object trajectories and textual task descriptions, with MF-MDM generating motion and refining hand-object interactions.MF-MDM G conditions on task text and object trajectories; MF-MDM R adds hand-object distances to refine generated motion.Source: §5.2 · pp. 6–7, Figure 5 · p. 6
  • MF-MDM uses sequential merging to aggregate object trajectories and diffused hand motion, while its refinement model also incorporates hand-object distances.The evaluation includes contact ratio, solid intersection volume, motion smoothness, FID, and a perceptual study; Table 3 reports generated-motion evaluations.Source: §5.2 · pp. 6–7, Table 3 · p. 7

5.3. Complex Task Completion (CTC)

  • CTC decomposes complex task completion into Primitive planning, object-trajectory retrieval, and hand-motion generation for each Primitive.GPT-4 generates a program that orders Primitives; retrieved expert object trajectories are re-targeted to the scene, and TaMF generates hand motion before the trajectories are joined by interpolation.Source: §5.3 · pp. 7–8
  • For CTC Primitive planning, GPT-4 creates a program encoding the Primitive Dependency Graph, and a dependency checker tests whether it completes the task without constraint violations.An Oracle retrieves and re-targets expert object trajectories; one TaMF model handles all Primitives, whose motions are connected by interpolation.Source: §5.3 · pp. 7–8, Figure 8 · p. 8

6. Future Works

The authors anticipate OAKINK2 supporting language-manipulation pre-training and, potentially, end-to-end text-to-manipulation learning. They also describe transferring its demonstrations and scenarios to other embodiments and simulation environments.

Future applications

  • The authors expect OAKINK2 to support large-scale language-manipulation pre-training and potentially end-to-end text-to-manipulation generation.They identify improving multimodal models for Complex Task Completion as a pre-training aim.Source: §6 · p. 8
  • OAKINK2 demonstrations of Primitives can be re-targeted to different embodiments, and its interaction scenarios can be integrated into simulation environments.The authors describe these as potential supports for embodied manipulation learning.Source: §6 · p. 8

Supplementary Materials

The platform calibration procedure covers the MoCap and multi-camera systems, which are synchronized using ROS2 software tools.

A.1. Platform Calibration & Synchronization

  • The MoCap system is calibrated with a vendor wand, while ArUco cubes calibrate the multi-camera system.Reflective markers on the cameras are tracked by the MoCap system.Source: §A.1 · p. 13
  • ROS2 software synchronization tools synchronize the MoCap and multi-camera systems.The passage identifies the tools as bundled with ROS2.Source: §A.1 · p. 13
  • The multi-camera system’s reflective markers are tracked by the MoCap system during calibration.The cameras are calibrated using ArUco cubes.Source: §A.1 · p. 13

A.2. Data Cleaning

OAKINK2 manually cleans MoCap marker data to address occlusion and labeling errors, then fits SMPL-X shape and pose in two stages to reconstruct human surfaces.

A.2. Data Cleaning

  • Three annotators remove ghost points, split overlapping markers, correct mislabeled markers, fill short trajectory gaps, and use multi-view triangulation to recover markers under extreme occlusion.They clean articulated parts, rigid bodies, the human body, and both hands in sequence.Source: §A.2 · p. 13

A.3. Human Pose and Surface

  • The two-stage fitting first estimates SMPL-X shape and marker-to-surface correspondence, then fits per-frame pose using the resulting shape and mapping.The pipeline uses PyTorch and gradient-based optimization.Source: §A.3 · pp. 13–14
  • Per-frame pose fitting combines observed-marker reconstruction error with body-pose prior, hand-anatomy, hand-object intersection, and auxiliary regularization costs.Regularization also includes velocity terms to smooth annotated trajectories.Source: §A.3 · p. 14

B.1. Task-specific Subsets

OAKINK2 forms task-specific subsets using visibility and grasp criteria, tailoring selected frames or views to different reconstruction and motion tasks. Its task demonstrations span multiple scenarios and Primitive Task types.

B.1. Task-specific Subsets

  • Subset selection uses hand/object visibility and defines a grasp as hand proximity of ≤5 mm plus object lift of ≥5 mm from its initial height.Combined and individual segmentation masks are checked to determine visibility.Source: §B.1 · p. 14
  • Single-view and multi-view hand subsets support hand mesh recovery, while the hand-object subset selects visible hands with grasped objects for pose estimation or reconstruction.The multi-view subset combines camera views when hands are visible in a majority of views.Source: §B.1 · p. 14
  • Approach-and-grasp frames provide auxiliary input for motion fulfillment and oracle object-trajectory retrieval, while grasp frames form a dedicated grasp subset.Primitive Task frames from grasp through task completion form the motion-task subset.Source: §B.1 · pp. 14–15

C. Dataset Evaluation

  • Figure 10 shows Primitive Task demonstration proportions across scenarios and lists recorded Primitive Tasks with their quantity distributions.The full Primitive and Complex Task lists are in Tables 7 and 8.Source: Figure 10 · p. 16

C.1. Cross-Dataset Validation

Cross-dataset validation reports consistent metric improvements when OAKINK2 training data is included. A separate physical-property assessment finds OAKINK2-Grasp annotation quality on par with OakInk subsets under the evaluated grasping conditions.

C.1. Cross-Dataset Validation

  • Consistent improvements in MPJPE and MPVPE after Procrustes analysis on OakInk-image (SP2) support OAKINK2's complementarity with existing datasets.The comparison trains a single-view hand mesh recovery model on FreiHAND, OAKINK2-H-SV, or their mixture.Source: §C.1 · p. 16, Table 5 · p. 16

C.2. Physical Property Assessment

  • OAKINK2-Grasp annotation quality is reported on par with OakInk-Core and OakInk-Shape despite relying primarily on MoCap rather than hybrid manual and MoCap annotation.Assessment samples were restricted to interactions where objects were grasped and lifted.Source: §C.2 · p. 16, Table 6 · p. 16

D.1. Task-aware Motion Fulfillment

The section defines physical and motion-quality metrics for assessing captured or generated interactions. These measures address contact, intersection, smoothness, and trajectory realism.

D.1. Task-aware Motion Fulfillment

  • Contact Ratio measures the share of trajectory frames with hand-object minimum distance within 5 mm, while Solid Intersection Volume measures object volume inside the hand surface.The object mesh is voxelized into 100^3 voxels for the intersection calculation.Source: §D.1 · p. 16
  • Power Spectrum KL divergence of Joints assesses motion smoothness through acceleration-distribution variance between predicted and ground-truth joint sequences, reported in both directions.The metric uses hand joints, so its values differ in range from full-body-joint measurements.Source: §D.1 · pp. 16–17
  • Fréchet Inception Distance evaluates trajectory realism using 64-dimensional motion embeddings from a transformer encoder trained to classify motion trajectories by category.The encoder is applied to both ground-truth and generated trajectories.Source: §D.1 · p. 17

E. Application: Complex Task Completion

The application evaluates LLM-generated Primitive plans against task dependencies and illustrates both planning outcomes and motion generation. Its results show lower planning success as tasks require more Primitives.

E. Application: Complex Task Completion

  • GPT-4 turns scene and task narratives into Python code that identifies objects, affordances, and trajectories, then sequences Primitive executions.An Entity-Component-System architecture separates object data or state from Primitive systems.Source: E. Application: Complex Task Completion · p. 17
  • Primitive Planning succeeds in 36% overall, 44% for tasks with three or fewer Primitives, 20% for tasks with three to five, and 0% for tasks with more than five.Test scenes perturb object positions while preserving task objectives and object-state descriptions.Source: E. Application: Complex Task Completion · p. 17
  • Generated plans can add unnecessary affordances that block execution; one example places the beaker before the asbestos mesh, violating the required order.The correct path requires placing the mesh before heating the beaker.Source: E. Application: Complex Task Completion · p. 17, §F.2 · p. 26
  • An alternative motion pipeline chains GNet static grasps, MNet approach and retreat trajectories, and INet in-between interaction motion.Figure 11 contrasts oracle trajectories with sequentially generated motion.Source: E. Application: Complex Task Completion · p. 18, Figure 11 · p. 18

F.1. Task List

  • Recorded tasks include short Primitive combinations such as scoop, unscrew, pour, and screw, as well as preparation goals such as hot soup and fruit tea.Source: §F.1 · p. 20

F.2. Visualization

  • The prompt template requests executable task code and demonstrates oracle trajectory queries; its entities, components, and systems correspond to objects, object state or affordances, and motion-generator interfaces.The visualization section also includes a successful planning example and a dependency-order failure example.Source: §F.2 · pp. 25–26

Related papers

Loading 2403.19417v1…