Source-linked AI summary
AKB-48: A Real-World Articulated Object Knowledge Base
Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Yang Han, Cewu Lu
TL;DR
Existing articulated-object resources emphasize structural annotations, with less attention to appearance, physics properties, and semantics. The paper builds AKB-48 with FArM and uses it to benchmark AKBNet; in manipulation, reported success is lower with predicted than ground-truth object state.
Problem
Existing articulated-object datasets focus more on structure than appearance, physics properties, and semantics, leaving object knowledge gaps relevant to tasks such as reconstruction and manipulation.
Source: §1 · pp. 1–2Method
The authors build AKB-48 from real-world articulated models organized with ArtiKG and FArM, then use the dataset to benchmark AKBNet for category-level pose estimation, reconstruction, and manipulation.
Source: §1 · p. 2Results
Manipulation success is lower with predicted than ground-truth object state: opening 72.5% vs 40.2%, and pulling 98.7% vs 44.6%.
Source: §5.4 · p. 8Takeaways & Limitations
The authors present AKB-48 as a platform for future articulation research in computer vision and robotics.
Source: §1 · p. 2For expensive objects, collection is limited to items available from homes or basic measurements propagated to existing simulated objects.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Human life is populated with articulated objects. A comprehensive understanding of articulated objects, namely appearance, structure, physics property, and semantics, will benefit many research communities. As current articulated object understanding solutions are usually based on synthetic object dataset with CAD models without physics properties, which prevent satisfied generalization from simulation to real-world applications in visual and robotics tasks. To bridge the gap, we present AKB-48: a large-scale Articulated object Knowledge Base which consists of 2,037 real-world 3D articulated object models of 48 categories. Each object is described by a knowledge graph ArtiKG. To build the AKB-48, we present a fast articulation knowledge modeling (FArM) pipeline, which can fulfill the ArtiKG for an articulated object within 10-15 minutes, and largely reduce the cost for object modeling in the real world. Using our dataset, we propose AKBNet, a novel integral pipeline for Category-level Visual Articulation Manipulation (C-VAM) task, in which we benchmark three sub-tasks, namely pose estimation, object reconstruction and manipulation. Dataset, codes, and models will be publicly available at https://liuliu66.github.io/articulationobjects/.
1. Introduction
Existing articulated-object datasets emphasize structure, while appearance, physics, and semantic knowledge are less represented. The paper introduces AKB-48 and FArM, then uses the dataset to benchmark AKBNet for category-level visual articulation manipulation.
1. Introduction
- Existing datasets emphasize structural information but pay less attention to appearance, physics properties, and semantics, limiting the object knowledge available for related tasks.The paper links this missing knowledge to unsatisfactory generalization for learning models.Source: §1 · pp. 1–2
- FArM combines sensors and turntables, a structural-semantic annotation GUI, and real-world physics experiments; annotation takes 10–15 minutes per object and saves 33-fold money and 5-fold time budgets.The paper estimates an object costs about $3 to buy and reports the savings relative to CAD modeling.Source: §1 · p. 2
- AKBNet combines category-level articulated-object pose estimation, part-shape reconstruction, and manipulation-policy reasoning for opening and pulling tasks.Its manipulation module reasons over perceived articulation information, including part segments, poses, joint properties, and full meshes.Source: §1 · p. 2
- The pose module defines categories semantically, allowing unseen objects in one category to have different kinematic structures.This extends prior category-level pose settings that assume a shared kinematic structure within a category.Source: §1 · p. 2
- AKBNet is evaluated both module by module, using ground-truth inputs from the preceding module, and systematically, using the preceding module’s outputs.The authors note that they cannot benchmark every task supported by AKB-48.Source: §1 · p. 2
- AKB-48 contains 2,037 real-world articulated models across 48 categories, with rich annotations organized in the ArtiKG knowledge graph.The authors describe it as the first large-scale articulation dataset with rich annotations collected from the real world.Source: §1 · p. 2
2. Related Work
Prior work ranges from large CAD repositories and annotated shape datasets to real-world instance collections and vision or robotics tasks. The paper positions AKB-48 against limits in articulated structure, category coverage, and dataset scale.
3D Model Repositories and Datasets
- ShapeNet [4] contains many shapes but treats its models as rigid; PartNet [24] adds hierarchical semantic parts, while PartNet-Mobility [31] and Shape2Motion [30] annotate joint properties.PartNet is described as annotating a subset of ShapeNet [4].Source: §2 · pp. 2–3
- Internet-sourced CAD datasets can scale quickly, but new categories or kinematic structures require artists to build models from scratch; real-world YCB and RBO collections are small-scale and instance-level.The paper identifies this scale as difficult for category-level articulation tasks requiring generalization across instances.Source: §2 · p. 3
Articulation-related Tasks
- Vision work addresses category-level recognition, segmentation, and pose, while robotics emphasizes interaction and manipulation; work bridging them still faces a small-scale data issue.The paper proposes AKBNet for category-level articulation in response to this context.Source: §2 · p. 3
3. Articulation Knowledge Base, AKB-48
This section describes AKB-48’s knowledge representation, real-world object selection, and FArM annotation workflow, then analyzes dataset coverage.
3.1. Articulated Object Knowledge Graph, ArtiKG
- ArtiKG organizes appearance meshes and textures, joint-and-part structure, semantic labels, and per-part physical attributes for articulated objects.Kinematic parts and semantic parts can differ: a mug handle may be semantic without being a separate kinematic part.Source: §3 · p. 4, §3.1 · p. 4
3.2. Object Selection: Real-world Scanning v.s CAD Modeling
- Object selection weighs annotation accuracy and cost: scanning captures appearance and physical properties, while CAD can represent internal structures that scanning focuses less on.For expensive objects, the authors sometimes measure basic information and propagate it to existing simulated objects; their survey estimates CAD modeling takes over 120 minutes and costs over $100 per object.Source: §3.2 · p. 4
3.3. Fast Articulation Knowledge Modeling (FArM) Pipeline
- FArM addresses annotation affordability through a guided 3D modeling workflow divided into three sub-processes.The section frames affordability as the remaining challenge after deciding what knowledge and objects to annotate.Source: §3.3 · p. 5, §3.3.2 · p. 5
3.3.1 Model Acquisition Equipment.
- The acquisition setup combines a 3D scanner, RGB-D camera, rotating turntables, and lift bracket; each object can be scanned within 5 minutes.Source: §3.3.1 · p. 5
3.3.2 Articulation Modeling
- Annotators align scans to category-canonical space, segment parts using multi-view boundary polygons, and arrange joints as a tree with animated motion for refinement.Joint locations and axes are annotated as a 6D vector.Source: §3.3.2 · p. 5
3.3.3 Physics Annotation
- FArM records per-part mass, inertia, material, and friction; inertia is computed from combinations of simplified primitive shapes.For inseparable parts, drainage estimates volume; material parameters are sourced from Machinery’s Handbook.Source: §3.3.3 · p. 5
3.4. Dataset Analysis
- AKB-48’s category selection prioritizes common daily objects, within-category variety, and objects with useful functions, with manipulation performance prioritized.Source: §3.4 · p. 5
- AKB-48 offers around 126K-triangle average surfaces and real textures versus synthetic models’ thousands of triangles and synthetic textures; its object count is comparable to PartNet-Mobility’s CAD models.The comparison also highlights part and joint annotations and physical information for each model.Source: §3.4 · pp. 5–6, Table 2 · p. 6
4. AKBNet
AKBNet is an integral pipeline for category-level visual articulation manipulation, taking an RGB-D image with detected 2D boxes to estimate part poses, reconstruct geometry, and reason about interaction policy. Its modules address pose, shape, and manipulation, with physics-property prediction also included.
4.1. Pose Module
- The pose module extracts point-cloud features to predict part segmentation, part-level NOCS, joint type, joint location, and joint axis, then recovers each rigid part’s 6D pose under kinematic constraints.Its loss combines segmentation and NOCS terms; the prose also describes losses for joint type, location, and axis.Source: §4.1 · pp. 6–7
4.2. Shape Module
- The shape module reconstructs full articulated geometry from a partial point cloud using shape and joint embeddings, with SDF values as supervision.At inference, predicted embeddings are used to reconstruct the full mesh.Source: §4.2 · p. 7
4.3. Manipulation Module
- The manipulation module trains reinforcement-learning agents for opening revolute joints and pulling prismatic joints using object and agent state representations.The object state includes pose, joint properties, and full geometry; actions specify end-effector translation and gripper width.Source: §4.3 · p. 7
- A physics-prediction submodule uses a three-layer MLP with parallel branches to estimate each part’s mass, friction, and inertia moment.The submodule is trained with L2 loss.Source: §4.3 · p. 7
5. Experiments
Experiments evaluate pose estimation, shape reconstruction, and manipulation, using both module-level and systematic settings. Results include strong pose and joint predictions, while manipulation success is lower with predicted object state than with ground-truth state.
5.1. Experimental Setup
- Pose and shape modules use 100K generated RGB-D images for training and 10K real-world images split evenly between fine-tuning and testing; manipulation agents train on 68 instances and test on 32.The evaluation reports part-based and joint-based pose metrics, Chamfer-L1 distance for shape, and success rate for manipulation.Source: §5.1 · p. 7
5.2. Pose Module Performance
- 9.8 rotation error, 0.021 translation error, and 53.6 3D IoU are reported for pose estimation; AKBNet is reported higher than NPCS and A-NCSH.Joint type accuracy is 94.6%, with joint-axis and joint-location errors of 8.1 and 0.019, respectively.Source: Table 3 · p. 8, §5.2 · p. 8
5.3. Shape Module Performance
- 4.2 Chamfer-L1 distance is reported with ground-truth joint state; using predicted joint state yields a distance 3.3 higher.Table 4 distinguishes predicted-state results from ground-truth joint-state input.Source: Table 4 · p. 8, §5.3 · p. 8
5.4. Manipulation Module Performance
- AKBNet predicts per-part mass, friction, and inertia moment; the authors state these predictions can enable force sensing for AKB-48 objects in simulation.They describe force controlling as a potential application.Source: §5.4 · p. 8
- 72.5% opening and 98.7% pulling success with ground-truth object state, versus 40.2% opening and 44.6% pulling with predicted state.The comparison uses TQC+HER; Table 5 reports success rates for the manipulation tasks.Source: Table 5 · p. 8, §5.4 · p. 8
6. Conclusion and Crowd-Sourcing Data-Collection Invitation
The conclusion presents AKB-48 as an articulated-object knowledge base and C-VAM benchmark, while noting that some categories may remain uncollected. It proposes releasing FArM and developing an online platform with crowdsourced data collection.
6. Conclusion and Crowd-Sourcing Data-Collection Invitation
- AKB-48 is presented as a large-scale articulated-object knowledge base and benchmark for the C-VAM task.The conclusion frames it as a resource for articulation problems.Source: §6 · p. 8
- The authors acknowledge that AKB-48 may not include some articulated-object categories, despite covering many categories found in daily life.Source: §6 · p. 8
- Future plans include releasing FArM to collect more articulated objects, supporting scanned shapes, and building an online platform for crowdsourced data collection.Source: §6 · p. 8