Source-linked AI summary
The KIT Motion-Language Dataset
Matthias Plappert, Christian Mandery, Tamim Asfour
TL;DR
Research linking human motion and natural language lacked a standardized, openly available dataset for development and evaluation. The paper introduces the open, extensible KIT Motion-Language Dataset, combining multi-source motion capture with crowdsourced language annotations and systematic perplexity-based selection. As of October 10, 2016, it contained 3911 motions and 6278 annotations, while the authors report that perplexity-based selection mitigates under-representation and erroneous or controversial annotations.
Problem
Existing research lacked a standardized, openly available dataset combining human motion and natural language for system development and evaluation.
Method
The authors aggregate multiple motion databases into a unified representation, collect language annotations with a web-based crowdsourcing tool, and use perplexity-based selection during annotation.
Results
3911 motions and 6278 natural-language annotations were available as of October 10, 2016.
Takeaways & Limitations
The dataset’s open, extensible, unified design supports more transparent and comparable research and systematic benchmarking in this area.
Takeaways & Limitations
The Motion Annotation Tool supports only single-subject motions without objects, limiting contextual coverage for manipulation tasks.
Abstract
from arXiv · showhide
Linking human motion and natural language is of great interest for the generation of semantic representations of human activities as well as for the generation of robot activities based on natural language input. However, while there have been years of research in this area, no standardized and openly available dataset exists to support the development and evaluation of such systems. We therefore propose the KIT Motion-Language Dataset, which is large, open, and extensible. We aggregate data from multiple motion capture databases and include them in our dataset using a unified representation that is independent of the capture system or marker set, making it easy to work with the data regardless of its origin. To obtain motion annotations in natural language, we apply a crowd-sourcing approach and a web-based tool that was specifically build for this purpose, the Motion Annotation Tool. We thoroughly document the annotation process itself and discuss gamification methods that we used to keep annotators motivated. We further propose a novel method, perplexity-based selection, which systematically selects motions for further annotation that are either under-represented in our dataset or that have erroneous annotations. We show that our method mitigates the two aforementioned problems and ensures a systematic annotation process. We provide an in-depth analysis of the structure and contents of our resulting dataset, which, as of October 10, 2016, contains 3911 motions with a total duration of 11.23 hours and 6278 annotations in natural language that contain 52,903 words. We believe this makes our dataset an excellent choice that enables more transparent and comparable research in this important area.
I. INTRODUCTION
Linking human motion with natural language is motivated by multimodal robot teaching, intuitive motion retrieval, and language-driven motion synthesis. The paper addresses the lack of a publicly available combined dataset by proposing the KIT Motion-Language Dataset and a systematic annotation process.
- Motivation: Natural language can express complex motions and execution parameters, supporting richer human-robot interaction than motion alone.The example “A person wipes the table with their right hand 5 times” specifies both the activity and its parameters.
- Applications: Natural-language descriptions could support full-text motion retrieval and, in generative systems, synthesis of requested motions.
- Research gap: No publicly available dataset previously combined human motion with natural-language descriptions, making results difficult to compare across studies.Existing whole-body motion databases provided tags or keywords rather than textual descriptions.
- Contribution: The KIT Motion-Language Dataset combines human motion and natural-language descriptions in an openly available dataset designed for systematic benchmarking.The paper also documents data acquisition, dataset contents, and annotation methods.
- Contribution: Perplexity-based selection prioritizes under-represented motions and motions with contradicting annotations for further annotation.The method was developed during data collection to make motion selection systematic and mitigate both problems.
III. HUMAN MOTION
This section introduces human motion as the dataset’s first modality and describes its acquisition procedures and unified whole-body representation.
- III. HUMAN MOTION: Human motion is the first dataset modality discussed in this section.
- III. HUMAN MOTION: The section briefly presents how motion data are acquired.
- III. HUMAN MOTION: The section describes a unified representation of whole-body motion used in the dataset.
A. Acquisition of Human Motion
The dataset aggregates optical marker-based motion from multiple sources and converts it into the unified Master Motor Map representation. This representation reduces dependence on capture setup and marker set, while supporting future converters.
- Acquisition: The dataset currently contains motion recorded with optical marker-based capture systems.
- Acquisition: The KIT and CMU databases use different marker sets and sampling frequencies, with recordings stored in industry-standard C3D files.KIT uses 56 markers at 100 Hz; CMU uses 41 markers at 60 Hz or 120 Hz.
- Unified representation: The Master Motor Map converts raw recordings from different capture systems into a standardized reference model of the human body.For marker-based data, virtual markers and inverse kinematics are used to minimize mean squared distances between corresponding markers.
- Unified representation: Updating marker placement on the 3D model adapts the conversion to data sources with different marker sets.
- Implementation: The open-source MMM framework stores converted recordings in an XML-based format and provides converters and visualization tools.
IV. NATURAL LANGUAGE ANNOTATIONS
The authors collect natural-language motion descriptions with a purpose-built web tool and design the annotation interface for accessibility and accurate observation. Interactive 3D visualization lets annotators inspect motions from suitable viewpoints and focus on relevant body parts.
- Annotation tool: The Motion Annotation Tool is a web-based system created specifically to collect natural-language motion annotations.
- Annotation tool: The tool is designed for broad accessibility through web technologies and support for smartphones and tablets.
- User interface: Interactive visualization is needed because motions differ in stationarity, spatial travel, body involvement, and laterality.
- User interface: A browser-based WebGL and three.js visualizer allows annotators to rotate and zoom the virtual motion from an appropriate perspective.
- Annotation interface: Annotators enter one complete English sentence describing the motion as accurately as possible, supported by example sentences.
B. Gamification
The Motion Annotation Tool uses gamification to motivate voluntary annotators, while motion selection combines random initialization with perplexity-based adaptive sampling.
- Gamification: Gamification was added to make voluntary annotation more interesting and fun.The tool used game-design elements alongside an easy-to-use interface.
- Gamification: The tool included progress tracking and a leveling system to make annotator progress more visible.Users advanced through levels from Novice toward Research Assistant and Junior Scientist.
- Gamification: The authors report that personal progress and comparison with other annotators strongly motivated most participants and supported data acquisition.They acknowledge that this effect was difficult to quantify.
- Motion Selection: The annotation process initially selects unannotated motions randomly with equal probability.This strategy initializes the process before every recorded motion has received an annotation.
- Motion Selection: Perplexity measures annotation surprise using a 4-gram language model, and motion perplexity averages the perplexities of its annotations.Higher perplexity indicates candidates for further annotation.
- Motion Selection: Motions are sampled from a discrete probability distribution weighted by motion perplexity and temporarily excluded after annotation.Perplexities are recomputed every hour, allowing selection to adapt continuously.
V. DATASET
This section presents the resulting dataset by describing its structure, contents, statistics, and properties relative to other datasets.
- The resulting dataset is examined through its structure and contents.
- The section reports statistics describing the data it contains.
- The dataset’s properties are compared with those of other datasets.
A. Structure
Each dataset entry combines raw and converted motion data, natural-language annotations, and metadata, with standardized formats and versioned distribution supporting reuse.
- Each entry contains raw motion, MMM-converted motion, natural-language annotations, and additional metadata.Figure 1 shows exemplary motions with their annotations.
- Raw motion formats depend on the source, while current recordings use industry-standard C3D files from optical marker-based capture.The format supports existing visualization and parsing tools.
- Annotations use a one-to-many relationship and are stored as human-readable arrays of strings in JSON files.
- Metadata links each entry and its annotations to the Motion Annotation Tool through unique, permanent identifiers.
- Researchers can use the identifiers to define their own training, validation, and test subsets rather than adopting a dataset-provided split.Perplexity scores are also available for pruning annotations.
- The dataset is distributed as downloadable ZIP archives, including dated older releases and current technical documentation.
B. Content
The dataset contains thousands of diverse motions and English annotations contributed by many volunteers, spanning multiple motion categories and variations.
- 3911 motions total 11.23 hours, sourced from KIT, CMU, and EKUT recordings.The average motion length is 10.33 ± 13.38 seconds.
- The motions cover gesticulation, locomotion, manipulation, and sports, with variations in speed, body part, direction, and repetition.
- 6278 English sentences contain 52 903 words, with an average sentence length of 8.43 ± 4.34 words and vocabulary size 1623.
- Annotations were written by 110 volunteers, including contributors located in Germany and France.
- The authors characterize the dataset as large and diverse across motions, subjects, and natural-language annotations.
C. Comparison with other Datasets
The dataset is presented as an open, extensible, unified, and thoroughly documented resource intended to improve transparency and comparability in motion-language research.
- The dataset is completely open and can serve as a benchmark for future methods and approaches.
- 3911 motions and 6278 sentences make the dataset, according to the authors, the largest and most diverse by far.The dataset is designed to grow through additional motion sources and continued crowd-sourced annotation.
- The unified MMM representation abstracts from capture methods, marker sets, and file formats.This supports updates when new data sources are added.
- Extensive methodological and technical documentation helps researchers work with and assess the dataset.
A. Annotation Perplexity
Annotation perplexity is used to characterize annotation quality and identify unusual or potentially erroneous annotations. Lower-perplexity annotations are typical and grammatical, whereas higher-perplexity annotations often indicate unusual wording, sparse motion types, or errors.
- The ten lowest-perplexity annotations all describe walking in various forms and contain no spelling or grammatical errors.This matches the large quantity of locomotion recordings in the motion data.
- The highest-perplexity annotations include note-form fragments, comments about broken motion, and a presumed premature submission.
- Annotation quality correlates with perplexity: walking descriptions generally worsen as perplexity increases.
- Walking keywords mostly occur at low perplexity, while “dance” and especially “waltz” occur at higher perplexity ranges.The authors relate this pattern to fewer dance recordings than walking recordings.
- The analysis identifies perplexity as a useful measure for selecting motion candidates for future annotation.
B. Perplexity-based Motion Selection
Perplexity-based selection replaces random selection with sampling biased toward motions whose annotations have higher mean perplexity. After the switch, both mean perplexity and its variability decrease, supporting more systematic annotation.
- The tool switched from random-based selection to perplexity-based selection on April 25 2016.Random selection chose uniformly among motions with the fewest annotations; perplexity-based selection weighted motions by mean perplexity.
- Before the switch, random selection produced growing variability because locomotion was overrepresented while other motions remained high-perplexity.
- The mean perplexity stabilizes around 1000 annotations, while the standard deviation remains high during the random-selection phase.
- After perplexity-based selection begins, mean perplexity and standard deviation decrease significantly as more annotations are collected.The decrease is rapid initially and slows over time.
- The results demonstrate the effectiveness of perplexity-based selection.
VII. CONCLUSION
The paper presents the KIT Motion-Language Dataset as a large, open, extensible resource for linking human motion and natural language, with a unified motion representation and systematically collected annotations. It also introduces perplexity-based motion selection to improve annotation coverage and stability, while identifying important directions for future dataset expansion.
- The KIT Motion-Language Dataset is freely available, larger and more diverse than previously used datasets, and designed for continuous growth.Its motion representation is independent of the concrete motion capture system and data source.
- As of October 10, 2016, the dataset contained 3911 motions and 6278 natural-language annotations.
- The dataset is intended as a candidate for systematic benchmarking in motion-language research, supporting more transparent evaluation and comparison.
- Perplexity-based motion selection detects under-represented or controversial annotations and decreases variance in perplexity.The authors report that the approach acts in a stabilizing manner during crowd-sourced annotation.
- Future work includes adding motion data and supporting multiple subjects and objects, especially because objects provide additional context for manipulation tasks.The authors also plan to incorporate user feedback and investigate motion-data measures for prioritizing annotation.