PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks

Matthias Plappert, Christian Mandery, Tamim Asfour

arXiv:1705.06400v2cs.LGcs.CLcs.ROstat.ML

TL;DR

The paper addresses the challenge of linking human whole-body motion and natural language without symbolic motion primitives or manual segmentation. It proposes deep recurrent sequence-to-sequence models for both directions and reports that they generate varied realistic motions from sentences and detailed descriptions from motion.

  • Problem

    Existing motion-language approaches often require symbolic motion representations defined in advance or obtained through complex segmentation, despite the importance of linking motion and language for robot interaction.

  • Method

    Deep recurrent sequence-to-sequence models learn a probabilistic bidirectional mapping between MMM whole-body motion and natural-language descriptions without explicit motion segmentation or manual feature engineering.

  • Results

    The system generates rich, detailed descriptions from many human motions and a similarly broad variety of realistic motions from natural-language descriptions.

  • Takeaways & Limitations

    The learned distributed representations support bidirectional motion-language generation across varied human whole-body behaviors.

  • Takeaways & Limitations

    The evaluation discusses ambiguity in judging generated motions and notes that user studies alone would make comparison with other systems difficult.

Abstract

from arXiv · show

Linking human whole-body motion and natural language is of great interest for the generation of semantic representations of observed human behaviors as well as for the generation of robot behaviors based on natural language input. While there has been a large body of research in this area, most approaches that exist today require a symbolic representation of motions (e.g. in the form of motion primitives), which have to be defined a-priori or require complex segmentation algorithms. In contrast, recent advances in the field of neural networks and especially deep learning have demonstrated that sub-symbolic representations that can be learned end-to-end usually outperform more traditional approaches, for applications such as machine translation. In this paper we propose a generative model that learns a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks (RNNs) and sequence-to-sequence learning. Our approach does not require any segmentation or manual feature engineering and learns a distributed representation, which is shared for all motions and descriptions. We evaluate our approach on 2,846 human whole-body motions and 6,187 natural language descriptions thereof from the KIT Motion-Language Dataset. Our results clearly demonstrate the effectiveness of the proposed model: We show that our model generates a wide variety of realistic motions only from descriptions thereof in form of a single sentence. Conversely, our model is also capable of generating correct and detailed natural language descriptions from human motions.

1. Introduction

The paper targets multimodal human–robot interaction by linking whole-body motion with natural-language instructions. It uses sequence-to-sequence deep learning to learn this bidirectional mapping without manual motion segmentation or feature engineering.

  • The approach is motivated by teacher–student interaction in which demonstrations and corrective language jointly convey a task.
  • Natural language can specify a motion, body part, and repetition count, enabling richer communication than motion observation alone.
  • The proposed system learns a bidirectional mapping between MMM joint-space motion and complete English descriptions using sequence-to-sequence learning.
  • The model can generate rich descriptions from observed motion and varied motions from language, including handedness and speed variations.

2. Related work

Prior work combines motion and language through symbolic motion representations, probabilistic models, and language models. The paper is situated within broader deep-learning research linking language to other modalities.

  • Related motion-generation methods include HMMs, conditional restricted Boltzmann machines, Gaussian mixture models, and dynamic movement primitives.
  • Earlier motion-language systems segment motion into primitives and map symbolic motion spaces to word spaces.
  • Other approaches use HMM-based motion primitives with latent semantic variables to connect complete sentences and human motion.
  • Parametric HMMs can use adverbs to parameterize motion, but one reported evaluation covered only motion generation from text.
  • Deep learning has also been used to combine natural language with images and videos, providing a broader multimodal context.

3. Data representation

The paper represents motion in MMM joint space and language as token sequences, with padding and special activity or sentence markers supporting variable-length generation. These choices reduce dependence on recording coordinates and enable neural sequence processing.

  • 3.1. Human whole-body motion: Marker trajectories are accurate but require normalization, vary with marker sets, and are high-dimensional because each marker contributes three coordinates.
  • 3.1. Human whole-body motion: MMM converts marker trajectories into a 44-joint kinematic representation that is independent of the recording coordinate system and abstracts marker details.
  • 3.1. Human whole-body motion: The motion representation adds a binary activity feature that identifies the active portion of padded sequences and signals whether generation is ongoing.
  • 3.1. Human whole-body motion: Motion sequences are down-sampled from 100 Hz to 10 Hz and augmented through offset-based splitting into ten sequences during training.
  • 3.2. Natural language descriptions: Each sentence is normalized, tokenized, indexed, and padded with SOS, EOS, and PAD symbols, while words are represented over a vocabulary of size V.
  • 3.2. Natural language descriptions: Word embeddings address the vocabulary-dependent dimensionality of one-hot representations, and this projection is learned end-to-end in the model.

4. Model

Both mapping directions use a shared sequence-to-sequence architecture that encodes an input sequence into a latent context vector and decodes the target sequentially. The motion-to-language model uses bidirectional encoding and word-by-word generation.

  • Motion-to-language model: The motion-to-language model encodes motion with stacked bidirectional RNNs, then generates words with stacked unidirectional RNNs until EOS.
  • Motion-to-language model: A fully connected layer produces output-distribution parameters, which the decoder converts into concrete generated words.
  • Language-to-motion model: The language-to-motion model reverses the modality roles, using a natural-language description to generate the corresponding whole-body motion.

4.1. Shared mode of operation

The shared sequence-to-sequence framework encodes either motion or language into a recurrent context vector, then probabilistically decodes the other modality step by step. Bidirectional recurrent encoders, recurrent decoders, beam search, and end-to-end training support both mappings.

  • 4.1. Shared mode of operation: The framework encodes an input sequence into a latent context vector and generates a target sequence through a probabilistic recurrent decoder.The encoder and decoder are jointly trained with back-propagation through time.
  • 4.1. Shared mode of operation: Bidirectional recurrent encoders process the input in both directions, while GRU-based recurrent networks implement the proposed model.The framework is compatible with other recurrent architectures, including LSTMs and vanilla RNNs.
  • 4.1. Shared mode of operation: Beam search converts probabilistic decoder outputs into concrete sequences while retaining a limited set of likely hypotheses at each step.The previous decoded output is fed back into the decoder together with the constant context vector.

4.2. Motion-to-language mapping

For motion-to-language mapping, a stacked bidirectional RNN encodes a motion sequence, and a recurrent decoder generates a natural-language description word by word. A vocabulary-wide softmax supplies conditional word probabilities, which are trained end-to-end and decoded with beam search.

  • 4.2. Motion-to-language mapping: A motion sequence M is encoded into a context vector, from which the decoder produces a natural-language description word by word.The encoder uses stacked bidirectional RNNs and takes the final processed output as the context vector.
  • 4.2. Motion-to-language mapping: A final softmax layer defines a probability distribution over the vocabulary conditioned on the encoded motion and previously generated words.The decoder connects each recurrent layer to the context vector, previous-word embedding, and preceding recurrent output.
  • 4.2. Motion-to-language mapping: The model is trained end-to-end with categorical cross-entropy computed only over the active, non-padded portion of each description.Back-propagation through time with mini-batches updates the network parameters.
  • 4.2. Motion-to-language mapping: Beam search expands W candidate probability vectors per timestep, retains the W most likely sequences, and ranks completed descriptions by accumulated sequence probability.Decoding continues until candidates emit the EOS token.

4.3. Language-to-motion mapping

For language-to-motion mapping, a bidirectional recurrent encoder conditions a recurrent decoder that predicts continuous joint values and a binary activity flag. A mixture-density output supports probabilistic motion generation, while a surrogate log-loss improves numerical robustness during training.

  • 4.3. Language-to-motion mapping: The language encoder produces a context vector, and stacked recurrent decoder layers use that vector, the previous motion timestep, and preceding-layer outputs.The architecture is designed for language-conditioned motion generation.
  • 4.3. Language-to-motion mapping: The decoder predicts multi-dimensional continuous motion frames rather than discrete words and uses a probabilistic output to generate non-deterministic sequences.The probabilistic decoder also provides likelihood scores for generated sequences.
  • 4.3. Language-to-motion mapping: Each motion frame combines J continuous joint values with a binary flag indicating which parts of the motion are active.The model represents the continuous component with a Gaussian mixture and the discrete component with a Bernoulli distribution.
  • 4.3. Language-to-motion mapping: The output layer predicts mixture weights, mean vectors, variance vectors, and an activity probability for each timestep.These parameters are produced using softmax, linear, softplus, and sigmoid activations, respectively.
  • 4.3. Language-to-motion mapping: The training loss combines the likelihood of ground-truth joint values under the Gaussian mixture with binary cross-entropy for the activity flag.The loss is computed only for the active portion of the motion, excluding padding.
  • 4.3. Language-to-motion mapping: Replacing products of per-joint likelihoods with sums of logarithms makes the surrogate loss more robust to numerical underflow and overflow.The authors report that minimizing the surrogate loss works well and yields the desired results.
  • 4.3. Language-to-motion mapping: Motion decoding samples mixture components, joint values, and activity indicators, then uses beam search with candidate truncation to maintain a fixed number of hypotheses.Unlike discrete decoding, likelihoods for every continuous possibility cannot be computed directly.

5. Experiments

Experiments evaluate the bidirectional model on a filtered KIT Motion-Language Dataset using fixed architectures and training settings. The model produces grammatical, semantically detailed descriptions, while corpus-level Bleu results remain comparable on unseen test motions.

  • Qualitative results: Generated descriptions are complete, grammatically valid English sentences and can express semantically identical motions with varied grammatical structure.Examples include present simple and present continuous descriptions of standing up, as well as interchangeable subject synonyms.
  • Qualitative results: The descriptions preserve motion details such as left-versus-right handedness, the acting foot, and perturbation direction.These distinctions are reported for wiping, waving, stomping, and pushing motions.
  • Quantitative results: Corpus-level Bleu scores correlate with hypothesis log-probability rankings, although scores are lower than typical machine-translation results because motion descriptions permit greater semantic variation.The same ranking pattern persists on the test split, where performance is slightly worse but still comparable.

5.4. Generating motion from language

The language-to-motion model learns to generate realistic whole-body motions from natural-language descriptions, including variations in handedness, speed, and repetitions. Its quantitative evaluation indicates correct motion generation and limited overfitting, while semantic evaluation remains difficult because motions and descriptions are inherently ambiguous.

  • 5.4.2. Qualitative results: The model adjusts generated motions to requested parameters, including right-, left-, and both-hand waving and different walking speeds.These behaviors are illustrated through generated motion examples.
  • 5.4.2. Qualitative results: The model generates unseen repetition counts, such as waving seven times, but language cannot reliably control the number of repetitions.The authors hypothesize that insufficient counting examples in the training data explain this limitation.
  • 5.4.2. Qualitative results: Generated motions lack global pose, dynamic stability, and object-contact constraints because the model represents motion only through joint angles.The wiping example diverges from the imagined table surface over time.
  • 5.4.3. Quantitative results: Quantitative evaluation is difficult because semantically correct motions can have different styles and joint-level characteristics.Squared error between reference and generated motions is therefore ill-suited to semantic correctness, while user studies are costly at dataset scale.
  • 5.4.3. Quantitative results: 65.5% ± 4.5 test performance shows the model generates correct human motion beyond the qualitative examples.Training performance was 71.6% ± 3.0, and the authors report comparable BLEU scores with minimal overfit and generalization to unseen examples.
  • 5.4.4. Analysis of learned representations: Context-vector visualizations form clusters by motion type and walking direction, suggesting that the models encode semantic meaning before decoding sequences.Wiping and mixing are confused when object information is absent from both the model and annotations.

6. Conclusion

The paper concludes that deep recurrent sequence-to-sequence models can map between whole-body motion and natural-language descriptions without explicit motion segmentation. The authors report rich generation and semantically meaningful representations, while identifying sequence compression, limited data, and missing dynamics and contact as remaining boundaries.

  • 6. Conclusion: The proposed probabilistic RNN models learn bidirectional motion-language mappings with distributed representations, minimal preprocessing, and no a-priori motion segmentation.The probabilistic formulation also produces multiple candidate hypotheses for ranking.
  • 6. Conclusion: Experiments demonstrate rich descriptions for varied motions and realistic motions generated from natural-language descriptions.Performance is quantified with corpus-level BLEU scores.
  • 6. Conclusion: The models learn distributed, semantically meaningful latent representations that support generating the desired output sequence.
  • 6. Conclusion: Encoding each input sequence into one context vector becomes problematic as sequence length increases.The authors identify attention mechanisms and hierarchical RNNs as directions for future improvement.
  • 6. Conclusion: The dataset should be enlarged with more complex multi-step motions to test generalization and reduce overfitting risk.The authors specifically mention sequences of distinct steps, such as cooking.
  • 6. Conclusion: Joint-value motion representations omit dynamic properties and environmental or object contact information needed for physically grounded multi-contact motions.

Appendix A. Detailed motion-to-language model architecture

The appendix specifies the detailed motion-to-language architecture by recording tensor shapes at each node. Motion sequences are padded to 300 timesteps at 10 Hz, while language sequences are padded to 41 words.

  • Appendix A. Detailed motion-to-language model architecture: Each architecture node specifies its input and output tensor shape, with batch size 128 as the first dimension.
  • Appendix A. Detailed motion-to-language model architecture: Motion tensors use a time dimension padded to 300 timesteps at 10 Hz, and the final dimension represents layer- or modality-specific features.
  • Appendix A. Detailed motion-to-language model architecture: Language tensors use a time dimension padded to 41 words in the detailed architecture.

Appendix B. Detailed language-to-motion model architecture

The appendix specifies the detailed language-to-motion architecture through tensor shapes at each node. It uses the same batch and sequence-padding conventions: batch size 128, 300 motion timesteps at 10 Hz, and 41 language words.

  • Appendix B. Detailed language-to-motion model architecture: Each architecture node specifies its input and output tensor shape, with batch size 128 as the first dimension.
  • Appendix B. Detailed language-to-motion model architecture: Motion tensors are padded to 300 timesteps at 10 Hz, while language tensors are padded to 41 words.
  • Appendix B. Detailed language-to-motion model architecture: The final tensor dimension denotes features whose meaning depends on the modality and network layer.
Loading 1705.06400v2…