Source-linked AI summary
Text2Action: Generative Adversarial Synthesis from Language to Action
Hyemin Ahn, Timothy Ha, Yunho Choi, Hwiyeon Yoo, Songhwai Oh
TL;DR
The paper addresses generating human actions from language descriptions. It proposes a GAN-based SEQ2SEQ model that encodes text and decodes actions using recurrent networks, trained on extracted 3D poses. The model generates diverse language-conditioned actions, transfers them to a Baxter robot, and is reported to produce more natural actions than prior work.
Problem
The paper studies how to learn the relationship between natural-language descriptions and human actions so robots can perform corresponding behaviors.
Method
A GAN-based SEQ2SEQ model uses an RNN text encoder, attention-conditioned action generator, random noise, and discriminator to synthesize 3D pose sequences.
Results
The model generates diverse human-like actions from the same sentence, transfers generated 3D sequences to a Baxter robot, and produces actions reported as more natural than a prior network.
Takeaways & Limitations
The results support using language-conditioned generative modeling to produce robot actions that correspond to input descriptions and can be understandable to users.
Abstract
from arXiv · showhide
In this paper, we propose a generative model which learns the relationship between language and human action in order to generate a human action sequence given a sentence describing human behavior. The proposed generative model is a generative adversarial network (GAN), which is based on the sequence to sequence (SEQ2SEQ) model. Using the proposed generative network, we can synthesize various actions for a robot or a virtual agent using a text encoder recurrent neural network (RNN) and an action decoder RNN. The proposed generative network is trained from 29,770 pairs of actions and sentence annotations extracted from MSR-Video-to-Text (MSR-VTT), a large-scale video dataset. We demonstrate that the network can generate human-like actions which can be transferred to a Baxter robot, such that the robot performs an action based on a provided sentence. Results show that the proposed generative network correctly models the relationship between language and action and can generate a diverse set of actions from the same sentence.
I. INTRODUCTION
The paper proposes a GAN-based SEQ2SEQ model that learns language–action relationships to generate human actions from sentences, with potential transfer to robots. It trains on pose sequences extracted from real-world annotated videos.
- Related work: Language–action mapping is studied using datasets containing human motions paired with word or sentence annotations.Prior work used HMMs for motion primitives and SEQ2SEQ models for language-to-action learning.
- Model motivation and design: GAN adversarial training is selected to synthesize realistic high-dimensional action data and improve realism over prior work.The generator creates actions to fool the discriminator, while the discriminator distinguishes generated from real data.
- Model motivation and design: The model combines a GAN with an RNN-based SEQ2SEQ architecture consisting of a text encoder and action decoder.The encoder processes a sentence into features, which the decoder converts into corresponding human actions.
- Dataset and evaluation: Because MSR-VTT lacks human pose information, 2D upper-body poses were extracted with CPM, converted to 3D, and used for training.The dataset provides web videos with annotation texts from real-world situations rather than laboratory recordings.
- Dataset and evaluation: 29,770 sentence–action pairs were gathered from MSR-VTT, containing 3,052 descriptions and 2,211 actions.Each sentence description is paired with about 10 to 12 actions.
II. TEXT2ACTION NETWORK
Text2Action encodes sentence embeddings into hidden states and uses attention, random noise, and adversarial discrimination to synthesize pose sequences conditioned on language.
- Text and action representations: Word one-hot vectors are converted into pretrained word embeddings before entering the text encoder.Each embedding is produced by multiplying a word vector by the word2vec-based embedding matrix V.
- Text and action representations: The encoder E maps embedded sentences to hidden states h containing processed language information.The hidden states are then used by both the generator and discriminator.
- Generator: The generator G uses attention-derived language features c and random Gaussian noise z to synthesize an action sequence x.The first pose input x0 is set to the mean of all first training poses.
- Discriminator: The discriminator D uses attention-derived features and an action sequence to determine whether the action is real or generated.Its output lies in [0, 1], returning 1 when the action is identified as real.
- Adversarial training: Generator and discriminator training is formulated as a two-player minimax game over the GAN value function.The generator seeks realistic actions that fool D, while D distinguishes generated actions from real data.
III. NETWORK STRUCTURE
The text encoder is an LSTM-based RNN that transforms the embedded input sequence into hidden states used to condition downstream generation.
- RNN-based Text Encoder: The RNN-based text encoder E encodes the input information e into LSTM hidden states h.These states represent the processed sentence information for subsequent network components.
- RNN-based Text Encoder: Each hidden state has dimension n and is computed by a nonlinear function implemented in an LSTM cell.The formulation uses sigmoid gating and parameterized matrices and vectors.
B. Generator
The generator applies attention-conditioned LSTM decoding with random noise to produce a human pose sequence from encoded language features.
- Generator: Attention converts encoder hidden states h into language feature vectors c, with c_t used to generate pose x_t.The attention mechanism supplies a time-specific feature vector for each output pose.
- Generator: The generator LSTM receives language features, previous poses, and random noise to compute hidden states across the output sequence.Its hidden states are denoted g = {g1, . . . gTo}.
- Generator: The generator computes the output pose at each time step from its recurrent state and conditioned inputs.The model uses learned parameters and an attention-based nonlinear function γ_t.
C. Discriminator
The discriminator evaluates generated or real action sequences against sentence-conditioned feature vectors, producing a scalar realness score. It uses the text encoder’s hidden states and attention-derived features when judging actions.
- The discriminator D takes an action sequence x and sentence-conditioned features c as input to determine whether x is fake or real.The features c are decoded from the text encoder’s hidden state h through attention.
- The discriminator output D(x, c) lies in [0, 1], returning 1 when the input action is identified as real.
- D processes the action and conditioning features through LSTM hidden states d and outputs its decision from the final hidden state.The discriminator’s hidden state uses the previous state, pose vector, feature vector, and a zero noise vector.
D. Implementation Details
Training first pretrains a language–action autoencoder, then initializes the GAN components from it and trains the generator and discriminator adversarially. The implementation uses RNN-based sequence processing, attention, Gaussian noise, and empirically selected optimization settings.
- Pretraining: The text encoder is pretrained in an autoencoder whose text-to-action encoder maps language to actions and whose action-to-text decoder reconstructs language.The autoencoder learns the relationship between natural language and human action before GAN training.
- GAN training: The generator uses attention-derived language features and sampled random vectors to generate fake actions, while the discriminator compares them with real actions.Training iterates over sampled sentence–action batches, noise vectors, feature encoding, fake-data generation, and discriminator evaluations.
- Pretraining: The autoencoder reconstructs both action sequences and word-embedding sequences using a loss whose coefficients control the two estimation objectives.The coefficients a1 and a2 determine the relative reduction of action-estimation and word-embedding-reconstruction losses.
- Pretraining: After autoencoder training, the text encoder initializes E and shared generator parameters are initialized from the trained autoencoder values.This initialization is intended to stabilize generator training, while E remains fixed during GAN optimization.
- GAN training: The GAN is trained for 400 epochs with batch size 32, hidden dimension n = 256, noise dimension nz = 16, and learning rates αD = 2e−6 and αG = 2e−6.The Adam optimizer is used, and the parameter values are selected empirically.
A. Dataset
The dataset is built from MSR-VTT videos by extracting and cleaning upper-body poses, converting them from 2D to 3D, and pairing them with action-focused annotations. Each action is represented as a normalized 24-dimensional pose sequence sampled over 32 frames.
- Pose extraction: MSR-VTT videos are processed with CPM to extract upper-body 2D poses, which are converted to 3D poses for training.Upper-body poses are used because lower-body regions are considerably occluded in the videos.
- Pose representation: Each pose vector xt ∈R24 contains the 3D neck position and vectors for seven other joints.Joint vectors are normalized so that ∥vi∥2 = 1 for i = 1, . . . , 7.
- Pose processing: Incorrectly extracted poses are manually corrected and then smoothed through Gaussian filtering.
- Sequence format: Each action sequence lasts 3.2 seconds at 10 fps, yielding 32 frames per sequence.
- Dataset scale: The dataset contains 29,770 sentence–action pairs from 3,052 descriptions and 2,211 actions, with each description paired with about 10 to 12 actions.The collection contains 2.713 hours of action sequences and a vocabulary of 1,627 words.
B. 3D Action Generation
The model generates action sequences by combining sentence-conditioned features with random noise. Varying noise changes actions for one sentence, while varying sentences changes actions with the same noise sequence.
- Noise variation: With one sentence fixed, three differently sampled noise sequences generate three distinct actions through G(z1, c), G(z2, c), and G(z3, c).The demonstrated sentence is ‘A girl is dancing to the hip hop beat’, which is absent from the training dataset.
- Rendering: The generated pose vectors are fitted to a human skeleton of predetermined size for visualization.
- Sentence variation: With the noise sequence fixed, changing the sentence-conditioned feature vectors produces actions corresponding to different input sentences.The setup uses G(z, c1), G(z, c2), and G(z, c3).
- Sentence variation: The generated sequences reflect sentence-specific motion patterns for drinking coffee, exercising with dumbbells, and cooking food.The examples include raising a hand toward the mouth, moving with dumbbells in both hands, and cooking while tasting a sample.
C. Comparison with [5]
The proposed network generates action sequences that more closely match human data than the comparison SEQ2SEQ network, including for novel combinations of training descriptions.
- Comparison on an included sentence: The proposed network synthesizes ballet actions more naturally and more similarly to the matching human action data than the network in [5].Both networks produce a ballet-player action with both arms open, but the proposed sequence is closer to the data.
- Generalization to an unseen combination: For the unseen sentence combining stumbling and lifting weights, the proposed network generates both behaviors, whereas [5] generates only weight lifting.The input combines ‘A drunk woman stumbling’ and ‘Woman is lifting heavy weights’ from the training data.
- Comparison with [5]: The comparison network produces behaviors that are somewhat corresponding but generally symmetric and less dynamic than the data.The paper attributes this to its likelihood-based loss when the training data contains asymmetric left- or right-sided poses.
- Diversity from adversarial training: The GAN-trained network generates varied human action sequences that remain close to the training data.Changing the random noise input provides a mechanism for generating diverse sequences.
D. Generated Action for a Baxter Robot
The generated 3D action trajectory can be transferred to a Baxter robot, which executes the language-conditioned behavior after trajectory slowing for joint-speed limits.
- Robot execution: The Baxter robot executes a generated 3D trajectory corresponding to the sentence ‘A man is throwing something out’.The trajectory is applied using Baxter-teleoperation code, and the resulting action takes the form of throwing something forward.
- Robot execution: The trajectory is slowed before execution because Baxter’s maximum joint speed is limited.The paper states that the generated trajectory is defined in a 3D Cartesian coordinate system.
- Language-conditioned action: The proposed model is designed to enable robots to execute various actions corresponding to input language descriptions.The generated 3D action sequence is transferred to a robot.