Source-linked AI summary
Convolutional Autoencoders for Human Motion Infilling
Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, Otmar Hilliges
TL;DR
Motion infilling completes gaps between known past and future human-motion sequences with plausible poses and smooth transitions. The paper trains a convolutional autoencoder on image-like motion representations, and reports handling variable-length gaps and transitions across activities without post-processing.
Problem
Motion infilling seeks to complete missing motion between known past and future sequences with plausible poses and smooth transitions, including when poses are only partially observed.
Source: §1 · p. 1Method
Results
The model fills gaps up to 120 frames between short sequences and produces transitions across motion types without post-processing.
Source: §1 · p. 2Takeaways & Limitations
One model can address variable-length and multiple gaps, partial poses, and noise removal while generating motion directly from its outputs.
Source: §1 · p. 2, §5 · p. 8Production use would require control over the global root trajectory, which the current method does not provide.
Source: §5 · p. 8
Abstract
from arXiv · showhide
In this paper we propose a convolutional autoencoder to address the problem of motion infilling for 3D human motion data. Given a start and end sequence, motion infilling aims to complete the missing gap in between, such that the filled in poses plausibly forecast the start sequence and naturally transition into the end sequence. To this end, we propose a single, end-to-end trainable convolutional autoencoder. We show that a single model can be used to create natural transitions between different types of activities. Furthermore, our method is not only able to fill in entire missing frames, but it can also be used to complete gaps where partial poses are available (e.g. from end effectors), or to clean up other forms of noise (e.g. Gaussian). Also, the model can fill in an arbitrary number of gaps that potentially vary in length. In addition, no further post-processing on the model's outputs is necessary such as smoothing or closing discontinuities at the end of the gap. At the heart of our approach lies the idea to cast motion infilling as an inpainting problem and to train a convolutional de-noising autoencoder on image-like representations of motion sequences. At training time, blocks of columns are removed from such images and we ask the model to fill in the gaps. We demonstrate the versatility of the approach via a number of complex motion sequences and report on thorough evaluations performed to better understand the capabilities and limitations of the proposed approach.
1. Introduction
Motion infilling completes gaps between known past and future sequences with plausible motion and smooth transitions. The paper proposes a convolutional autoencoder that handles varied gaps and motion types without output post-processing.
1. Introduction
- Motion capture occlusions and tracking failures can lose valuable data, while keyframe interpolation can reduce artists’ manual work.Infilling is also relevant to completing partially observed poses and missing joints.Source: §1 · p. 1
- Motion infilling completes a gap between known past and future sequences with plausible poses and smooth transitions at both ends.It also includes cases with missing joints across a sequence or only partial poses.Source: §1 · p. 1
- The proposed convolutional autoencoder fills long gaps and supports multiple variable-sized gaps, partial poses, missing joints, and Gaussian-noise removal.Its outputs require no further post-processing.Source: §1 · pp. 1–2, Figure 2 · p. 2
- The method treats motion infilling as image inpainting on matrix representations, using a large receptive field to capture spatial relationships and temporal dependencies.Training uses reconstruction alone and a curriculum of increasingly large gaps; the model fills up to 120 frames between short sequences.Source: §1 · p. 2
- A single model generates smooth transitions across different motion types, including walking, jumping, and kicking, while closely satisfying known-sequence constraints.The model does not require post-processing.Source: §1 · p. 2
2. Related Work
Prior work addresses motion prediction, infilling, de-noising, and keyframing with recurrent, convolutional, or manifold-learning approaches. This paper’s related-work discussion emphasizes recurrent infilling limits and the challenge of capturing spatial structure in motion matrices.
2. Related Work
- Earlier recurrent infilling methods use fixed, short gaps or require post-optimization or inverse-kinematics post-processing, while this work targets smooth outputs from a neural network alone.The cited recurrent approaches include a bidirectional RNN and later methods for larger gaps.Source: §2 · p. 3
- Motion matrices pose a spatial challenge because neighboring values need not be neighboring skeletal joints, but a sufficiently large receptive field can capture spatio-temporal relationships.The paper contrasts this with approaches using dense layers across the input height or larger filters.Source: §2 · p. 3
- Prior convolutional work applies matrix representations to motion prediction, while this paper uses small filters and reports learning bone-length consistency and smoothness from reconstruction data alone.The comparison includes approaches using larger filters or bone-length regularization.Source: §2 · p. 3
- Earlier convolutional autoencoders recover poses from partial inputs or fixed gaps, but some require careful layer-wise training or separate activity models; this work trains one model across motion types.The cited work also includes a frame-wise ResNet approach requiring post-optimization for smoothness.Source: §2 · p. 3
- Keyframing seeks to reduce artists’ manual labor while preserving control, but this paper focuses on neural motion synthesis rather than a production-ready animation tool.The paper situates keyframing among physics-based animation, space-time optimization, and motion graphs.Source: §2 · pp. 3–4
- The approach draws on image-inpainting autoencoders but uses a custom architecture for motion data and its task-specific properties.The paper represents each frame as a column in a motion matrix.Source: §2 · p. 4, Figure 3 · p. 4
3. Method
The method treats motion infilling as image inpainting over joint-position maps and trains a deep convolutional denoising autoencoder to reconstruct missing pose information from sequence constraints.
3. Method
- Randomly removing joint information during curriculum training avoids explicit skeletal-structure modeling and supports robustness to input perturbations, including noise reduction.Source: §3 · p. 4
3.1. Data Representation
- Poses use 3D joint positions relative to the root, concatenated with root translational and rotational velocities, yielding 69 dimensions for 22 joints.Source: §3.1 · p. 4
3.2. Model Architecture
- Encoding units each contain two 3 × 3 convolutional layers activated with leaky ReLU, and max-pooling halves spatial and temporal dimensions; a mirrored, weight-decoupled decoder reverts pooling with strided de-convolutions.Source: §3.2 · p. 5
3.3. Training
- Training removes complete column blocks, progressively increases gap spans, and also masks random joints; the ℓ1 loss covers the full sequence.Source: §3.3 · pp. 5–6
4. Evaluation
The evaluation finds that the full model improves on the Vanilla AE, while curriculum learning trades some accuracy on shorter tasks for broader training. Qualitative tests show plausible infilling across variable gaps and motion types, but very long gaps, little context, and rare motions can limit detail or consistency.
4.1. Quantitative Results
- Table 1 shows that the full model improves on the Vanilla AE, while the curriculum model loses some accuracy on 60-frame and reconstruction tasks but still ranks second.The comparison covers reconstruction and 60- or 120-frame infilling; reported errors are in centimeters and infilling error is computed only over the gap.Source: Table 1 · p. 6, §4.1 · p. 6
- For de-noising, the model beats the baseline by a large margin on randomly dropped joints, but the baseline performs better on Gaussian-noise removal.Neither model was trained specifically for these tasks, and joint error may not track perceived smoothness or naturalness.Source: §4.1 · p. 6, Table 2 · p. 6
- In pure reconstruction, the model’s predicted bone lengths have a smaller spread than Holden et al.’s and are typically closer to ground truth at the median.Figure 5 compares six selected bones over 610′080 validation-set estimates.Source: §4.1 · p. 6, Figure 5 · p. 7
4.2. Qualitative Results
- Blending 16 test-set seed sequences produced 1,927 frames, with the model filling about 67% using variable gaps between motion categories.Generation takes one forward pass; on a low-end GPU, 1,927 frames take 2.8 seconds, or 1.45 ms per frame.Source: §4.2 · p. 7
- Tertiary constraints can blend an arm action into a walking sequence, but they must span multiple frames or the model treats them as noise.The demonstrated setup places arm-joint positions from another test clip across 15 frames within a 60-frame gap.Source: §4.2 · p. 7
- The model produces convincing motions after entire joints are removed, including when missing joints are close in the skeleton hierarchy, such as the elbow and hand.The experiment masks joints over full 240-frame test clips.Source: §4.2 · pp. 7–8
- Gaps of 20 and 80 frames yield smooth, plausible motion, while a 250-frame gap trends toward a mean pose and a five-frame gap can erase fine detail between very different motions.The authors identify the five- and 250-frame cases as extremes; the latter is roughly twice the gap size seen during training.Source: §4.2 · p. 8
- A fully masked motion cycle is recoverable only when it recurs; one-off masked turns cannot be reconstructed, and rare training motions may appear jittery.The authors also note that bone lengths are not guaranteed consistent and foot skating may occur.Source: §4.2 · p. 8
5. Conclusion
The paper uses a convolutional autoencoder and an inpainting formulation to complete large gaps in 3D human motion. It produces smooth transitions, while global trajectory control and consistent bone lengths or foot contact remain limitations.
5. Conclusion
- An inpainting-trained convolutional autoencoder uses curriculum learning to fill large motion gaps with plausible poses and smooth transitions between different motion sequences.Training masks increasingly large blocks to support variable gap lengths and robustness to input noise.Source: §5 · p. 8
- Production use would require control over global root trajectories, which the method currently obtains by integrating predicted root velocities.The authors suggest decoupling root trajectory control from pose generation.Source: §5 · p. 8
- The method cannot guarantee consistent bone lengths or the absence of foot-skating artifacts.The authors note these issues may not majorly degrade visual quality in their visualizations.Source: §5 · p. 8
6. Appendix
The appendix shows motion infilling across varied sequences and tests denoising under Gaussian noise and missing joints. The examples demonstrate plausible reconstructions, alongside some physically infeasible poses.
6. Appendix
- One model produces plausible, smooth transitions across varied sequences, including rare dancing and floor-lying motions, though some raw poses float above the ground.One example combines five motion sequences; the displayed results receive no further processing to enforce physical plausibility.Source: §6 · p. 11, Figure 12 · p. 12
- The model recovers plausible motion from Gaussian-corrupted inputs and inputs with joints independently masked at p = 0.3, despite not being trained specifically for denoising.The Gaussian noise has zero mean and unit variance; the examples show reconstructed poses in blue.Source: §6 · p. 11, Figure 9 · p. 11, Figure 10 · p. 11