Source-linked AI summary
Audio to Body Dynamics
Eli Shlizerman, Lucio M. Dery, Hayden Schoen, Ira Kemelmacher-Shlizerman
TL;DR
The paper addresses whether natural body movement can be predicted from music, a question requiring suitable recital data, pose estimation, and audio-motion modeling. It trains LSTMs on piano and violin videos to predict body landmarks and uses them to animate avatars. The resulting skeletons and body dynamics are reported as promising, while the approach remains limited to visible 2D keypoints and does not model exact piano-key correspondence.
Problem
The paper asks whether body movement can be predicted computationally from music, requiring training videos, accurate body poses, and a model of audio-motion correlation.
Method
The method trains LSTM networks on piano and violin recital data to map audio features to body landmarks, then applies predicted landmarks to animate an avatar.
Results
The paper reports promising initial results, with separate piano and violin networks producing skeletons and body dynamics that move according to the audio.
Takeaways & Limitations
The results support the hypothesis that natural body gestures can be predicted from audio and motivate applications and further research in VR/AR and recognition.
Takeaways & Limitations
The current approach uses 2D keypoints, predicts only visible points, and does not learn correspondence between music and exact piano keys.
Abstract
from arXiv · showhide
We present a method that gets as input an audio of violin or piano playing, and outputs a video of skeleton predictions which are further used to animate an avatar. The key idea is to create an animation of an avatar that moves their hands similarly to how a pianist or violinist would do, just from audio. Aiming for a fully detailed correct arms and fingers motion is a goal, however, it's not clear if body movement can be predicted from music at all. In this paper, we present the first result that shows that natural body dynamics can be predicted at all. We built an LSTM network that is trained on violin and piano recital videos uploaded to the Internet. The predicted points are applied onto a rigged avatar to create the animation.
1. Introduction
The paper asks whether body movement can be predicted from music and explores this question using recital videos, an LSTM, and avatar animation. Its initial results produce promising body dynamics for piano and violin performance.
- Research question: The central question is whether body movement can be predicted computationally from a music signal.The problem requires suitable videos, accurate pose estimation, and a model that learns correlations between music and body movement.
- Motivation: Laboratory motion-capture approaches are difficult to execute and generalize, motivating the use of publicly available recital videos.Online videos could provide more diverse data, while recent pose-estimation methods made learning from such footage more feasible.
- Goal: The paper explores creating natural, logical body and finger dynamics from music without using MIDI information about exact piano keys.The focus is an avatar whose hands and fingers move like those of a pianist.
- Data: The method collects piano and violin recital videos and processes visible upper-body and finger landmarks, totaling 50 points per frame.The point representation includes 21 finger points per hand and 8 upper-body points.
- Method: An LSTM learns correlations between audio features and body skeleton landmarks, after which predicted landmarks animate an avatar driven by the input audio.The pipeline is audio to predicted movement points to an animated avatar.
- Results: Separate neural networks are trained for piano and violin, producing promising skeletons and interesting body dynamics.The authors direct readers to supplementary videos with audio to experience the results.
2. Related Work
Prior work established relationships between audio and facial or bodily motion, but did not directly explore animating body pose from music. This paper builds on those findings while targeting natural-looking, non-unique motion rather than exact reproduction.
- Audio-driven motion: Speech-driven facial animation has shown that mouth landmarks can be predicted from audio, providing a precedent for audio-to-motion modeling.These results include high-quality talking-face videos and speech-driven avatar animation.
- Research gap: Animating body pose directly from music remained comparatively unexplored, despite related work on music responses, pose prediction, and multimodal recognition.The related areas include behavioral studies, video-based body-pose learning, and combined audio-visual analysis.
- Multimodal evidence: Multimodal studies report higher accuracy when audio and visual inputs are combined than when either modality is used alone.Examples include facial-expression recognition and body-pose estimation improved with audio input.
- Audio-motion correspondence: Studies of speech and body rhythms, emotional movement, and pianist performance found correspondences between audio characteristics, movement, and perceived music-motion relationships.These findings motivate investigating audio-to-body correlations for musical performance.
- Pose prediction: LSTMs have been used to predict or edit future poses from visual inputs, while earlier motion-style animation work did not use audio as input.This paper differs by using music as the driving signal for body-pose animation.
- Inverse direction: Related research also studies the inverse direction, predicting sounds from faces, objects, photos, or video montages.These efforts provide complementary context for the paper’s goal of predicting video from audio.
- Scope: The authors argue that prior work provides enough evidence of audio-motion correlation to motivate estimation, while not expecting exact body-motion reproduction.They assume the transformation from music or speech to body motion is not unique and instead target natural-looking movement that fits the input.
3. Body Dynamics Representation
The paper constructs body-dynamics representations from recital videos by selecting suitable in-the-wild recordings, extracting keypoints, factorizing motion, and reducing aligned keypoints with PCA.
- Data and preprocessing: The selected videos favor clear solo audio, high resolution, fixed cameras, bright lighting, and long continuous performances.These conditions were chosen to support reliable processing of videos “in the wild.”
- Keypoint extraction: Three detection libraries are combined because their failures and successes are complementary on in-the-wild recital videos.OpenPose, MaskRCNN, and DeepFace are used before frame and detection filtering.
- Keypoint extraction: Frames are filtered using reference-box distance, face-signature mismatch, or excessive consecutive-frame keypoint displacement.The retained hand points come from OpenPose within the selected person box, with additional detector selection described in the preprocessing procedure.
- Motion factorization: The representation separates audio-related motion from camera, rigid-person, and non-audio body transformations under a fixed-camera assumption.The authors set non-audio person motion to zero and aim to predict the audio-related component, such as bow drawing or piano-key striking.
- Dimensionality reduction: Aligned keypoints are arranged in a 2p × f matrix, reduced by PCA to capture 90% of the data, and linearly upsampled to 4 × videofps.The resulting PCA coefficients represent body motion and are paired with MFCC coefficients representing audio.
4. Audio to Body Keypoints Prediction
The audio-to-keypoints module uses a delayed unidirectional LSTM to learn mappings from audio MFCC features to PCA coefficients representing body keypoints.
- Network architecture: The prediction network learns a correlation between audio features and body movements using a Long-Short-Term Memory network.Audio inputs are MFCC features, while outputs are PCA coefficients of body keypoints.
- Network architecture: The architecture is a unidirectional single-layer LSTM with time delay and a fully connected layer that improved performance in the authors’ experiments.The LSTM memory is denoted m, audio features are x_i, and body-keypoint PCA coefficients are y_i.
- Training configuration: The model uses a hidden state of 200, truncated backpropagation with 400 time steps, time delay 5, dropout 0.4, and learning rate 5e −3.Training runs for 300 epochs with ADAM; typically 10 PCA components are used.
5. Body Keypoints to Avatar
Predicted 2D keypoints are transferred to a rigged 3D avatar, with separate procedures for body parts, arms, hands, fingers, and violin-related constraints.
- Avatar initialization: An ARKit application applies sequences of predicted 2D points to a rigged body avatar in real time.The avatar is initialized by aligning predicted points with 3D world coordinates.
- Avatar initialization: The rig is aligned using average shoulder positions, while body, arms, head, and fingers are animated independently when their points are available.A root rotation offset matches the avatar to the piano pose, and wrist position determines piano depth.
- Body and arms: The body is represented by an inverse-kinematics chain from the average hips through the average shoulders, with spine scaling based on average spine length.This defines the avatar’s spine from the predicted body landmarks.
- Body and arms: The arm inverse-kinematics chain uses the wrist as its reference and estimates forward offset from forearm length across frames.Maximum forearm length corresponds to a straight arm on the source plane, while zero length corresponds to a perpendicular arm.
- Hands and fingers: Hand rotation is inferred from the relative positions of the pointer and pinkie roots, while finger angles are applied individually.The pointer–pinkie ordering determines palm orientation in the avatar.
- Violin rigging: The violin is constrained to the head and left hand, while the bow is constrained to the right hand and oriented toward the violin bridge.Four points provide constraint references for violin rigging.
6. Experiments
The experiments train separate violin and piano networks, evaluate parameter choices and data preparation, and produce promising avatar animations from predicted body points. Random audio produces no movement, while representative results expose limitations from 2D pose training and difficult visual conditions.
- Data and setup: Separate violin and piano networks were trained and tested independently on recital datasets.The datasets used 80% of frames for training and 20% for validation, after removing approximately 10% of frames with inaccurate keypoints.
- Parameter evaluation: Errors drop significantly after filtering training frames with incorrect skeleton, person detection, or person recognition.Using less training data or fewer PCA coefficients improves training fit but increases test error; dropout does not improve results, while time delay helps.
- Control experiment: Random test audio produces no correlation and no predicted movement from either network.This experiment serves as a control for the audio-to-motion relationship.
- Qualitative results: Representative predictions show similar hand and finger movement rather than exact agreement with ground-truth keypoints.The ground truth comes from a 2D pose detector and can itself contain mistakes.
- Failure cases: 2D training poses limit prediction of occlusions and invisible points, while fast motion and motion blur can introduce detector errors.These errors affect what the network learns for high-frequency audio.
- Avatar animation: The system produces avatar animations from predicted points in augmented real scenes.The piano, violin, and avatar are synthetic objects placed in a real scene using ARKit.
7. Discussion and Limitations
The paper presents audio-to-body animation as a promising initial hypothesis, while identifying extensions needed for broader and more accurate motion modeling. These include 3D movement, occlusion prediction, richer training data, and activity-general models.
- Contribution: The paper reports promising initial results for predicting body gestures from audio and identifies VR/AR and recognition as potential application areas.The authors frame the work as an initial result and invite further research.
- 3D movement: A 3D keypoint estimator could extend the approach from 2D movement to 3D motion and more diverse human modeling.The current OpenPose and MaskRCNN estimators provide 2D points.
- Occlusions: The current system predicts only visible points, so occluded-keypoint prediction remains an open direction.In piano performance, one hand can occlude the other even though the audio contains music from both hands.
- Training data: Training uses only YouTube videos, with sensor information or MIDI files proposed as complementary data sources for greater realism and accuracy.MIDI could provide information about the exact piano keys associated with the music.
- Generalization: Obtaining good training data for each activity class is difficult, motivating models that handle varied poses without prior activity classification.The paper also suggests incorporating video activity recognition into the learning framework.