PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Mesh Recovery from Multiple Shots

Georgios Pavlakos, Jitendra Malik, Angjoo Kanazawa

arXiv:2012.09843v1cs.CV

TL;DR

Edited-media videos offer diverse human behavior but challenge 3D recovery with abrupt shot changes and heavily truncated people. The paper uses multi-shot optimization to recover 3D sequences and create pseudo-ground-truth training data. It reports improved robustness for single-image recovery and proposes a transformer temporal model for video.

  • Problem

    Edited-media videos contain shot changes and close-ups with heavy truncation, limiting the applicability of existing human 3D understanding methods.

    Source: §1 · p. 2, §2.4 · p. 3

  • Method

    The paper uses multi-shot optimization to recover consistent 3D motion across shots and generates pseudo-ground-truth human meshes for training image and video recovery models.

    Source: §1 · p. 2

  • Results

    Training on the recovered data improves single-frame robustness against heavy truncation, while the proposed pure transformer-based t-HMMR is competitive and suited to films with missing observations.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The workflow produces long 3D motion sequences and training data from movies, and the paper presents multi-shot recovery as a new task with applications across human mesh recovery.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Videos from edited media like movies are a useful, yet under-explored source of information. The rich variety of appearance and interactions between humans depicted over a large temporal context in these films could be a valuable source of data. However, the richness of data comes at the expense of fundamental challenges such as abrupt shot changes and close up shots of actors with heavy truncation, which limits the applicability of existing human 3D understanding methods. In this paper, we address these limitations with an insight that while shot changes of the same scene incur a discontinuity between frames, the 3D structure of the scene still changes smoothly. This allows us to handle frames before and after the shot change as multi-view signal that provide strong cues to recover the 3D state of the actors. We propose a multi-shot optimization framework, which leads to improved 3D reconstruction and mining of long sequences with pseudo ground truth 3D human mesh. We show that the resulting data is beneficial in the training of various human mesh recovery models: for single image, we achieve improved robustness; for video we propose a pure transformer-based temporal encoder, which can naturally handle missing observations due to shot changes in the input frames. We demonstrate the importance of the insight and proposed models through extensive experiments. The tools we develop open the door to processing and analyzing in 3D content from a large library of edited media, which could be helpful for many downstream applications. Project page: https://geopavlakos.github.io/multishot

1. Introduction

Movies offer diverse human behavior over long temporal contexts, but shot changes fragment sequences and close-ups heavily truncate people. The paper treats shots of the same scene as multi-view evidence, using multi-shot optimization and resulting data to improve human mesh recovery.

1. Introduction

  • Shot changes are usually treated as independent scenes, although they depict one underlying dynamic scene from different viewpoints and can help disambiguate 3D pose and motion, especially for truncated people.The paper frames this as a lost opportunity in movie-derived video.Source: §1 · p. 2
  • The paper proposes multi-shot optimization to recover a consistent 3D motion sequence across shot changes, addressing temporal fragmentation and partial humans.Source: §1 · p. 2
  • The optimization recovers long 3D motion sequences from movies as pseudo-ground-truth training data for image- and video-based human mesh recovery.The paper reports improved robustness for single-frame recovery and proposes t-HMMR, a pure transformer temporal model suited to films with missing observations.Source: §1 · p. 2
  • Applying the approach to AVA [12] yields MS-AVA, a dataset of over 350k frames with pseudo-ground-truth 3D, used to train single-image and video recovery models.Experiments evaluate the optimization and collected data on MS-AVA and common benchmarks.Source: §1 · p. 2
  • The paper introduces multi-shot human mesh recovery and a dataset for training more robust single-frame and pure transformer-based temporal models.The authors state that code and MS-AVA will be released upon publication.Source: §1 · pp. 2–3

2. Background

The background reviews body models, optimization and regression approaches, video temporal encoders, and existing video datasets. It positions the work around edited-media conditions—shot changes and truncated people—that differ from conventional performance-capture data.

2. Background

  • The background presents the literature review as context for the proposed approach, focusing on the most relevant prior methods.Source: §2 · p. 3

2.1. Human body modelling

  • SMPL represents a body as a function of pose and shape parameters that returns a 3D mesh and joints; other body models vary in modeling approach and expressivity.Source: §2.1 · p. 3

2.2. 3D pose and shape from single image

  • Single-image reconstruction either optimizes a body model to fit detected features with reprojection and prior objectives or directly regresses SMPL parameters from image features.HMR [20] is described as a canonical direct-regression approach; prior work also identifies heavy truncation as a challenge [18] [37].Source: §2.2 · p. 3

2.3. 3D pose and shape from video

  • Video methods add a temporal encoder that maps per-frame features to sequence features, using convolutional, recurrent, hybrid, or self-attention architectures.The paper proposes a pure transformer based exclusively on self-attention to handle missing identities in films.Source: §2.3 · p. 3

2.4. Common video datasets

  • Common video datasets provide 3D supervision or outdoor ground truth, but are designed for performance capture and favor viewpoints showing the full body.Films and other edited media instead contain shot changes and close-ups that induce substantial truncation.Source: §2.4 · p. 3

3. Multi-shot optimization

The workflow links identity tracklets across shots, then optimizes a consistent 3D reconstruction across shot boundaries. Applied to AVA, it produces MS-AVA sequences and supports cross-shot evaluation of reconstructed shape.

3. Multi-shot optimization

  • The workflow first prepares pseudo-ground-truth inputs, describes multi-shot optimization, and then constructs the MS-AVA dataset.Source: §3 · p. 4

3.1. Preprocessing steps

  • 2D pose tracklets are extended across shot boundaries using shot detection and movie-trained person re-identification, then supplied to multi-shot optimization.Pose trackers such as OpenPose [7] or Alpha-Pose [10] detect joints but fail across shot boundaries.Source: §3.1 · p. 4

3.2. Multi-shot optimization

  • The optimization uses smoothness of canonical-frame body structure across shots rather than camera-frame smoothness, since viewpoint changes can be abrupt.It optimizes objectives over the sequence and initializes reconstruction with pose and shape estimates from a regression network [23].Source: §3.2 · p. 4

3.3. Multi-Shot AVA dataset

  • MS-AVA extends tracklets by connecting identities across shots and re-identifying people after absences, yielding longer, diverse sequences for temporal-model training.Figure 3 distinguishes continuous sequences, shot changes, and noncontinuous identities; the dataset is built from AVA at 1 fps.Source: §3.3 · p. 4, Figure 3 · p. 5
  • Cross-shot PCK evaluates recovered 3D shape by projecting predictions across a shot boundary in both directions and measuring 2D reprojection against available keypoints.The evaluation addresses AVA, where only 2D keypoints are present.Source: §3.3 · p. 4, Figure 4 · p. 5
  • Table 1 reports tracklet counts and raw durations for all tracklets and those longer than 20 seconds; MS-AVA provides many long sequences.The caption attributes the long sequences to multi-shot continuity and re-identification across disjoint subsequences.Source: Table 1 · p. 5

4. Human Mesh Recovery

The section uses pseudo-ground-truth 3D bodies from offline multi-shot optimization to train direct single-image and video mesh-recovery models. It combines more robust single-frame HMR with t-HMMR, a transformer temporal encoder designed for discontinuous sequences and absent-person frames.

4. Human Mesh Recovery

  • Offline multi-shot optimization supplies pseudo-ground-truth 3D bodies for direct single-image and video mesh-recovery training without relying on keypoint detections.These recovered sequences provide the supervision source for both model settings.Source: §4 · p. 5

4.1. Single-frame model

  • The single-frame model predicts body and camera parameters from an image and is supervised by optimized body parameters plus detected 2D joints.Its projected model joints are compared with detected 2D locations.Source: §4.1 · p. 5
  • MS-AVA's diverse, challenging visuals, including truncation, help improve the single-frame model's robustness.Source: §4.1 · p. 5

4.2. Temporal model

  • Convolutional and recurrent temporal encoders are poorly suited to shot changes and absent-person frames, which require padding or concatenation that discards timing.Such discontinuities arise from shot changes, occlusion, tracking failures, or re-identification failures.Source: §4.2 · p. 5
  • t-HMMR uses a pure transformer encoder to select relevant sequence elements, masking invalid frames while retaining temporal context.Per-frame presence values prevent invalid inputs from contributing to self-attention; positional encodings preserve frame timing.Source: §4.2 · pp. 5–6, Figure 5 · p. 6

5. Experiments

The experiments evaluate multi-shot optimization, the MS-AVA training data, and the temporal transformer across reconstruction and human mesh recovery tasks. Results show gains over optimization baselines, improved robustness for single-frame recovery, and stable temporal modeling that handles missing observations.

5. Experiments

  • The experiments assess optimization quality, single-frame model performance and robustness, and the temporal architecture’s value.The baseline HMR is retrained on standard datasets; the final HMR adds MS-AVA, while t-HMMR freezes the image encoder for computational reasons.Source: §5 · p. 6, §5.1 · p. 6

5.2. Quantitative evaluation

  • Multi-shot optimization outperforms single-frame and single-shot optimization baselines on cross-shot PCK.Figure 6 reports results across varying thresholds α.Source: §5.2 · pp. 6–7, Figure 6 · p. 7
  • On the uncropped Partial Humans subset, the MS-AVA-trained model outperforms state-of-the-art approaches and is more robust to truncation.The comparison is reported in Table 2 and illustrated qualitatively in Figure 8.Source: §5.2 · p. 7, Table 2 · p. 7, Figure 8 · p. 7
  • On 3DPW, MS-AVA training improves over synthetic cropping augmentation, and the evaluation reports the same trends as Partial Humans.Table 3 reports single-frame regression results using PA-MPJPE in millimeters.Source: §5.2 · p. 7, Table 3 · p. 7
  • The transformer encoder is more stable than convolutional and recurrent encoders, while achieving PA-MPJPE of 56.1mm versus 56.5mm for VIBE on 3DPW.Across runs, the other encoders tend to diverge after a few epochs; the transformer is trained on less data than the full VIBE model.Source: §5.2 · p. 8, Figure 9 · p. 8
  • With missing-frame sequences in training, the transformer benefits most, and it performs best when tested on all sequences.The comparison uses cross-shot PCK on AVA across training and testing protocols in Table 4.Source: Table 4 · p. 8, §5.2 · p. 8
  • The temporal t-HMMR predictions are more consistent than single-frame HMR predictions because they use a larger temporal context.Figure 10 illustrates a pose for frame 1 that is consistent with frame 2.Source: Figure 10 · p. 8, §5.2 · p. 8

6. Conclusion

The paper introduces 3D human reconstruction from multiple shots and a framework for generating training data from edited media. It applies that data to multiple human mesh recovery tasks and reports experiments demonstrating the contributions.

6. Conclusion

  • The paper introduces 3D human reconstruction from multiple shots and a framework for generating edited-media training data for different human mesh recovery tasks.Its contributions span offline iterative optimization, single-view prediction, and temporal prediction.Source: §6 · p. 8
Loading 2012.09843v1…