PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Person Human Motion Forecasting in Complex Scenes

Serdar Ozsoy, Lars Doorenbos, Juergen Gall

arXiv:2608.27039v1cs.CVcs.AI

TL;DR

Human motion forecasting in complex scenes must handle ambiguous futures while jointly modeling motion history, social interactions, and object context. OCSD addresses this with conditional diffusion, timestep-wise object conditioning, and social encoding, achieving state-of-the-art benchmark performance and realistic, diverse long-term forecasts. Its main scope limitation is that predictions cannot adapt to objects moving during the forecast interval.

  • Problem

    Complex-scene motion forecasting requires jointly modeling motion history, social interactions, and objects, while the same context can support multiple plausible futures and short- versus long-horizon methods remain difficult to unify.

  • Method

    OCSD is a conditional diffusion framework that integrates past motion, social context, and object signals, modulating denoising with object cues at every timestep and encoding per-person social context.

  • Results

    OCSD achieves state-of-the-art performance on HiK and HOI-M3, including 121.5 mm (31.3%) and 130.5 mm (33.2%) lower two-second path error, respectively, while improving long-term realism and diversity.

  • Takeaways & Limitations

    Integrating social and object information in one generative framework supports accurate short-term forecasts and plausible, diverse long-term futures across varying group sizes and interaction complexity.

  • Takeaways & Limitations

    Using only the last observed object positions prevents predictions from adapting to objects that move during the forecast interval, affecting some HOI-M3 interactions.

Abstract

from arXiv · show

Accurately forecasting the movement of people in complex scenes requires reasoning over the past and present state of the entire environment. In this context, effectively incorporating object information and social interactions into a unified framework remains particularly challenging. To address this, we propose Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model that integrates motion history, multi-person interactions, and object cues into a single framework. OCSD uses an object-conditioning mechanism that modulates denoising at every timestep, enabling fine-grained human-object reasoning, and a social encoder that models the interactions between all humans in the scene. As a result, our model naturally handles varying group sizes, complex social interactions, and supports sampling multiple plausible futures. Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks. It reduces the two-second path error by 121.5 mm (31.3%) on HiK and 130.5 mm (33.2%) on HOI-M3 compared to prior work, and produces more realistic long-term forecasts.

1 Introduction

Forecasting people in complex scenes is ambiguous and requires jointly reasoning about motion history, social interactions, and objects across time horizons. OCSD unifies these factors with conditional diffusion, achieving strong short-term accuracy and long-term realism and diversity.

  • Accurate complex-scene forecasting requires reasoning about motion history, interactions with other people, and objects.
  • Multiple futures can be correct for the same context, making human motion forecasting inherently ambiguous.
  • Short- and long-horizon methods typically trade off local realism against stochastic, diverse predictions, leaving unified modeling open.
  • OCSD uses object-conditioned denoising at every timestep and a social encoder to integrate human-object dependencies and social context.
  • 121.5 mm (31.3%) lower two-second path error on HiK and 130.5 mm (33.2%) lower error on HOI-M3 demonstrate strong benchmark performance.
  • OCSD unifies past conditioning, social context, and object signals for multi-person motion forecasting in complex scenes.

2 Related Works

Prior work has addressed single-person motion, social dynamics, and object context through largely separate approaches. More recent unified models combine social and scene context, while OCSD targets both short-term accuracy and long-term plausibility and diversity.

  • Single-person forecasting includes deterministic neural models and probabilistic generative approaches for diverse future motions.
  • Multi-person forecasting adds interdependent social dynamics, motivating transformer architectures with local individual and global social encoders.
  • Object-aware methods use contact maps, scene distance fields, or segmented object instances to condition human-motion forecasts.
  • Unified models have only recently merged social and scene context, often emphasizing either long-term diversity or short-term accuracy.
  • OCSD bridges these directions by targeting short-term path and pose accuracy alongside long-term plausibility and diversity.

3 Method

OCSD forecasts multi-person motion from observed poses, social interactions, and object information using a conditional diffusion framework. It represents context relative to each person, integrates social features at the U-Net bottleneck, and applies object conditioning throughout denoising.

  • 3 Method: The forecasting task models past motion, person-to-person interactions, and object type and location to predict all people’s future poses.The observed sequence and object set condition the forecast over the next Tout frames.
  • 3 Method: OCSD treats future motion as a conditional distribution because identical observations can admit multiple plausible futures, especially over longer horizons.The model therefore uses conditional diffusion rather than predicting only a single deterministic continuation.
  • Input Representations: Pose inputs are represented in person-centered local coordinates, while self-stream and social-stream features separate individual motion from other people’s relative poses.The local representation supports stable learning, and the social stream restores relative-position information lost by canonicalization.
  • Object Context Integration: Object context is canonicalized per person, converted into semantic object tokens, and integrated at multiple U-Net stages through cross-attention and FiLM modulation.The resulting object-conditioned features can influence motion generation across the network rather than at a single fusion point.
  • Social Integration: Social context is encoded from pairwise relative poses, mean-pooled after masking padded people, and merged into each reference person’s U-Net bottleneck features.This produces a fixed-dimensional representation despite dynamically varying group sizes.
  • Conditional Diffusion Model: During training, the diffusion network predicts clean full motion from noised future motion and observed past, using timestep and object context with a masked L1 reconstruction objective.Inference uses DDPM reverse sampling with inpainting; reusing a noised past prevents drift and discontinuities at the observed–predicted boundary.

4 Experiments

OCSD is evaluated on HiK and HOI-M3 across short- and long-term forecasting, with strong accuracy, realism, and diversity. Ablations show that object and social context, combined object conditioning, and loss weighting contribute to performance.

  • Experimental setup: OCSD is evaluated on HiK and HOI-M3 for short-term and long-term multi-person motion forecasting.HiK contains up to 16 people and 50 objects, while HOI-M3 contains up to 5 people and at least 5 objects per scene.
  • Experimental setup: The evaluation measures short-term path and pose errors and long-term realism and diversity.The experiments use Path error, Pose error, NDMS, and UMWR.
  • Main results: 121.5 mm (31.3%) is the two-second path-error improvement over HUMOF on HiK.OCSD improves over HUMOF by 44.7 mm after one second and 121.5 mm after two seconds; it also performs better at all four pose-error time points.
  • Main results: OCSD outperforms baselines across short-term accuracy and long-term motion realism and diversity on HiK and HOI-M3.On HiK, it is best on all short-term metrics except the 0.5-second path error, where it trails HUMOF by 0.1 mm.
  • Ablation studies: The combined cross-attention and FiLM object-conditioning mechanism achieves the best ablation results on HiK.The study compares the combined mechanism with only Cross-Att and only FiLM variants.
  • Main results: 25.9% is the NDMS improvement over SiMLPe, increasing realism from 0.27 to 0.34.OCSD maintains high UMWR across ten seconds, while some baselines produce frozen motions.
  • Ablation studies: Context increases long-term NDMS realism, while UMWR diversity is slightly higher without context because environmental and social constraints are absent.The diversity of the context variants remains similar overall.

5 Conclusion

OCSD unifies individual dynamics, social interactions, and object influence in a generative framework for complex multi-person scenes. Experiments on HiK and HOI-M3 show improved forecasting and support diverse plausible futures.

  • OCSD integrates individual dynamics, social interactions, and object influence into one unified generative framework.Object cues modulate denoising at every timestep, while per-person social context is integrated into the model.
  • OCSD maintains flexibility across varying group sizes, interaction complexity, and diverse possible futures.
  • OCSD demonstrates substantial improvements over existing methods on the HiK and HOI-M3 benchmarks.The model also correctly continues stand-up and walking motions in a qualitative HOI-M3 example.

A Datasets

The evaluation uses two multi-person, multi-object indoor datasets with tracked humans and objects, differing in scene scale and group size. Standard or prior-work-based splits and processing pipelines define their evaluations.

  • Humans in Kitchens: HiK contains kitchen recordings with dynamic groups of up to 16 people and 50 simultaneously visible annotated objects.The dataset covers four kitchen environments with minimally scripted everyday activities and dense scene context.
  • Humans in Kitchens: HiK training uses kitchens A, B, and C, while evaluation uses curated transitional sequences from kitchen D under the standard protocol.The official test split and evaluation script are used.
  • HOI-M3: HOI-M3 contains 199 long indoor sequences with up to 5 people and at least 5 tracked objects per scene.All humans and objects are tracked in 3D, with object vertices used as the object representation.
  • HOI-M3: HOI-M3 evaluation uses 49 living-room sequences, a 20% test split, and the data-processing pipeline of prior work.Future motions are predicted for every sequence frame using a one-second sliding-window stride.

B Metrics

The metrics separate global trajectory accuracy from local pose accuracy and evaluate long-term stochastic forecasts through realism, directional similarity, speed, and diversity measures.

  • Accuracy metrics: Path Error measures the Euclidean distance between predicted and ground-truth 3D root-joint positions at each future timestep.
  • Accuracy metrics: Pose Error uses MPJPE across aligned non-root joints to measure local body configuration independently of global translation.The joint positions are the predicted and ground-truth 3D locations.
  • Realism metrics: NDMS evaluates long-term stochastic motion realism against a distribution of real-world examples rather than a single ground-truth sequence.It uses overlapping short motion segments and penalizes visible discontinuities between observed and forecast motion.
  • Realism metrics: Directional Similarity measures alignment between predicted and reference joint velocity vectors.
  • Realism metrics: Magnitude Ratio compares velocity magnitudes to assess whether generated motions have similar speeds.
  • Diversity metrics: UMWR measures the ratio of unique nearest neighbors to total motion words, with higher values indicating greater motion diversity.The metric uses eight-frame motion words; low scores indicate repetitive or frozen motion.

C Additional Implementation Details

Training uses a single RTX-4090 GPU with horizon-dependent batch sizes and learning-rate adjustment, while qualitative comparisons use SAST because HUMOF code was unavailable.

  • Training: Models are trained on one RTX-4090 GPU with batch sizes of 64 for two-second and 16 for ten-second prediction.For long-term training, the learning rate is adjusted to 1 × 10^-4 to accommodate the smaller batch size.
  • Training: Additional long-term training changes include processing each bottleneck temporal feature sequence separately.
  • Qualitative evaluation: Qualitative results on both datasets are compared with SAST because HUMOF code was unavailable when the manuscript was prepared.

D.1 HiK dataset: 2-second prediction

On HiK, OCSD accurately predicts future poses for all people, including the scene’s largest mover, and better infers a walking trajectory than SAST.

  • D.1 HiK dataset: 2-second prediction: OCSD simultaneously predicts future motions for all individuals, with predicted poses closely aligning to ground truth over the 2-second horizon.Even the purple/yellow skeleton with the largest motion remains difficult to distinguish from ground truth at frame 74.
  • D.1 HiK dataset: 2-second prediction: OCSD correctly forecasts the purple/yellow person walking from the left group to the right group, whereas SAST keeps the person near the last observed position.

D.2 HOI-M3 dataset: 2-second prediction

On HOI-M3, OCSD captures pose dynamics and object-centered interactions over 2 seconds, while its long-horizon samples remain diverse and plausible compared with SAST.

  • D.2 HOI-M3 dataset: 2-second prediction: OCSD captures the central person’s walking poses and a left person’s continuing chair interaction, although the walking motion is slightly slower.The model identifies the chair interaction affordance and generates a valid walking path.
  • D.2 HOI-M3 dataset: 2-second prediction: OCSD’s predicted motions overlap significantly with ground truth in a 2-second qualitative comparison, indicating accurate forecasting.
  • D.2 HOI-M3 dataset: 2-second prediction: For a single 10-second scenario, OCSD generates diverse but plausible futures in which the purple/yellow person walks to different spatial locations.The two samples show that the model does not collapse to a single deterministic mean.
  • D.2 HOI-M3 dataset: 2-second prediction: Over a 10-second prediction window, OCSD maintains motion plausibility, whereas SAST predicts sitting at a location where no chair exists.

E Limitations

The paper reports limitations involving moving objects, object semantics, dataset scale, and context-dependent behavior, while supplementary analyses also compare architectures and components.

  • E Limitations: OCSD cannot adapt to objects that move during prediction because it uses their last observed positions instead of forecasting future object locations.This causes deviations in human-object interaction for a subset of HOI-M3, while HiK is largely unaffected because its objects are mostly static.
  • E Limitations: One-hot object categories work for HiK and HOI-M3’s closed vocabularies but are unsuitable for open-vocabulary generalization.Preliminary CLIP text embeddings did not improve the approach; vision-language embeddings are proposed as future work.
  • E Limitations: Supplementary ablations report that mean pooling and bottleneck injection outperform cross-attention and multi-layer social-context integration.
  • E Limitations: Context consistently improves path error, but its effect on pose error depends on the activity.Context slightly increases pose error for walking while improving predictions for object-centered activities such as coffee-machine interaction and sink activity.
  • E Limitations: On the relatively small datasets, DiT overfits and performs worse than the U-Net backbone under identical conditioning and training protocols.

F.4 Additional motion quality metrics

Additional analyses indicate that OCSD preserves realistic motion quality, benefits from more samples, and improves best-of-K accuracy while maintaining diverse predictions.

  • F.4 Additional motion quality metrics: OCSD’s foot sliding remains within 5% of ground truth, while interpenetration statistics are likewise close to ground truth.Ground-truth motion also contains penetrations because object surfaces are approximated.
  • F.4 Additional motion quality metrics: Increasing evaluation samples from 4 to 16 raises UMWR, especially for 2-second predictions, showing diversity is not saturated at four samples.
  • F.4 Additional motion quality metrics: Best-of-K path and pose errors on HiK improve as K increases from 1 to 20, while predictions remain diverse and consistently high quality.
Loading 2608.27039v1…