PaperlayerView exact revision on arXiv

Source-linked AI summary

MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillation

Yuxiang Fu, Qi Yan, Lele Wang, Ke Li, Renjie Liao

arXiv:2503.09950v1cs.CVcs.AIcs.LG

TL;DR

Human trajectory forecasting must represent multiple plausible futures while maintaining accuracy, but diffusion-based approaches can suffer from limited diversity and slow sequential sampling. MoFlow jointly predicts K correlated trajectories with a flow-matching objective and distills the teacher into a one-step IMLE student. The resulting models achieve competitive or state-of-the-art performance across three datasets, while the student samples 100 times faster.

  • Problem

    Human trajectory forecasting must capture inherently multi-modal future motion while producing accurate and diverse trajectories.

  • Method

    MoFlow jointly predicts K correlated future trajectories with a diversity-promoting flow-matching loss, then uses sample-based IMLE distillation to train a one-step student.

  • Results

    The teacher and student achieve competitive or state-of-the-art performance across NBA SportVU, ETH-UCY, and SDD, while the student provides a 100-fold sampling-speed increase.

  • Takeaways & Limitations

    One-step IMLE distillation preserves the final performance across all three datasets while eliminating iterative denoising during sampling.

  • Takeaways & Limitations

    The paper identifies extending the approach with semantic maps and scene graphs as future work for encoding lanes, sidewalks, and obstacles and broadening applicability.

Abstract

from arXiv · show

In this paper, we address the problem of human trajectory forecasting, which aims to predict the inherently multi-modal future movements of humans based on their past trajectories and other contextual cues. We propose a novel motion prediction conditional flow matching model, termed MoFlow, to predict K-shot future trajectories for all agents in a given scene. We design a novel flow matching loss function that not only ensures at least one of the $K$ sets of future trajectories is accurate but also encourages all $K$ sets of future trajectories to be diverse and plausible. Furthermore, by leveraging the implicit maximum likelihood estimation (IMLE), we propose a novel distillation method for flow models that only requires samples from the teacher model. Extensive experiments on the real-world datasets, including SportVU NBA games, ETH-UCY, and SDD, demonstrate that both our teacher flow model and the IMLE-distilled student model achieve state-of-the-art performance. These models can generate diverse trajectories that are physically and socially plausible. Moreover, our one-step student model is $\textbf{100}$ times faster than the teacher flow model during sampling. The code, model, and data are available at our project page: https://moflow-imle.github.io

1. Introduction

Human trajectory forecasting must balance precise prediction with diverse future movements because human motion is intrinsically multi-modal. MoFlow addresses diffusion-model limitations by jointly generating diverse trajectories and distilling flow-based inference into a faster student model.

  • Motivation: Human trajectory forecasting must predict precise futures while representing diverse movements arising from inherently multi-modal human motion.The task supports applications including autonomous driving, drones, and human-robot interaction.
  • Limitations of prior methods: Diffusion models can produce overlapping independently sampled trajectories, limiting spatial diversity when futures should diverge across directions and regions.Their sequential sampling process also requires many neural function evaluations.
  • MoFlow: MoFlow jointly models multiple future trajectories for each agent and uses a flow-matching loss to promote diverse, multi-modal predictions.The model combines a social temporal encoder with a motion decoder that predicts K-shot trajectories for all scene agents.
  • Distillation: IMLE distillation trains a one-step flow-model student using teacher samples, avoiding the need for teacher-model gradients or a full iterative sampling process.The supplied contribution passage characterizes the method as efficient because it only requires samples from the teacher model.
  • Reported outcome: MoFlow and its distilled student achieve state-of-the-art performance across SportVU NBA, ETH-UCY, and SDD datasets while balancing trajectory accuracy and diversity.The paper reports this cross-dataset outcome as a principal contribution.

2. Related work

Prior work uses generative models to represent uncertain human futures, while diffusion and flow models improve distribution modeling but retain diversity and sampling-efficiency challenges. Existing distillation methods either approximate iterative dynamics or align distributions, with LED reported to trade faster inference for sample quality.

  • Generative trajectory forecasting: GANs, variational methods, conditional normalizing flows, and diffusion models have been applied to capture multi-modal human trajectories.Diffusion models are described as achieving strong performance and representing future multi-modality.
  • Diffusion and flow models: Diffusion and flow-matching models transform noise into data through learned reverse processes or ODE-based dynamics across generative tasks.The related-work discussion presents them as closely related model families inspired by nonequilibrium thermodynamics.
  • Distillation strategies: Distillation approaches either train students to approximate teacher sampling dynamics or match the teacher’s output distribution.Examples include knowledge distillation, progressive distillation, rectified flow, adversarial objectives, and distribution-matching objectives.
  • Known trade-offs: LED accelerates human-motion diffusion inference with a deterministic initializer but is often associated with discernible deterioration in sample quality.The method predicts trajectories at an intermediate diffusion time τ much smaller than the total time T.

3. Background

The forecasting problem models future multi-agent trajectories conditioned on observed motion and contextual information. Flow matching linearly connects Gaussian noise to data, learns a contextual velocity field, and samples trajectories by numerically integrating that field.

  • Motion prediction: Observed trajectories X contain each agent’s past positions, while Y contains each agent’s ground-truth future coordinates across Tf frames.Context C summarizes features such as absolute and relative coordinates and velocity.
  • Motion prediction: The goal is to learn pϕ(Y|C), a probabilistic model of multi-agent future trajectories conditioned on contextual information, capturing motion uncertainty and multi-modality.Training uses a maximum-likelihood objective over the data distribution of contexts and future trajectories.
  • Flow matching: Flow matching linearly interpolates data trajectories Y1 and Gaussian noise Y0 to form the intermediate flow state Yt.The flow time t belongs to [0,1) and is distinct from motion-prediction time.
  • Flow matching: The target velocity field Ut defines the probability path from noise to data, while a neural network vθ learns to approximate it using contextual information C.For the linear flow, the target field has constant velocity and is determined by samples from the prior and data distributions.
  • Sampling: Sampling begins with standard-normal noise and iteratively updates the state toward t=1 using vθ, equivalent to Euler integration of the underlying ODE.The learned velocity field is conditioned on the motion context.

4. Method

MoFlow combines attention-based scene encoding, correlated K-shot flow predictions, and a multi-modal objective for diverse trajectory sets. Conditional IMLE distills the teacher into a one-step student while preserving trajectory-set matching and offering flexible, efficient generation.

  • Context Encoding: An attention-based context encoder processes each agent’s historical state and models inter-agent dynamics with Transformer self-attention.The encoder produces agent features for the motion decoder and can use agent-specific positional characteristics.
  • Multi-modal Learning Objective: The combined regression and classification objective trains at least one accurate trajectory while promoting diverse, plausible motion patterns across the K predictions.This addresses the inability of independently sampled trajectories to enforce coherence, reduce self-collision, and ensure diverse mode coverage.
  • Flow Matching for Motion Prediction: The data-prediction network Dθ maps a noisy trajectory, context, and flow time to a one-step future-trajectory estimate that can be reparameterized for ODE sampling.Its formulation is equivalent to the flow-matching objective in data space.
  • Flow Matching for Motion Prediction: The model predicts K correlated scene-level waypoint trajectories rather than independent samples, alongside classification logits for the K predictions.The shared-noise construction keeps the K interpolated components aligned during training, while sampling initializes all components from standard normal distributions.
  • IMLE Distillation: Conditional IMLE trains a student generator from teacher samples by selecting the nearest of m samples and minimizing a Chamfer distance between correlated trajectory sets.The student generates K-component trajectories conditioned on context, while Chamfer distance compares trajectory sets across all agents.
  • IMLE Distillation: IMLE provides an architecturally flexible and comparatively stable distillation framework that avoids the mode collapse and vanishing-gradient issues associated with adversarial models.The student can share the teacher’s architecture apart from time-step embeddings.

5. Experiments

MoFlow is evaluated on NBA, ETH-UCY, and SDD against established baselines, with strong accuracy across datasets and substantially faster one-step sampling after IMLE distillation.

  • Datasets and baselines: The experiments cover NBA, ETH-UCY, and SDD, using established trajectory-forecasting baselines where comparable results are available.NBA forecasts 20 future frames from 10 past frames; ETH-UCY and SDD forecast 12 future frames from 8 past frames.
  • Quantitative results: 0.71 minADE and 0.87 minFDE on NBA improve over the previous state of the art by more than 12% and 25%, respectively.Both the teacher and student methods achieve these NBA results.
  • Quantitative results: On ETH-UCY, MoFlow matches the best available methods, while on SDD it achieves state-of-the-art minADE and competitive minFDE within 0.5 of the best model.The paper attributes the ETH-UCY performance plateau to dataset size and rapidly advancing expressive models.
  • Distillation and efficiency: 100 times faster sampling preserves largely unchanged performance across all three datasets after IMLE distillation.The one-step student removes the time-consuming denoising ODE-solving process.
  • Qualitative results: Qualitative results show closer ground-truth alignment than LED on NBA and frequent alignment with pedestrian ground truth on ETH-UCY.NBA visualizations also showcase diverse predictions from professional basketball scenes.
  • Distillation ablation: IMLE distillation is reported as fast and stable, whereas DCGAN- and WGAN-based distillation methods fail to deliver satisfactory performance.The comparison is conducted on the NBA dataset using min20ADE and min20FDE.

6. Conclusion

The paper introduces MoFlow for jointly generating diverse multimodal trajectories and an IMLE-based one-step distillation approach. Qualitative results support accurate, diverse predictions across NBA and ETH-UCY, while future work targets richer environmental context.

  • Contributions: MoFlow jointly models multiple future trajectories with a flow model and uses a flow-matching loss designed to capture human-motion multimodality.The objective is to learn a diverse set of future trajectories rather than a single trajectory per agent.
  • Contributions: The IMLE distillation method strategically matches multimodal future trajectories through a latent variable for one-step generation.The paper presents this as its first one-step IMLE distillation model.
  • Qualitative evidence: NBA visualizations compare best-of-20 MoFlow IMLE predictions with LED and ground truth, while also showing diverse samples from the IMLE generator.The pink trajectory is the closest of K = 20 predictions to ground truth in L2 distance.
  • Qualitative evidence: ETH-UCY best-of-20 predictions selected by lowest FDE closely match ground-truth trajectories and capture important motion nuances.The qualitative comparison concerns the MoFlow IMLE distillation model.
  • Training objective: The method addresses the single-observed-future limitation by using data-space prediction and a multimodal learning loss.The paper explains that vanilla flow matching cannot be directly adopted when training data contain only one future trajectory per context.
  • Training objective: The reparameterized model Dθ predicts data at time t = 1 while implicitly learning the vector field vθ through a linear transformation.The paper states that this rearranges network modules without changing the vanilla framework’s loss functions.

A.2. MoFlow Teacher Model Training

MoFlow trains a flow-based teacher to generate K correlated trajectory predictions, using shared noise, time-dependent masking, and a loss that balances multimodality with accurate mode coverage. The training design addresses overfitting near clean data and uses iterative denoising during sampling.

  • Multi-modal teacher training: MoFlow trains K correlated scene-level trajectory predictions jointly rather than independent samples, supporting coherent multimodal forecasts.The model uses tied noise across components because untied noise introduces excessive variability and makes convergence more difficult.
  • Overfitting mitigation: The data-space loss can overfit when t approaches 1 because the noisy input becomes highly similar to the clean trajectory.As t approaches 1, the Gaussian variance (1 − t)^2 decreases quadratically, encouraging the model to rely excessively on the input.
  • Overfitting mitigation: A flow-time-dependent masking mechanism masks the noisy embedding with probability fm(t), using an S-shaped logistic schedule with k = 20 and m = 0.5.The mechanism acts as training-only embedding-level dropout to encourage the model to extract useful context signals.
  • Teacher sampling: Teacher sampling starts each component from standard normal noise and iteratively updates the K trajectories toward t = 1 using the predicted vector field.The implementation uses a nonlinear time mapping with T = 100 sampling steps and p = 5.

A.4. Network Architecture

MoFlow uses dataset-specific encoders to model temporal and social interactions, followed by a decoder that models interactions among K scene predictions and agents. Both MoFlow variants are smaller than the LED initializer.

  • Encoder: MoFlow uses a spatio-temporal transformer encoder for ETH-UCY and SDD, while NBA uses a PointNet-like encoder.The encoder jointly processes context and noise embeddings to capture complex interactions over time and space.
  • Decoder: The motion decoder adds factorized self-attention over the sample and agent dimensions to capture interactions among K predictions.The decoder contains four blocks whose dimensions and attention heads match the encoder.
  • Model size: ∼1M fewer parameters than the LED initializer are used by both the MoFlow and IMLE models.

A.5. Model Training Details

Training normalizes relative motion and uses positional encodings plus alternating attention to strengthen spatial relationships among agents and predicted trajectories.

  • Preprocessing: Future relative motion is min-max normalized to [−1, 1] to facilitate flow-matching training.A basic transformer serves as the backbone without requiring special training tricks.
  • Positional encoding and attention: Sinusoidal positional encodings are applied across agents and prediction levels, with alternating self-attention reinforcing inter-agent interactions.

B. Further Qualitative Results

Qualitative NBA comparisons show MoFlow producing diverse, contextually plausible trajectories that more closely follow basketball movement patterns and ground truth than LED. IMLE distillation preserves empirical performance while substantially reducing sampling cost.

  • NBA qualitative results: MoFlow avoids implausible opposite-direction trajectories and observed backcourt violations that appear in LED predictions on NBA scenes.The qualitative comparison reports more realistic and contextually appropriate basketball trajectories from MoFlow.
  • NBA qualitative results: MoFlow’s best-of-20 predictions more closely align with ground-truth futures than LED predictions in NBA visualizations.The comparison highlights subtle differences among MoFlow IMLE predictions, LED predictions, and ground truth.
  • Prediction plausibility: MoFlow generates K scene-level predictions with classification logits that distinguish the plausibility of trajectories within the empirical distribution.The predicted trajectories may be widely distributed while still receiving differentiated plausibility estimates.
  • Sampling speed-up: 100x fewer NFEs reduce student-model runtime by 98% on the same hardware, while IMLE distillation does not compromise empirical performance in this setup.Both teacher and student models are reported to sample faster than state-of-the-art methods while delivering superior performance.

C.2. Ablations on Flow Matching Configurations

The ablation evaluates MoFlow’s flow-matching configuration and finds that prediction-level positional encoding and shared noise provide the strongest configuration without variance explosion.

  • The ablation study evaluates the contributions of different MoFlow components.
  • Table 5 presents the superiority of the selected input-output dimension adaptation and time-schedule configuration.
  • Prediction-level positional encoding provides a greater advantage than agent-level positional encoding.
  • Shared noise yields strong final results without the variance explosion observed in another configuration.

C.3. Ablations on IMLE Configurations

The IMLE ablations examine sampling and distillation behavior, including the choice of student samples and preservation of the teacher model’s sample quality.

  • m = 20 is selected because it delivers superior empirical performance, while larger m values increase training-time memory usage and may cause out-of-memory issues.
  • The IMLE configuration is evaluated using multiple generated trajectories and qualitative comparisons on ETH-UCY and NBA scenes.
  • IMLE preserves the teacher model’s MASD with a deviation of only ∼0.1m, whereas LED deviates by ∼9m.
Loading 2503.09950v1…