PaperlayerView exact revision on arXiv

Source-linked AI summary

From Where to How: Continuous 4D Interaction Forecasting from Egocentric Video

Qiaohui Chu, Haoyu Zhang, Meng Liu, Haoxiang Shi, Dongmei Jiang, Liqiang Nie

arXiv:2609.08636v1cs.CVcs.AI

TL;DR

Egocentric 4D forecasting must anticipate both future interaction locations and the body motion that realizes them, but prior approaches insufficiently capture their continuous temporal and geometric correspondence. The paper introduces Coherent4D and HIGFlow, a cascaded where-to-how framework that predicts locations first and conditions pose forecasting on them. Experiments report consistent improvements on both location and pose forecasting across three domains, with ablations validating the proposed components.

  • Problem

    Existing methods often model interaction localization and body motion separately, leaving their continuous temporal and geometric correspondence insufficiently captured.

  • Method

    Coherent4D pairs synchronized continuous future 3D locations with full-body poses, while HIGFlow uses predicted locations to condition a deterministic motion anchor and residual Flow Matching.

  • Results

    HIGFlow achieves consistent improvements over representative baselines on interaction location and full-body pose forecasting across the evaluated domains.

  • Takeaways & Limitations

    Jointly modeling future interaction locations and full-body motion provides a continuous framework for coordinated 4D interaction forecasting.

Abstract

from arXiv · show

Egocentric 4D interaction forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize them, providing an important capability for assistive robotics and human-computer interaction. Existing methods struggle to translate semantic understanding into precise continuous 3D localization and to balance motion diversity with structural consistency in pose forecasting. More fundamentally, these tasks are often modeled separately, leaving the continuous geometric and temporal correspondence between interaction locations and body motion insufficiently captured. To address these challenges, we introduce Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting, comprising approximately 233K samples across three domains. Each sample pairs a sequence of future 3D interaction locations with corresponding full-body poses, aligned in time and expressed in a shared coordinate system. We also provide evaluation metrics in continuous space. Building on this formulation, we propose HIGFlow, a Hand Interaction Guided Residual Flow framework that models forecasting as a cascaded where-to-how process. HIGFlow first forecasts continuous future interaction locations by combining semantic grounding with short-horizon visual dynamics, and then uses the predicted location sequence to condition a deterministic motion anchor and residual Flow Matching for diverse yet structurally consistent full-body motion forecasting. Extensive experiments across all three domains demonstrate consistent improvements over representative baselines on both location and pose forecasting, while ablations validate the contributions of the proposed components. The project page is available at https://corrineqiu.github.io/from-where-to-how/.

I. INTRODUCTION

The paper identifies a gap in continuous, jointly grounded forecasting of future interaction locations and body motion. It addresses this gap with Coherent4D and HIGFlow, coupling temporally aligned location and pose prediction in a shared coordinate system.

  • Motivation: Existing methods largely treat interaction localization and full-body pose forecasting as separate or loosely connected problems.This limits explicit propagation of continuous interaction locations as geometric constraints for motion forecasting.
  • Limitations of prior work: Prior datasets rarely pair ordered continuous 3D interaction locations with full-body poses at matched future timestamps in a shared metric space.FIction links the tasks but uses discrete voxel occupancy and candidate-location conditioning instead of continuous synchronized sequences.
  • Coherent4D: Coherent4D provides approximately 233K samples pairing ordered future 3D interaction locations with temporally aligned full-body poses in a shared coordinate system.The dataset spans Cooking, Health, and Bike Repair and supports continuous-space evaluation.
  • HIGFlow: HIGFlow uses a cascaded where-to-how formulation, forecasting continuous future locations before conditioning full-body motion on those predictions.This explicitly couples interaction localization with motion realization.
  • HIGFlow: Semantic-Dynamic Location Forecasting separates semantic grounding from continuous metric localization, while Hand-Conditioned Residual Flow Matching models diverse motion around a deterministic anchor.The motion module is designed to produce diverse yet structurally consistent full-body poses.
  • Results: Experiments show consistent improvements for HIGFlow on interaction location and full-body pose forecasting across the evaluated domains.The authors also report benefits from jointly modeling future locations and full-body motion.

B. 3D Interaction Location Forecasting

The paper frames 3D interaction location forecasting as predicting ordered metric hand-interaction locations from egocentric observations. Coherent4D supports this task with synchronized data construction, semantic-geometric context, and a shared coordinate representation.

  • Task formulation: 3D interaction location forecasting predicts ordered metric hand-interaction locations indicating where upcoming hand-environment interactions will occur.The task extends beyond image-space trajectories toward continuous metric 3D prediction.
  • Related methods: Recent methods combine semantic or VLM-assisted grounding with predictive visual dynamics, but continuous metric regression remains insufficiently integrated with both.The paper identifies semantic reasoning, metric regression, and short-horizon dynamics as distinct ingredients for localization.
  • Dataset support: Coherent4D pairs ordered 3D interaction locations with SMPL-parameterized full-body poses at matched future timestamps in a shared sample-local frame.Each sample includes observed histories and future location-pose targets from a 30-second observation window.
  • Annotation pipeline: The annotation pipeline grounds scene objects, constructs shared coordinates, builds location sequences, attaches SMPL states, and generates forecasting samples.This preserves temporal and geometric coupling between locations, body motion, and scene context.
  • Scene context: Object context includes semantic categories and continuous 3D bounding-box attributes obtained by lifting egocentric detections through SLAM reconstruction.The retained box attributes include center, size, and orientation.
  • Coordinate construction: Spatial quantities are transformed into a shared sample-local coordinate frame and normalized by dividing by 5 m before clipping each coordinate to [−1, 1].The transformation is applied to interaction locations, object centers, SMPL root translations, and body joints.

3) Location Sequence Construction:

The dataset construction converts sparse interaction annotations into temporally ordered continuous 3D location sequences, aligns them with body states, and forms forecasting samples that preserve event timing.

  • Interaction events are identified from narration timestamps, object matching, and hand-object geometric consistency, then assigned continuous 3D locations from hand meshes.
  • Each interaction location is paired with the full-body SMPL state nearest its timestamp and transformed into the sample-local coordinate frame.
  • Samples use a preceding 30-second egocentric observation window and retain nonuniform future-event timing through absolute and relative timestamps.
  • 233,828 samples span 787 takes, three domains, 535 interaction-object categories, and forecasting horizons of 10, 5, and 4 steps for Cooking, Health, and Bike Repair.
  • Continuous-space evaluation converts normalized locations to metric coordinates and reports ADE, ADE90, and FDE for sequence, upper-tail, and endpoint localization accuracy.

2) Full-body Pose Forecasting Metrics:

Pose forecasting is evaluated with complementary geometric and rotational metrics, while Single and Best-5 protocols assess individual and multimodal predictions.

  • MPJPE, PA-MPJPE, Root Trans., and Body Geo. measure absolute joint accuracy, aligned pose accuracy, global displacement, and local rotational error.
  • MPJPE averages Euclidean distances between corresponding predicted and ground-truth joints across valid future poses.
  • PA-MPJPE evaluates pose accuracy after aligning predicted joints to ground truth with an optimal similarity transformation.
  • Root Trans. measures Euclidean error in SMPL root translations, reflecting global body-displacement accuracy.
  • Body Geo. measures rotational discrepancy across the 23 non-root body joints using geodesic angular distance.
  • Single evaluates the first generated pose sequence, whereas Best-5 selects the candidate with the lowest joint-position error across five forecasts and applies it consistently to all pose metrics.

A. Problem Formulation

The formulation treats continuous 4D interaction forecasting as a coupled where-to-how problem: predict future interaction locations, then forecast aligned body motion conditioned on them.

  • The model first predicts an ordered sequence of future interaction locations from egocentric context.
  • Egocentric location context includes observed frames, environment descriptors, location history, and a task prompt.
  • Full-body pose forecasting takes observed pose history together with a future interaction-location sequence.
  • Training conditions pose prediction on ground-truth locations, while inference conditions it on locations predicted by the first stage.
  • Each predicted SMPL body state has 147 dimensions: 6D root orientation, 3D root translation, and 23 local joint rotations in continuous 6D form.

B. HIGFlow Framework

HIGFlow uses a cascaded where-to-how architecture: it forecasts continuous future interaction locations, then conditions full-body motion forecasting on the predicted location sequence. The location stage combines semantic grounding with visual dynamics and decodes normalized continuous 3D coordinates directly.

  • Framework overview: HIGFlow separates forecasting into Semantic-Dynamic Location Forecasting and Hand-Conditioned Residual Flow Matching, with predicted locations explicitly conditioning motion.This establishes a direct dependency between interaction-location progression and body motion.
  • Semantic-Dynamic Location Forecasting: The location stage forecasts ordered continuous 3D interaction locations by combining high-level semantic grounding with short-horizon visual dynamics.Qwen3-VL provides semantic representations, while frozen V-JEPA features provide complementary motion-sensitive information.
  • Semantic-Dynamic Location Forecasting: Qwen3-VL encodes egocentric frames, task prompts, location history, and environment descriptors into step-specific future representations.The representations are extracted at indexed future-step hand-trajectory placeholders.
  • Semantic-Dynamic Location Forecasting: V-JEPA features are projected into the language-model hidden space and fused through gated residual adapters that regulate dynamic information per future step.The gate adaptively controls the residual contribution, whose magnitude is bounded by α.
  • Continuous coordinate decoding: A coordinate decoder maps each fused representation directly to a normalized continuous 3D location, avoiding metric-coordinate generation as text tokens.The mask-aware ADE loss averages errors over valid future interaction targets, while Smooth L1 losses operate coordinate-wise in normalized space.

2) Hand-Conditioned Residual Flow Matching:

The pose stage conditions on observed pose history and future interaction locations, predicts a deterministic location-grounded anchor, and models bounded stochastic residuals around it with conditional Flow Matching. Multiple residual samples are integrated and composed with the anchor to produce plausible future motion sequences.

  • Pose forecasting design: The pose stage first predicts a location-conditioned deterministic anchor and then models stochastic residuals around it using conditional Flow Matching.This combines a structured reference motion with diverse residual variations.
  • Spatiotemporal conditioning and anchoring: A 24-node SMPL graph preserves skeletal topology while graph propagation captures dependencies among physically connected body parts before pose-history Transformer processing.The graph contains one root joint and 23 articulated body joints.
  • Spatiotemporal conditioning and anchoring: The future condition encoder represents interaction location, displacement, and relative future step for each forecasted time point.Location differences are defined between consecutive future steps, with the first displacement set to zero.
  • Deterministic anchor: The deterministic anchor estimates root translation, root orientation, and articulated body configuration as a location-grounded reference rather than the final prediction.It combines global pose history with the corresponding future-location representation.
  • Residual Flow Matching: Residual Flow Matching represents rotational and translational corrections in a 75-dimensional anchor-relative residual space.The residual comprises root rotation, root translation, and 23 body-joint rotation corrections.
  • Residual regulation: Step-specific gates and component-wise bounds regulate stochastic deviations from the anchor while allowing their magnitude to adapt across future interaction steps.The regulation separately controls root rotation, root translation, and body rotation residuals.
  • Inference: At inference, S residual priors are independently integrated with NODE Heun steps and composed with the deterministic anchor to produce S plausible future motion sequences.Kinematic composition applies rotational corrections through the exponential map and adds root translation corrections.

A. Experimental Settings

Experiments compare HIGFlow with specialized baselines for both forecasting stages under shared 3D settings. Pose comparisons use the same interaction-location conditioning protocol and report both Single and Best-5 results.

  • Baseline design: HIGFlow is compared with separate baseline groups corresponding to interaction-location forecasting and full-body pose forecasting.The evaluation is organized around the two stages of the cascaded framework.
  • Interaction location forecasting: Location forecasting baselines include FIction, Qwen3-VL-2B, V-JEPA 2.1, Diff-IP3D, and MMTwin.Diff-IP3D extends Diff-IP2D to continuous 3D interaction locations, while MMTwin is reproduced in its 3D configuration.
  • Evaluation protocol: Pose results are reported under Single and Best-5 protocols, while location methods are evaluated in a shared sample-local continuous 3D setting.The compared methods use matched prediction targets for the Coherent4D evaluation.
  • Full-body pose forecasting: Pose forecasting is compared against FIction, SkeletonDiffusion, and SLD-HMP under the same interaction-location conditioning protocol.All methods receive ground-truth future interaction locations during training, with interfaces adapted to the SMPL representation where needed.

2) Implementation Details:

The implementation uses multimodal semantic and dynamic features, staged training, and domain-specific pose-flow settings. Experiments show that location guidance, coordinate decoding, V-JEPA fusion, and the anchor–flow combination contribute to performance, with gains varying by domain.

  • Implementation details: The location stage uses Qwen3-VL and V-JEPA with M = 64 memory tokens, α = 0.1, and a three-layer coordinate decoder.The location loss weights are (λADE, λS1, λCE) = (1.5, 0.6, 0.01).
  • Implementation details: The pose anchor and residual-flow stages use separate optimization settings, with the flow stage freezing the pose-history encoder, future-condition encoder, and anchor predictor.The flow stage uses σr = 0.003 and domain-specific residual gates and bounds.
  • Representation and coordinate decoding: CoordDec provides the clearest and most consistent gains, while LocEnc and EnvEnc generally improve cross-domain consistency by preserving structured inputs.The full model achieves the most balanced performance across domains.
  • Input ablations: Removing location history or frame input substantially degrades forecasting, while environment context is particularly important for Bike Repair; V-JEPA mitigates degradation in most cases.The result supports complementary semantic and dynamic visual context.
  • V-JEPA memory budget: M = 64 achieves the best results on Bike Repair and remains competitive on Health and Cooking, providing the strongest overall trade-off across domains.Performance does not improve monotonically as the V-JEPA memory budget increases.
  • Location conditioning: Ground-truth future interaction locations generally improve pose forecasting accuracy relative to predicted locations, showing that downstream pose quality depends on localization quality.HIGFlow benefits from more accurate location guidance under both conditioning settings.
  • Pose ablations: Removing future-location conditioning causes the largest overall degradation, while the deterministic anchor improves structural stability and residual flow matching adds motion variation.Their combination achieves the strongest and most balanced performance across domains and metrics.
  • Integration steps: Four integration steps achieve the best result in 9 of 12 domain-metric comparisons, while increasing integration steps does not consistently reduce forecasting error.The analysis evaluates NODE step counts from 1 through 16 under the Best-5 protocol.

C. Qualitative Analysis

Qualitative examples show that HIGFlow uses future interaction locations to guide pose forecasting and can preserve task-relevant motion structure. They also expose error propagation over longer horizons and remaining mismatches in detailed contact and body configuration.

  • Qualitative comparison: HIGFlow preserves task-relevant pose structure and transitions more closely to reference motion in representative Bike Repair and Cooking examples.The comparison includes observed pose histories, synchronized exocentric frames, future references, ground truth, and predictions from HIGFlow and FIction.
  • Limitations: Interaction-location errors can grow over the forecast horizon and propagate through cascaded inference, leaving the predicted body too upright during leaning and reaching.
  • Limitations: Correct interaction regions do not always determine contacting-hand placement, contact height, arm configuration, or torso orientation.The examples indicate that location conditioning improves coarse spatial alignment but does not fully capture human intent, object affordances, or detailed contact constraints.
  • Method: HIGFlow combines continuous interaction localization with residual flow matching to produce diverse yet structurally consistent full-body pose forecasts.The framework is presented as a cascaded where-to-how model that uses future interaction locations as geometric conditions for pose prediction.
Loading 2609.08636v1…