PaperlayerView exact revision on arXiv

Source-linked AI summary

EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting

Wen Wang, Ruibing Hou, Hong Chang, Shiguang Shan, Xilin Chen

arXiv:2608.22449v1cs.ROcs.AI

TL;DR

Egocentric hand-motion forecasting must capture manipulation progression, but existing methods leave planning implicit and jointly optimize planning representations with motion synthesis. EMPIRE learns explicit plans first, then generates bimanual motion from frozen planner representations, achieving strong accuracy and efficiency while introducing a large plan-annotated dataset.

  • Problem

    Existing VLM-based forecasters directly predict future motion and jointly optimize semantic learning with motion generation, leaving manipulation planning implicit and risking disruption of manipulation-aware representations.

  • Method

    EMPIRE decomposes forecasting into a planner that learns explicit, temporally ordered hand-specific manipulation plans and an actor that generates motion from frozen planner hidden states.

  • Results

    EMPIRE consistently outperforms existing methods under a unified protocol, reducing MPJPE by 19.8% versus VITRA and matching Being-H0-14B accuracy with 83.5× faster end-to-end inference.

  • Takeaways & Limitations

    Explicit manipulation planning provides a structured intermediate interface for long-horizon bimanual hand-motion forecasting, with the reported benefits extending to accuracy, efficiency, and fine-grained finger articulation.

Abstract

from arXiv · show

Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to future motions, overlooking the underlying manipulation process that governs hand-object interactions. Moreover, end-to-end optimization couples manipulation learning with motion synthesis, causing motion-generation gradients to interfere with the pre-learned manipulation-aware representations. To overcome these limitations, we propose EMPIRE, a two-stage framework that introduces Explicit Manipulation Planning as an Intermediate Representation for Egocentric hand-motion forecasting. Stage I: Learn to Plan. EMPIRE first learns explicit manipulation plans from multimodal context to capture the progression of hand-object interactions. Stage II: Learn to Act. A motion generator synthesizes future bimanual hand motions conditioned on frozen planner representations, preventing motion-generation gradients from affecting manipulation planning. To support our method, we further construct EMPIRE-651K, a bimanual hand-motion forecasting dataset comprising 650,910 training windows across 111 tasks, each paired with an explicit per-hand manipulation plan. Under identical training and evaluation protocols, EMPIRE achieves state-of-the-art forecasting accuracy, with an MPJPE of 84.53 mm and a finger-relative error of 38.97mm. We release the code and dataset at https://github.com/wangwen-banban/EMPIRE.

Introduction

EMPIRE addresses two limitations in egocentric hand-motion forecasting: implicit manipulation planning and coupled optimization of semantic representations with motion generation. It introduces a two-stage plan-then-act framework and reports improved forecasting efficiency and accuracy under a unified protocol.

  • Method: Stage II synthesizes future hand motion using frozen planner representations, separating manipulation planning from motion generation.Figure 1 summarizes this design as Stage I learning to plan and Stage II learning to act with the planner frozen.
  • Motivation: Existing methods directly map VLM representations to future hand motion, leaving the manipulation process implicit.This can produce locally plausible motion that diverges from intended interaction progression, especially over long horizons.
  • Motivation: Motion-generation gradients can update the semantic representation used for conditioning, weakening manipulation-aware structure learned during VLM pretraining.The coupled optimization creates a dynamically changing feature space for the motion generator.
  • Method: EMPIRE first learns explicit, motion-oriented manipulation plans from egocentric RGB, monocular depth, a coarse instruction, and camera field of view.The plan captures the progression of hand-object interactions before motion synthesis.
  • Dataset and Results: 650,910 training windows across 111 manipulation tasks form EMPIRE-651K, each augmented with explicit per-hand manipulation plans.The dataset is constructed from EgoDex to support the proposed planning supervision.
  • Dataset and Results: 19.8% lower MPJPE than VITRA and 38.8% lower optimization time are reported, while matching Being-H0-14B accuracy with 83.5× faster end-to-end inference.These comparisons use a unified training and evaluation protocol.

Related Work

Related work connects multimodal perception to embodied planning and hand action generation. Existing systems include VLM-based planners, robot-action policies, and flow-based action experts, while EMPIRE targets egocentric bimanual hand-motion forecasting.

  • Vision-Language Planning: VLMs provide open-vocabulary scene understanding from visual and linguistic inputs but remain limited in metric-scale perception and fine-grained 3D spatial reasoning.These limitations matter for spatial interaction and motivate using VLMs as high-level planners in embodied systems.
  • Vision-Language Planning: Embodied systems such as SayCan and PaLM-E use VLMs for high-level or multimodal sequential planning grounded in environmental context.The cited examples illustrate planning-oriented uses of multimodal models before execution.
  • Hand Motion Generation: Hand-motion and robot-action generators map visual-language inputs to actions across settings including robot control, heterogeneous embodiments, and shared latent action spaces.Examples include RT-2, OpenVLA, Octo, π0, and XL-VLA.

Method

EMPIRE separates manipulation planning from motion synthesis by learning an explicit plan from multimodal egocentric observations, then conditioning a motion generator on frozen planner representations. Its two-stage pipeline uses plan hidden states as the interface between perception and future bimanual motion generation.

  • Overview: EMPIRE uses a two-stage plan-then-act pipeline in which a Planner produces explicit manipulation plans and an Actor transforms their hidden states into future two-hand motion.The Stage-I VLM is frozen during Stage-II motion-generator training.
  • Problem Formulation: The forecasting target is a continuous future bimanual trajectory conditioned on egocentric RGB observations, a coarse action caption, camera field of view, and the current two-hand state.The trajectory must remain continuous with the current state and consistent with the manipulation context.
  • Problem Formulation: EMPIRE introduces a structured plan of temporally ordered, hand-specific sub-actions as an intermediate representation between multimodal observations and motion.The plan makes the manipulation process explicit rather than learning p(A | O, s0) directly.
  • Stage I: Learn to Plan: Stage I constructs a multimodal prefix from RGB, inferred monocular depth, caption, and horizontal and vertical FoV tokens, then autoregressively predicts a motion-oriented plan.RGB supplies appearance, depth supplies complementary geometry, and caption and FoV specify the task and camera configuration.
  • Stage II: Learn to Act: Stage II caches planner-generated plans, freezes the Stage-I Planner, projects the generated plan hidden states, and conditions a flow-matching DiT on them and the current hand state.The DiT cross-attends to plan conditioning while the current-state encoding supplies the initial-pose condition.
  • Stage II: Learn to Act: Freezing the Planner prevents the motion-generation objective from altering the learned planning representation and avoids back-propagation through the Planner.During deployment, the frozen Planner predicts the plan online once, and the Actor uses its hidden-state span to generate future motion.

Dataset: EMPIRE-651K

EMPIRE-651K converts EgoDex recordings into aligned bimanual forecasting windows and augments them with explicit, temporally ordered manipulation plans. The dataset construction includes MANO fitting, episode segmentation, VLM annotation, consistency filtering, and human quality auditing.

  • Dataset construction: The dataset contains 650,910 forecasting windows across 111 manipulation tasks, constructed from EgoDex using the official training partitions.The held-out test partition is excluded from dataset construction for training.
  • Skeleton-to-MANO conversion: EgoDex recordings are converted into MANO-based hand representations by wrist-frame transformation, chirality correction, and sequence-level fitting.The conversion supplies the parametric hand representation required for motion forecasting.
  • Temporal window construction: Recordings are divided into task-specific episodes and non-overlapping five-second windows to preserve correspondence between observations, instructions, and future motion.This avoids unrelated task transitions within forecasting sequences.
  • Caption and motion-plan annotation: Qwen2.5-VL generates segment captions and caption-grounded, temporally ordered hand-specific plans, while Qwen3 filters inconsistent caption–plan pairs.The pipeline first identifies manipulation intent, then decomposes it into sub-actions.
  • Annotation quality: A human audit of 100 stratified training windows reports annotation quality of 90.41/100, with usable caption–plan pairs defined by both means reaching at least 4/5.The audit evaluates grounding, attribution, coverage, coordination, and usefulness for forecasting.
  • Annotation quality: The audited captions and plans provide supervision for the planning interface between perception and motion generation.This conclusion is tied to the reported human audit of annotation quality.

Experiment

Experiments train EMPIRE and a reimplemented VITRA baseline on the same five training partitions and evaluate them on a held-out test partition. The protocol also analyzes seen and unseen tasks in Part 1 ablations.

  • Data splits: EMPIRE and the reimplemented VITRA baseline are trained on 650,910 windows from all five training partitions, with the test partition excluded.The training data span 111 manipulation tasks.
  • Implementation details: The model uses a PaliGemma-2-3B backbone with a 182M DiT-Base flow-matching generator trained in two stages.The planner is fine-tuned for 1 epoch, then the VLM is frozen while the action generator is optimized for 4 epochs.
  • Evaluation protocol: Evaluation uses a held-out test partition containing 6,836 five-second forecasting windows from 111 manipulation tasks.The same test set is used for full training and Part 1 ablation models.

26 training

EMPIRE consistently improves forecasting accuracy and efficiency, with explicit planning especially beneficial for difficult tasks. Ablations show that plans, direct depth integration, frozen planner representations, and balanced actor capacity each matter.

  • Forecasting accuracy: EMPIRE achieves the best overall performance, reducing MPJPE by 19.8% versus VITRA while improving finger articulation over larger Being-H0 variants.The largest Being-H0 model localizes wrists better, but EMPIRE produces more accurate finger motion over long horizons.
  • Training and inference efficiency: 71 hours on 8 A40 GPUs reduces training time by 38.8% versus VITRA’s 116 hours.EMPIRE generates each forecasting window in 1.0 second and is 83.5× faster than Being-H0-14B.
  • When does motion planning help?: 49.7 mm average MPJPE reduction on hard tasks exceeds the 4.6 mm reduction on easy tasks, while planning improves 89 of 111 tasks.The planning gain correlates negatively with baseline MPJPE (r = −0.567) and extends to 74 of 85 unseen tasks.
  • Model components: 22.26 mm lower overall MPJPE results when explicit motion plans replace caption-only conditioning, with unseen-task error falling from 129.73 mm to 104.80 mm.The ablation uses the VITRA re-implementation as the caption-only starting point.
  • Model components: 95.42 mm overall MPJPE is achieved by direct depth concatenation, whereas cross-attention over the same depth features does not improve the plan-only model.The depth-augmented model mainly benefits unseen tasks, indicating a trade-off between generalization and fitting the smaller ablation subset.
  • DiT capacity: DiT-B achieves the best overall and unseen-task performance, balancing motion capacity and generalization better than DiT-M or DiT-L.DiT-L improves seen-task performance but degrades unseen-task generalization, while DiT-M lacks sufficient capacity.

Conclusion

EMPIRE combines explicit manipulation planning with decoupled motion learning for long-horizon bimanual hand-motion forecasting. Its benchmark and experiments support this plan-then-act paradigm as accurate and efficient.

  • Conclusion: EMPIRE inserts an explicit motion plan between vision-language understanding and long-horizon bimanual hand-motion forecasting.Stage I learns manipulation-aware plans, while Stage II trains a flow-matching motion generator with the VLM frozen.
  • Conclusion: EMPIRE-651K contains 650,910 training windows across 111 manipulation tasks with temporally aligned MANO trajectories, instructions, and motion-plan supervision.The benchmark is constructed from EgoDex by converting skeleton annotations into aligned forecasting data.
  • Conclusion: Experiments demonstrate state-of-the-art forecasting accuracy for EMPIRE under the reported evaluation protocols.The conclusion attributes the result to explicit manipulation planning combined with decoupled motion learning.

Data-Annotation Prompts

EMPIRE-651K annotations are built through staged caption generation, caption-grounded motion-plan construction, and consistency filtering. The prompts enforce hand-specific, temporally ordered descriptions while removing incompatible caption–plan pairs.

  • Caption generation: Qwen2.5-VL first generates a coarse caption from each five-second egocentric video.The caption-generation prompt focuses separately on left- and right-hand actions and requires a fixed output format.
  • Caption–plan consistency filtering: Qwen3 labels each caption–plan pair CONSISTENT or INCONSISTENT according to task, object, hand-role, and goal compatibility.Inconsistent samples are removed before aligned supervision tuples are constructed.
  • Caption-grounded motion-plan construction: The motion-plan prompt decomposes visible hand actions into one temporally ordered sequence of fine-grained sub-actions consistent with the video and caption.It requests contact, grasp, release, lift, placement, opening, closing, rotation, and stabilization events when visible.
  • Annotation constraints: The prompts require 40 to 120 words for generated content and prohibit invented objects or actions contradicting the caption.The motion-plan output uses an XML-like <cot> tag with numbered steps.

Human Audit of Annotation Quality

The human audit evaluates caption and manipulation-plan quality across 100 windows from distinct episodes. Results indicate high annotation quality, with remaining issues concentrated in hand attribution and plan details.

  • Audit design: 100 windows from distinct episodes were manually inspected using six criteria covering caption grounding, hand attribution, plan grounding, plan coverage, temporal coordination, and forecasting usefulness.The audit sampled 20 windows from each of five training partitions, using the same 60-frame, five-second configuration as training.
  • Audit results: 4.62/5 overall audit quality was achieved, with annotation quality remaining high across all five training partitions.Partition-level scores ranged from 4.46 to 4.76.
  • Audit results: 84.0% of caption-plan pairs were considered usable under the criterion C ≥4 and P ≥4.C averages the two caption-related criteria, while P averages the four plan-related criteria.
  • Error analysis: The most frequent annotation errors involved caption hand/None attribution, plan hand-role assignment, and hallucinated plan steps.These occurred in 9/100, 8/100, and 6/100 cases, respectively.
  • Qualitative examples: Qualitative audit examples show captions identifying visible hand roles and tasks while plans decompose interactions into temporally ordered, semantically consistent steps.Figure 5 presents ordered ground-truth frames, captions, and motion-plan steps from audited windows.

Training and Evaluation

Training and evaluation use common five-second, bimanual forecasting protocols, while EMPIRE separates planning from motion generation. This design reduces optimization and inference costs while preserving fine-grained motion quality.

  • Evaluation protocol: 6,836 test windows across all 111 tasks are evaluated with 60 future frames at 12 FPS and K=8 stochastic predictions under best-of-8 MPJPE selection.Predictions cover both hands and are evaluated after recovering joint trajectories.
  • Optimization cost: 39% lower optimization cost is achieved than the single-stage baseline, requiring 71 hours instead of 116 hours.Stage I updates the 3B VLM for one epoch, while Stage II freezes it and trains the 182M DiT.
  • Optimization cost: The efficiency gain comes from decoupling semantic learning from motion generation, so only the lightweight DiT is updated during motion-synthesis training.The expensive VLM backbone is optimized only during plan learning.
  • Inference cost: 1.0 s per test window is required by EMPIRE, compared with 15.6–83.5 s for Being-H0 variants under the same end-to-end protocol.The timing includes preprocessing, inference, and trajectory decoding for one GPU-generated best-of-8 prediction.
  • Inference cost: The model reuses one VLM-generated plan representation to condition eight DiT samples, avoiding repeated autoregressive motion-token generation.The plan contains at most 96 autoregressive tokens, while motion synthesis uses horizon-parallel flow matching with four Euler steps.
  • Inference cost: The caption-only baseline requires 0.4 s per window, making EMPIRE’s added cost the short plan-generation step rather than autoregressive motion synthesis.The comparison frames explicit planning as a balance between semantic guidance and inference efficiency.

Ablation Analysis

The ablation analysis tests predicted-plan conditioning and examines where explicit planning helps. Gains are strongest for difficult, unseen, and long-horizon tasks, but are not uniform across the benchmark.

  • Plan conditioning: Stage II is trained with Stage-I predicted plans to match deployment conditioning and avoid the train–test mismatch caused by using cleaner ground-truth plans.Stage I remains frozen, so predicted-plan training changes conditioning data without adding optimization cost.
  • Task-level gains: 89/111 tasks improve with self-generated plans, yielding a 22.3 mm sample-weighted average gain and a 14.3 mm median per-task improvement.Negative per-task MPJPE differences indicate improvement.
  • Seen and unseen tasks: Self-generated plans improve 74/85 unseen tasks versus 15/26 seen tasks, with weighted improvements of −24.8 mm and −10.9 mm, respectively.The unseen-task improvement is more than twice the seen-task improvement.
  • Long-horizon tasks: The largest reported gains occur on complex, long-horizon tasks such as wash_fruit, stock_unstock_fridge, wash_put_away_dishes, and flip_pages.Their MPJPE changes are 343→154 mm, 303→185 mm, 234→126 mm, and 164→58 mm, respectively.
  • Limitations of gains: Performance degradation is relatively rare and occurs mainly on simple low-error tasks or highly challenging tasks with potentially inaccurate predicted plans.The results therefore indicate that planning improves average performance without benefiting every task.
  • Qualitative forecasts: EMPIRE better preserves temporal bimanual interactions and finger articulations over long horizons than VITRA and Being-H0 in qualitative comparisons.The comparisons use five future RGB frames from the 60-frame, 12 FPS forecasting horizon.
Loading 2608.22449v1…