LiMA: Bridging Long-term Imagination to Real-time Dexterous Manipulation via Asynchronous Diffusion

1 Introduction

• VLA models lack an explicit mechanism for anticipating and responding to continuous physical-world changes needed for robust, generalizable dexterous manipulation.The passage describes VLA progress from large egocentric manipulation datasets and broad visual, linguistic, and action priors.Source: §1 · p. 2
• IDM-based WAMs loosely couple imagined observations with action prediction, while integrated WAMs face prohibitive inference latency from long latent feature spaces.The passage identifies these as two current WAM framework paths.Source: §1 · p. 2
• LiMA combines asynchronous dual-system generation with Latent Schrödinger Bridge Coupling and Spatiotemporal Adaptive Modulation to align intent, refinement, and real-time observations.Its slow system generates long-horizon intent while its fast system refines motion at high frequency.Source: §1 · p. 2
• LiMA reports a stronger balance between task success and execution efficiency, with gains especially on long-horizon and contact-intensive tasks and lower latency than integrated WAM baselines.The evaluation covers diverse dexterous manipulation tasks across varying temporal horizons.Source: §1 · p. 2

2.1 Dexterous Manipulation

• VLA models [30] [31] [32] [33] [34] bring large-scale reasoning priors to control but often fail to ground abstract intent in fine-grained physical interactions.Earlier point-cloud methods [28] [1] [29] handled contact physics but lacked semantic depth.Source: §2.1 · p. 2
• WAMs [35] [36] [37] [38] [40] use video synthesis to model pixel-level dynamics, but sequential IDMs have sparse feature coupling and unified models incur joint-denoising costs.LiMA is presented as an asynchronous hierarchy that combines video-based foresight with high-frequency execution.Source: §2.1 · p. 3

2.2 Hierarchical Planning and Control in Robotics

• Synchronous dual-system methods [44] [45] [46] [47] are limited by the deliberative module’s lower sampling frequency, while asynchronous designs [48] [49] [50] [51] can couple the systems only superficially.LiMA links the systems at the noise level through a distributional diffusion bridge.Source: §2.2 · p. 3

3.1 Preliminaries

• The policy models a synchronized future action chunk conditioned on language, proprioceptive state, and multi-view observations.The action chunk represents coordinated trajectories for both arms and all dexterous fingers.Source: §3.1 · p. 3

3.2 Model Architecture

• The Future Dreamer generates long-horizon imagined visual latents and a coarse macro-action prior, which serve as the structured prior boundary for bridge refinement.It is initialized from pretrained Cosmos-Predict2-2B and fine-tuned on task demonstrations.Source: §3.2 · p. 4
• The 1B-parameter Motion Refiner uses intent and real-time proprioceptive state to predict near-term actions and short-horizon visual latents.Imagined visual priors are fused with online visual tokens through localized adaptive modulation.Source: §3.2 · pp. 4–5

3.3 Latent Schrödinger Bridge Coupling

• The latent bridge treats Dreamer intent as a structured prior and trains the Refiner to recover clean execution latents from intent-conditioned intermediate states.This replaces refinement from an unconditional Gaussian prior and is intended to keep the action trajectory aligned with long-horizon intent.Source: §3.3 · p. 5

3.4 Asynchronous Dual-System Coordination

• The Future Dreamer refreshes intent periodically while the Motion Refiner continues processing current observations at full frequency using the cached prior.This couples sparse planning with dense motion refinement.Source: §3.4 · pp. 5–6

3.5 Training Objective

• Training jointly optimizes the Future Dreamer and Motion Refiner, with a boundary-noise regularizer perturbing the intent latent on the fast system’s conditioning path.The Refiner is trained to recover clean joint execution latents, and the total objective combines dream and refinement losses.Source: §3.5 · p. 6

4.1 Experiment Setup

• The evaluation uses a bimanual UR5–SharpaWave platform and six tasks, comparing LiMA with two VLA and two WAM baselines using SR and PSR.Each task has 100 expert demonstrations and 20 physical evaluation trials.Source: §4.1 · pp. 6–7

4.2 Results and Analysis

• 45.8% lower inference latency than Cosmos-Policy (600ms to 325ms) accompanies strong performance across six tasks, with clearer advantages on long-horizon and contact-intensive tasks.The latency is amortized per Refiner update for generating a 32-step action chunk, not per individual control action.Source: §4.2 · p. 7, §4.1 · p. 7
• Figure 3 shows Dreamer attention focused on global task context and Refiner attention on local manipulation regions, unlike Cosmos-Policy’s more dispersed attention.The comparison is illustrated for the Cook Rice task across three camera views.Source: §4.2 · p. 7

4.3 Ablation Study

• SR falls from 80% to 60% without adaptive modulation and to 55% without multi-action tokens; removing asynchrony raises latency from 0.325s to 0.45s without improving success.The authors associate multi-action-token removal with jittery, unstable fine-grained grasping and conclude that the design choices and computation allocation both matter.Source: Table 3 · p. 8, §4.3 · pp. 8–9
• Ten bridge denoising steps offer a more practical performance–efficiency trade-off than five or fifteen steps.Five steps leave the intent prior coarse, while fifteen slightly improve success but substantially increase latency.Source: Table 3 · p. 8, §4.3 · p. 8
• The 3:7 slow–fast configuration achieves the best overall result, allocating more computation to high-frequency Refiner correction.Source: Table 3 · p. 8, §4.3 · p. 8
• Removing future visual intent causes the largest degradation; I2SB outperforms Gaussian-initialized cross-attention and Flow Matching variants.The authors describe I2SB as anchoring the refinement trajectory to paired intent and execution boundary states.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Generalization

• On Cook Rice, LiMA achieves higher success across four OOD visual perturbations, including a 70% success rate on novel object instances.The tested perturbations are unseen backgrounds, novel object instances, unseen illumination, and cluttered scenes.Source: §4.4 · p. 9

• Severe visual occlusion and low visual variance degrade performance where vision-based planning is constrained.Source: §5 · p. 9
• Camera-only inputs limit the Dreamer’s ability to maintain accurate trajectories when key visual features are obscured.The authors identify tactile feedback as a next step for compensating for visual uncertainty in contact-rich interactions.Source: §5 · p. 9
• Reported latency may not reflect LiMA’s maximum achievable efficiency because deployment-specific acceleration was not used.The measurement omits operator fusion, compilation, quantization, and multi-GPU parallelism.Source: §5 · p. 9

• LiMA is an asynchronous diffusion-based framework for bridging long-term imagination and high-frequency reactive execution.Source: §6 · p. 9
• The Future Dreamer and Motion Refiner coordinate long-horizon foresight with real-time responsiveness, while Latent Schrödinger Bridge Coupling aligns strategic intent with clean execution latents.Source: §6 · p. 9
• Real-world experiments show improved complex long-horizon manipulation while maintaining robustness and efficient closed-loop control.Source: §6 · p. 9

A.1 Additional Preliminaries

• The Schrödinger Bridge constructs a bridge process between informative boundary distributions, unlike standard diffusion generation from an unstructured Gaussian prior.I2SB provides an analytic posterior over intermediate states between structured source and clean target endpoints.Source: §A.1 · p. 14
• Intermediate bridge states are sampled from a Gaussian posterior whose mean interpolates between endpoints and whose covariance adds controlled stochasticity.The posterior has zero variance at both endpoints, with X0 at t = 0 and X1 at t = 1.Source: §A.1 · p. 14
• LiMA uses the Dreamer’s strategic intent as source boundary X1 and the Refiner’s clean joint execution latent as target boundary X0.This intent-to-execution coupling initializes refinement from a structured intent prior while retaining stochastic refinement capacity.Source: §A.1 · p. 14

A.2 Additional Implementation Details

• The real-world platform uses two UR5 arms with SharpaWave dexterous hands, three RealSense D435 camera views, and VIVE Trackers with MetaGlove Pro gloves.The cameras comprise one head-mounted and two wrist-mounted views.Source: Figure 6 · p. 15
• Each latent tensor has 16 channels, 8 slots, and 28 × 28 spatial resolution, with a five-dimensional batch-channel-slot-height-width layout.The Dreamer and Refiner use compatible eight-slot sets, allowing the intent latent to serve as a structured prior without an extra projection or alignment network.Source: §A.2 · pp. 14–15
• The Future Dreamer starts from a pretrained Cosmos-Predict2-2B checkpoint, while the Motion Refiner is randomly initialized and trained from scratch.Training uses batch size 14, a 1.0 × 10^-4 peak learning rate, and a staged warmup and decay schedule.Source: §A.2 · p. 15

B.1 Robot System Setup

• Two UR5 arms with 22-DoF SharpaWave hands use one head-mounted global camera and two wrist cameras for local views around contact regions.VIVE Trackers and MetaGlove Pro gloves record wrist trajectories and finger configurations for synchronized demonstrations.Source: §B.1 · pp. 15–16
• Each policy produces short-horizon arm and hand action chunks, and trials begin from a predefined robot posture and task instruction.Methods share the robot platform, camera setup, initialization protocol, and success criteria.Source: §B.1 · p. 16

B.2 Baseline Settings

• The baselines cover integrated world-action modeling, VLA, inverse-dynamics models, and video-prediction-based policy learning, with Cosmos-Policy the closest integrated generative comparison.All methods use the same demonstrations when task-specific training is required, with outputs adapted to the shared bimanual action interface.Source: §B.2 · pp. 16–17
• GR00T N1.6 is reported after 50,000 training steps because the recommended 2,000 steps did not fully expose its best performance in this setting.InternVLA-A1 is extended to the 56-DoF dual-arm and dual-hand action configuration.Source: §B.2 · p. 16

C.1 Task Details

• The six tasks cover short-horizon single-arm manipulation and long-horizon bimanual coordination under multi-view observations and language instructions.Progress Success Rate is computed from task-specific progress stages.Source: §C.1 · p. 17, Table 4 · p. 18
• Stack Cup requires the right hand to grasp a cup, align it above another cup, and place it to complete the stack.The task evaluates basic grasping, alignment, and precise placement.Source: §C.1 · p. 17
• Roll T-shirt requires bimanual folding before the right hand places the garment in a basket, testing localization under cloth deformation and recognition of task stages..Source: §C.1 · p. 17
• Cook Rice combines spoon grasping, scooping and transferring rice, closing the cooker, and returning the spoon across multiple stages.The task requires smooth long-horizon motion and maintenance of task-level intent.Source: §C.1 · p. 17
• Make Sandwich combines tong-based lettuce placement with returning the tongs and placing a bread slice, testing tool use and precise multi-step interaction.The task also involves coordinated use of both hands.Source: §C.1 · pp. 17–18
• Make Coffee sequences coffee-bag pouring, kettle pouring, and handing a cup to a person, but uses simulated pouring rather than real liquid transfer for safety and hardware protection.The task evaluates long-horizon multi-object interaction.Source: §C.1 · p. 18
• Assemble Package requires both hands to align and combine two components, testing bimanual coordination, spatial alignment, and execution stability..Source: §C.1 · p. 18

• Success Rate (SR) measures whether the policy completes all predefined task goals in the correct order.It reflects final task-level success.Source: §C.2 · pp. 18–19
• Progress Success Rate (PSR) measures the proportion of progress stages completed, with per-stage success rates reported in Table 5.The staged measure enables analysis of where methods fail during execution.Source: §C.2 · p. 19, Table 5 · p. 19
• Figure 8 visualizes LiMA's predicted ego-centric future observations alongside ground-truth observations for three long-horizon tasks.Source: Figure 8 · p. 18, §C.2 · p. 19
• Inference time is the average time to generate one executable action chunk, measured under the same robot platform and protocol.For LiMA, the effective average combines Future Dreamer and Motion Refiner inference times across an asynchronous update cycle.Source: §C.2 · p. 19

C.3 Generalization Evaluation

• 70% success rate on unseen objects and 75% in cluttered scenes; LiMA maintains task intent despite appearance shifts and visual distractions.Cosmos-Policy shows more frequent grasping failures in the perturbed scenes.Source: §C.3 · p. 20
• LiMA shows larger success-rate fluctuations under unseen backgrounds and lighting than in the base setting, yet outperforms baselines.Source: §C.3 · p. 20

D Failure Cases

• Limited camera coverage can hide manipulation steps, and transparent objects are harder to perceive reliably from RGB observations.These factors partially explain why Assemble Package has lower success than other tasks.Source: D Failure Cases · p. 20
• Vision-only contact perception is limited; low-contrast garments can make LiMA miss the contact region and pinch empty space.The authors propose wider-field-of-view cameras and tactile sensing as possible mitigations.Source: D Failure Cases · p. 20