Learning to Act Robustly with View-Invariant Latent Actions

1. Introduction

• Multi-view data can expose policies to viewpoint variation, but acquisition costs are prohibitive and scaling is poor.Viewpoint brittleness is identified as a barrier to robust real-world deployment.Source: §1 · p. 1
• Scene-level approaches enforce image-level invariance, asking one compact feature to encode both static context and task-relevant motion.This can make the invariant representation broader than control requires and does not explicitly distinguish context from dynamics.Source: §1 · pp. 1–2
• VILA enforces invariance on latent actions that capture observation changes, aligning them across views with ground-truth action sequences.Its weighted contrastive loss and global similarity-alignment term shape the dynamics-centered latent space.Source: §1 · p. 2

2.1. Viewpoint Robustness in Visuomotor Policies

• Observation-level approaches use multi-view data, expanded views through synthesis or collection, or camera geometry as policy input.The cited approaches include [13] [17] [20] [36], [12], and, respectively.Source: §2.1 · p. 2

2.2. Latent Actions

• Latent action models encode changes between observations as compact dynamics codes, and VILA adds multi-view, action-guided regularization to align codes for the same motion.Earlier view-robust encoders focus on scene-level features, which can conflate task motion with static context or visual appearance.Source: §2.2 · p. 2, §2.1 · p. 2

3. Methods

• VILA has two stages: latent action learning creates a compact, action-guided, view-invariant representation, and latent behavior cloning trains a policy to predict latent actions.The inverse dynamics model is reused in the behavior-cloning stage, where the latent policy serves as an encoder.Source: §3 · p. 2

3.1. Base Latent Action Learning

• The base learner encodes observations at times t and t+k into features, infers a latent action, and predicts the future feature under a temporal-consistency loss.The target feature comes from a separate encoder whose parameters are an exponential moving average of the online encoder; the loss is mean squared error to a stable target.Source: §3.1 · p. 3

3.2. Action-Guided Latent Action Invariance

• Action-guided invariance brings latent actions closer when their ground-truth action sequences are similar, using weighted InfoNCE locally and structural alignment globally.The method converts normalized squared sequence distances into soft weights, then aligns pairwise similarity structures of latent actions and ground-truth action vectors.Source: §3.2 · p. 3

3.3. Latent Behavior Cloning

• Latent behavior cloning predicts latent actions from the current observation, which then conditions a downstream policy that outputs low-level actions.Because prediction uses the current observation, no future frames are needed at test time.Source: §3.3 · p. 3

4.2. Unseen View Generalization

• VILA achieves the best fine-tuned unseen-view success on all five simulated tasks and alone retains non-trivial frozen performance where other methods often approach zero.The comparison covers seen and unseen views in the frozen and fine-tuned settings.Source: Table 1 · p. 4, §4.2 · p. 5

4.1. Viewpoint Setups

• Simulation uses five tasks and 25 camera poses per trajectory, with 10 views for training and 15 held out; eight additional poses test extrapolation beyond the sampled range.The sampled azimuth and elevation offsets span −90° to +90° and −15° to +15°, respectively.Source: §4.1 · pp. 3–4, Table 2 · p. 5
• For real-world Pick & Place and Drawer data, ZeroNVS augments videos to 27 viewpoints; four are used for training and three held out for evaluation.The augmented views vary camera azimuth and vertical and optical-axis translations.Source: §4.1 · pp. 4–5

4.2. Unseen View Generalization

• The evaluation trains a visuomotor policy per dataset and task, then measures success separately on seen and unseen views with either a frozen or jointly fine-tuned encoder.Baselines share the downstream policy, preprocessing, and training schedule, isolating differences in encoder pretraining strategy and camera conditioning.Source: §4.2 · p. 5
• On real-world unseen views, VILA averages 63.3% success on Pick & Place and 85.0% on Drawer, versus baseline averages below 13.3% and 0%, respectively.The table evaluates Pick & Place on three unseen views and Drawer on two, with 10 episodes per setting.Source: Table 3 · p. 5, §4.2 · p. 6
• Across fine-tuned and frozen settings, VILA's success degrades more slowly as the nearest training-camera gap grows; under extrapolated poses, it retains meaningful success while baselines largely fail.Coffee is excluded from the extrapolated-view table because all methods failed on that task.Source: Table 2 · p. 5, §4.2 · p. 6

4.3. Unseen Task Adaptation

• Across Coffee data budgets, VILA provides a stronger prior than Vanilla, while other encoders often match or underperform Vanilla trained from scratch.The encoders are transferred from multi-view Stack Three pretraining and fine-tuned with a small subset of Coffee demonstrations.Source: §4.3 · p. 7

5.1. Ablation Studies

• Action-guided weighting and structural alignment are complementary: removing either degrades unseen-view success, while alternatives and direct action regression underperform the full objective.The base latent-action loss alone performs worse, and adding an action-regression head further harms performance.Source: §5.1 · p. 7, Table 4 · p. 7
• Uniformly sampling temporal offsets from 1–10 steps gives the best unseen-view generalization; smaller, larger, or fixed offsets degrade performance.The authors attribute this to a moderate offset range capturing dynamics most effectively.Source: §5.1 · p. 7
• L2 distance for weighted contrastive learning and cosine similarity for structural alignment outperform Soft-DTW in this setting.The authors note their resampled sequences are at most 10 steps; Soft-DTW’s approximation and higher cost do not improve performance.Source: §5.1 · p. 7
• A latent action dimension of 128 gives the best unseen-view generalization; both smaller and much larger spaces hurt performance.The authors characterize 128 as a trade-off between expressivity and regularization.Source: §5.1 · pp. 7–8

5.2. Representation Quality Analysis

• VILA has the highest view entropy on seen and unseen views and the lowest action entropy, combining viewpoint mixing with tight grouping by action outcomes.The entropy analysis uses nearest neighbors among sampled transitions; higher view entropy and lower action entropy are favorable.Source: §5.2 · p. 8, Table 5 · p. 8
• In Lift UMAP plots, baselines cluster unseen views separately, especially views 10–14, while VILA mixes them with seen views.The passage reports this pattern both before and after policy training.Source: §5.2 · p. 8, Figure 7 · p. 8

• VILA enforces invariance on latent actions rather than scene-level visual features, aligning its representation with control dynamics.Source: §6 · p. 9
• Across five simulated tasks and a real-world SO-ARM setup, VILA yields consistent gains in unseen-view generalization and data-efficient adaptation to unseen tasks.Source: §6 · p. 9
• Experiments assume multi-view observations; real-robot augmentation uses ZeroNVS, and robustness is studied primarily for camera pose.The authors note that extending the principle to lighting, backgrounds, and object appearance is a next step.Source: §6 · p. 9

A. Implementation Details

• The appendix provides implementation details for view generation, NVS generation, VILA training, baselines, and diffusion-policy training.Source: A. Implementation Details · p. 11

A.1. View Configurations

• Tables 6 and 7 list world-frame camera positions and MuJoCo quaternions for the main and extrapolated views.Source: §A.1 · p. 11

A.2. ZeroNVS Configurations

• Real-world SO-ARM101 novel views are generated with the official ZeroNVS implementation and checkpoint, using four selected view combinations alongside the original view.The selected combinations vary azimuth, vertical translation, and optical-axis translation.Source: §A.2 · p. 11

A.3. VILA Configurations

• VILA uses two stages: latent-action pretraining from multi-view videos, followed by latent behavior cloning in the learned latent space.Source: §A.3 · p. 11

A.4. Baseline Implementations

• CLASS and ReViWo are configured to match VILA’s representation dimensions within each simulation or real-world setting; KYC adds per-pixel Plücker rays from camera extrinsics to the image input.Source: §A.4 · p. 11
• Tables 6 and 7 give exact camera poses for the 25 main views and extrapolated views, while Table 8 lists ZeroNVS hyperparameters for real-world data augmentation.Source: Table 6 · p. 11, Table 7 · p. 11, Table 8 · p. 12

A.5. Diffusion Policy Configurations

• Simulation experiments use the Robomimic diffusion-policy implementation, and all methods share its architecture and training settings.Table 10 summarizes simulation hyperparameters; real-world diffusion-policy training uses LeRobot, with configurations in Table 11.Source: §A.5 · p. 12, Table 10 · p. 12, Table 11 · p. 12

B.1. Entropy Results

• Across Square, Stack Three, Coffee, and Mug Cleanup, VILA consistently attains the highest seen and unseen view entropy and the lowest action entropy, before and after policy fine-tuning.The reported trend matches Lift.Source: §B.1 · p. 12
• Seen and unseen view entropies are computed over 25 views, with higher values indicating better representation quality.The Square and Stack Three tables report these metrics.Source: Table 12 · p. 13, Table 13 · p. 13
• Action entropy is computed over 10 clustered action classes, with lower values indicating better representation quality.The Square and Stack Three tables report this metric before and after policy fine-tuning.Source: Table 12 · p. 13, Table 13 · p. 13

B.2. UMAP Visualization

• View-based UMAPs visualize encoder representations across 25 camera views on Square, Stack Three, Coffee, and Mug Cleanup.The figures cover each of these four tasks.Source: §B.2 · p. 13, Figure 8 · p. 14, Figure 9 · p. 14
• Action-cluster UMAPs color encoder representations by 10 clusters derived by k-means from 10-step ground-truth action sequences.Figures cover Lift, Square, Stack Three, Coffee, and Mug Cleanup.Source: §B.2 · p. 13, Figure 12 · p. 15, Figure 13 · p. 15