3D Neural Scene Representations for Visuomotor Control

Yunzhu Li; Shuang Li; Vincent Sitzmann; Pulkit Agrawal; Antonio Torralba

Source-linked AI summary

3D Neural Scene Representations for Visuomotor Control

Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, Antonio Torralba

arXiv:2107.04004v2Jul 8, 2021cs.ROcs.CVcs.LG

TL;DR

Robots need models that infer dynamic 3D scenes from 2D observations for manipulation across viewpoints and object types. This paper combines NeRF, time contrastive learning, and autoencoding to learn viewpoint-invariant scene states and action-conditioned dynamics. The resulting system supports control with goals specified from unseen viewpoints, including rigid-object and fluid tasks, while also enabling future prediction and novel-view synthesis.

Problem
Robotic visuomotor systems need 3D-aware state representations for dynamic manipulation from 2D observations, especially when views or scene configurations are unseen.
Method
The framework combines neural radiance fields, autoencoding, time contrastive learning, an action-conditioned dynamics model, and test-time auto-decoding optimization.
Results
The learned representations support visuomotor control of rigid objects and fluids, including goals specified from viewpoints outside training, and enable future prediction and novel-view synthesis.
Takeaways & Limitations
The framework provides a learned 3D dynamics model for control and prediction in complex dynamic manipulation scenes.
Takeaways & Limitations
The convolutional encoder is not camera-pose equivariant, so unseen viewpoints require test-time auto-decoding optimization for accurate state estimation.

Abstract

from arXiv · show

Humans have a strong intuitive understanding of the 3D environment around us. The mental model of the physics in our brain applies to objects of different materials and enables us to perform a wide range of manipulation tasks that are far beyond the reach of current robots. In this work, we desire to learn models for dynamic 3D scenes purely from 2D visual observations. Our model combines Neural Radiance Fields (NeRF) and time contrastive learning with an autoencoding framework, which learns viewpoint-invariant 3D-aware scene representations. We show that a dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks involving both rigid bodies and fluids, where the target is specified in a viewpoint different from what the robot operates on. When coupled with an auto-decoding framework, it can even support goal specification from camera viewpoints that are outside the training distribution. We further demonstrate the richness of the learned 3D dynamics model by performing future prediction and novel view synthesis. Finally, we provide detailed ablation studies regarding different system designs and qualitative analysis of the learned representations.

1 Introduction

The paper targets 3D-aware state representations for visuomotor control from 2D observations, addressing viewpoint changes and dynamic objects. It combines neural scene representations, time contrastive learning, and dynamics modeling to support control and broader scene prediction.

3D-aware representations are needed because pixel-based systems lack explicit world structure, limiting amodal completion and operation from different views.
Prior image-space dynamics methods struggle with high-dimensional modeling, while time-contrastive representations can fail on unseen configurations and camera poses.
The proposed framework embeds neural radiance fields in an autoencoder, adds time contrastive learning for viewpoint-invariant states, and trains action-conditioned dynamics in the learned state space.
Inference-via-optimization enables state estimation for out-of-distribution camera poses, supporting control when the goal is specified from an unseen perspective.
The work extends 3D-aware representations to dynamic control involving rigid objects and fluids, with future prediction and novel-view synthesis across environments.

2 Related Work

Related work develops learned scene representations and visual dynamics models using autoencoders, voxelgrids, particles, image-space states, keypoints, and latent spaces. These approaches motivate combining 3D structure with viewpoint-invariant dynamics for control.

3D Scene Representation Learning: Autoencoder-like latent spaces, voxelgrids, particle sets, and explicit 3D representations have been used to infer scene structure and synthesize views from images.
3D Scene Representation Learning: The training overview combines image encoding, state aggregation, viewpoint-invariant time contrastive learning, view-conditioned decoding, and action-conditioned latent dynamics.
Model-Based RL in Robotic Manipulation: Model-based reinforcement learning methods differ by using physics-based or data-driven models and by assuming full state access or visual observations.
Model-Based RL in Robotic Manipulation: Visual model-based methods learn dynamics in image space, keypoint representations, or low-dimensional latent spaces, sometimes using reconstruction, joint dynamics, or time-contrastive losses.

3 3D-Aware Representation Learning for Dynamics Modeling

The framework learns viewpoint-invariant, 3D-aware scene states from multi-view visual observations by combining neural radiance-field decoding, time contrastive learning, and predictive dynamics modeling. A latent dynamics model then predicts future states from actions for downstream control.

3.1 3D-Aware Scene Representation Learning: A neural radiance-field encoder-decoder represents dynamic scenes as latent states that support rendering observations from specified viewpoints.The decoder predicts density and color from 3D locations, viewing directions, and the scene state, while the encoder maps observations to that state.
3.1 3D-Aware Scene Representation Learning: The scene representation averages viewpoint-conditioned image features, producing a normalized state intended to capture shared scene content across cameras.Each image feature combines ResNet features with camera-view information before multi-view aggregation.
3.1 3D-Aware Scene Representation Learning: Time contrastive learning pulls together features from different viewpoints at the same time while separating features from different times.The training samples use a same-time, different-view positive and a different-time, same-view negative, with a margin-based loss.
3.2 Learning the Predictive Model: The dynamics model iteratively predicts future latent states from the current state and action sequence over a horizon H.It is implemented as an MLP and trained with multi-step prediction loss.
3.2 Learning the Predictive Model: Training combines reconstruction, time contrastive, and dynamics losses, first learning the encoder-decoder and then fitting dynamics with the encoder fixed.The final objective is L = Lrec + Ltc + Ldyn.

4 Visuomotor Control

The learned latent dynamics model supports online planning and closed-loop visuomotor control, while auto-decoding adapts state estimates for goals observed from unfamiliar viewpoints. Qualitative evaluations show successful control across viewpoint conditions and better results than the tested non-auto-decoding and baseline methods.

4.1 Online Planning for Closed-Loop Control: Online planning finds actions that minimize the distance between the predicted future state and the goal state, enabling closed-loop control with MPC.The planner uses the latent dynamics model to predict state sequences, and MPPI performed best among the tested planners.
4.2 Auto-Decoder for Viewpoint Extrapolation: Out-of-distribution camera poses can cause conventional visual encoders to produce inaccurate state estimates and degraded visuomotor performance.Auto-decoding addresses this by refining the latent state against the observed image while keeping decoder parameters fixed.
4.2 Auto-Decoder for Viewpoint Extrapolation: Auto-decoding updates the scene representation by backpropagating reconstruction error through the fixed decoder for K iterations.The optimized state is then used as the goal representation for online planning.
4.2 Auto-Decoder for Viewpoint Extrapolation: The method performs well when goal images come from encountered, interpolated, or extrapolated viewpoints relative to training cameras.The evaluation compares goals specified from all three viewpoint conditions in FluidPour and FluidShake.
4.2 Auto-Decoder for Viewpoint Extrapolation: On FluidPour and FluidShake, the full method outperforms Ours w/o AD and TC+AE when the goal view is outside training distribution and differs substantially from the agent view.Both comparison methods fail to accomplish the task, with 3D fluid or cube configurations deviating from the target.

5 Experiments

The experiments evaluate 3D-aware visuomotor control across fluid and rigid environments, including targets specified from training, interpolated, and extrapolated viewpoints. The full method outperforms baselines, supports novel-view goal specification, and predicts future dynamics for visual reconstruction.

Environments: The evaluation covers FluidPour, FluidShake, RigidStack, and RigidDrop, spanning fluid manipulation, floating rigid objects, stacking, and dropping.FluidPour and FluidShake use 1,000 trajectories of 300 steps; RigidStack uses 800 trajectories of 80 steps, and RigidDrop uses 1,000 trajectories of 50 steps.
Evaluation Metrics: The control benchmarks specify goals from viewpoints seen during training, interpolated between training views, or extrapolated outside the training distribution.Performance is quantified using Chamfer distance plus object-specific position and angle errors.
Goal Specification from Novel Viewpoints: Test-time auto-decoding enables the agent to achieve a target configuration specified from a viewpoint outside the training distribution.The demonstrated extrapolated view moves the camera closer, higher, and downward relative to the container.
Baseline Comparisons: The full model significantly outperforms all baselines across FluidPour and FluidShake, evaluation metrics, and training, interpolated, and extrapolated target viewpoints.Figure 6 reports mean performance with standard-error bars; the prose likewise states superiority under all scenarios and metrics.
Dynamic Prediction and Novel View Synthesis: The dynamics model iteratively predicts latent scene states from actions, while the decoder reconstructs future observations from different viewpoints across four environments.Figure 7 compares open-loop predictions with novel-view ground truth at the final time step.

6 Conclusion

The paper concludes that NeRF-based autoencoding with time contrastive learning produces viewpoint-invariant 3D scene representations for visuomotor control. Auto-decoding extends goal specification beyond the training viewpoints, while future prediction and novel-view synthesis demonstrate broader model utility.

6 Conclusion: The framework learns viewpoint-invariant 3D-aware representations from visual observations using an autoencoder, NeRF rendering, and time contrastive learning.The representation is used for model-based visuomotor control in dynamic scenes.
6 Conclusion: Auto-decoding test-time optimization enables goal specification from viewpoints outside the training distribution.The conclusion places this capability within control tasks involving rigid objects and fluids.
A Model Details: The decoder combines 3D points, camera-ray directions, and latent state features to output scene density and RGB color.Its architecture follows NeRF and uses the learned state representation to reconstruct observations.

C Training Details

Training uses Adam with NeRF-style decoder settings and environment-specific camera-distance bounds. Viewpoint extrapolation is evaluated using images generated outside the fixed-distance training viewpoint distribution.

Training Configuration: Adam training starts at 5e−4 and decreases to 5e−5, with batch size 2 and NeRF-derived decoder hyperparameters.The decoder uses environment-specific near and far camera-distance values.
Viewpoint Extrapolation: Training observations use fixed-distance cameras, whereas extrapolation experiments generate viewpoints outside that training distribution.Figure 9 contrasts the fixed-distance training setup with novel evaluation viewpoints.

D Control Details

The controller uses MPPI to sample, evaluate, and iteratively reweight action trajectories with the learned dynamics model. Auto-decoding test-time optimization uses 500 updates, while MPPI samples 1,000 trajectories and applies configured iteration, weighting, filtering, and horizon settings.

Control Optimization: MPPI samples action trajectories, rolls them out through the learned model, and reweights them by reward before selecting the best sequence.The reward is based on the squared distance between predicted terminal state and the goal state.
Control Optimization: Each sampled trajectory adds filtered Gaussian noise to the current action sequence across the control horizon.Noise samples are generated for every trajectory and time step using filtering coefficient β.
Control Optimization: The action sequence is updated for L iterations, after which the best action sequence is selected for control.The procedure repeats the sampling, rollout, and reward-weighting cycle at each control step.
Implementation Settings: Auto-decoding test-time optimization uses K=500 updates, while MPPI uses M=1,000 sampled trajectories and γ=50.The number of action-sequence updates is L=100 at the first time step and 10 thereafter; β=0.7.
Auto-Decoding Evaluation: Figure 10 evaluates auto-decoding test-time optimization when the input viewpoint lies outside the training distribution.The supplied passage identifies the figure’s setting but does not report its complete qualitative outcome.

E Additional Experimental Results

Auto-decoding test-time optimization improves the scene representation for goal images captured from viewpoints outside the training distribution. The optimized representation can then serve as the goal embedding for control.

Auto-Decoding Test-Time Optimization: For viewpoints outside the training distribution, auto-decoding derives a more accurate representation of the scene’s 3D contents.The optimized representation is used to compute the goal embedding that the agent must achieve.
Auto-Decoding Test-Time Optimization: The optimized representation can be used as s_goal in Equation 4 for planning toward the specified goal.This connects viewpoint extrapolation at test time to the downstream control objective.

E.2 Validation of the Dynamics Prediction on Real-World Data

Real-world pouring experiments evaluate dynamics prediction and fluid-aware control using synchronized RGBD recordings. The model predicts future latent states and rendered frames closely resembling ground truth, while outperforming a robot-state-only PID baseline on fluid-shape matching.

Experimental Setup: The real-world dataset contains 50 pouring episodes recorded at 15 Hz, with 45 episodes for training and five for testing.The recordings comprise 43,400 frames of a human pouring water between cups.
Real-World Dynamics Prediction: The model predicts future latent states and rendered frames that closely resemble ground-truth pouring trajectories.The evaluation uses four synchronized RGBD cameras and holds out five of 50 recorded episodes for testing.
Real-World Dynamics Prediction: Figure 11 compares ground-truth and predicted future pouring sequences across four camera views.The caption states that the prediction identifies when fluid pours and begins filling the receiving container.
Representation Validation: Nearest-neighbor retrieval produces reasonable images whose learned representations preserve scene contents and camera-pose invariance.The supplied caption presents this as qualitative evidence from Figure 12.
Fluid-Aware Control: The model learns to pour a target amount of fluid before repositioning the container, unlike PID control that only matches the robot state.The PID baseline leaves the top container without pouring fluid into the bottom container in the described trial.
Fluid-Aware Control: Chamfer distance is 0.048237 for Ours versus 0.085727 for PID when matching the final fluid shape.The reported comparison evaluates fluid shape in 3D point space.

E.4 Nearest Neighbor Search Using the Learned Representations

Nearest-neighbor retrieval evaluates whether learned representation distances reflect 3D scene similarity across viewpoints. The method achieves retrieval errors of 1.772718 and 2.584928 time steps on FluidPour and FluidShake, respectively.

Representation Interpretation: The representation is designed so scenes with similar 3D contents remain close even when their images differ substantially in pixel space.This property supports planning when the goal and agent viewpoints are different.
Quantitative Evaluation: Table 1 reports L1 distance between ground-truth and retrieved time indexes and identifies the learned representation as the best-performing method.The retrieved frame is described as approximately two time steps from the ground truth on average.
Quantitative Evaluation: The learned representation achieves average time-step errors of 1.772718 in FluidPour and 2.584928 in FluidShake.The evaluation selects the closest frame from a randomly selected viewpoint along a 300-frame trajectory.
Quantitative Evaluation: The method outperforms random selection and pixel-difference retrieval for nearest-neighbor search across viewpoints.Random selection averages 100 time steps from the ground truth, while pixel difference performs worse than random selection.

F Limitations and Future Works

The model performs reasonably in regions supported by training data but is not expected to generalize to unseen cases such as additional containers.

Generalization is limited in cases absent from training, such as scenes containing more containers than the model has seen.The authors suggest expanding training diversity or using a structured, compositional representation instead of a single vector as future directions.