PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels

Long Le, Ryan Lucas, Chen Wang, Chuhao Chen, Dinesh Jayaraman, Eric Eaton, Lingjie Liu

arXiv:2508.17437v2cs.CV

TL;DR

Inferring physical properties from 3D visual information is difficult because existing approaches often rely on slow, per-scene optimization. PIXIE learns a supervised, feed-forward mapping from visual features to material fields, achieving higher realism scores with much faster inference and zero-shot transfer to real scenes.

  • Problem

    Existing methods for acquiring 3D material properties rely on manual specification or slow test-time optimization from sparse observations.

  • Method

    PIXIE distills CLIP visual features into 3D and uses a feed-forward 3D U-Net with supervised per-voxel material prediction, trained on PIXIEVERSE.

  • Results

    PIXIE improves Gemini realism scores by 1.46-4.39x over DreamPhysics and OmniPhysGS while reducing inference time by three orders of magnitude.

  • Takeaways & Limitations

    PIXIE enables fast, plausible 3D physics simulation by coupling predicted material fields with learned scene representations and off-the-shelf MPM solvers.

  • Takeaways & Limitations

    PIXIE focuses on single-object interaction and predicts a point estimate despite real-world material uncertainty.

Abstract

from arXiv · show

Inferring the physical properties of 3D scenes from visual information is a critical yet challenging task for creating interactive and realistic virtual worlds. While humans intuitively grasp material characteristics such as elasticity or stiffness, existing methods often rely on slow, per-scene optimization, limiting their generalizability and application. To address this problem, we introduce PIXIE, a novel method that trains a generalizable neural network to predict physical properties across multiple scenes from 3D visual features purely using supervised losses. Once trained, our feed-forward network can perform fast inference of plausible material fields, which coupled with a learned static scene representation like Gaussian Splatting enables realistic physics simulation under external forces. To facilitate this research, we also collected PIXIEVERSE, one of the largest known datasets of paired 3D assets and physic material annotations. Extensive evaluations demonstrate that PIXIE is about 1.46-4.39x better and orders of magnitude faster than test-time optimization methods. By leveraging pretrained visual features like CLIP, our method can also zero-shot generalize to real-world scenes despite only ever been trained on synthetic data. https://pixie-3d.github.io/

1 Introduction

Existing 3D reconstruction methods model geometry and appearance but not physical properties, while automated material discovery remains slow and difficult. PIXIE addresses this gap with supervised, feed-forward prediction from visual features, supported by the PIXIEVERSE dataset and designed for fast, generalizable physics inference.

  • 1 Introduction: Existing scene-reconstruction methods capture photorealistic geometry and appearance but remain blind to underlying physical properties.Neural Radiance Fields [28] and Gaussian Splatting [18] are described as visual representations rather than physical models.
  • 1 Introduction: Manual material assignment burdens users, while test-time optimization can require hours to infer hundreds of thousands of particle parameters from sparse signals.These limitations motivate a generalizable alternative to per-scene optimization.
  • 1 Introduction: PIXIE uses CLIP features and a feed-forward model to associate visual patterns with physical behavior, enabling fast inference and cross-scene generalization.The paper motivates this mapping using examples such as vegetation that shares visual and physical characteristics across contexts.
  • 1 Introduction: PIXIE predicts discrete material types and continuous physical parameters directly from visual features using supervised learning.The predicted quantities include Young’s modulus, Poisson’s ratio, and density.
  • 1 Introduction: PIXIEVERSE contains 1624 paired 3D objects with annotated materials spanning 10 semantic classes.The dataset is presented as the largest open-source dataset of 3D objects with physical material annotations.
  • 1 Introduction: PIXIE achieves 1.46-4.39x higher realism scores than prior approaches and runs orders of magnitude faster than test-time optimization methods.Its predicted material fields can also be coupled with Gaussian splatting for simulation under forces such as wind and gravity.

2 Related Work

Related work spans 2D world models, manually or heuristically assigned physics, and test-time material optimization. These approaches either lack explicit physically grounded 3D modeling, depend on user or model-driven assignments, or require observations and costly iterative optimization.

  • 2D World Models: 2D world models predict dynamics or material labels in images but do not explicitly model 3D geometry or a physically grounded world.The cited examples include 2D material prediction and next-frame generation conditioned on latent actions [3] [2].
  • Manual Assignment or Assignment of Physics using LLMs: Manual and heuristic approaches assign materials through user specifications, scripted object-to-material dictionaries, or language and vision-language models.These methods combine learned 3D representations such as Gaussian splatting with physics solvers.
  • Test-time material optimization using videos: Video-based methods optimize material parameters by matching simulated deformations to ground-truth observations or using video diffusion priors.Several methods require ground-truth multiview videos or particle positions under known forces [15] [23] [45] [16] [43], while others use video diffusion models [14] [26] [44].

3 Method

PIXIE maps multi-view visual features to spatially varying material properties through supervised learning, then couples those predictions with learned scene representations for physics simulation. PIXIEVERSE provides diverse annotated objects for training this generalizable mapping.

  • 3 Method: NeRF distills view-independent CLIP features from posed multi-view RGB images, which PIXIE voxelizes into a regular 3D feature grid.The resulting grid preserves geometric and semantic visual information for material prediction.
  • 3 Method: PIXIE predicts discrete material types and continuous Young’s modulus, Poisson’s ratio, and density directly from 3D visual features.The target is a fine-grained material field assigning semantic labels and physical parameters at each spatial location.
  • 3 Method: PIXIEVERSE contains 1624 annotated 3D objects spanning 10 semantic classes and diverse material categories.Annotations include spatially varying material types, Young’s modulus, Poisson’s ratio, and density.
  • 3 Method: A feature projector and 3D U-Net jointly map the voxelized CLIP grid to the material grid using cross-entropy and mean-squared-error losses.The projector reduces CLIP features from R768 to R64 before U-Net prediction.
  • 3 Method: Masked supervision is applied only to occupied voxels because roughly 98% of the voxel grid is background.Occupancy is obtained by thresholding NeRF densities at α = 0.01, and the loss combines discrete classification with continuous regression.
  • 3 Method: Predicted material fields are attached to Gaussian-splat particles and simulated with an MPM solver under forces such as gravity and wind.Gaussian splatting supplies a learned static scene representation for physics simulation.

4 Experiments

PIXIE is evaluated on synthetic and real scenes against test-time optimization and language-model-based baselines. It achieves substantially better realism and reconstruction metrics with much faster inference, while CLIP features support zero-shot transfer to real scenes.

  • 4 Experiments: The evaluation uses 38 synthetic test scenes and six real-world scenes, with Gemini-2.5-Pro judging animation realism on a 0-to-5 scale.PSNR and SSIM are additionally measured against reference videos in PIXIEVERSE.
  • 4 Experiments: PIXIE-CLIP achieves 1.62-5.91x VLM improvement and 3.6-30.3% PSNR and SSIM gains, leading across the reported metrics.The table compares reconstruction quality, VLM realism, material accuracy, and continuous errors over E, ν, and ρ.
  • 4.1 Synthetic Scene Experiments: 4.35 ± 0.08 VLM realism score, 1.46-4.39x above baselines, while reducing inference time to 2 s from minutes or hours.PIXIE also improves PSNR and SSIM by 3.6−30.3% and VLM scores by 2.21−4.58x over prior works.
  • 4.2 Zero-shot Generalization to Real-World Scenes: PIXIE zero-shot generalizes to many real-world scenes without real-scene supervision, assigning rigid vase bases and flexible leaves plausibly.The paper reports that no other baseline generalizes under this setting.
  • 4.3 PIXIE’s Feature Type Ablation: Replacing CLIP with RGB or occupancy features drops VLM score by 40-60% and nearly doubles parameter MSE.The ablation also reports substantially weaker material-class prediction and real-scene generalization for RGB and occupancy features.

5 Conclusion and Limitations

PIXIE jointly reconstructs geometry, appearance, and physical material fields, enabling fast, plausible 3D animation and zero-shot real-scene generalization. Its limitations include single-object interactions and uncertainty in visually inferred materials.

  • PIXIE jointly reconstructs geometry, appearance, and explicit physical material fields from posed RGB images.
  • PIXIE distills CLIP features into 3D and uses a feed-forward 3D U-Net trained with per-voxel material supervision to avoid test-time optimization.The resulting material fields can be coupled with Gaussian Splatting and MPM for simulation under forces such as gravity and wind.
  • PIXIE improves Gemini realism scores by 1.46-4.39x over DreamPhysics and OmniPhysGS while reducing inference time by three orders of magnitude.
  • CLIP visual priors enable zero-shot transfer to real scenes despite training only on synthetic data, supporting physically plausible animation with off-the-shelf MPM solvers.
  • The method focuses on single-object interaction and predicts point estimates despite real materials having unresolved uncertainty from visual information alone.The authors identify multi-object scenes and distributional material prediction as future directions.

A.3 Material Point Method (MPM) for Physics Simulation

MPM simulation combines conservation laws with material-specific constitutive models and parameters to determine how objects deform under external forces. PIXIE predicts both the discrete material model and continuous parameters needed by this physics pipeline.

  • MPM models motion through mass and momentum conservation, with external forces such as gravity or user interactions entering the momentum equation.
  • Material-specific constitutive laws define how stress depends on deformation, with elastic materials recovering strain and plastic materials flowing beyond a yield threshold.
  • Constitutive simulation separates an elastic stress law from return mapping that projects trial updates onto a feasible yield surface when plastic flow occurs.
  • PIXIE jointly predicts the discrete material model and continuous parameters including Young’s modulus, Poisson’s ratio, and density.These choices determine material behavior at both the class and within-class parameter levels.

B PIXIEVERSE Dataset Details

PIXIEVERSE is built from Objaverse through a semi-automatic pipeline that retrieves candidate assets, filters unsuitable scenes, and assigns material labels using visual features and VLM assistance.

  • Dataset construction: PIXIEVERSE is curated from Objaverse across 10 semantic classes, using manually tuned physical parameter ranges and segmentation queries to guide labeling.The curation removes multi-object scenes, missing textures, duplicates, ambiguous labels, and physically implausible objects.
  • Material annotation: For each retained object, multi-view images support semantic-part, material-class, parameter-range, and constraint annotations before values are sampled and voxelized.Material properties are applied uniformly within parts, and the resulting fields become regular-grid supervision.
  • Asset retrieval: 500 candidate objects per class are retrieved by cosine similarity between class search terms and Objaverse asset names.The pipeline uses an all-MiniLM-L6-v2 sentence transformer for this matching step.
  • Asset filtering: Gemini filters low-quality assets, after which a human scans the results and corrects mistakes through a web interface.This adds human review after automated filtering.
  • 3D segmentation: CLIP-based 3D segmentation assigns each point to the most similar queried part, but query wording can substantially change segmentation quality across objects.This motivates object-dependent query proposals rather than relying on one fixed vocabulary.

B.4 VLM Actor-Critic Labeling

The VLM actor-critic pipeline combines manually tuned physics examples with object-specific proposals, rendered segmentation candidates, and critic selection to improve PIXIEVERSE labels.

  • Human prior: Current VLMs may lack robust physical understanding, so manually tuned class-specific parameter examples and query terms are supplied as in-context guidance.The examples cover semantic classes such as trees and rubber toys.
  • Actor-critic labeling: The actor proposes material parameters and alternative semantic segmentation queries from multi-view images, while the critic selects queries after viewing rendered 3D masks.This separates proposal generation from segmentation-quality evaluation.
  • Physics proposals: Material uncertainty is represented by plausible parameter ranges, sampled uniformly and constrained for consistency across object parts.For example, the actor can output E ∈{1e4, 1e5} and impose relative-density or stiffness constraints.
  • Object-specific rules: The prompts encode object-specific structural rules, such as stationary bases, stiffness ordering, visible-part selection, and whether an object should be segmented as one part.Examples distinguish rigid vases from flexible flowers and prohibit stationary labels when no fixed base is present.

D VLM As a Physics Judge

PIXIE evaluates rendered physics animations with a vision-language judge and supplements realism judgments with executionability and reconstruction metrics.

  • VLM evaluation: The VLM evaluator compares candidate animations and scores their realism on a 1–5 scale, optionally using a reference video and scene prompt.The evaluator’s system prompt is provided in Fig. 15.
  • Evaluation metrics: The PIXIEVERSE ablation reports executionability and Gemini realism, with executionability measuring simulations that run successfully without numerical explosion.These metrics assess both physical-run validity and perceived realism.

E Model architecture

PIXIE uses a 3D U-Net for both discrete material segmentation and continuous parameter regression, with a projector that standardizes input features for spatial processing.

  • Architecture: PIXIE predicts discrete material classes and continuous material parameters with a 3D U-Net architecture.The architecture contains a feature projector and a spatial 3D U-Net backbone.
  • Feature projection: The feature projector maps RGB, CLIP, or occupancy inputs to a unified 32-channel conditioning representation.It uses three Conv3D, GroupNorm, and SiLU layers, with hidden dimension 128 when input channels exceed 32.

E.3 3D UNet Architecture

PIXIE uses a 3D U-Net encoder–decoder to transform 32-dimensional 3D feature grids into discrete material classifications or continuous physical-parameter predictions.

  • E.3 3D UNet Architecture: The 3D U-Net processes 32-dimensional feature grids through four encoder–decoder resolution levels with skip connections.The architecture uses base dimension 64 and channel multipliers [1] [2] [4].
  • E.3 3D UNet Architecture: The encoder progressively downsamples features from 64^3 to 8^3 while increasing channels from 64 to 256.The first two levels maintain 64 channels; later levels reach 128 channels at 16^3 and 256 channels at 8^3.
  • E.3 3D UNet Architecture: At the bottleneck, residual blocks surround an attention block operating at 8^3 resolution with 256 channels.Attention blocks are disabled in the implementation by setting attention resolutions to empty.
  • E.3 3D UNet Architecture: The decoder mirrors the encoder, using nearest-neighbor upsampling followed by 3D convolution and skip connections.Decoder levels match the channel dimensions and residual-block counts of corresponding encoder levels.
  • E.3 3D UNet Architecture: Residual blocks use additive skip connections, normalization, LeakyReLU activations, convolutions, dropout, and zero-initialized final convolutions.A 1 × 1 × 1 convolution matches channels when input and output dimensions differ.
  • E.3 3D UNet Architecture: The output layer produces 8 channels for discrete material classification or 3 channels for continuous material-parameter regression.It applies layer normalization, LeakyReLU activation, and a final 3D convolution.

F Additional Results

Additional results visualize PIXIE’s material predictions and compare CLIP features with RGB and occupancy ablations across synthetic and real scenes.

  • F Additional Results: PIXIE simultaneously recovers material class, Young’s modulus, Poisson’s ratio, and mass density with high accuracy in visualized predictions.Examples include elastic foliage, rigid metal, and realistic stiffness and density gradients within objects.
  • F Additional Results: CLIP features outperform RGB and occupancy on per-class accuracy across synthetic PIXIEVERSE scenes.Figure 17 reports the comparison across semantic classes.
  • F Additional Results: Replacing CLIP features with RGB or occupancy severely degrades material prediction on real scenes.Shown errors include foliage mislabeled as metal and Young’s modulus becoming uniform within an object.
Loading 2508.17437v2…