PaperlayerView exact revision on arXiv

Source-linked AI summary

ContPhy: Continuum Physical Concept Learning and Reasoning from Videos

Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua B. Tenenbaum, Chuang Gan

arXiv:2402.06119v2cs.CV

TL;DR

Existing benchmarks leave limited coverage of varied continuum properties and language-based dynamics questions. This paper introduces ContPhy and evaluates AI models on it, finding that their performance remains unsatisfactory while ContPRO achieves the best overall performance.

  • Problem

    Many physical-reasoning benchmarks focus on simple rigid scenes or narrow contact prediction, leaving varied physical properties and broader language-based dynamics questions less assessed.

    Source: §1 · p. 1

  • Method

    ContPhy uses simulated videos and templated questions to test physical-property inference and dynamics across diverse scenarios, and the paper introduces ContPRO, combining particle-based dynamics models with large language models.

    Source: §1 · p. 2

  • Results

    Traditional AI models and multimodal large language models perform far from satisfactorily on ContPhy, while ContPRO achieves the best overall performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ContPhy broadens physical-reasoning assessment to continuum properties and dynamics, especially for soft bodies and fluids, where evaluated models still struggle.

    Source: §1 · p. 2, §7 · p. 9

    The synthesized questions rely on a predefined vocabulary, limiting language diversity relative to natural language, and the scenarios omit some real-world objects and physical factors.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce the Continuum Physical Dataset (ContPhy), a novel benchmark for assessing machine physical commonsense. ContPhy complements existing physical reasoning benchmarks by encompassing the inference of diverse physical properties, such as mass and density, across various scenarios and predicting corresponding dynamics. We evaluated a range of AI models and found that they still struggle to achieve satisfactory performance on ContPhy, which shows that the current AI models still lack physical commonsense for the continuum, especially soft-bodies, and illustrates the value of the proposed dataset. We also introduce an oracle model (ContPRO) that marries the particle-based physical dynamic models with the recent large language models, which enjoy the advantages of both models, precise dynamic predictions, and interpretable reasoning. ContPhy aims to spur progress in perception and reasoning within diverse physical settings, narrowing the divide between human and machine intelligence in understanding the physical world. Project page: https://physical-reasoning-project.github.io

1. Introduction

The paper asks whether AI can infer continuum physical properties from observations and predict the resulting dynamics, addressing gaps in benchmarks that emphasize rigid objects or narrow prediction tasks. It introduces ContPhy and evaluates models on it, while proposing ContPRO as an oracle approach.

1. Introduction

  • Existing benchmarks often use simple rigid-object scenes or narrow contact prediction, leaving varied physical properties and language-based counterfactual and goal-planning questions untested.The paper notes that Physion uses objects with uniform physical parameters and asks whether two objects will make contact after the video.Source: §1 · p. 1
  • ContPhy combines diverse physical scenarios with video questions about properties and dynamics, including counterfactual predictions and goal-oriented problem-solving.Its question engine uses dense simulated annotations and predefined textual templates.Source: §1 · p. 2
  • Traditional AI models and multimodal large language models perform far from satisfactorily on ContPhy, while ContPRO achieves the benchmark’s best overall performance.ContPRO combines particle-based models for dynamic prediction with large language models for complex language reasoning, and requires more supervision signals.Source: §1 · p. 2
  • The paper’s benchmark spans properties such as mass, density, elasticity, and deformability, and its oracle combines symbolic representation with particle-based dynamic models.The authors present these as three contributions alongside evaluating multiple machine models.Source: §1 · p. 2

2. Related Work

Prior physical-reasoning benchmarks range from simple visual primitives to more complex scenes, but the paper distinguishes ContPhy by its varied soft-object properties and broader language-based dynamics tasks. It also positions ContPhy beyond conventional video question answering and simulation work on deformable bodies.

2. Related Work

  • Earlier benchmarks test physical intuition with visual primitives, while later work adds language questions about rigid collisions or predicts future contact in more complex scenes.The paper describes ContPhy as the only benchmark among those compared to include soft objects with varying physical parameters and language-based counterfactual and goal-planning questions.Source: §2 · p. 3
  • Typical video question-answering datasets focus on visual attributes, actions, and human-centered events rather than comprehensive physical reasoning.ContPhy is designed to evaluate physical reasoning about video scenes.Source: §2 · p. 3
  • Prior soft-body research has emphasized simulating deformable objects and fluids for robotic manipulation and cognitive experimentation, including simulations with varied physical parameters.The paper connects those simulation tools with natural-language physical-commonsense reasoning.Source: §2 · p. 3

3. Dataset

ContPhy is constructed to assess physical reasoning across diverse continuum scenes and question types. Its dataset combines simulated videos and structured questions, with question generation and scenario coverage designed to test property inference and dynamics reasoning.

3. Dataset

  • ContPhy targets reasoning across continuum scenes, including rigid bodies, soft bodies, and fluids, with physical properties treated as variables.The dataset construction covers scene diversity, question structure, simulation, question generation, and bias analysis.Source: §3 · p. 3

3.1. Diverse Physical Scenarios

  • The scenarios combine varied materials and properties, including liquids, cloths, ropes, plastoelastic balls, and articulated bodies, whose interactions produce different physical behaviors.The four scenarios cover liquid trajectories, cloth-covered object behavior, pulley tensions and coordinated motion, and soft-ball bouncing or permanent deformation.Source: §3.1 · pp. 4–5

3.2. Diverse Structured Questions

  • Questions cover physical-property inference and dynamics, including counterfactual, goal-driven, and predictive questions, with the available dynamics types varying by scenario.Fluids and balls include all three dynamics types; ropes include counterfactual and goal-driven questions, while cloth questions are predictive.Source: §3.2 · p. 5

3.3. Generation Setup and Statistics

  • Videos are simulated and rendered through a pipeline that records annotations, then a question engine uses templates and simulation results to generate question-answer pairs.The dataset contains 6,500 questions from 2,000 videos, split into 50% training, 20% validation, and 30% testing.Source: §3.3 · pp. 5–6

4. ContPRO

ContPRO is an oracle neural-symbolic framework that combines video perception, physical simulation, LLM-based program parsing, and symbolic execution. It pairs particle-based dynamic models with an LLM parser to answer physical reasoning questions with precise dynamic predictions.

4. ContPRO

  • ContPRO combines video perception, physical simulation, LLM program parsing, and symbolic execution to turn videos and questions into predicted answers.The perception module detects objects and static attributes; the simulator predicts dynamics, the parser produces executable programs, and the executor uses object attributes and dynamics to answer questions.Source: §4 · pp. 6–7
  • ContPRO marries LLMs with particle-based physical dynamic models, avoiding handcrafted design and in-domain parser training while providing precise dynamic prediction.Because the framework requires point clouds as input, the authors call it an oracle model.Source: §4 · p. 6
  • DPI-Net predicts rope and cloth dynamics, while MPM is used for fluids and soft balls because it can estimate properties and dynamics through differentiable inference.DPI-Net is trained on rope and cloth data; MPM properties are initialized to fixed values and optimized using gradient descent toward a Chamfer-loss target.Source: §4 · pp. 6–7
  • ChatGPT parses natural-language queries into executable Python programs using provided perception and dynamics APIs and a physical-reasoning prompt.The programs execute over extracted static attributes, physical properties, and dynamic trajectories to produce answers.Source: §4 · p. 7

5. Experiments

ContPhy evaluates physical reasoning across diverse video-question tasks using blind, visual, physical, and multimodal language-model baselines. Humans perform strongly, while current machine models remain inconsistent across question types and scenarios and struggle to match human physical video understanding.

5.1. Experimental Setup

  • The setup compares blind, visual, physical, and multimodal models, scoring property questions as classification and dynamic choices as binary classification.For dynamics, accuracy is reported per option and per question, with a multiple-choice question counted correct only when all choices are correct.Source: §5.1 · p. 7

5.2. Evaluation of Physical Reasoning

  • Blind models generally underperform other model families, though they perform similarly in some rope goal-driven questions that challenge machine physical reasoning.The authors attribute the broader weakness to the importance of visual information; humans still achieve high performance.Source: §5.2 · pp. 7–8
  • ALPRO performs robustly overall, especially on cloth and rope, but no visual model achieves top accuracy across every scenario.Visual models excel on rope counterfactual and cloth predictive questions but fall short on goal-driven questions, which may require reverse reasoning from a goal.Source: §5.2 · p. 8
  • Physical models are competitive in some settings, including PhyDNet on rope and ball counterfactuals and PIP on fluid goal-driven questions, but have limitations such as cloth.The authors hypothesize that models designed around simple primitives struggle with ContPhy’s continuum objects, varied environments, and question types.Source: §5.2 · p. 8
  • Gemini and GPT-4V fall short on cloth and fluid questions but perform best in some rope and ball settings; ContPRO excels on most questions and surpasses human accuracy on predictive questions.The authors suggest discrete-frame visibility and unfamiliar soft objects may contribute to MLLM weaknesses; ContPRO uses MPM for fluid and ball dynamics and achieves 96.9% on fluid stick-number inference and 63.5% on fluid density inference.Source: §5.2 · pp. 8–9
  • Humans outperform visual and foundation models in all scenarios, based on 460 valid answers from 16 participants.The study sampled video-question pairs across all scenarios and question types.Source: §5.2 · p. 9
  • Current state-of-the-art models struggle with ContPhy, while human results show strong physical-property inference and counterfactual reasoning about unseen information.Across question types and scenarios, model performance varies; the authors characterize the benchmark as a significant challenge for vision-language models seeking human-like physical video understanding.Source: §5.2 · pp. 7, 9

6. Limitations

ContPhy has limitations in language diversity and scenario complexity. Its question vocabulary is predefined, and its four scenarios do not cover all real-world physical interactions or factors.

6. Limitations

  • ContPhy questions use a predefined vocabulary, leaving their language diversity limited relative to natural language.The authors identify this as a limitation of the synthesized questions produced by the question engine.Source: §6 · p. 9
  • The dataset covers four designed scenarios, while real-world interactions can involve additional objects and physical factors not included.The scenarios feature solids, ropes, cloths, and fluids.Source: §6 · p. 9

7. Conclusion

ContPhy broadens physical-reasoning evaluation to continuum settings, especially soft bodies and fluids, but evaluated AI models still struggle on the benchmark.

  • Evaluated AI models still struggle with continuum physical reasoning, especially for soft bodies and fluids.ContPhy assesses physical-property inference across dynamic contexts and prediction of corresponding dynamics.Source: §7 · p. 9

Impact Statement

The work aims to evaluate current machine-learning models’ physical reasoning and facilitate development of models with physical commonsense.

  • The work aims to evaluate current models’ physical reasoning and facilitate development of AI models with physical commonsense.The authors report no potential negative societal consequences requiring specific attention.Source: Impact Statement · p. 10

8. Dataset Details

ContPhy’s dataset details describe multimodal simulation outputs and annotations, varied video scenarios, and a balanced question-generation process. The questions combine visual, physical-property, and dynamics reasoning, with LLM rephrasing increasing lexical diversity and balance.

8.1. Video Details.

  • Each scenario provides sensor outputs and semantic annotations, with sensor data describing objects’ 4D states at multiple levels.Outputs include Full HD RGB, object-level data, meshes and particles, and collision or touch events; fluid and rope meshes are resampled each frame, while particle outputs track points across frames.Source: §8.1 · p. 13, §8.1.1 · p. 13

8.1.2. ANNOTATION DATA STRUCTURE.

  • Annotations record camera and sampling information alongside scenario-specific object properties and simulation outcomes.Fluid annotations include density and collision records; rope annotations cover objects, motion direction, and segment tension; cloth and ball annotations capture sampled properties and simulation outcomes.Source: §8.1.2 · p. 13

8.1.3. PHYSICAL VIDEO DIVERSITY

  • The 2,000 videos span four scenarios, with 500 per scenario and randomized configurations tailored to each scenario.Fluid densities are selected to produce discernible stratification; rope layouts and object attributes vary; cloth parameters and object arrangements vary; ball properties and wall positions vary.Source: §8.1.3 · pp. 13–14

8.2.1. QUESTION DISTRIBUTION

  • Question counts are balanced by type, and the dataset compares their distributions across fluid, rope, cloth, and ball scenarios.Source: §8.2.1 · p. 14, Figure 5 · p. 15

8.2.2. QUESTION TEMPLATES AND EXAMPLES

8.2.3. LOGICAL STEPS TO INFER ANSWERS

  • Most question types require two or three reasoning steps spanning visual attributes, physical properties, and dynamics across solids, soft objects, and liquids.Source: §8.2.3 · p. 15

8.2.4. LLM-BASED QUESTION REPHRASING

  • Gemini automatically rephrases questions, and lexical diversity is evaluated with TTR and word distribution alongside syntactic and question-type measures.The evaluation also reports sentence length and variance, question-type counts and distributions, and Flesch-Kincaid grade level for readability.Source: §8.2.4 · p. 15

8.2.5. WORD DISTRIBUTION

  • After LLM rephrasing, question wording is more diverse and its word distribution more balanced.Source: §8.2.5 · p. 15

9. The Continuum: Liquids, Soft Bodies, Rigid Bodies, and Articulated Bodies

ContPhy uses “continuum” for a broad range of bodies and examines their physical dynamics and properties, focusing particularly on fluids and soft bodies.

  • ContPhy’s continuum includes liquids, soft materials, rigid bodies, and articulated bodies, extending beyond datasets focused on rigid visual primitives.Source: §9 · p. 16
  • The scenarios couple body types: rope-and-pulley scenes combine ropes, rigid bodies, and articulated bodies, while cloth scenes combine cloth and rigid bodies.Fluid scenes feature liquids and ball scenes focus on soft balls; the dataset includes dynamics reasoning and parameter or concept reasoning such as density, tension, elasticity, and mass.Source: §9 · p. 16
  • The paper focuses predominantly on fluids and soft bodies, while retaining rigid bodies across scenarios and articulated bodies in rope scenarios.Source: §9 · p. 16

10. More Implementation Details

The appendix details foundation-model evaluation and ContPRO’s implementation across its perception and physical simulation modules. It also specifies scenario-specific physical inference settings for fluids, balls, rope, and cloth.

10.1. Foundation Models Evaluation Details

  • Gemini and GPT4-V were evaluated on 10-frame video samples using prompts tailored to question groups.Source: §10.1 · p. 16

10.2. Oracle Model ContPRO Details

  • ContPRO uses a Mask R-CNN ResNet-R101-FPN perception module trained for 50k iterations on full-HD images.The batch size is 16 across 8 GPUs, with 1,000 proposals per image.Source: §10.2 · p. 16
  • ContPRO uses MPM for ball and fluid simulations and DPI-Net for rope and cloth simulations.Source: §10.2 · p. 16
  • Fluid inference runs in 2D, resamples 30 or 100 points per flow branch, and estimates properties from frame 190 through frame 250.The video frame step is 1/60 and the simulation step is 1/3000.Source: §10.2 · p. 16
  • Ball inference resamples 200 points per ball and uses the von Mises formula, with a simulation step of 1/6000/32 to capture rapid collisions.Property inference runs from the video start to the first collision.Source: §10.2 · p. 16
  • Rope inference adds object mass to GNN training and predicts 30 frames from frame 0; cloth inference begins at frame 15 and predicts 115 frames.For rope counterfactuals, the input mass is revised; cloth movement is set to ground truth because object motion is caused by cloth movement.Source: §10.2 · pp. 16–17

11. Experiments of More Baselines

The experiments examine point-cloud features for vision baselines and compare additional prompting methods for multimodal language models. Point clouds improve vision-model performance in almost all tested settings.

11.1. Experiments of Multi-modalities

  • Point-cloud features improve CNN-LSTM and MAC performance in almost all settings.The authors attribute the gains to added object-location and spatial-relationship information, which is important for predicting dynamics.Source: §11.1 · p. 17

11.2. Experiments of More Prompting Methods

  • Prompting comparisons include scenario-specific guidelines, in-context examples, and human-explained examples; Figure 6 plots average, maximum, and minimum scores alongside human performance.The chart values subtract random-choice scores for normalization.Source: §11.2 · p. 17

12. Qualitative Examples

The section presents qualitative ContPRO program examples across scenarios and question types, with the full API described in Listing 7.

12. Qualitative Examples

  • The examples show Python-style ContPRO programs, with one case for each scenario and question type.The listing presentation omits some comments, spaces, and blank lines for clarity.Source: §12 · p. 17
  • The rope property example queries two objects’ masses and tests whether one exceeds half the other’s mass.Source: §12 · p. 26
  • Listings illustrate predictive cloth-contact, counterfactual ball-drop, and goal-driven fluid-entry programs.Source: §12 · p. 26
  • The API registers events to initiate predictive or counterfactual simulations, then checks whether an action occurs in the resulting dynamic scene.The API also supports goal-driven simulations and querying object properties.Source: §12 · pp. 29–32
Loading 2402.06119v2…