Source-linked AI summary
Physically Grounded Vision-Language Models for Robotic Manipulation
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, Dorsa Sadigh
TL;DR
Current VLMs have limited object-centric physical reasoning, restricting their usefulness for robotic manipulation. The paper introduces PHYSOBJECTS and fine-tunes a VLM on its annotations, then uses that model in LLM-based robotic planning. The resulting model improves physical reasoning, generalizes to held-out concepts, and improves planning and real-robot task success.
Problem
Current VLMs have limited understanding of physical concepts such as material and fragility in common objects, limiting their usefulness for robotic manipulation.
Method
The paper builds PHYSOBJECTS and fine-tunes InstructBLIP on physical-concept annotations, then integrates the fine-tuned VLM into an LLM-based robotic planner.
Results
Fine-tuning improves physical reasoning, including held-out concepts, while the integrated system improves planning performance and real-robot task success rates.
Takeaways & Limitations
Physically grounded VLMs can provide useful visual physical reasoning for robotic planning without relying solely on interaction data.
Takeaways & Limitations
The fine-tuned VLM still makes errors relative to human judgment.
Abstract
from arXiv · showhide
Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PhysObjects, an object-centric dataset of 39.6K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, including generalization to held-out concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically grounded VLMs. We additionally illustrate the benefits of our physically grounded VLM on a real robot, where it improves task success rates. We release our dataset and provide further details and visualizations of our results at https://iliad.stanford.edu/pg-vlm/.
I. INTRODUCTION
The paper addresses limited VLM understanding of physical object concepts by using scalable visual reasoning and physical-concept annotations, then integrates the resulting model into robotic planning. The approach improves physical reasoning and planning performance, including on a real robot.
- Motivation: LLM-based robotic planners need sensory grounding to reason about physical object properties relevant to manipulation.These properties include material composition and fragility, which can help identify objects to interact with and their affordances.
- Motivation: Human visual perception can provide high-level physical judgments from static images without object interactions.Examples include judging a glass cup as more fragile than a plastic bottle or a bowl as better suited than a shallow plate for holding water.
- Motivation: Pre-trained VLMs offer a scalable alternative for physical reasoning, but their out-of-the-box performance remains limited because internet-scale data contains few object-centric physical reasoning examples.PHYSOBJECTS is designed to add more suitable data about common real-world objects.
- Contributions: PHYSOBJECTS contains 39.6K crowd-sourced and 417K automated physical concept annotations of real household objects for VLM fine-tuning.The dataset is the paper’s central resource for improving physical reasoning.
- Robotic application: The fine-tuned VLM is queried by an LLM about scene objects before the LLM produces a plan, and the plans are evaluated on a real Franka Emika Panda robot.The system improves planning performance on tasks requiring physical reasoning and improves real-robot task success rates.
II. RELATED WORK
Prior work studies physical reasoning from interaction data, physical representations, VLMs, and LLM-based robotic planning. This paper combines pre-trained VLMs with human annotations as a scalable alternative for object-centric physical reasoning.
- Physical reasoning: Prior physical-reasoning methods learn object properties or representations from interaction data.Other approaches focus on representations that capture physical concepts rather than directly estimating properties.
- Positioning: PHYSOBJECTS differs from prior interaction-based approaches by using pre-trained VLMs and human annotations as a more scalable alternative to learning from interaction.Its focus is physical reasoning about common real-world objects for robotics.
- VLMs and robotic planning: VLMs provide broad visual-language reasoning, while prior work has applied LLMs and VLMs to robotic planning and scene feedback.The paper uses InstructBLIP as its base VLM and positions PHYSOBJECTS as focused object-centric physical-reasoning supervision.
III. PHYSOBJECTS DATASET
PHYSOBJECTS benchmarks physical concepts on real household-object images using automated and crowd-sourced annotations. Its concepts, splits, and preference-based labels are designed to support scalable visual physical reasoning while preserving challenging examples.
- Dataset construction: PHYSOBJECTS contains 39.6K crowd-sourced and 417K automated physical concept annotations for images of real household objects.The image source contains 117,424 images, 225,466 object bounding boxes, 277 object categories, and 4,203 object instance IDs.
- Dataset construction: The dataset uses object-instance-based train, validation, and test splits comprising 73.0%, 14.8%, and 12.2% of objects, respectively.Splitting by object instance helps define separate object instances across the partitions.
- Physical concepts: The dataset covers eight main physical concepts and two held-out concepts, excluding friction and volume because they require interaction or geometric reasoning outside the study’s focus.Held-out concepts are density and liquid capacity.
- Physical concepts: Main concepts include continuous mass, fragility, and deformability; categorical material and transparency; and three container-specific concepts.The container concepts are contents, ability to contain liquid, and being sealed, with 956 container object instances.
- Annotation scheme: Continuous concepts are annotated as pairwise preferences indicating which object has a higher level, whether levels are roughly equal, or whether the relationship is unclear.Preference comparisons are used because precise continuous values are difficult to specify from static images.
- Annotation quality: Crowd-sourced annotations cover 13.2K examples, with 93.7% reaching at least 2/3 annotator agreement and 58.1% reaching unanimity.The annotations were collected from 573 crowd-workers, and automated labels add 417K examples.
IV. PHYSICALLY GROUNDING VISION-LANGUAGE MODELS
The method fine-tunes InstructBLIP to predict physical concepts from object images and prompts, using likelihood-based scores to learn both categorical labels and pairwise preferences.
- Fine-Tuning VLMs: InstructBLIP receives a single object bounding box and concept-specific question prompt, then predicts text output for physical-concept reasoning.The model uses the FlanT5-XXL version of InstructBLIP.
- Learning From Preferences: Categorical concepts are trained by maximum likelihood of annotated labels, while continuous concepts require a continuous preference score.Preference annotations compare two objects as higher, equal, or unclear on a concept.
- Learning From Preferences: The score s(o, c) equals the VLM likelihood of “yes” conditioned on object o and concept c.This score is non-negative, and its logarithm represents the difference between yes and no log-likelihoods.
- Learning From Preferences: The Bradley-Terry model converts two object scores into the probability that object o1 exceeds object o2 on concept c.The probability uses s(o1, c) divided by the sum of both object scores.
- Learning From Preferences: Fine-tuning minimizes a binary cross-entropy objective over preference labels, using score differences as logits.The logits are log s(o, c) = log p(yes | o, c) − log p(no | o, c).
V. EXPERIMENTAL RESULTS
The evaluation measures physical reasoning at the dataset, planning, and real-robot task levels.
- Evaluation: The study evaluates VLMs using PHYSOBJECTS test accuracy, planning accuracy in real scenes, and real-robot task success rate.These metrics cover object-level reasoning, robotic planning, and physical task execution.
A. Dataset Evaluation
PG-InstructBLIP improves physical-concept recognition over InstructBLIP, including held-out concepts and paraphrased prompts, while benefiting from relatively small annotated datasets.
- Dataset Evaluation: PG-InstructBLIP outperforms InstructBLIP on all evaluated main concepts, with the largest improvement on contents.It also performs slightly better than single-concept fine-tuned models, suggesting possible positive transfer.
- Dataset Evaluation: PG-InstructBLIP improves held-out-concept accuracy by 11.9% despite never seeing those concepts or object instances during fine-tuning.The held-out concepts are density and liquid capacity.
- Dataset Evaluation: PG-InstructBLIP remains superior to InstructBLIP on unseen paraphrased prompts, with limited degradation from the original prompts.This result indicates robustness to question variety in the evaluated setting.
- Dataset Scaling: Performance scales positively with PHYSOBJECTS dataset size, yet models still benefit significantly from using only 10% of the dataset.The result suggests that relatively small amounts of annotated data can improve VLM physical reasoning.
- Additional Results: The appendix reports limited degradation on general VQA benchmarks compared with InstructBLIP.It also includes results for alternative prompts, smaller models, automatically annotated data, held-out concepts, and fine-tuning ablations.
B. Real Scene Planning Evaluation
The evaluation tests language-instructed planning on unseen real scenes, comparing physically grounded and baseline VLMs across task categories. PG-InstructBLIP outperforms InstructBLIP across all categories, with the largest advantage on Multi-Concept tasks.
- Evaluation setup: The evaluation covers 51 task scenarios across 8 scenes, divided into Single Concept, Multi-Concept, and Common Knowledge categories.Single Concept uses one physical concept, while Multi-Concept requires reasoning about multiple concepts.
- Results: PG-InstructBLIP outperforms InstructBLIP on all task categories, especially Multi-Concept.
- Results: PG-InstructBLIP performs slightly better than InstructBLIP on Common Knowledge tasks.
- Results: Planning without VLM interaction performs substantially worse, indicating that the tasks require grounding beyond object detection.
C. Real Robot Evaluation
The real-robot evaluation measures plan quality on two scenes with five pick-and-place tasks per scene. PG-InstructBLIP produces successful executions more often, but remains vulnerable to out-of-distribution question formats.
- Evaluation setup: The evaluation uses a Franka Emika Panda robot across two scenes with five tasks per scene.Success rates reflect plan quality because the primitive executions are ensured to succeed.
- Task design: The tasks include sorting objects by material, liquid-carrying ability, transparency, mass, and deformability.
- Results: PG-InstructBLIP leads to successful robot executions more often than InstructBLIP.
- Caveat: PG-InstructBLIP can still fail on questions that differ from its training prompt formats.Both models incorrectly classify the glass jar as translucent, despite the physically grounded model’s lower error on the ceramic bowl’s material question.
VI. DISCUSSION
The discussion presents PHYSOBJECTS as a large-scale resource for physically grounded VLMs and reports improved physical reasoning and robotic planning. It also identifies limits in physical-quantity grounding and scope beyond the selected concepts.
- Summary: Fine-tuning a VLM on PHYSOBJECTS improves physical reasoning, including performance on held-out physical concepts.
- Summary: Using the fine-tuned VLM for real-world robotic planning improves performance on tasks requiring physical reasoning.
- Limitations: The model’s outputs are relational rather than grounded in real physical quantities, limiting applications such as identifying whether an object is too heavy to pick up.
- Future work: Future work could expand the dataset toward geometric and social reasoning for broader downstream robotic applications.
- Physical concepts: The dataset defines ten physical concepts, including mass, fragility, deformability, liquid capacity, material, transparency, contents, liquid transportability, and sealing.
D. Training Details
Training balances crowd-sourced and automated data while varying image and prompt inputs to improve robustness. The reported setup fine-tunes InstructBLIP with Q-Former-only updates and removes Q-Former text conditioning.
- Optimization: Fine-tuning updates only the Q-Former parameters and uses mixed-precision bfloat16 training with linear learning-rate warmup and decay.
- Dataset balancing: Continuous-concept sub-datasets are sampled at rates proportional to the square root of their annotation counts to address data imbalance.
- Image sampling: Training randomly samples among an object’s bounding-box images, except for crowd-sourced contents and is sealed data whose labels may vary across images.
- Prompt design: Half of training questions include object category labels and half omit them to promote robustness to different VLM queries.
- Ablation: Removing Q-Former text conditioning improves general VQA and held-out paraphrased-prompt evaluation but does not significantly change the main crowdsourced evaluation.
E. Evaluation Details
The evaluation measures physical reasoning on PHYSOBJECTS, held-out concepts, VQA benchmarks, and planning-related settings. Results indicate robustness across fine-tuning choices, comparable performance across model sizes, and limited degradation on general VQA.
- Evaluation protocol: At least 2/3 annotator agreement and majority labels define crowdsourced test ground truth.
- Evaluation protocol: Continuous-concept predictions compare paired objects, while categorical predictions select the highest-likelihood PHYSOBJECTS label.
- Generalization and model size: Fine-tuning on PHYSOBJECTS does not significantly degrade general VQA performance, while the two evaluated model sizes have comparable physical-reasoning performance.
- Dataset evaluation: Automatically annotated examples are easier because object categories alone can determine them, producing generally higher performance than crowdsourced data.
- Held-out concept scaling: Held-out-concept experiments fine-tune on all concepts except one, then assess additional fine-tuning on the held-out concept against base-model baselines.
- Ablations: Ablated models show no significant overall performance change, and automatically annotated data does not significantly affect crowdsourced-data performance.
F. Real Scene Planning Evaluation Details
The real-scene evaluation tests whether an LLM planner benefits from querying a VLM about detected objects’ physical concepts. The framework compares interactive VLM-assisted planning with planning without VLM interaction under constrained robot primitives.
- Planning framework: OWL-ViT detects scene objects and categories, then GPT-4 receives detections, task instructions, primitives, and a few-shot chain-of-thought prompt.
- Planning framework: The interactive planner asks object-centric questions, receives the VLM’s top responses with likelihoods, and continues until it produces a primitive-based plan or declares the task impossible.
- Robot primitives: The planner’s available actions include going to, picking up, bringing, putting down, and completing an action sequence.
- Example planning: Example reasoning selects a carryable lock instead of the heavier countertop when the task requests the heaviest object.
- Example planning: For deformable-container retrieval, the planner first identifies likely containers and then selects the object with the highest deformability confidence.
- Evaluation setup: The evaluation compares plans from an interactive VLM prompt and a no-interaction prompt, judging whether plans satisfy instructions while respecting heavy-object constraints.
- Evaluation limitation: The planner sometimes emits primitives not provided in the prompt, and evaluators do not specially handle these cases because the evaluation targets physical reasoning benefit rather than functional planning.
G. Real Robot Evaluation Details
The real-robot evaluation uses object-moving primitives and evaluates whether generated plans succeed on physical manipulation tasks. Execution retries isolate planning quality from primitive-execution failures.
- Robot setup: The real robot provides move-[X]-to-the-side, move-[X]-into-[Y], and done primitives parameterized by detected-object letters.
- Robot setup: Experiments use a 7-DoF Franka Emika Panda with a Robotiq 2F-85 gripper and Polymetis real-time control.
- Task constraints: Prompts restrict side-moving and container-moving tasks to their corresponding primitive and prohibit moving furniture.
- Evaluation procedure: Plans are retried until all primitives execute successfully, so reported success rates reflect planning quality rather than primitive execution.
- Evaluation procedure: A non-author human evaluator judges task success from the instruction and a video of the robot executing each generated plan.
- Evaluation data: The evaluation tables include scene images, detected objects, and task instructions, with occasional more precise labels supplied when OWL-ViT labels are inaccurate or ambiguous.