Source-linked AI summary
PIQA: Reasoning about Physical Commonsense in Natural Language
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi
TL;DR
Physical commonsense reasoning remains difficult for language models despite its importance for understanding and manipulating everyday objects. The paper introduces PIQA, a benchmark built from goal-solution pairs, and finds that pretrained models fall short of human performance while revealing gaps in their physical knowledge.
Problem
Language models lack reliable physical commonsense about everyday objects, properties, affordances, and manipulation, although this knowledge is important for daily problem solving and AI systems that interact with the world.
Method
The paper introduces PIQA, a benchmark of goal-solution pairs constructed from crowdsourced everyday instructions and alternative approaches targeting concrete physical reasoning.
Results
The best pretrained models still fall roughly 18 points short of human performance, and analyses identify which physical-knowledge facets challenge them.
Takeaways & Limitations
PIQA provides a benchmark and diagnostic resource for advancing language representations that capture physical knowledge traditionally learned through experience.
Takeaways & Limitations
Performance on PIQA is not intended to establish that models cannot learn physical knowledge from language alone, since in-domain data and fine-tuning could improve results.
Abstract
from arXiv · showhide
To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains - such as news articles and encyclopedia entries, where text is plentiful - in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical common-sense questions without experiencing the physical world? In this paper, we introduce the task of physical commonsense reasoning and a corresponding benchmark dataset Physical Interaction: Question Answering or PIQA. Though humans find the dataset easy (95% accuracy), large pretrained models struggle (77%). We provide analysis about the dimensions of knowledge that existing models lack, which offers significant opportunities for future research.
Introduction
Physical commonsense knowledge supports everyday reasoning, but reporting bias limits its textual expression and challenges pretrained language models beyond abstract tasks. PIQA addresses this gap by benchmarking physical commonsense through everyday situations and analyzing model shortcomings.
- Motivation: Children form physical categories and concepts before language, acquiring knowledge of objects’ properties, affordances, and manipulability that supports everyday problem solving.This knowledge contributes to day-to-day human activities such as deciding what objects can be used for.
- Motivation: Reporting bias means commonsense facts about inappropriate object use are rarely directly reported, limiting learning from text.The passage gives applying eyeshadow with a toothpick as an example of a rarely reported physical-commonsense fact.
- Problem: Pretrained language models have succeeded on abstract tasks, but their ability to robustly answer questions requiring a physical world model remains unclear.The passage contrasts encyclopedia question answering and named-entity recognition with the unresolved challenge of physical commonsense reasoning.
- Benchmark: PIQA evaluates language representations on physical commonsense through everyday situations, with a preference for atypical solutions.The benchmark is inspired by instructables.com and focuses on building, crafting, baking, or manipulating objects with everyday materials.
- Benchmark: The dataset targets physical knowledge by collecting semantic perturbations and alternative approaches, then removing basic artifacts with AFLite.The alternatives are designed to remain syntactically and topically similar while testing physical knowledge rather than superficial cues.
- Contributions: The work shows that large-scale language-pretraining approaches fall short when physical-world modeling is required and uses error and corpora analyses to guide future research.The paper’s stated goal is to develop language representations capturing details of the real world.
Dataset
PIQA is a multiple-choice benchmark for physical commonsense understanding, built from how-to instructions and Goal–Solution pairs. Its design emphasizes concrete reasoning, subtle physical errors, balanced linguistic cues, and reduced annotation artifacts.
- Task format: PIQA frames physical commonsense understanding as multiple-choice QA in which a model or human selects the single correct solution from two candidates.The dataset uses questions q and two possible solutions s1 and s2, exactly one of which is correct.
- Data construction: Annotators use instructables.com prompts spanning costume, outside, craft, home, food, and workshop categories to create concrete goals, valid procedures, and plausible but incorrect tricks.The tricks often exploit subtle misunderstandings of preconditions or physics, while the source instructions encourage atypical uses of familiar objects.
- Validation: Validation removes low-agreement examples and uses qualification and batch-level review, excluding cases requiring expert domain knowledge rather than commonsense.Participants needed over 80% qualification accuracy, and users with low agreement were removed from participation.
- Dataset statistics: Over 16,000 training QA pairs, approximately 2K development pairs, and approximately 3K test pairs comprise the dataset, totaling over 3.7 million training lexical tokens.Goals average 7.8 words, while both correct and incorrect solutions average 21.3 words.
- Dataset properties: At least 85% vocabulary overlap between correct and incorrect solutions and nearly identical sequence-length distributions reduce simple linguistic cues and keep the task centered on physical phenomena.The training data contains 6,881 unique nouns, 2,493 verbs, 2,263 adjectives, and 604 adverbs; AFLite is also used to remove stylistic artifacts and trivial examples.
Experiments
Experiments evaluate pretrained transformer models on PIQA using independent two-way classification, finding that RoBERTa modestly improves over GPT but remains substantially below human performance. Fine-tuning required grid-search tuning because some configurations, particularly for BERT, were unstable.
- Models: The experiments evaluate GPT, BERT, and RoBERTa-Large on PIQA using pretrained transformer models.RoBERTa-Large has 355M parameters.
- Evaluation setup: Models classify the two solution choices independently from the goal, choice, and [CLS] representation, followed by a linear transformation and two-option softmax.This setup approximates the probability of each solution choice.
- Optimization: Fine-tuning was often unstable, particularly for BERT, so the experiments used grid search over learning rates, batch sizes, and training epochs and reported the best validation configuration.Some hyperparameter settings produced validation performance around chance.
- Results: The model results show a roughly 20-point absolute gap between state-of-the-art systems and human performance on PIQA.Human performance was calculated by majority vote among annotators who achieved ≥90% on the qualification task.
- Results: 8pt performance gain separates RoBERTa from GPT, yet RoBERTa remains roughly 18 points short of human performance on PIQA.The paper attributes the limited gain despite RoBERTa’s larger training setup to the dataset’s difficulty and adversarial construction for BERT.
Analysis
The analysis probes what physical commonsense RoBERTa learns from language, finding weaknesses in simple relations, versatile concepts, and novel combinations of familiar objects. PIQA remains difficult even when answer choices differ only by small edits.
- Analysis: PIQA enables probing pretrained language models’ physical knowledge, a domain with less linguistic literature than syntax.The analysis uses PIQA to examine the physical knowledge learned through language alone.
- Single-word edits: RoBERTa performs nearly at chance on questions distinguished by flexible relations such as “before,” “after,” “top,” and “bottom.”These relations are highlighted among the concepts where the model struggles most.
- Common replacements in PIQA: 75% accuracy on “water” replacements contrasts with 90% for “spoon,” while “freeze” reaches 66%, showing difficulty with versatile nouns and verbs.The analysis attributes the contrast partly to water’s broad properties and affordances, whereas spoons have fewer challenging properties.
- Qualitative analysis: Qualitative examples show that RoBERTa recognizes ridiculous generations but struggles with subtle relations, non-prototypical situations, and novel combinations of common objects.Humans can visualize uncommon actions, such as using a gluestick as coaster feet, and verify their results.
Related Work
Related work on physical understanding spans NLP, computer vision, and robotics. Prior studies address procedural and causal knowledge, visual relationships and commonsense, intuitive physics, and embodied learning for tool use and construction.
- Physical understanding connects scientific knowledge with interactive knowledge acquisition by embodied agents across NLP, computer vision, and robotics.
- Language: NLP research studies causal implications, knowledge extraction, and large language models’ failures on tools and elided procedural knowledge in recipes.
- Vision: Vision research predicts visual relationships, actions, and dependent objects, while HAKE annotates essential objects and body parts for completing or defining actions.
- Vision: Visual commonsense research examines intuitive physics, cause-effect relationships, and inferences beyond a single image.
- Robotics: Robotics research uses interaction, intuitive-physics priors, and internal models of physics, shape, and material strength to advance tool use and construction.
Conclusion
The paper introduces PIQA as a benchmark for physical commonsense understanding and finds that pretrained models lack basic physical knowledge. It frames these shortcomings as evidence that learning about the world from language alone is limiting, while aiming to support progress toward more useful language models.
- The authors argue that learning about the world from language alone is limiting, despite evaluating large-scale pretrained models as NLP’s de facto progress standard.They state that matching humans through extensive in-domain data and fine-tuning would miss the point.
- PIQA is introduced as a benchmark for evaluating and studying physical commonsense understanding in natural language models.
- The best available pretrained models lack understanding of some of the most basic physical properties of the world around us.
- PIQA aims to provide insight and a benchmark for language representations that capture traditionally seen or experienced knowledge and enable language models useful beyond NLP.