PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarks for Automated Commonsense Reasoning: A Survey

Ernest Davis

arXiv:2302.04752v2cs.AI

TL;DR

Commonsense benchmarks are often flawed, and the paper notes that there is no clear independent measure of commonsense reasoning. It examines how to identify commonsense use and benchmark artifacts, concluding that absolute measurement remains aspirational and that carefully vetted test sets matter.

  • Problem

    Commonsense task boundaries are difficult to define, and benchmarks may measure artifacts rather than commonsense reasoning, complicating assessment.

    Source: §1.1 · p. 4, §2.1 · p. 11

  • Method

    The paper surveys criteria for identifying commonsense reasoning, ways to test whether systems use commonsense resources, and common benchmark purposes and flaws.

    Source: §1.1 · p. 4, §2 · p. 10, §2.1 · p. 11

  • Results

    An absolute measure of commonsense reasoning remains a pipe dream because no independent, well-defined measure is currently known.

    Source: §3 · p. 12

  • Takeaways & Limitations

    For practical test sets, small and clean is better than large and noisy because expert vetting is costly and limited.

    Source: §5.1 · p. 16

    Benchmark results cannot currently provide an absolute measure independent of a specific benchmark.

    Source: §3 · p. 12

Abstract

from arXiv · show

More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of artificial intelligence (AI) systems. However, these benchmarks are often flawed and many aspects of common sense remain untested. Consequently, we do not currently have any reliable way of measuring to what extent existing AI systems have achieved these abilities. This paper surveys the development and uses of AI commonsense benchmarks. We discuss the nature of common sense; the role of common sense in AI; the goals served by constructing commonsense benchmarks; and desirable features of commonsense benchmarks. We analyze the common flaws in benchmarks, and we argue that it is worthwhile to invest the work needed ensure that benchmark examples are consistently high quality. We survey the various methods of constructing commonsense benchmarks. We enumerate 139 commonsense benchmarks that have been developed: 102 text-based, 18 image-based, 12 video based, and 7 simulated physical environments. We discuss the gaps in the existing benchmarks and aspects of commonsense reasoning that are not addressed in any existing benchmark. We conclude with a number of recommendations for future development of commonsense AI benchmarks.

1 Commonsense Reasoning

Common sense is a vague, multidimensional category, so the paper surveys proposed characteristics rather than seeking a tight definition. Its boundaries and expressions across language, culture, cognition, and modality complicate attempts to identify it cleanly.

1.1 Claims about common sense that seem true to me

  • Common sense spans tasks and modalities, and its scope includes physical, biological, psychological, social, spatial, and temporal realities.It can support inferences such as recognizing that a long-distance walk cannot be completed in fifteen minutes.Source: §1.1 · p. 4
  • Commonsense knowledge is generally foundational and rarely explicitly taught, unlike common, encyclopedic, or expert knowledge.Benchmarks often cross these boundaries, including both common knowledge and obscure encyclopedic or expert facts.Source: §1.1 · p. 4
  • Commonsense reasoning goes beyond linguistic or perceptual interpretation and is integrated with abilities such as language, vision, and manipulation.The trophy-and-suitcase example depends on size constraints, not just understanding the words or syntax.Source: §1.1 · pp. 4–5

1.2 Half-truths about commonsense knowledge

  • Language independence is a proposed characteristic, but text-trained language models tie knowledge to surface forms and commonsense language bias remains largely untested.Most resources and benchmarks exist only in English or in symbolic forms using English words.Source: §1.2 · p. 6
  • Commonsense beliefs vary across cultures and historical periods, while widespread and reasonably stable beliefs form a more central core.The paper treats beliefs confined to a small group or short time period as more peripheral.Source: §1.2 · pp. 6–7
  • Commonsense reasoning may be fast and intuitive, but System 1 and System 2 are not cleanly separated and their proposed criteria do not align perfectly.Some processes can be automatic by default yet deliberately performed with greater attention.Source: §1.2 · p. 7
  • Simple language is often associated with commonsense knowledge, but spatial knowledge is an important exception because precise spatial representations may require technical or non-linguistic forms.Describing the possible shapes of a string bag illustrates this challenge.Source: §1.2 · p. 8

1.3 An untrue claim about commonsense knowledge

  • Commonsense knowledge is not necessarily logically simple; understanding a “vain threat” or a string bag's possible configurations involves complex structure.Characterizing the shapes a string bag can attain is complex, especially when reasoning about bags in general.Source: §1.3 · pp. 8–9

2 What is a commonsense task? What AIs possess commonsense knowledge?

Identifying a commonsense task is difficult because common-sense ability and task definitions can become circular. The paper proposes combining task characteristics and evidence of actual reasoning use, while warning that benchmark artifacts can mimic success.

2 What is a commonsense task? What AIs possess commonsense knowledge?

  • Commonsense task and ability definitions are circular and do not connect cleanly to the proposed characteristics of commonsense knowledge.Unlike well-defined tasks such as translation, commonsense reasoning typically occurs within other tasks and can apply across task types.Source: §2 · p. 9
  • A multi-pronged assessment can consider whether tasks are simple across people, cultures, and languages and whether systems actually use commonsense knowledge.Training-set analysis, ablation, or the effect of removing a commonsense resource can provide evidence of such use.Source: §2 · pp. 9–10
  • A system's benchmark success may reflect resource features only weakly related to commonsense, as in KVL-BERT's use of ConceptNet connections without arc labels or directions.This makes it important to examine what knowledge a system actually uses.Source: §2 · p. 10

2.1 Artifacts

  • Artifacts can let systems succeed without solving the intended task; in SNLI, they enabled solving two thirds of problems, and in MultiNLI, more than half.Negation words in the hypothesis were more common when it contradicted the premise, and removing the premise exposed reliance on this clue.Source: §2.1 · p. 11

3 Purposes

Commonsense benchmarks can serve several purposes, from comparing systems and tracking progress to drawing attention to neglected problems. The paper also treats benchmark construction itself as a goal, while noting that absolute measurement remains out of reach for now.

3 Purposes

  • Benchmarks can enable comparisons between systems and times and characterize performance across different kinds of tasks..Source: §3 · p. 11
  • Absolute measurement is currently a pipe dream because there is no clear independent measure of commonsense reasoning, though a well-defined measure may become possible with deeper understanding..Source: §3 · p. 12
  • Benchmarks may also serve as resources, draw attention to overlooked problems, demonstrate technology's limitations, and provide vivid examples..Source: §3 · p. 12
  • Advancing benchmark-construction practice is itself a purpose, as past successes and failures can teach how to build high-quality benchmarks.ImageNet demonstrated that crowd workers can be used to create a high-quality AI benchmark.Source: §3 · p. 12

4 Desiderata for Benchmarks

The desiderata treat benchmark use as testing, distinguishing the quality of individual items from the coverage of a benchmark collection. These goals are not all compatible, even in principle or practice.

4.1 Quality of individual items

  • Individual items should have correct answers, clear evaluation criteria, and a natural solution path involving commonsense reasoning.Free-form answers are harder to evaluate than true/false, multiple-choice, or fixed-form answers.Source: §4.1 · pp. 12–13
  • Benchmarks should target commonsense rather than expert or encyclopedic knowledge, connect to end-user tasks, and include some rich, complex reasoning.Everyday situations and ordinary discourse can combine time, space, and social relations.Source: §4.1 · pp. 12–13
  • Items should be manageable for people, ecologically valid, and naturally worded, while avoiding undesirable biases and excessive cultural or linguistic specificity.Artificial tasks can still be reasonable, and cultural or linguistic independence is framed as a preference.Source: §4.1 · p. 13

4.1.1 Defects

  • 17 of the first 100 OpenBookQA test problems examined were substantially flawed, excluding misspellings and punctuation errors.OpenBookQA was one of three benchmarks selected as exemplifying commonsense knowledge in HELM.Source: §4.1.1 · p. 14

4.2 Desirable characteristics of a benchmark or collection of benchmarks overall

  • Benchmark collections should cover the range of commonsense domains and reasoning forms; simulated environments should be realistic, rich, and interactive.The authors describe simpler or passive environments as useful but limited testbeds for commonsense physical reasoning.Source: §4.2 · pp. 14, 16
  • A well-defined, replicable construction method can support generalization beyond a benchmark, whereas catch-as-catch-can sets make generalization difficult.The authors note that benchmark sets made by automatic synthesis are a rare exception to this practical pattern.Source: §4.2 · p. 16

5 Four general issues

The section examines four benchmark-design dilemmas: test-set size and quality, adversarial filtering, human performance as a standard, and whether test sets should be secret. It argues that each choice can constrain what benchmark results reliably show.

5.1 Should test sets be large or clean?

  • Because expert vetting is scarce, the author favors small, clean test sets that localize failures and avoid penalizing systems for rejecting flawed choices.Clean examples can also make unresolved weaknesses usable for further study; the author notes that noisy datasets make long-tail examples difficult to distinguish from defects.Source: §5.1 · pp. 16–18
  • 100,000 examples with 10% noise distinguish systems separated by around 0.3%, while 1,000 clean examples require around 3% separation, assuming noise is independent of both systems.The author cautions that this comparison depends on the independence assumption and argues that broadly defined commonsense problems lack a precise underlying distribution.Source: §5.1 · p. 17

5.2 Should beating the current AI systems be an explicit goal?

  • Adversarial filtering can make benchmarks less representative and exaggerate later gains by hiding questions an earlier system solved but a later one misses.The author reports individual-question regressions between GPT-3 versions despite an overall change from 45% completely correct to 50%.Source: §5.2 · p. 18

5.3 What is “human level” performance?

  • Where human judgments disagree, human performance is not an absolute gold standard; AI agreement cannot reasonably be required to exceed human-human agreement.The author also says comparisons require knowledgeable humans working carefully and unhurriedly, rather than rushed crowd-worker judgments.Source: §5.3 · p. 19

5.4 Should test sets be secret or published?

  • A conventional secret-test setup publishes training data and withholds test items, but extra test-only vetting can break their shared-distribution assumption.Out-of-distribution evaluation also weakens the training set’s usefulness as a description of the test set.Source: §5.4 · p. 20
  • Foundation models complicate hidden evaluation because task-specific training is less central, models often cannot be uploaded, and training data are generally proprietary.The author notes that such models may need to be evaluated on their owners’ machines, risking exposure of a hidden test set.Source: §5.4 · p. 20
  • Public examples may have entered LLM training, though a February 2023 ChatGPT probe recalled only one side each of two well-known Winograd schemas.When probed, ChatGPT instead produced two defective original examples, according to the author’s test.Source: §5.4 · p. 21

6 Constructing Benchmarks

The survey identifies six broad construction techniques, often combined, and distinguishes them by their scale, quality controls, and characteristic trade-offs.

6 Constructing Benchmarks

  • Six techniques construct commonsense benchmarks: expert construction, crowdsourcing, gamification, web mining, automated synthesis, and adaptation from existing resources.These techniques can be combined, and further methods may improve a benchmark’s quality, difficulty, or size.Source: §6 · p. 21
  • Expert-built benchmarks are necessarily small, and although consistent quality should be achievable, it is not always attained.Examples include many math word-problem collections, most BIG-bench benchmarks, and the Winograd Schema Challenge.Source: §6 · p. 22
  • Crowdsourcing can lower costs and diversify examples, but speed incentives, weak task design, and occasional bot use can undermine quality.Crowd work may collect or produce examples, questions, and answers, or validate examples; protocols can range from simple to complex.Source: §6 · p. 22
  • Gamification has been attempted in only three commonsense benchmarks, and players may find encyclopedic knowledge easier to use than commonsense reasoning.Designing an entertaining, competitive game in which commonsense knowledge leads to winning is described as extremely difficult.Source: §6 · p. 22
  • Web mining yields large, naturally occurring collections, but they may not represent anything beyond the collected examples themselves.Human or automated work is generally needed to turn harvested web material into AI tasks.Source: §6 · p. 22
  • Automated synthesis can create large datasets with controlled properties and avoid some flaws, but synthetic benchmarks tend to have limited scope.Generated text and images have often been stilted or schematic, and generation processes may introduce artifacts unrelated to the underlying examples.Source: §6 · p. 23
  • Adapting existing collections can preserve, exacerbate, remove, or introduce flaws, while dataset ancestry may be difficult to trace.For large image and mathematical-problem collections, locating the original source of an individual example may be impossible.Source: §6 · p. 23

7 The Existing Benchmarks for Automated Commonsense Reasoning

The survey catalogues benchmarks by modality and describes its inclusion criteria, then examines BIG-bench and elementary mathematical word-problem collections as examples of the landscape.

7 The Existing Benchmarks for Automated Commonsense Reasoning

  • The annotated catalogue organizes benchmarks as text-, image-, video-, or simulated-environment based and records each entry’s description, examples, size, construction method, and links.The website is continually updated, so the paper reflects a time-specific snapshot.Source: §7 · p. 23

7.1 BIG-bench (Beyond the Imitation Game)

  • BIG-bench includes 48 datasets tagged as commonsense; 22 of its 38 purpose-built datasets were manually constructed, and the median dataset has 232 examples.The manually constructed datasets are comparatively small; expert raters reviewed the BIG-bench-created datasets, but they were not tested on naive human subjects.Source: §7.1 · p. 24

7.2 Collections of Elementary Mathematical Word Problems

  • Elementary mathematical word-problem collections are generally designed to test mathematics rather than commonsense, and commonsense-requiring problems are rarely distinguished from others.Their examples rarely involve incorrect answers, meaningless questions, or encyclopedic knowledge, though keyword-recognition artifacts sometimes occur and can be reduced through systematic rewording.Source: §7.2 · pp. 24–25

8 Twelve Illustrative Benchmarks

The paper uses twelve selected benchmarks to illustrate the variety of tasks, construction methods, and quality issues found across commonsense benchmark research.

8 Twelve Illustrative Benchmarks

  • The twelve illustrative benchmarks span text, images, video, and simulated environments and were selected to represent diverse issues, examples, and construction methods.The authors do not present them as the best benchmarks or as unusually flawed ones.Source: §8 · p. 33

8.1 CommonsenseQA 2.O

  • CommonsenseQA 2.0 contains 14,343 yes/no questions, and its reported human accuracy is 90.3% versus 78.1% for the top leaderboard entry.Its gamified construction used crowd-written questions based on topic and relation prompts, an AI adversary, worker validation, and an automated quality filter.Source: §8.1 · pp. 33–34
  • The survey judges the CommonsenseQA 2.0 examples unsatisfying overall, citing ambiguous wording, incorrect explanations, obscure factoids, and potentially incorrect or encyclopedic claims.The authors report that 92% of examples were correctly tagged, but the survey identifies problems in several featured items.Source: §8.1 · pp. 33–34

8.2 SWAG and HellaSWAG

  • HellaSWAG poses 70,000 grounded multiple-choice continuation questions, but several reviewed validation examples have questionable labels, misspellings, and grammatical errors that could confound evaluation.The correct continuation is a real next-event sentence, while three adversarially generated incorrect options are human verified.Source: §8.2 · p. 34

8.3 McTaco

  • McTaco tests five kinds of temporal reasoning by asking whether candidate answers are possible or impossible, with more than one answer sometimes correct.Its questions were generated from MultiRC texts, verified by other workers, expanded with automated methods, then retained only when four workers agreed on true/false labels.Source: §8.3 · pp. 34, 38
  • McTaco’s examples can contain nearly identical answers and unclear context, although those issues are not necessarily defects for a task distinguishing possible from impossible answers.The survey also notes that the displayed examples contain minor typos and grammatical errors present in the original.Source: §8.3 · p. 34, Table 14

8.4 evaluating information essentiality

This BIG-bench dataset tests whether paired statements provide enough information to solve a problem, and the survey characterizes it as generally high quality despite a few issues.

8.4 evaluating information essentiality

  • The BIG-bench information-essentiality dataset contains 68 problems, each followed by statements 1 and 2, with five answer choices about whether statement 1 alone, statement 2 alone, either, both together, or neither is sufficient.Source: §8.4 · p. 38
  • The survey describes the manually constructed dataset as generally high quality, while noting one mislabeled problem and two or three unclear wordings.It was manually constructed by two authors.Source: §8.4 · p. 38
  • Some problems would probably challenge the average high school student, even though solving them is not required for the sufficiency judgment.The survey notes that many questions involve elementary mathematics.Source: §8.4 · p. 38

8.5 misconceptions

These sections describe benchmarks spanning misconception judgments, dialogue-speaker attribution, and richly annotated images, while noting limitations in what some tasks measure and how their data can be used.

8.5 misconceptions

  • The 219-example misconceptions dataset mixes popular-misconception judgments with encyclopedic facts, obscure misquotations, misspellings, and claims that may not be common.False examples came from pre-existing collections, and true examples are their corrections, though some misconceptions lack a correction.Source: §8.5 · pp. 38, 40

8.6 movie dialog same or different and play dialog same or different

  • Movie-dialog attribution uses 50,000 automatically generated problems asking whether two consecutive dialogue sentences were spoken by the same person.The author suggests that clues such as discourse markers could yield strong scores without demonstrating understanding.Source: §8.6 · p. 40
  • 68% average human accuracy versus 63% for the best AI on movie-dialogue speaker attribution; most AIs scored at chance (50%).The dialogues are online, so a model able to retrieve them may exploit that access; Shakespeare-based examples depend heavily on familiarity with the plays and language.Source: §8.6 · p. 40

8.7 Visual Genome

  • Visual Genome contains 108,000 complex annotated images with scene graphs, object relations, regions, and question-answer pairs, but is primarily a computer-vision resource rather than a commonsense benchmark.Its QA pairs can be used directly as a test set; other annotations can support tests more indirectly.Source: §8.7 · pp. 40, 42
  • Visual Genome’s annotation pipeline used a multi-step protocol, including iterative region descriptions and WordNet synset mapping validated by crowd workers.Workers generated descriptions until 50 were collected per image, after which overly similar descriptions were filtered.Source: §8.7 · p. 42

8.8 Adversarial VQA

These benchmarks test image and video reasoning through human-written adversarial questions or synthetic scenes and questions, with task construction ranging from human validation to extensive filtering.

8.8 Adversarial VQA

  • Adversarial VQA combines 81,000 images with 150,000 adversarial human-written questions and 1.5 million human-written questions requiring commonsense beyond image interpretation.Annotators tried to stump an existing visual question-answering system, and other humans screened successful questions for reasonableness and answer correctness.Source: §8.8 · pp. 42, 44

8.9 PTR (ParT Reasoning)

  • PTR pairs 80,000 synthetic images with about 710,000 questions across concept, relational, analogy, arithmetic, and physics categories.Objects are generated and positioned from parametrized models, while questions follow fixed forms based on each image’s symbolic structure.Source: §8.9 · pp. 44, 46

8.10 AGQA (Action Genome Question Answering)

  • AGQA tests temporal and spatial relations in video using questions with 1–3-word answers, generated from augmented scene graphs and filtered for biases and artifacts.Filtering reduced 192 million generated question-answer pairs to a balanced set of 3,900,000.Source: §8.10 · pp. 46–47

8.11 Animal-AI

Animal-AI and OGRE use simulated physical environments to assess learning across varied tasks and reasoning about mechanical interactions.

8.11 Animal-AI

  • Animal-AI’s expert-designed testbed was used in an Olympics challenge with 60 teams given little task information to encourage general-purpose agents.The simulated arena uses simple physics and objects that can be combined into environments modeled on animal experiments.Source: §8.11 · pp. 47–48

8.12 OGRE (Object-based Generalization for Reasoning Environment)

  • OGRE is a two-dimensional physics-puzzle environment where agents choose where to release a green ball to achieve a specified goal as objects move and interact.Its 37 expert-created task templates each define 100 synthetically generated tasks, extending PHYRE to test object generalization.Source: §8.12 · p. 48

9 Examples unsystematically created by the community in reaction to ChatGPT

Community members responded to ChatGPT with individually crafted probes that exposed apparent commonsense failures, but these examples were not a systematic benchmark and some probing was non-replicable.

9 Examples unsystematically created by the community in reaction to ChatGPT

  • Individually crafted social-media tests exposed weaknesses not covered by earlier benchmarks, including cases where AI systems could not count.The paper presents examples of ChatGPT failures in commonsense reasoning from December 2022 and January 2023.Source: §9 · p. 49, Table 18
  • The examples website is not a commonsense benchmark: many items are not commonsensical, are screenshots, and lack correct answers.The site records ChatGPT’s wrong answers but does not give the correct answers.Source: §9 · p. 49
  • Extended chatbot dialogues can expose weaknesses, but user responses to the chatbot make these experiments inherently non-replicable.The author notes that this probing may not be feasible with AI systems other than chatbots.Source: §9 · p. 49

10 Uncovered forms of commonsense reasoning

Commonsense benchmark coverage is uneven: taxonomic reasoning is well covered, while many temporal, spatial, physical, psychological, and social forms remain untested or only partly tested. The section illustrates how richer narratives and experiment descriptions can elicit questions requiring broader commonsense reasoning.

10 Uncovered forms of commonsense reasoning

  • Many forms of temporal, spatial, physical, psychological, and social commonsense reasoning remain untested systematically or are covered only partially and haphazardly.The paper suggests using more complex materials, such as narratives and descriptions of experiments, to elicit richer questions.Source: §10 · pp. 51–52
  • Taxonomic reasoning at common-sense, common-knowledge, and encyclopedic levels is very well covered in existing benchmarks and resources.Source: §10 · p. 51
  • Benchmarks test simple exact computations over given values reasonably well, but poorly cover comparisons, combined comparisons, counting, and combined quantitative reasoning.Source: §10 · p. 51
  • McTaco tests several temporal-reasoning dimensions, but temporal intervals, hypothetical timelines, and combinations of temporal knowledge remain poorly covered.Source: §10 · p. 51
  • ATOMIC encodes extensive but not deep naïve psychological causal knowledge as a resource, while psychological-reasoning benchmarks are absent beyond haphazard coverage.Source: §10 · p. 51
  • Narrative and experiment-description questions can probe reasoning beyond the short texts and simple photographs commonly used in benchmarks.The narrative example asks about event likelihood, knowledge, locations, and causal connections; the chemistry example asks about experiment setup and consequences of changes.Source: §10 · pp. 52–53, Table 20

11 Recommendations

The recommendations call for reliable, foundational, and structurally richer commonsense benchmarks, supported by careful quality checks and clear examples. They also address puzzle familiarity, human-test transfer, language coverage, and protection against benchmark training-data contamination.

11 Recommendations

  • New benchmarks should prioritize high quality and foundational temporal, spatial, physical, psychological, and social reasoning over common, encyclopedic, or expert knowledge.Source: §11 · p. 54
  • Benchmarks should extend existing coverage to reasoning with greater structural richness.Source: §11 · p. 54
  • For reasonably small test sets, creators should try to eliminate flawed examples; for very large sets, they should inspect a random sample and report flaw frequency and types.Source: §11 · p. 54
  • Benchmark papers should include specific examples—at least ten for text benchmarks and four for image benchmarks—and reviewers should flag significant unacknowledged flaws.Source: §11 · p. 54
  • Famous puzzles can invite puzzle-mode solving or memorized answers; GPT-3's apparent success on familiar false-belief problems failed when details changed.Source: §11 · p. 54
  • Human-designed tests require caution as AI benchmarks because passing them does not establish understanding comparable to a human's.Source: §11 · p. 55
  • Testing commonsense tasks in languages beyond English can reveal whether success is language-bound rather than evidence of underlying understanding.Source: §11 · p. 55
  • AI organizations should let benchmark creators mark benchmarks and interface queries to prevent their use as training data.Source: §11 · p. 55

Appendix A: Flawed examples in OpenBookQA

The appendix examines 17 examples from the first 100 OpenBookAI test items and flags errors in wording, answer keys, ambiguity, and knowledge demands. The cases illustrate why benchmark examples need close review.

Appendix A: Flawed examples in OpenBookQA

  • The appendix samples 17 of the first 100 OpenBookAI test examples, excluding additional spelling, punctuation, and phrasing problems.It also flags examples relying on obscure biological species and terms.Source: Appendix A · p. 67
  • Several items rely on expert or encyclopedic knowledge, including Punnett-square boxes and the habitat of Mola mola.Source: Appendix A · pp. 67–68
  • Some examples have questionable or inaccurate premises, including the friction-speed relationship and claims about electric-car motors or plant energy needs.Source: Appendix A · pp. 67–68
  • Other items have ambiguous or flawed answer choices: weather can affect several listed outcomes, and squirrels do not all stockpile pecans.Source: Appendix A · pp. 67–69

Appendix B: Commonsense questions about a chemistry experiment, from

The appendix presents questions about a chemistry experiment to illustrate commonsense reasoning over a detailed description and diagram. Its examples ask readers to reason about apparatus changes, measurements, and pressure, while the text cautions that the experiment description omits necessary safety precautions.

Appendix B: Commonsense questions about a chemistry experiment

  • The experiment description traces oxygen collection and explains that equal water levels inside and outside the beaker make its pressure atmospheric.Source: Appendix B · p. 69
  • The questions test consequences of changing the apparatus, including tube placement, holes, material, and a stopper.Source: Appendix B · p. 69
  • The appendix warns not to perform the experiment from this description because potassium chlorate is explosive and omitted safety precautions are necessary.Source: Appendix B · p. 69
  • The illustration assumes oxygen generation, whereas the description assumes measurement of oxygen produced from a measured amount of potassium chlorate.Source: Figure 10 · p. 70
  • Other questions probe accurate measurements, graduated-beaker use, setup procedure, pressure adjustment, and whether the beaker can hold the produced oxygen.Source: Appendix B · p. 71
Loading 2302.04752v2…