Source-linked AI summary
DREAM: Improving Situational QA by First Elaborating the Situation
Yuling Gu, Bhavana Dalvi Mishra, Peter Clark
TL;DR
This paper asks whether language models can answer socially situated questions more accurately when given pertinent details about the situation. It trains DREAM to generate scene elaborations and supplies them as QA context; these elaborations improve downstream accuracy beyond further fine-tuning alone.
Problem
The paper investigates whether language models can answer situated questions more accurately when given pertinent details about the situation, since how they answer such questions is not known.
Source: §1 · p. 1Method
Results
DREAM’s scene elaborations improve downstream QA accuracy, including beyond what further fine-tuning on DREAM’s training data achieves.
Source: §9 · p. 9Takeaways & Limitations
The results suggest focused scene elaborations using social constructs such as motivations may improve reasoning and inject scenario-based knowledge into QA.
Source: §1 · p. 2Some generated scene-elaboration components do not accurately reflect the situation, including implausible motivations or emotions.
Source: §7.1 · p. 8
Abstract
from arXiv · showhide
When people answer questions about a specific situation, e.g., "I cheated on my mid-term exam last week. Was that wrong?", cognitive science suggests that they form a mental picture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the question situation, elaborating the "scene". To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elaborations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context improves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further finetuning the QA system on DREAM's training data. These results suggest that adding focused elaborations about a situation can improve a system's reasoning about it, and may serve as an effective way of injecting new scenario based knowledge into QA models. Finally, our approach is dataset-neutral; we observe improved QA performance across different models, with even bigger gains on models with fewer parameters. We make our dataset and model publicly available at https://github.com/allenai/dream.
1 Introduction
The paper tests whether pertinent scene details can help language models answer socially situated questions, without claiming that those details reflect models’ internal reasoning. It trains DREAM to generate scene elaborations and finds that adding them improves downstream QA beyond further fine-tuning alone.
1 Introduction
- DREAM’s scene elaborations are more accurate, useful, and consistent than the elaborations elicited from zero-shot Macaw.Macaw is poor at answering elaboration queries despite good performance on the QA end-task itself.Source: §1 · pp. 1–2
- Drawing on the idea that mental models support understanding and question-answering, the paper tests whether added situational details improve answers without making claims about LM reasoning internally.The added details elaborate the scene rather than pre-generating a proof or answer-centered explanation.Source: §1 · pp. 1–2
- DREAM is trained to answer scene-elaboration questions, and its outputs are supplied as additional context to a downstream QA model.The paper describes this as a dataset-neutral approach that improves QA across different tasks and models, including models of different sizes.Source: §1 · pp. 1–2
- In the mushroom example, Macaw changes from an incorrect to a correct answer when DREAM’s safe-food and helpfulness details are added as context.The example illustrates how scene elaborations can alter a downstream answer.Source: §1 · p. 2
- DREAM-generated scene elaborations improve downstream QA accuracy more than further fine-tuning the QA system on DREAM’s training data.The approach leaves the downstream QA system unchanged and allows different QA systems to be substituted.Source: §1 · p. 2
2 Related Work
Prior work connects situated problem-solving with mental models and shows that generated or retrieved contexts can assist QA. This paper’s focus is elaborating the general social scene, rather than constructing an answer-centered proof or explanation.
2 Related Work
- Cognitive-science accounts describe mental models as coherent internal representations and suggest that people elaborate a situation rather than construct only a deductive proof.Other AI work has explored mental models, questions people ask about text or images, and scene elaboration for QA.Source: §2 · p. 2
- Prior QA methods add retrieved passages, explanations, self-talk, intermediate computations, chains of thought, generated facts, or causal graphs as context.These approaches motivate the broader use of generated or retrieved contexts to improve QA.Source: §2 · p. 2
- DREAM’s distinctive focus is articulating the general scene with social constructs such as motivations, rather than pre-generating an answer-centered proof or explanation.Its scene can prefix the question without requiring context to depend on answer choices or downstream-task tuning.Source: §2 · pp. 2–3
- Table 1 juxtaposes Macaw-11B-probed and DREAM-generated scene elaborations, with colors marking crowdworker judgments of component accuracy.The caption identifies green, orange, and red as accurate, partially accurate, and inaccurate judgments, respectively.Source: Table 1 · p. 3
3 Scene Elaborations (SE)
The paper studies whether question-agnostic details about a social situation can improve situated QA. It represents those details across four social dimensions and probes language models with questions aligned to them.
3 Scene Elaborations (SE)
- The task asks whether a question-agnostic elaboration of a described social situation can improve question-answering about it.The input is a situation S and question Q; the model answers A, and the elaboration adds details about S.Source: §3 · p. 3
3.1 Representing Scene Elaboration
- A scene elaboration is a four-part representation of motivation, emotion, social norms, and likely consequences.Motivation concerns characters before the situation, emotion follows it, the rule of thumb concerns social acceptability, and consequences describe likely outcomes.Source: §3.1 · p. 3
3.2 Probing for Scene Elaboration
- The probing questions ask about an entity’s motivation and emotion, a relevant rule of thumb, and what is likely to happen next.For motivation and emotion, the method identifies entities in the situation and asks about each one when multiple entities are present.Source: §3.2 · p. 4
4 Our Model: DREAM
DREAM is trained to map a situation to a four-part scene elaboration using examples assembled from existing commonsense resources. Its training data pairs situations with targets for motivation, emotion, social norms, and consequences.
4 Our Model: DREAM
- DREAM maps a situation S to a scene elaboration SE, and the model is fine-tuned from a T5-11B Macaw checkpoint.Training interleaves examples for the four components and runs for 50K steps, selecting a checkpoint by validation score.Source: §4 · p. 4, §4.2 · p. 5
4.1 Training Data
- The Scene Elaborations Dataset combines Story Commonsense, Social Chemistry, and Moral Stories, with examples converted into question-answering format.Training examples pair a situation with a component-specific target, and the four generated results are concatenated with component identifiers.Source: §4.1 · pp. 4–5
- Story Commonsense supplies situation-to-emotion and situation-to-motivation mappings, including cases annotated with no significant emotion or motivation.The mappings use a story sentence as the situation and a character’s emotional response or prior motivation as the target.Source: §4.1 · p. 4
- Social Chemistry maps situations to relevant rules of thumb, which can be more situation-specific than a single general moral norm.The paper contrasts its situation-linked rules with Moral Stories’ general norm field.Source: §4.1 · p. 4
- Moral Stories supplies consequences for paired moral and immoral actions, yielding contrastive good- and bad-outcome targets.Each story produces examples using its moral and immoral consequence fields.Source: §4.1 · pp. 4–5
5 Experiments
The experiments examine Macaw’s scene elaborations, compare them with DREAM’s, and test whether DREAM’s elaborations help downstream QA. They use situational QA datasets and assess elaboration quality as well as QA accuracy.
5 Experiments
- The experiments ask whether Macaw produces accurate and consistent elaborations, whether DREAM improves them, and whether DREAM’s elaborations improve QA.Macaw is the representative language model used in the experiments.Source: §5 · p. 5
5.1 Representative LM: Macaw
- The representative system is the off-the-shelf, T5-based Macaw question-answering model, evaluated in its 11B version.Macaw is built on UnifiedQA, which is built upon T5.Source: §5.1 · p. 5
5.2 Test Datasets
- The evaluation uses three situational QA datasets, generally using test partitions; CODAH uses all splits because it is smaller.For ETHICS-CS, the experiments report both test and adversarially selected test-hard results.Source: §5.2 · p. 5, Table 3 · p. 6
5.3 Metrics
- Crowdworkers rate elaboration components for accuracy and usefulness, rate whole elaborations for consistency, and QA accuracy is compared with and without elaborations.Accuracy and usefulness are averaged across the four components; consistency concerns agreement among sentences, independently of answer support.Source: §5.3 · p. 6
6 Results
The results show that DREAM produces better scene elaborations than Macaw and that adding them as context improves downstream QA, while DREAM-data fine-tuning alone does not. The elaborations also help across model sizes, and each component contributes to QA performance.
6.1 Q1: How good are Macaw’s Scene Elaborations of a situation S?
- Macaw’s elaborations average about 48% accurate, 49% consistent, and 27% useful for the QA end task.Crowdworkers rated a random sample of 100 questions from each dataset.Source: §6.1 · p. 6
6.2 Q2: Does DREAM generate improved Scene Elaborations?
- DREAM’s elaborations score 16–26% higher in accuracy, 12–16% higher in usefulness, and 15–29% higher in consistency than Macaw’s across three situational QA tasks.The authors note that DREAM’s elaborations are not perfect.Source: §6.2 · p. 6
6.3 Q3: Can the scene elaborations produced by DREAM help improve QA?
- Adding DREAM-generated scene elaborations as context significantly improves Macaw’s zero-shot QA, whereas further fine-tuning on DREAM’s data does not, even with task-specific few-shot training.The comparison covers the targeted test sets for ETHICS, Social IQA, and CODAH.Source: §6.3 · p. 7, Table 5 · p. 7
6.4 Ablation of SE Components
- Each scene-elaboration component independently improves Macaw’s QA scores, and using all components outperforms any single component.The component comparison uses the Social IQA dataset.Source: §6.4 · p. 7, Table 6 · p. 7
6.5 DREAM SE improves QA performance across different models
- DREAM’s scene elaborations similarly improve answer accuracy for Macaw-3B, Macaw-large, and UnifiedQA-large across all three QA tasks, with higher absolute improvements for models with fewer parameters.The authors say this illustrates the portability of DREAM’s scene elaborations and an advantage of leaving end-task QA models unchanged.Source: §6.5 · p. 7
7 Analysis and Discussion
A crowdworker review of 400 elaborations found that most contained some true or useful content, though consistency and usefulness varied. The analysis also identifies generation errors and cases where QA predictions improve, worsen, or remain unchanged after adding elaborations.
7 Analysis and Discussion
- In a random sample of 400 questions rated by crowdworkers, 98.75% of DREAM scene elaborations had something true; of those, 78.75% had something useful to justify the correct answer.Turker votes indicated 94% were somewhat consistent (at least 2 of 4 components consistent with each other), and 55% were largely consistent (at least 3 of 4).Source: §7 · pp. 7–8
7.1 Failures while generating SEs:
- Generation failures include inaccurate situation reflections (≈12%), true but unhelpful content (≈25%), and inconsistency (6% with half or more inconsistent statements; 45% with at least one).Examples include a motivation that mischaracterizes a harmful act and a true motivation that does not identify the correct answer.Source: §7.1 · p. 8
7.2 Influence of SE on QA:
- After seeing scene elaboration (SE), Macaw changed its prediction from wrong to correct in 9.25% of cases, and from correct to wrong in 7.75% of cases.Source: §7.2 · p. 8
- Macaw retained a wrong answer on 14.75% of the sample; for the 59 such cases, elaborations averaged 64% accuracy, 66% consistency, and 32% usefulness.The authors infer that QA models remain imperfect at exploiting relevant input context.Source: §7.2 · p. 8
8 Future Directions
The authors outline directions for making elaborations more adaptable and useful to QA, including varied components, task-specific filtering, joint answer-and-elaboration models, and retrieval-based systems.
8 Future Directions
- A future DREAM could generate a wider variety of scene-elaboration components and dynamically select those most salient to each situation.The current model uses a fixed set of four components.Source: §8 · p. 9
- DREAM could be further fine-tuned on task-specific examples whose elaborations crowdworkers mark as both true and useful for QA.The proposed annotation process is described as easier and cheaper than creating elaborations from scratch.Source: §8 · p. 9
- A joint situational-QA model could produce answers and elaborations together, potentially making elaborations question-salient and answers consistent with them.The authors also propose using elaborations as answer justifications.Source: §8 · p. 9
- On ETHICS-CS, comparing BERT encodings of situation plus elaboration improved kNN QA accuracy by 17% over encoding the situation alone.The authors identify interpretable nearest-neighbor models as a possible direction for using prior elaborations on new problems.Source: §8 · p. 9
9 Conclusion
The conclusion reports that DREAM-generated scene elaborations improve downstream QA accuracy beyond fine-tuning on DREAM’s data alone. It presents the approach as question-agnostic and portable because it leaves end-task QA models unchanged.
9 Conclusion
- DREAM-generated scene elaborations improve downstream QA accuracy, including beyond what further fine-tuning on DREAM’s training data achieves.The conclusion describes focused elaborations as a possible way to add scenario-based knowledge to QA models.Source: §9 · p. 9
- The approach improves existing QA systems in a question-agnostic way while leaving the end-task models unchanged.The authors say this helps prevent interference across task capabilities and catastrophic forgetting.Source: §9 · p. 9
- The authors make the DREAM dataset and model publicly available to facilitate future research.Source: §9 · p. 9
B Building a Dynamic Memory of Scene Elaborations for Better Reasoning
The paper tests DREAM scene elaborations as added input to KNN, reporting improved ETHICS-CS accuracy and semantically closer retrieved situations.
B Building a Dynamic Memory of Scene Elaborations for Better Reasoning
- The KNN setup encodes situations with optional scene elaborations using BERT, retrieves k=5 neighbors by Euclidean distance, and majority-votes their gold answers.It uses the ETHICS-CS training partition to populate the space and evaluates on the test partition.Source: B Building a Dynamic Memory of Scene… · p. 12
- KNN answer accuracy improves by 17% when DREAM-generated scene elaborations are included in the question encoding.The evaluation uses the ETHICS-CS test set, excluding long questions with AITA context.Source: B Building a Dynamic Memory of Scene… · pp. 12–13, Table 8 · p. 13
- With scene elaborations, retrieved training situations can be semantically closer to the query; a handpicked example contrasts an educational experience with merely visiting a farm.Table 9 describes the query and top-five retrieved situations, marking retrieved situations that match the query’s gold label in green and others in red.Source: B Building a Dynamic Memory of Scene… · p. 13, Table 9 · p. 13
- These preliminary findings suggest opportunities for interpretable nearest-neighbor models to reuse scene elaborations in new problems.The paper describes scene elaborations as old experiences that such models could use and adapt.Source: B Building a Dynamic Memory of Scene… · p. 13
C On the use of our dataset and model
The authors release the scene-elaboration dataset and DREAM model for research purposes and note a risk associated with large-scale language models.
C On the use of our dataset and model
- DREAM may produce biased or offensive statements, and its dataset and model are released for research purposes only.Source: C On the use of our dataset and model · p. 13