PaperlayerView exact revision on arXiv

Source-linked AI summary

SocialIQA: Commonsense Reasoning about Social Interactions

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi

arXiv:1904.09728v3cs.CL

TL;DR

Social reasoning remains difficult for language models, in part because large-scale resources for training and evaluation are lacking and written-text reporting bias limits learned commonsense. The paper introduces SOCIAL IQA, a crowdsourced benchmark with question-switching to reduce answer-style artifacts. Models lag human performance on the benchmark, while sequential finetuning on it produces state-of-the-art results on multiple commonsense challenges.

  • Problem

    Large-scale resources for training and evaluating social and emotional reasoning are lacking, and reporting bias in written-text corpora limits the commonsense knowledge language models can learn.

    Source: §1 · pp. 1–2

  • Method

    The authors create SOCIAL IQA, a 38k-question crowdsourced benchmark, and use handwritten negatives plus question-switched answers to reduce stylistic artifacts.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    Models remain below human performance on SOCIAL IQA, while sequential finetuning on it achieves state-of-the-art results on COPA and Winograd datasets.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    SOCIAL IQA serves both as a benchmark for social commonsense and as a resource for transfer learning to other commonsense challenges.

    Source: §1 · p. 2

    The reported results indicate that current models still struggle with social situations compared with humans, and the authors suggest they may benefit from more complex entity-state reasoning or more explicit commonsense.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

We introduce Social IQa, the first largescale benchmark for commonsense reasoning about social situations. Social IQa contains 38,000 multiple choice questions for probing emotional and social intelligence in a variety of everyday situations (e.g., Q: "Jordan wanted to tell Tracy a secret, so Jordan leaned towards Tracy. Why did Jordan do this?" A: "Make sure no one else could hear"). Through crowdsourcing, we collect commonsense questions along with correct and incorrect answers about social interactions, using a new framework that mitigates stylistic artifacts in incorrect answers by asking workers to provide the right answer to a different but related question. Empirical results show that our benchmark is challenging for existing question-answering models based on pretrained language models, compared to human performance (>20% gap). Notably, we further establish Social IQa as a resource for transfer learning of commonsense knowledge, achieving state-of-the-art performance on multiple commonsense reasoning tasks (Winograd Schemas, COPA).

1 Introduction

Social reasoning about others’ mental states and likely actions remains challenging for machine-learning models, partly because large-scale training and evaluation resources are lacking. The paper introduces SOCIAL IQA as a benchmark and resource for transfer learning.

1 Introduction

  • SOCIAL IQA is a large-scale multiple-choice benchmark for social and emotional intelligence, with 38k questions about the pragmatic implications of everyday social events.The authors describe it as the first large-scale resource of its kind.Source: §1 · p. 2
  • Question-switching combines handwritten negative answers with answers collected by asking workers a different but related question to minimize crowdsourcing stylistic artifacts.Source: §1 · p. 2
  • BERT-large reaches 64.5% on SOCIAL IQA, below human performance, leaving social reasoning challenging for existing models.Source: §1 · p. 2
  • Sequential finetuning on SOCIAL IQA yields new state-of-the-art results on COPA and the original and extended Winograd datasets.The authors frame this transfer as implicitly endowing models with social commonsense knowledge.Source: §1 · p. 2

2 Task description

SOCIAL IQA tests whether models can infer social causes and effects from a context and question. Its task design draws on ATOMIC to cover motivations, reactions, and likely actions.

2 Task description

  • Models choose the correct answer from three options for a question explicitly tied to an observed context.Source: §2 · p. 2
  • Questions probe motivations, emotional reactions, and likely preceding or following actions, linking the task to reasoning about others’ beliefs, motivations, and needs.Source: §2 · p. 2

ATOMIC

  • ATOMIC seeds contexts and question types with event-centered commonsense knowledge, including causes, effects on the agent, and effects on other participants.Its triples pair an event phrase with one of nine inference dimensions and an inference object.Source: ATOMIC · p. 3
  • Contexts instantiate ATOMIC event phrases, yielding questions about diverse motivations, reactions, and likely preceding or following actions.Source: ATOMIC · p. 3

3 Dataset creation

SOCIAL IQA’s dataset is built through crowdsourcing contexts, questions, and answer choices seeded by ATOMIC events and inference dimensions. Its negative-answer design combines handwritten errors with answers to related but different questions, and the final tuples are validated and filtered.

3.1 Event Rewriting

  • Workers rewrite ATOMIC event prompts into grammatical sentences with names and filled-in placeholders to cover varied social situations.For example, “PersonX spills all over the floor” can become “Alex spilled food all over the floor.”Source: §3.1 · p. 3

3.2 Context, Question, & Answer Creation

  • Workers turn event sentences into detailed contexts, refine questions generated from ATOMIC’s nine inference dimensions, and provide two potentially correct answers.An example question asks what Alex will want to do next after spilling food and making a mess.Source: §3.2 · p. 3

3.3 Negative Answers

  • The pipeline collects four incorrect options and filters out two, using distinct approaches intended to reduce annotation artifacts and make model reliance on data biases harder.The approaches are designed to produce options that are adversarial for models but easy for humans.Source: §3.3 · p. 3
  • Handwritten Incorrect Answers (HIA) are context-relevant but subtly wrong responses written to resemble correct answers in topic, length, and style.Two are collected alongside the original context, question, and correct answers.Source: §3.3 · p. 3
  • Question-Switching Answers (QSA) use correct answers to a different, related question about the same context as adversarial incorrect options.QSA and correct answers differ less stylistically than handwritten incorrect answers (HIA), with |d|≤.1; all three answer types have comparable arousal.Source: §3.3 · p. 4, Figure 4 · p. 4

3.4 QA Tuple Creation

  • The final three-way tuples combine a randomly selected correct answer with incorrect answers least entailed by it, then undergo human validation and lightweight adversarial filtering on dev and test.Final examples require majority agreement on the correct answer; cases without a majority are discarded.Source: §3.4 · p. 4

3.5 Data Statistics

  • The dataset has around 33k training, 2k dev, and 2k test tuples, with SOCIAL IQA contexts assigned to the same partition as their source ATOMIC events.Questions about what someone will feel afterwards or likely do next are more common; effects of situations on people are less frequent.Source: §3.5 · pp. 4–5

4 Methods

The authors establish baselines by fine-tuning OpenAI-GPT and BERT (Devlin et al., 2019). They classify context-question-answer choices using model-specific input formats, an MLP score, and softmax normalization.

4 Methods

  • OpenAI-GPT and BERT are fine-tuned as SOCIAL IQA baselines, with the language-model representations tuned and classifier-specific parameters learned from scratch.OpenAI-GPT is unidirectional and trained on BookCorpus; BERT is bidirectional and trained on BookCorpus and English Wikipedia.Source: §4 · p. 5
  • Each choice is formed by concatenating context, question, and answer with model-specific separator tokens before classification.BERT places its classifier token before the context; its sequence uses [CLS], [UNUSED], and [SEP].Source: §4 · p. 5
  • Agent and theme ATOMIC dimensions are grouped together, so xReact and oReact are treated as the reactions question type.Source: §4 · p. 5
  • The classifier maps the classifier-token representation through an MLP, normalizes scores across choices with softmax, and predicts the highest-probability triple.The reported ablation removes contexts and questions and states that both are necessary for BERT-large’s performance; the table caption also says human evaluation uses 900 randomly sampled examples.Source: §4 · p. 5, Table 2 · p. 5

5 Experiments

The experiments evaluate SOCIAL IQA with pretrained language models and show that performance depends on substantial training data, while errors reveal challenges in tracking events and participants. The models remain far behind human performance on social reasoning.

5.1 Experimental Set-up

  • The experiments finetune models on 33k training instances and select hyperparameters using dev performance before reporting test results.The search varies learning rate, batch size, and training epochs.Source: §5.1 · p. 5

5.2 Results

  • BERT-large outperforms other tested models, but SOCIAL IQA remains challenging for computational models compared with human performance.The authors also report that reasoning over both the context and question is necessary.Source: §5.2 · p. 6
  • Performance begins to converge only after around 20k training examples, and exceeding 80% would require nearly 1 million examples.The learning curve improves over the 33% random baseline with only a few hundred examples; Figure 5 marks human performance at 86.9%.Source: §5.2 · p. 6, Figure 5 · p. 7
  • Questions about motivations and actions needed before a context are less challenging than questions about effects, stative descriptions, and what people will want to do next.Figure 6 summarizes BERT-large dev accuracy across question types.Source: §5.2 · p. 6
  • Errors suggest models may rely on lexical associations, choosing answers with incorrect timing or answers about the wrong participants.Examples include selecting something that would happen before a context or attributing a likely feeling to someone other than the person asked about.Source: §5.2 · pp. 6–7

6 SOCIAL IQA for Transfer Learning

The paper tests whether SOCIAL IQA can transfer social commonsense to other tasks by sequentially finetuning models before task-specific training. This approach improves performance and stability across COPA, Winograd Schema Challenge, and DPR.

6 SOCIAL IQA for Transfer Learning

  • SOCIAL IQA is presented as a resource for improving downstream commonsense performance on the Winograd Schema Challenge and COPA.The paper reports state-of-the-art performance on both through sequential finetuning before the downstream task.Source: §6 · p. 7
  • COPA is a 1,000-question, two-choice task about causes and effects of a premise; Winograd Schema Challenge tests pronoun resolution using 273 short sentences.The paper also evaluates on DPR, a collection of 943 Winograd-style sentence pairs.Source: §6 · p. 7
  • For WSC evaluation, the authors use DPR as training data because WSC has only 273 test instances and no training set.DPR contains Winograd-style sentence pairs.Source: §6 · p. 7

6.1 Sequential Finetuning

  • Sequential finetuning on SOCIAL IQA yields state-of-the-art results on COPA, Winograd Schema Challenge, and DPR, with 2.6–5.5% maximum performance increases over the BERT-only baseline.The reported results also show generally lower standard deviations; Table 4 covers all three tasks.Source: §6.1 · p. 8, Table 4 · p. 8
  • COPA dev accuracy improves as the model is finetuned on more SOCIAL IQA instances, while same-sized SWAG yields lower COPA performance at 76.2%.The authors attribute gains to both dataset scale and the social and emotional nature of SOCIAL IQA's knowledge.Source: §6.1 · p. 8, Figure 7 · p. 8

7 Related Work

Related work includes expert-curated commonsense benchmarks, crowdsourced question collections, knowledge repositories, and methods for reducing annotation artifacts. SOCIAL IQA differs in its focus on social inferences and its use as a natural-language QA resource.

Commonsense Benchmarks

  • WSC and COPA are expert-curated commonsense QA benchmarks, but their small sizes—150 and 1,000 examples, respectively—challenge modern models that need many training instances.Source: §7 · p. 8
  • CommonsenseQA has 12k multiple-choice questions that mostly probe factual and physical commonsense, whereas SOCIAL IQA separates contexts from questions and targets social inferences.Source: §7 · p. 8

Commonsense Knowledge Bases

  • Unlike taxonomic knowledge bases, SOCIAL IQA is formatted as natural-language QA, yet the authors demonstrate its use as a resource for external tasks.Source: §7 · p. 9

Constrained or Adversarial Data Collection

  • Prior artifact-reduction approaches restrict incorrect responses or use adversarial filtering; SOCIAL IQA uses a multistage pipeline including adversarial question-switching.Source: §7 · p. 9

8 Conclusion

The conclusion presents SOCIAL IQA as a large-scale benchmark for social commonsense and a transfer-learning resource. Pretrained language models remain well below human performance, while transfer improves results on COPA and Winograd Schema Challenge.

8 Conclusion

  • SOCIAL IQA is a 38k-question multiple-choice benchmark covering inferences about people’s actions in situational contexts.Source: §8 · p. 9
  • Pretrained language models achieve accuracies up to 65%, compared with human performance close to 90%, indicating that social inferences remain challenging for AI systems.Source: §8 · p. 9
  • Transfer learning from SOCIAL IQA yields significant improvements and state-of-the-art performance on COPA and Winograd Schema Challenge.Source: §8 · p. 9

Related papers

Loading 1904.09728v3…