Source-linked AI summary
Large Language Models Can Be Easily Distracted by Irrelevant Context
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou
TL;DR
Existing evaluations largely assume that all context is relevant, leaving language-model distractibility insufficiently tested despite the prevalence of mixed-relevance information in real-world problems. The paper introduces GSM-IC to evaluate prompting techniques under irrelevant context and finds substantial degradation, alongside improvements from self-consistency, irrelevant-context exemplars, and ignore-context instructions.
Problem
Most language-model benchmarks provide only relevant problem information, leaving distractibility from irrelevant context insufficiently evaluated.
Method
The paper introduces GSM-IC, an arithmetic reasoning dataset derived from GSM8K by adding solution-irrelevant sentences, and evaluates multiple prompting techniques and mitigation strategies.
Results
All investigated prompting techniques are sensitive to irrelevant information, while self-consistency, irrelevant-context exemplars, and ignore-context instructions improve GSM-IC performance.
Takeaways & Limitations
Robust reasoning evaluation should include sensitivity to irrelevant context alongside performance on challenging problems.
Takeaways & Limitations
The study leaves distractibility on other tasks and different language models for future work.
Abstract
from arXiv · showhide
Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information.
1. Introduction
The paper studies how irrelevant context affects language-model prompting by introducing GSM-IC, an arithmetic benchmark that adds solution-irrelevant sentences to grade-school math problems. Across prompting techniques, irrelevant information substantially reduces performance, while self-consistency and prompt-based demonstrations or instructions offer mitigation strategies.
- GSM-IC measures language-model distractibility by adding one solution-irrelevant sentence to an otherwise unchanged grade-school math problem.The dataset is derived from GSM8K and uses irrelevant context that does not affect the problem’s solution.
- The study evaluates Codex and GPT-3.5 with chain-of-thought, zero-shot chain-of-thought, least-to-most, and program-based prompting.
- Adding irrelevant information to prompt exemplars or instructing models to ignore irrelevant context consistently improves performance.
- All investigated prompting techniques are sensitive to irrelevant information, and their GSM-IC performance greatly decreases relative to clean GSM8K problems.
- 18% or fewer of baseline-solvable problems can be consistently solved across all irrelevant-information types with greedy decoding.
- Self-consistency improves every prompting technique, reaching 99.7% correct-answer recall with 20 samples per problem.
- The evaluation indicates that irrelevant-context sensitivity reveals weaknesses in context understanding and identifying relevant information for real-world reasoning tasks.
2. Related Work
Related work has documented failures from distracting or perturbed input across question answering, factual reasoning, arithmetic, logical reasoning, and prompting settings. Prior mitigation often relies on training or finetuning, whereas this paper examines prompting-based robustness to irrelevant context in arithmetic reasoning.
- Prior natural-language benchmarks use model-agnostic perturbations and adversarial examples to test sensitivity to altered inputs.
- Question-answering and pretrained language models have been shown to be affected by distracting or irrelevant context in adversarial, factual-reasoning, and related tasks.
- Knowledge-aware finetuning on counterfactual and irrelevant context strengthens robustness, while this work tests mitigation without training or finetuning.
- Logical-reasoning benchmarks containing irrelevant content often motivate specially trained models, and prompting alone has generally underperformed finetuned models.
- Related prompting studies examine sensitivity to wrong answers, misleading instructions, irrelevant instructions, and incorrect reasoning steps in exemplars.
3. The GSM-IC Dataset
GSM-IC is built from selected GSM8K problems by adding controlled irrelevant sentences, then evaluates distractibility with micro, macro, and normalized accuracy.
- Dataset Creation: The benchmark selects 100 GSM8K development problems solvable by at least one investigated prompting technique, forming an easy base set.Each problem requires two to seven reasoning steps, and 60 require two steps.
- Dataset Creation: Each derived example adds one irrelevant sentence generated by varying topic, role-name overlap, and number range.Templates produce in-topic or off-topic sentences, optionally overlapping original role names and using similar- or different-magnitude numbers.
- Dataset Creation: 58,052 examples comprise GSM-IC after manual verification that generated sentences are acceptable and do not change the base problem’s standard solution.The three generation factors are treated as orthogonal to create combinations of derived examples.
- Evaluation Metrics: Micro accuracy averages method accuracy over individual test problems, whereas macro accuracy averages over base-problem classes requiring every derived variant to be solved correctly.Macro accuracy therefore measures consistent solving across irrelevant-sentence variations.
- Evaluation Metrics: Normalized accuracy measures distractor impact relative to a method’s accuracy on the base problems.For either micro or macro accuracy aM, normalization uses the corresponding base-problem accuracy nM.
4. Investigated Solutions
The paper evaluates several prompting strategies under irrelevant context and tests mitigation through self-consistency, exemplar design, and explicit instructions to ignore distractors.
- 4.1. Base Techniques: The investigated techniques include chain-of-thought, zero-shot chain-of-thought, least-to-most, and program prompting.They differ in whether they use reasoning exemplars, decompose problems, or represent arithmetic reasoning as executable Python.
- 4.1. Base Techniques: Self-consistency samples multiple solutions and selects the majority result, allowing combination with the other prompting techniques.It marginalizes over intermediate reasoning paths that share a final answer.
- 4.2. Prompt Design: Few-shot prompts place exemplar problems and solutions before the target problem, using one simple exemplar in the main experiments.The exemplar can contain either the original problem or a version with irrelevant context.
- 4.2. Prompt Design: The irrelevant-context exemplar adds a sentence unrelated to the requested answer, as illustrated by inserting a peach fact into an apple-counting problem.Prompt building blocks substitute the original or modified problem into the investigated formats.
- 4.3. Instructed Prompting: Instructed prompting prepends an instruction to solve grade-school math while ignoring irrelevant information in the questions.This tests whether an explicit natural-language directive helps models filter distractors.
5. Experiments
Experiments on GSM-IC-4K show that irrelevant context substantially disrupts prompting-based arithmetic reasoning, while self-consistency, distractor-containing exemplars, and targeted instructions improve robustness. LTM is generally the strongest baseline, especially as reasoning difficulty increases, but important sensitivity remains.
- Experimental Setup: The evaluation uniformly samples 4,000 GSM-IC examples and uses greedy decoding without self-consistency or 20 responses at temperature τ = 0.7 with self-consistency.The experiments primarily use code-davinci-002 and also evaluate text-davinci-003.
- Main Results: Irrelevant context causes significant performance drops for both models and all prompting techniques, with fewer than 30% of base problems consistently solved after distractors are added.A single irrelevant sentence can trigger numerical misuse or errors arising from its presence in the reasoning steps.
- Main Results: LTM is generally most robust to irrelevant context and has about double code-davinci-002 macro accuracy of COT, although text-davinci-003 shows lower LTM macro accuracy than COT.For text-davinci-003, role-overlap distractors reduce macro accuracy to 0 on problems requiring more than two reasoning steps.
- Robustness Interventions: Few-shot prompts using exemplars with distractors consistently outperform original exemplars on GSM-IC without reducing accuracy on the original GSM8K dataset.This prompt design therefore improves robustness on distractor-containing problems while preserving clean-task performance.
- Main Results: Self-consistency improves overall micro accuracy by more than 11 percentage points, including a 35.5-point gain for 0-COT, but its best macro accuracy is only 45%.The correct answer appears among 20 samples for 99.7% of COT and LTM problems and 96.5% of 0-COT problems.
- Reasoning Difficulty: LTM performance remains fairly consistent across reasoning difficulty, whereas COT and PROGRAM exhibit significant drops on problems requiring four or more reasoning steps.LTM is less sensitive to irrelevant context on complicated problems requiring more steps.
- Robustness Interventions: Adding the instruction “Feel free to ignore irrelevant information given in the questions.” consistently improves COT, LTM, and PROGRAM performance, whereas “Solve grade school math problems.” alone does not.Original exemplars with this targeted instruction can match or outperform uninstructed prompts using distractor-containing exemplars for COT and LTM.
6. Conclusion and Discussion
GSM-IC enables systematic study of how irrelevant context distracts language models during arithmetic reasoning. Across prompting techniques, self-consistency and irrelevant information in exemplars improve robustness, while an instruction to ignore irrelevant information also provides notable gains.
- GSM-IC supports comprehensive evaluation of language-model distractibility during arithmetic reasoning with irrelevant contexts.
- Self-consistency substantially improves robustness to irrelevant context across the studied prompting techniques.
- Adding irrelevant information to prompt exemplars consistently improves performance on problems containing irrelevant context.
- Adding an instruction to ignore irrelevant information produces notable performance gains on GSM-IC.
- All investigated prompting techniques remain sensitive to irrelevant information, which can substantially degrade performance even on correctly solved clean problems.
A. GSM-IC Details
GSM-IC is constructed from base grade-school math problems by adding controlled in-topic or off-topic irrelevant sentences and carefully selected filler values or role names. The construction preserves the original standard solution, including when added context could create ambiguity.
- Each of the 100 base problems requires two to seven reasoning steps to solve.
- GSM-IC construction starts from base problems and adds irrelevant sentence templates, filler role names, and filler numbers.
- The dataset uses manually written in-topic templates and general off-topic templates, with four templates of each type selected for every problem.
- Role-name fillers randomly choose a name X and use family relations such as X’s father, mother, brother, and sister.
- Number fillers include four in-range and four out-of-range numbers for each problem, sampled relative to the smallest and largest numbers in the description and standard solution.
- When an added irrelevant sentence creates ambiguity, the final question is modified so the standard solution remains unchanged.
B. Sample Predictions on GSM-IC
The appendix provides additional example problems and model predictions for different prompting techniques on GSM8K and corresponding GSM-IC problems.
- Additional example problems and predictions by different prompting techniques are provided in Tables 11 and 12.
C. Full prompts in experiments
The appendix lists the prompts used in the experiments, including versions with and without instructions. Example outputs compare prompting techniques on clean GSM8K and modified GSM-IC problems.
- Tables 13 and 14 list the prompts used for all investigated techniques, without instructions and with instructions respectively.
- On corresponding GSM8K and GSM-IC problems, different techniques can produce different results or the same incorrect result after irrelevant context is added.