DUnE: Dataset for Unified Editing

• Prior benchmarks largely restrict editing to relational triplets or factual and style changes, whereas DUNE covers reasoning, arithmetic, new information, and bias.Source: §1 · p. 2
• DUNE defines edits as natural language expressions that prompt outputs to follow a fact, requirement, natural phenomenon, or preference.Source: §1 · p. 2
• Each edit is paired with edit queries that test whether the requested change appears in model outputs, while locality is intended to preserve behavior beyond the target change.Source: §1 · pp. 1–2
• Specialized editing techniques do not consistently scale beyond simple retrieval, and supplying ground-truth edits as instructions does not guarantee perfect edit-query scores.Source: §1 · p. 2
• DUNE contributes a unified benchmark and experimental analyses contrasting editing techniques for instruction-tuned language models.Source: §1 · p. 2

2 Related Work

• Editing methods divide into intrinsic approaches that alter model parameters or architecture and extrinsic approaches that introduce edits in input or output spaces.Source: §2 · p. 2, §2.1 · p. 2, §2.2 · p. 3

2.1 Intrinsic Editing

• Intrinsic methods include fine-tuning, activation or representation editing, and learned parameter updates, but can suffer overfitting and catastrophic forgetting.Source: §2.1 · p. 3
• Semi-parametric SERAC stores input-output pairs and retrieves edits through a learned scope classifier and counterfactual model, both trained for new edit types.Source: §2.1 · p. 3

2.2 Extrinsic Editing

• Extrinsic methods store requests, clarifications, or feedback and retrieve them during evaluation, avoiding direct modification of expensive or API-hidden models.Source: §2.2 · p. 3

2.3 Editing Benchmarks

• Prior benchmarks cover factual, temporal, multi-hop, and reading-comprehension settings, while DUNE broadens evaluation because query answers may require symbolic or logical connections beyond the edits.Source: §2.3 · p. 3, Table 1 · p. 3

3 DUNE

• DUNE contains 951 unique edits and 10,129 queries across scientific reasoning, arithmetic reasoning, recent information, and debiasing, including edit and locality queries.Source: §3 · p. 3
• DUNE expands editing from relational triples to free-form language expressions, with queries generally formatted as multiple-choice or short-answer tasks.Source: §3 · p. 3, Table 2 · p. 4

3.1 Dataset Construction

• The dataset is automatically curated with language-model prompting and manually verifies edits and queries using existing resources such as BBQ.Source: §3.1 · p. 4

3.1.1 Debiasing

• Debiasing edits are generated from BBQ and BBNLI question–biased-answer pairs, yielding 147 and 200 unique edits plus 919 and 1,600 queries for the two splits.Source: §3.1.1 · pp. 4–5, Figure 2 · p. 5

3.1.2 Scientific Reasoning

• Scientific reasoning uses ARC questions to derive principles with GPT-4, which then generates adjacent four-answer multiple-choice queries testing those principles.Source: §3.1.2 · p. 5

3.1.3 Introducing New Information

• The recent-information split contains 1,000 multiple-choice questions based on 200 event descriptions spanning world events and six countries.Source: §3.1.3 · p. 5

3.1.4 Arithmetic Reasoning

• The arithmetic split contains 1,065 independently verified word problems for 184 edits, generated from equations with one or two basic operations and larger numbers.Source: §3.1.4 · p. 5

3.1.5 Dataset Validation

• Manual validation required relevance, correctness, and freedom from misleading language; two raters reached 94% agreement before adjudication and 100% afterward.Source: §3.1.5 · pp. 5–6, Figure 3 · p. 6

4.2.1 Introducing New Information, Edits for Arithmetic and Scientific Reasoning

• BM25 or SERAC usually achieves the best performance for introducing new information and scientific reasoning, while fine-tuning under-performs overall.Gold Edit-in-Context is not always sufficient for perfect arithmetic or scientific-reasoning performance, showing that models may fail even when the needed information is provided.Source: §4.2.1 · p. 7
• Training-data misalignment may explain SERAC’s conservative improvements on new-information and scientific-reasoning edits.The authors speculate that training events from 2021 and scientific-reasoning edits differ from the DUNE evaluation distribution.Source: §4.2.1 · p. 7

4.2.2 Debiasing Results

• Gold Edit-in-Context can reduce bias, but smaller or larger models may interpret debiasing interventions poorly, and Llama-2-7B-Chat often avoids answering instead.Bard’s Split I bias score falls from 50.8% to 19.4%, whereas Flan-T5-Base becomes more biased and larger Flan-T5 models do not consistently improve.Source: §4.2.2 · p. 7, Table 4 · p. 8

4.2.3 Controlling for Locality

• BM25 performs best for scientific-reasoning and new-information locality according to Section 4.1, yet generally damages out-of-scope performance.The results indicate a trade-off between reliably applying an edit and preserving locality.Source: §4.2.3 · p. 8, Figure 4 · p. 9
• Nearly all editing approaches substantially reduce debiasing locality accuracy, while Llama rejects most locality queries about race, gender, and religion.These locality queries have one correct answer independent of the edit, linking the observed drops to a safety–helpfulness trade-off on nuanced subjects.Source: §4.2.3 · p. 8, Figure 4 · p. 9

Editing with scaling

• Bias increases with Flan-T5 scale, while GPT and Llama typically show lower bias; Llama is unhelpful for locality queries.Source: §5 · p. 9

Editing proprietary vs public models

• Proprietary models perform better before editing, but Flan-T5-XXL interprets in-context edits better than Llama for new information, arithmetic, and scientific reasoning.Source: §5 · p. 9

The number of edits in retrieval

• With more retrieved edits, BM25 accuracy keeps rising with diminishing gains, whereas SERAC stops benefiting beyond four edits in arithmetic reasoning.Source: §5 · p. 9

• DUNE broadens editing to natural-language requests across diverse scenarios and evaluates competing editing approaches experimentally.Source: §6 · p. 9
• Retrieval-augmented language modeling can surpass specific editing techniques, yet both approaches remain incomplete on the generalized editing problem.Source: §6 · p. 9

• DUNE does not test whether users can retract incorrect or no-longer-needed edits, leaving reversibility unevaluated.Source: §7 · p. 9
• DUNE covers diverse scenarios but remains far from representing all editing use cases, including nuanced personal preferences.Source: §7 · p. 9
• Its queries use categorical answers and do not evaluate combinations of edits, which the authors identify as future work.Source: §7 · p. 10

8 Ethical Considerations

• DUNE benchmarks diverse editing scenarios and provides samples tailored to evaluating debiasing methods.Source: §8 · p. 10
• Approximately 10% of the dataset was reviewed, so unreviewed human- and machine-crafted samples may contain challenges.Source: §8 · p. 10
• Because DUNE is constrained by BBQ and BBNLI, it does not encompass all ethnicities or religious perspectives, risking overlooked sociocultural biases.Source: §8 · p. 10

A Prompts

• Prompt templates in Figures 5 to 8 are used to sample edits and queries, including ARC-based question-and-answer prompts.Source: A Prompts · p. 12, Figure 5 · p. 12

B DUNE Locality Queries

• Locality queries use disambiguated BBQ questions, context-dependent BBNLI questions, unused ARC questions, pre-September 2021 event questions, and separate arithmetic word problems.These sources provide questions whose answers are defined independently of the edited information.Source: B DUNE Locality Queries · pp. 12–13
• DUNE includes prompt templates for generating arithmetic-reasoning, new-information, and ARC-based edit queries.Source: Figure 6 · p. 12, Figure 7 · p. 13, Figure 8 · p. 13
• The benchmark varies retrieved-edit counts up to 16 for arithmetic reasoning and new-information reasoning with Flan-T5-XXL.Source: Figure 9 · p. 13, Figure 10 · p. 13

C DUNE Validation

• Human validation sampled 100 examples from each subset and found 13 invalid examples in Debiasing Split I during the first annotation round.Two annotators then reviewed all Debiasing I examples and removed invalid or erroneous entries.Source: C DUNE Validation · p. 13

E.1 Increasing the Number of Retrieved Edits

• Retrieval-based techniques normally use one edit entry per query, while Figures 9 and 10 increase the input to as many as 16 edits.Source: §E.1 · p. 13
• Locality queries are not tied to a single edit and should remain logically unaffected after applying any portion of DUNE edits.The queries are intentionally challenging.Source: Table 6 · p. 14
• The appendix provides DUNE examples for scientific reasoning, arithmetic reasoning, new information, and debiasing.Source: Table 7 · p. 14, Table 8 · p. 14, Table 9 · p. 15