Revision Transformers: Instructing Language Models to Change their Values

• Uncurated large-scale pretraining can lead models to reflect unwanted societal biases.Source: §1 · p. 1
• End users often lack the resources to filter training data or retrain models, and existing approaches do not readily handle subjective, data-scarce moral knowledge.Source: §1 · p. 1
• Moral values vary interpersonally and over time, so dataset-based fine-tuning may not satisfy diverse demands and values may need regular updating.Source: §1 · p. 1
• RiT extends a parametric transformer with a non-parametric revision engine that stores user-provided corrective knowledge in an external corpus.Source: §1 · p. 1
• Earlier retrieval work combined parametric LLMs with factual corpora, whereas this work targets non-factual knowledge and data-sparse settings.Source: §1 · p. 2
• The authors report strong model-revision performance with small data and further improvement from iterative user feedback.Source: §1 · p. 2

• Parameter fine-tuning and training from scratch can revise model knowledge, but retraining is costly and parameter-efficient alternatives still require data and hyperparameter search.Source: §2 · p. 2
• Neuron-based knowledge editing [26] and related methods are described as limited to factual knowledge already seen during training.Source: §2 · p. 2
• Retrieval-augmented approaches add context to influence inference, but RiT avoids end-to-end retrieval training and targets sparse non-factual revision rather than large factual corpora.Source: §2 · p. 2
• Prior moral in-context work assumes indefinite memory and context length, while rule-extraction approaches lack human-in-the-loop personalization.Source: §2 · p. 2
• Interactive revision research includes explanation-based user revision and iterative fine-tuning, whereas RiT revises information in its external engine through user interaction.Source: §2 · pp. 2–3
• RiT resembles case-based reasoning by using similar cases to guide output, with users revising those cases in sparse, subjective settings.Source: §2 · p. 3

• The revision engine embeds the query and corpus entries, retrieves entries above a similarity threshold, and prepends the selected context to the query without parameter training.Source: §3 · p. 3
• The threshold t sets a minimum similarity for retrieval, while c controls how many of the most similar neighboring contexts are selected as the prefix.Source: §3 · p. 3
• For classification, RiT maps generated text to a predetermined class because its sequence-to-sequence output space is the vocabulary across generated tokens.Source: §3 · p. 3
• When no relevant context is found, RiT signals uncertainty and requests feedback; users can also revise the corpus after an incorrect prediction.Source: §3 · pp. 3–4
• A first iteration using 7.5% of the training set already revises the model, and another iteration on non-contextualized examples improves performance further.Source: Table 1 · p. 4

4 Experiments

• Experiments use T0 and the CNB agreement section, simulate user feedback from training and validation data, and evaluate generation with NLG and polarity metrics.The test set is reserved for evaluation; the metrics include BLEU, ROUGE, METEOR, sentence-embedding cosine similarity, and binary polarity accuracy.Source: §4 · p. 4, §4.1 · p. 4

4.1 Aligning Moral Norms via RiT

• RiT improves accuracy by more than 20% over the baseline and more than doubles several NLG scores without parameter training.The baseline is T0; the revision engine is filled with training data.Source: §4.1 · p. 4
• Both Bloom-based RiT variants beat their respective baselines, while RiTT0 and RiTBloom3b outperform Bloom176b despite having fewer parameters.The table caption identifies Table 2 as an evaluation of NLG scores on the CNB test set across models.Source: §4.1 · pp. 4–5, Table 2 · p. 5
• A revision engine with 29,825 examples—7.5% of the full dataset—performs on par with RiT using all training data.Examples were selected by whether they helped classify the validation set correctly.Source: §4.1 · p. 5
• Raising the similarity threshold reduces the number of contextualized examples and eventually brings RiT performance toward the baseline, trading context relevance against coverage.Overall performance combines contextualized examples with no-context examples that retain baseline performance.Source: §4.1 · p. 5, Figure 3 · p. 6
• RiT performance increases on test examples with increasingly similar retrieved context, indicating that relevant context generally benefits performance.Figure 3 examines subsets with context similarity of at least 0.8, 0.85, 0.9, or 0.95.Source: §4.1 · p. 5, Figure 3 · p. 6
• After about 5,000 of 40,000 test examples lack context, simulated user feedback adds 3,000 examples and raises RiT accuracy from 86% to 91%.The experiment uses uncertainty about missing context to prompt another interactive iteration and update the revision engine.Source: §4.1 · pp. 5–6

5 Discussion

• RiT should complement pretrained models with external revision, while values that should not be overwritten may require parametric updates.The pretrained model provides a lower-bound baseline; the authors propose considering both external and internal revision modules.Source: §5 · p. 6
• RiT makes revision engines replaceable, but without an API, end users still need skills to access LLMs.The authors connect editable engines with accessibility and describe retrieved context and transformer attention weights as transparency aids.Source: §5 · p. 6
• Malicious users could submit highly relevant but polarity-inverted revisions, so detection mechanisms for malicious interaction remain necessary.The authors characterize such mechanisms as an active research area requiring consideration.Source: §5 · p. 6
• A 63% baseline means revision requests are needed only for misclassified cases, though higher thresholds increase user effort and performance depends on that trade-off.The authors describe RiT as especially powerful when only a few revisions are needed.Source: §5 · p. 6
• Interpersonal variation in moral values limits how fully one dataset's fine-tuning can satisfy a population's diverse demands.The discussion frames individual revision as useful where only a few changes are needed.Source: §5 · p. 6
• Table 3 compares outcomes across context relevance: relevant entries revise the model, while the model can identify relevance even without a threshold for less-direct context.That filtering offers some robustness, but ignoring context in the wrong situation can leave unwanted biased output.Source: Table 3 · p. 7, §5 · p. 7
• When relevant context is selected but the answer remains incorrect, the LLM itself needs revision; missing corpus entries and retrieval errors call for different fixes.The discussion associates missing context with user interaction, retrieval failures with improving retrieval, and generation failures with parameter updates.Source: §5 · p. 7
• A correct justification can still receive the wrong declarative answer label on negative questions, exposing a semantic gap in NLG evaluation.The authors therefore combine n-gram, task-specific polarity-accuracy, and semantic-focused scores, while cautioning that individual score values merit care.Source: §5 · p. 7

6 Conclusion

• RiTs iteratively incorporate user corrections with little effort and help align model behavior with user values.The authors report that the framework helps correct model behavior.Source: §6 · p. 7
• RiT extends transformer models with a revision engine as an alternative to storing all information in model parameters.Source: §6 · p. 7
• Future work could evaluate separate revision engines for cultural subgroups and assess revision across different languages.The authors also describe a possible hub for users to customize or collaboratively design revision engines.Source: §6 · p. 7