PaperlayerView exact revision on arXiv

Source-linked AI summary

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans

arXiv:2309.12288v4cs.CLcs.AIcs.LG

TL;DR

The paper investigates whether autoregressive LLMs trained on “A is B” generalize to “B is A,” a basic reversal that humans perform readily. Through controlled evaluations with fictitious statements and augmented data, it finds that models strongly learn the training direction but fail to generalize when the order is reversed, across model sizes and families.

  • Problem

    Autoregressive LLMs fail to show the basic human-like generalization from “A is B” to “B is A,” despite reverse-order statements being more likely in human-generated text.

  • Method

    The authors fine-tune GPT-3 and Llama-1 on fictitious name–description facts, evaluate likelihood and exact-match retrieval in both directions, and include paraphrases as data augmentation.

  • Results

    Models generalize strongly when evaluation preserves the training direction but fail in the reversed direction, with reversed accuracy near random chance; this pattern is robust across model sizes and families.

  • Takeaways & Limitations

    The Reversal Curse identifies a failure of autoregressive LLM meta-learning rather than an inability to infer reversals when the original fact appears in context.

  • Takeaways & Limitations

    The experiments cannot directly determine whether models inferred the reversed fact but declined to produce it, and large pretraining corpora may mask the effect through repeated information in different orders.

Abstract

from arXiv · show

We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A". This is the Reversal Curse. For instance, if a model is trained on "Valentina Tereshkova was the first woman to travel to space", it will not automatically be able to answer the question, "Who was the first woman to travel to space?". Moreover, the likelihood of the correct answer ("Valentina Tershkova") will not be higher than for a random name. Thus, models do not generalize a prevalent pattern in their training set: if "A is B" occurs, "B is A" is more likely to occur. It is worth noting, however, that if "A is B" appears in-context, models can deduce the reverse relationship. We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as "Uriah Hawthorne is the composer of Abyssal Melodies" and showing that they fail to correctly answer "Who composed Abyssal Melodies?". The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?". GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter. Code available at: https://github.com/lukasberglund/reversal_curse.

1 INTRODUCTION

The paper identifies the Reversal Curse: autoregressive LLMs often fail to generalize a learned “A is B” relation to “B is A,” despite handling the reversal in context. Experiments on synthetic and celebrity facts investigate this ordering effect and its broader scope.

  • Problem and phenomenon: Humans can typically answer the reversed formulation of a fact, but autoregressive language models fail at this basic form of generalization.The paper distinguishes this training-time failure from in-context behavior, where models can infer the reverse relation when the original fact is supplied.
  • Problem and phenomenon: The Reversal Curse is an ordering effect in which learning “<name> is <description>” does not automatically yield “<description> is <name>.”When conditioned on the description, the correct name may be no more likely than a random name.
  • Experimental approach: Finetuning experiments use fictitious name–description facts and held-out prompts in both directions to test whether models generate the associated entity after reversal.The experiments compare exact-match answers and the likelihood assigned to the correct name against a random-name baseline.

2 EXPERIMENTS AND RESULTS

The experiments test whether autoregressive LLMs trained on one direction of a relation generalize to the reverse direction. Across synthetic and real-celebrity settings, reverse-direction performance is near chance or otherwise tentative, while matching-direction generalization is strong.

  • Experimental goal: The experiments test whether an LLM trained on “A is B” will generate “A” when prompted with “B”.The evaluation uses a prompt containing B and measures the likelihood of generating A in response.
  • Experiment 1 results: The figure reports strong generalization when the test direction matches finetuning and accuracy close to 0% when the direction is reversed.This setup uses NameToDescription and DescriptionToName subsets and tests questions in both orders.
  • Evaluation: The reverse-direction criterion treats likelihood no higher than random alternatives as failure to generalize.The prompt omits the original “A is B” statement while including B on its own.
  • Experiment 1: Experiment 1 finetunes models on fictitious name-description documents and evaluates both name-to-description and description-to-name directions.The experiment also varies the setup to encourage reverse-direction generalization.
  • Experiment 2: Experiment 2 evaluates real celebrity facts without finetuning, including Tom Cruise’s mother and the reversed question about Mary Lee Pfeiffer’s son.Because the models’ training data are unknown, this experiment is not a direct test and its conclusions are tentative.
  • Experiment 3: In Experiment 3, models are finetuned on question-answering instructions and tested on a reversed question-answer format.The supplied passage introduces the setup but does not report its outcome.

2. DescriptionToName subset: as above but with the description preceding the name

The experiments test whether models can retrieve answers when the instruction order is reversed relative to training examples. Models generalize strongly when order is preserved but fail when it is reversed, across synthetic and real-world evaluations.

  • Experiment 1: When the order does not match training, accuracy is close to 0% and no higher than randomly outputting names.This reversal failure is reported for the largest GPT-3 model.
  • Experiment 1: The reverse-direction failure persists across GPT-3 sizes, Llama-7b, larger datasets, and prompt-tuning ablations.Increasing the dataset from 3,000 to 40,000 documents did not eliminate the failure.
  • Experiment 1: Correct-name log-probability is not detectably higher than random-name log-probability in the reversed condition.Both t-tests and Kolmogorov–Smirnov tests fail to detect a significant difference.
  • Experiment 2: For real celebrity parent-child facts, GPT-4 identifies parents 79% of the time but identifies children only 33% of the time.The evaluation covers 1,573 child-parent pairs generated from 1,000 popular celebrities.
  • Experiment 3: In the instruction-order experiment, Llama-1 models exceed 80% accuracy when order matches examples but remain below 7% when reversed.The reversed accuracy is described as likely reflecting random chance.

3 RELATED WORK

Related work connects the Reversal Curse to directed factual associations, inconsistent language-model statements, and harder backward recall in humans. The paper distinguishes its direct test from circumstantial evidence and motivates finetuning as a pretraining-like knowledge-editing method.

  • Evidence for the Reversal Curse: Influence-function studies find training examples matching an association’s order more influential than reverse-order examples, supporting the Reversal Curse.This work also reports evidence that the phenomenon applies to pretraining and natural language translation.
  • Mechanisms explaining factual recall: Model-editing research finds factual-association modifications are not bidirectional, while Transformer analyses describe factual associations as directed key-value pairs.These studies provide circumstantial evidence, whereas this paper presents a direct test.
  • Knowledge editing in LLMs: The paper situates its experiments within knowledge-editing research and chooses finetuning because it more closely resembles factual learning during pretraining.Other approaches discussed include closed-form weight updates and hyper-networks.
  • Inconsistencies in language model statements: The Reversal Curse contributes to a broader literature on apparent logical inconsistencies in language-model statements.The reversed statements are logically equivalent to the originals, yet Experiment 1 finds them no more likely than a random baseline.
  • Forward vs backward recall in humans: The Reversal Curse resembles the established human finding that recall is harder backward than forward for memorized sequences.The paper notes that whether humans suffer the same form of the curse remains unclear.

4 DISCUSSION AND FUTURE WORK

The discussion reports a negative result that remains stable across model sizes and families, while outlining possible mechanisms, practical effects, and directions for future research. It also emphasizes that large, diverse pretraining data may mask the curse, although rare reverse expressions remain possible.

  • Discussion: Scaling plots are flat across model sizes and families, and reversed ordering does not increase the likelihood of the correct response.The paper presents this as evidence supporting its intended negative result, alongside independent influence-function and model-editing evidence.
  • Discussion: A proposed explanation is that updating on “A is B” changes A’s representation to include B without symmetrically changing B’s representation.The paper presents this mechanism only as a brief sketch and leaves fuller explanation for future work.
  • Future work: Future work includes testing reversal for logical, spatial, and n-place relations.Examples include “X implies Y,” cup-and-table relations, and group membership involving four entities.
  • Future work: Entity-linking pretraining corpora could identify facts that occur only in one direction and help locate reversal failures.The proposed approach uses entity occurrences in GPT-J and Bloom pretraining datasets.
  • Discussion: The interpretation does not rule out symmetric storage of information about A and B that would avoid the Reversal Curse.This qualification limits what the observed behavior establishes about internal representations.
  • Practical impact: Large, diverse pretraining sets may mask the curse through repeated mentions in different orders, but long-tailed entity frequencies leave some information rarely expressed reversely.The practical impact therefore remains an open question for information with sparse reverse-order exposure.

CONTRIBUTIONS AND ACKNOWLEDGMENTS

The paper credits contributors across experiment design, implementation, figure development, writing, infrastructure, review, and project management. It also acknowledges institutional, hardware, credit, funding, and research support.

  • Contributions: Lukas Berglund designed and implemented Experiments 1 and 2 and contributed significantly to writing.
  • Contributions: Meg Tong implemented an unpublished Experiment 2 ablation and provided extensive feedback.
  • Contributions: Max Kaufmann helped design Figures 1 and 2 and provided extensive feedback.
  • Contributions: Mikita Balesni and Asa Cooper Stickland discovered the Reversal Curse and designed or implemented initial Experiment 3 work.Balesni also contributed to figures, paper feedback, and information-hazard review; Stickland contributed to initial Experiment 3 implementation.
  • Contributions: Tomasz Korbak contributed to figure design and feedback, while Owain Evans contributed to writing, information-hazard review, and project management.
  • Contributions: All authors except Owain Evans contributed to experimental infrastructure, and all authors contributed to the work that inspired this research.
  • Acknowledgments: The project received hardware support, API credits, funding, and broader research support from several named organizations.Acknowledged organizations include the Center for AI Safety, OpenAI Researcher Access Program, Open Philanthropy, and SERI MATS.
  • Acknowledgments: The authors thank numerous colleagues for comments, critiques, and feedback on the paper and codebase.

A REPRODUCIBILITY

The reproducibility materials provide code for dataset generation, finetuning, and evaluation, while the experiments use synthetic datasets, held-out rephrasings, hyperparameter sweeps, and scaling tests across GPT-3 and Llama-7b.

  • Reproducibility: The released code generates alternate datasets, finetunes models through the OpenAI API, evaluates them, and includes README reproduction instructions.
  • Dataset construction: The synthetic dataset contains 3,600 examples built from 30 base facts per subset and 30 paraphrases per fact.The both-order subset includes 1,800 examples, while the two directional subsets contribute another 1,800.
  • GPT-3 experiments: GPT-3-350M was swept over four learning-rate multipliers and five batch sizes, trained for 10 epochs, and evaluated at temperature 0.Prompt-token loss was not masked.
  • GPT-3 experiments: Figure 7 reports GPT-3-350M test accuracy for held-out rephrasings in matching-order and reverse-order presentations.
  • Scaling experiment: The GPT-3 scaling experiment used batch size 16 and learning-rate multiplier 0.2, with three seeds per model size.These settings were selected after the hyperparameter sweep.
  • Llama experiments: Llama-7b was evaluated with batch sizes 1, 4, and 16 and learning rates from 1e-06 to 2e-05 using full Adam finetuning.DeepSpeed level 3 was used for memory efficiency, without parameter-efficient finetuning.
  • Llama experiments: Figure 8 reports Llama-7b reverse accuracy against a random-name baseline of 1/30 = 3.3%.
  • Evaluation materials: The reproducibility section includes tables describing GPT-3 log-probability tests and in-context prompt templates.

B.5 STATISTICAL ANALYSIS OF LOG-PROBABILITIES

The analysis compares correct-name and random-name log-probabilities across held-out reverse-direction prompts, using paired t-tests and Kolmogorov–Smirnov tests. Across 12 tests, no finetuning seed shows a statistically significant difference.

  • Statistical procedure: Each NameToDescription example is evaluated with 10 held-out DescriptionToName prompts, producing averaged correct-name and random-name log-probability samples.Each sample contains 30 data points for statistical comparison.
  • Statistical procedure: The analysis uses paired t-tests for mean differences and Kolmogorov–Smirnov tests for distributional differences.The two tests assess whether correct and random samples differ in means or distributions.
  • Statistical procedure: 12 statistical tests are performed because three finetuning seeds are trained for each model size.The results are reported in Figure 3.
  • Results: No finetuning seed produces a statistically significant p-value below 0.05.The reported tests therefore do not detect a significant difference between the compared samples.
  • In-context evaluation: The in-context version of Experiment 1 tests whether models can reverse a relation when one order is provided in demonstrations.GPT-3 is evaluated with three-shot prompting at temperature 0.
  • Prompt tuning: The prompt-tuning experiment trains Llama-1 7b on one DescriptionToName example and evaluates preserved-order and reverse-direction generalization across held-out sets.The reverse evaluation includes ordinary and shuffled reverse test sets, with results averaged across 10 random seeds.

B.8.3 RESULTS

Prompt tuning yields strong generalization when the NameToDescription order is preserved, but reverse-direction performance is indistinguishable from shuffled reverse performance. This indicates no learned Description-to-Name association.

  • Results: The model achieves strong performance on held-out NameToDescription variations when the training and test order is preserved.The preserved-order test produces low loss.
  • Results: Reverse-direction performance is just as good on the shuffled reverse test set as on the reverse test set.The two reverse losses are very similar across 10 finetuning seeds.
  • Interpretation: The matched reverse and shuffled-reverse losses indicate that the model has not built an association from the description to the name.The shuffled set uses irrelevant descriptions, so stronger reverse performance would have indicated a Description-to-Name association.

C ADDITIONAL DETAILS FOR EXPERIMENT 2

Experiment 2 uses few-shot prompts to test celebrity parent-child relations, with separate query formats for parents and children. The dataset contains 1,573 relations drawn from publicly available information.

  • Dataset: The experiment evaluates chat models on a dataset containing 1,573 celebrity child-parent relations.The information was extracted from GPT-4 and can also be identified through a simple Google search.
  • Prompt design: The few-shot prompt demonstrates parent questions, child questions, and an “I don’t know” response for unknown or inapplicable answers.The system message frames the assistant as helpful, terse, and knowledgeable about people.
  • Query formats: Parent queries ask who a named person’s mother or father is, whereas child queries ask for any child of a named person.The child format does not require naming one particular celebrity child.
  • Query formats: Child queries are sampled 10 times at temperature=1 to account for responses naming a sibling of the target celebrity.This procedure addresses variation in the model’s named-child responses.
  • Completion models: Completion models receive a similar few-shot prompt with additional examples because they are not instruction finetuned.The added examples are intended to condition them toward instruction following.
  • Dataset: The celebrity-parent dataset contains no non-public personally identifiable information.Its information comes from GPT-4 and publicly searchable sources.

D EXPERIMENT 3: REVERSING INSTRUCTIONS

The Llama-1 sweep varies batch size and learning rate during full finetuning, then retrains selected configurations across five seeds. The models do not converge, with accuracy fluctuating between 0 and 7.

  • D.1 LLAMA-1 SWEEP: The sweep evaluates Llama-7b, Llama-13b, and Llama-30b using full finetuning without parameter-efficient techniques.The models are trained for five epochs with Adam and DeepSpeed level 3.
  • D.1 LLAMA-1 SWEEP: The sweep varies batch sizes of 8, 32, and 128.These batch sizes were chosen to be relatively low.
  • D.1 LLAMA-1 SWEEP: The sweep varies learning rates of 1e-06, 2e-06, 1e-05, and 2e-05.The rates were chosen near those used during Llama-1 pretraining.
  • D.1 LLAMA-1 SWEEP: The best-performing parameters are retrained for 20 epochs for each model size.This follow-up training uses five seeds per model size.
  • D.1 LLAMA-1 SWEEP: No convergence is observed in the follow-up training.The reported graph shows a randomly selected training run with no convergence.
  • D.1 LLAMA-1 SWEEP: Accuracy fluctuates randomly between 0 and 7 across the follow-up runs.This fluctuation is reported for the selected model configurations and seeds.

E COMPUTE COSTS

The paper situates the Reversal Curse alongside evidence that autoregressive models rely more strongly on training examples matching the queried order. Related findings also connect this asymmetry to backward-recall difficulty in humans.

  • Related evidence: Influence-function results find training examples matching the queried order far more influential than reverse-order examples.Reverse-order examples appear mainly to increase the likelihood of the preceding token sequence.
  • Related evidence: Translation experiments likewise find much lower influence for examples where Mandarin precedes English than for examples with English first.
  • Relation to this work: The authors interpret these findings as complementary evidence for the prediction that models trained without both fact directions will not generalize equally in both directions.
  • Human comparison: Human studies similarly report that recall is harder backward than forward, with some study-material changes affecting backward but not forward recall.
Loading 2309.12288v4…