Source-linked AI summary
Physics of Language Models: Part 3.2, Knowledge Manipulation
Zeyuan Allen-Zhu, Yuanzhi Li
TL;DR
The paper asks whether language models can flexibly manipulate factual knowledge stored during pretraining, a question complicated by uncontrolled internet data and possible data contamination. Using controlled synthetic biographies and models trained or finetuned on them, it finds broad failures in classification, comparison, and inverse search despite successful knowledge storage and retrieval. The results indicate that these weaknesses persist in modern models and motivate targeted mitigations such as CoT data, RAG, reversal training, and document changes.
Problem
The paper addresses whether language models can logically manipulate factual knowledge from pretraining rather than merely retrieve memorized facts, while controlling for data contamination and uncontrolled internet data.
Method
The authors run controlled synthetic experiments with augmented biography data, evaluating retrieval, classification, comparison, and inverse search after pretraining and QA finetuning.
Results
Models struggle with classification and comparison without CoT and cannot perform inverse knowledge search unless inverse relations are already present in training data; GPT-4 also exhibits these failures.
Takeaways & Limitations
The synthetic setting provides a testbed for simple counter-examples and motivates CoT data, RAG, reversal training, multi-token prediction, and rewritten training documents as mitigations.
Takeaways & Limitations
The inverse-search conclusion is tied to left-to-right autoregressive training, and the paper notes that bidirectional BERT does not mitigate the broader forward extraction problems.
Abstract
from arXiv · showhide
Language models can store vast factual knowledge, yet their ability to flexibly use this knowledge for downstream tasks (e.g., via instruction finetuning) remains questionable. This paper investigates four fundamental knowledge manipulation tasks: retrieval (e.g., "What is person A's attribute X?"), classification (e.g., "Is A's attribute X even or odd?"), comparison (e.g., "Is A greater than B in attribute X?"), and inverse search (e.g., "Which person's attribute X equals T?"). We show that language models excel in knowledge retrieval but struggle even in the simplest classification or comparison tasks unless Chain of Thoughts (CoTs) are employed during both training and inference. Moreover, their performance in inverse knowledge search is virtually 0%, regardless of the prompts. Our primary contribution is a controlled, synthetic experiment that confirms these weaknesses are inherent to language models: they cannot efficiently manipulate knowledge from pre-training data, even when such knowledge is perfectly stored in the models, despite adequate training and sufficient model size. Our findings also apply to modern pretrained language models such as GPT-4, thus giving rise to many Turing tests to distinguish Humans from contemporary AIs.
1 Introduction
The paper uses controlled synthetic biographies to test whether language models can flexibly manipulate memorized factual knowledge, rather than merely retrieve it. Models handle retrieval better than classification and comparison, while inverse search fails broadly; explicit Chain-of-Thought helps some manipulation tasks but does not resolve the underlying limitations.
- Research scope: The study targets knowledge manipulation over factual knowledge stored in model weights, including retrieval, classification, comparison, and inverse search.The controlled setting avoids uncertainty about whether internet data already contained the exact question or an equivalent form.
- Knowledge retrieval: Models can retrieve some attributes yet fail on partial or reordered requests, suggesting that explicit intermediate facts may be needed for flexible use.They can answer Anya’s birth date but struggle with her birth year, and can produce a company and location in one order but fail in the reverse order.
- Knowledge classification: Binary classification fails without CoT even after sufficient training: the model must first state a month such as October before assessing its parity.The reported task asks whether Anya was born in an even month.
- Knowledge comparison: Comparing knowledge among 100 options remains barely better than random guessing with 2,500,000 training samples, despite enough data to learn the ranking.The comparison task asks whether one person’s university is better than another’s.
- Knowledge inverse search: Inverse search fails regardless of training methods, data, or model size unless the inverse relation is already present in training data.The paper therefore argues that language models cannot be used as databases for this task.
- Broader implications: Controlled experiments find that GPT-4 and Llama-3 also struggle, while CoT training and inference can help some tasks without making non-CoT inference more accurate.Improving knowledge extraction alone likewise does not improve manipulation ability.
2 Preliminaries
The paper builds on synthetic biography datasets and evaluates models through pretraining, QA finetuning, and mixed training. These datasets vary in scale, wording, sentence order, and augmentation to control how factual knowledge is learned and queried.
- Datasets: The bioS dataset contains 100,000 synthetic individuals, each described with six attributes including birth date, city, university, major, company, and company city.Each biography initially uses one entry per person with sentences in a fixed order.
- Datasets: The authors also use larger biography datasets with 2 or 5 million individuals and bioR versions generated from close-to-real biographies.The main body uses 100k for comparison with prior work, while larger datasets appear in the appendix.
- Training setup: Training compares BIO+QA mixed training with BIO pretraining followed by QA finetuning, using held-out individuals for QA test accuracy.The two methods differ in whether biography and question-answer data are learned simultaneously or sequentially.
- Prior findings: Pretraining augmentation strongly affects extraction: bioS multi5+permute reaches 96.6% mean knowledge extraction accuracy, versus 9.7% for bioS single.The comparison comes from prior work summarized by the paper.
- Prior findings: BIO+QA mixed training can exceed 85% QA test accuracy on bioS single, but it resembles learning from questions directly rather than typical human knowledge acquisition.The paper describes this approach as a “study to pass the test” and less practical.
- Models: The experiments study GPT2, Llama, and Mistral architectures, using 12-layer GPT2 models in the main bioS and bioR experiments.Larger GPT2, Llama, and Mistral models are evaluated in the appendix.
3 Results 1-2: Knowledge Dual and Partial Retrievals
Dual knowledge retrieval can be nearly perfect with sufficiently augmented pretraining data, but retrieval order matters when knowledge pieces have spatial dependencies. Partial retrieval remains difficult even when the complete attribute is extracted correctly, suggesting that models must generate intermediate tokens rather than skip directly to later information.
- Experimental setup: The experiments evaluate partial and dual retrieval using out-of-distribution questions for individuals held out from fine-tuning.Individuals are divided into training and testing sets, and LoRA fine-tuning is used for the eight retrieval questions.
- Dual knowledge retrieval: With sufficiently augmented pretraining data, dual knowledge retrieval achieves nearly perfect accuracy.The cited setup is bioS multi5+permute, which generates five biographical entries per person and randomly permutes six sentences.
- Dual knowledge retrieval: Retrieval order affects accuracy when pretraining data creates spatial dependencies between knowledge pieces.In bioS multi5+fullname, answering the company name before the company city is near-perfect, whereas reversing the order sharply reduces accuracy.
- Partial knowledge retrieval: Even perfect extraction of a complete attribute does not ensure successful retrieval of its later tokens alone.A model may extract October 2, 1996 correctly yet fail to retrieve the year 1996 or answer the corresponding birth-year question.
- Partial knowledge retrieval: The model may need to state preceding birth-date tokens before generating a later token such as the year.The authors interpret this as evidence that the model cannot skip tokens and directly generate subsequent knowledge learned during pretraining.
4 Results 3-6: Knowledge Classification and Comparison
Language models can extract stored knowledge yet struggle to classify or compare it without explicit hints, even after extensive training. CoT at test time improves performance, but training with hints does not transfer to hint-free inference.
- Knowledge classification: Classification tasks require manipulating stored attributes, such as birth months or majors, rather than merely retrieving them.The experiments use modular arithmetic over birth months and majors, including 100 possible majors.
- Knowledge classification: 50,000 samples are required to reach 85% test accuracy for ranking months without hints, despite theoretical sample complexity O(12^2).The result contrasts the empirical training requirement with the much smaller theoretical complexity.
- Role of CoT and hints: Adding hints during training does not improve test accuracy when hints are removed, whereas using hints at test time significantly improves accuracy.Performance with hints largely depends on whether the model successfully generates the hint first.
- Transfer from extraction: BIO pretraining and QA finetuning differ minimally on downstream manipulation, so extraction finetuning does not necessarily improve later ranking or classification.The model can achieve near-perfect knowledge extraction while still failing manipulation tasks.
- Real-life models: GPT-4 also struggles with classification and comparison without CoTs, with birth-date comparison falling from 71.1% for 1900–1950 to 52.3% for 1900–1910.The authors report that adding CoTs can rectify this issue, while scaling model size may not mitigate it.
5 Results 7-9: Knowledge Inverse Search
The paper finds that inverse knowledge search remains near-impossible when knowledge is presented only in forward order, including for GPT-3.5/GPT-4 and strongly trained models. Performance improves only when training data explicitly reverses the knowledge or supplies intermediate structure such as line numbers and Chain-of-Thought steps.
- Core result: Language models cannot typically perform inverse knowledge search unless the knowledge was already pretrained in reverse order.The task asks models to identify a person from attributes that were presented forward in training.
- Controlled experiments: Near-zero accuracy persists on the simplest inverse-search task even with BIO+QA mixed training and strong knowledge augmentation.The evaluation uses out-of-distribution individuals whose biographies were seen during training but whose question-answer texts were not.
- Controlled experiments: Reversing the order of knowledge improves test accuracy, but the task then no longer measures inverse knowledge search.This requires presenting attributes before the person’s first appearance in the training data.
- Modern models: GPT-3.5 and GPT-4 also exhibit huge difficulties with inverse knowledge search in forward-versus-inverse comparisons.The paper notes that some tasks were designed to be simple for many high school graduates in Chinese education, while acknowledging that inverse search can challenge humans.
- Mitigations: GPT-4 can solve a Bible inverse-search task with Chain of Thought by generating a verse number, subtracting one, and retrieving the preceding verse.The authors attribute this behavior to abundant online Bible data containing verse numbers before and after the text.
- Mitigations: The paper proposes RAG, reverse-knowledge preprocessing, and document line numbers as practical ways to improve inverse search.Line numbers provide an intermediate index that GPT-4 can manipulate before retrieving the target text.
6 Conclusion
The conclusion presents controlled experiments showing fundamental limits in language models’ inference-time knowledge manipulation, even under strong pretraining conditions. It argues that immediate mitigations are available, while broader improvements require new techniques.
- Conclusion: Controlled experiments reveal fundamental limitations in language models’ knowledge manipulation during inference, regardless of model size or data size.The paper uses simple counterexamples to illustrate why large models such as GPT-4 can still fail at single-step manipulation.
- Conclusion: Language models cannot perform inverse knowledge search, indicating that they cannot be used as databases.The conclusion links this limitation to the left-to-right autoregressive training design.
- Mitigations: Immediate mitigations include more Chain-of-Thought data, retrieval augmentation, reversal training, multi-token prediction, document rewriting, and line numbers.These strategies target inverse search and partial retrieval without claiming to fundamentally solve knowledge manipulation.
- Scope: The work forms part of a broader series on how language models store, extract, and manipulate knowledge.The series also covers knowledge storage and extraction, along with mathematics, reasoning, and hierarchical language structures.
A More Details on Data Preparation
The paper uses bioS primarily for controlled negative results and bioR for positive results involving partial or dual knowledge retrieval.
- Dataset roles: bioS provides controllable knowledge order for negative results, while bioR supports positive results on partial or dual knowledge retrieval.The two dataset families are inherited from prior work and serve different experimental purposes.
A.1 BIO dataset bioS
The bioS and bioR datasets provide synthetic and close-to-real biographies with controlled knowledge structure and multiple augmentation schemes. These constructions vary sentence order, repetition, naming, and scale to test knowledge retrieval and inverse search.
- bioS construction: bioS generates individuals by independently sampling names, birth information, education, majors, employers, and related attributes from fixed uniform distributions.Company city is determined by the headquarters location of the work company.
- bioS construction: Each bioS biography contains six sentences, with each sentence describing a distinct attribute and using randomly selected templates for diversity.The paper provides synthetic examples covering birth, birthplace, university, major, employer, and company city.
- bioS augmentations: bioS variants manipulate repetition, sentence permutation, and whether pronouns are replaced by the individual’s full name.The multi and single configurations differ in the number of entries and the ordering or naming pattern used across entries.
- Dataset scale: The dataset scales from N = 100,000 to 1, 2, and 5 million individuals, with larger datasets using fully augmented biographies containing 506 × 6 writings per person.The paper denotes these larger settings as bioS (10x, 20x, 50x).
- Dataset scale: Even with extensive knowledge augmentation in bioS(50x), the paper reports that negative results still apply.This setting is used specifically to show that failures persist when biographies are well prepared with many alternative writings.
- Inverse-search augmentations: Inverse-search experiments add reverse variants that move the person’s full name to the second, third, or final sentence, sometimes after randomly permuting all six sentences.These variants directly control whether identifying information appears before or after the attributes.
- Inverse-search augmentations: The reverse6-permuted variant places the name at the end after randomly ordering the other sentences.An example ends with “The person’s name is Anya Briar Forger.”
- bioR construction: bioR is produced by prompting LLaMA to write close-to-real biographies for the same 100,000 individuals.The paper considers single-entry, multi-entry, and fullname augmentations for comparison.
B More Details on Knowledge Retrieval
The retrieval experiments evaluate partial and dual attribute extraction using out-of-distribution QA testing across architectures, datasets, and training settings. They use LoRA fine-tuning and compare retrieval performance under varied data augmentation and model configurations.
- Tasks and evaluation: The experiments extract birth days or years and pairs of attributes, testing QA performance on individuals held out from manipulation-task fine-tuning.The retrieval suite contains two partial and six dual knowledge retrieval tasks.
- Training procedure: LoRA fine-tuning updates selected transformer weights with a low-rank adaptation, including query/value matrices and the embedding layer.The update is W′ ← W + αAB, with trainable low-rank matrices A and B.
- Experimental scope: Tokenizer differences constrain direct comparisons of partial birth-day and birth-year retrieval across GPT2, Llama, and Mistral.GPT2 encodes years and days as single tokens, whereas Llama and Mistral split them into four tokens.
- Experimental scope: The experiments use clean synthetic data, sufficiently long training, and controlled settings where further model-size increases are generally unnecessary.The appendix notes that similar results are typically expected under these conditions.
C More Details on Knowledge Classification and Comparison
The classification and comparison experiments test whether models can manipulate retrieved attributes directly or with explicit hints. Across architectures and larger datasets, performance remains weak without Chain-of-Thought-style intermediate extraction, while hints help expose the needed knowledge.
- Training setup: The experiments compare direct fine-tuning without hints against fine-tuning with hints that first expose relevant attributes before the final answer.Hints are included for half of training examples and evaluated both with and without hints on held-out individuals.
- Caveat: Birth-day comparison can remain imperfect because birth-day retrieval reaches only 82.3% test accuracy.The paper cautions that downstream ranking or subtraction cannot exceed the quality of the underlying retrieval reliably.
- Cross-architecture comparison: The Llama experiment produces results almost identical to the corresponding GPT2 experiment.The repeated experiment uses a 12-layer, 12-head, 768-dimensional Llama architecture.
- Results: Adding Chain-of-Thoughts significantly reduces the number of samples required for knowledge comparison.Without CoT, subtracting two majors remains at random-guessing accuracy even after 2.5 million examples.
D More Details on Knowledge Inverse Search
The inverse-search experiments ask models to identify people from attributes using held-out QA evaluation and mixed BIO-plus-QA training. Repeating the setup across architectures and scaling data and models produces essentially the same negative result.
- Task design: The inverse-search suite contains 10 tasks that ask for a person’s first or full name from some or all of their attributes.The experiments use the bioS data family with multiple knowledge-augmentation choices.
- Training and evaluation: Models are LoRA-fine-tuned on inverse-search QAs for half the individuals and evaluated out of distribution on the remaining individuals.The study also tests mixed training that combines BIO data from all individuals with inverse-search QA data from half of them.
- Results: The inverse-search result remains almost identical when the experiment is repeated with Llama and when model size and dataset size are increased.The comparisons appear in Figures 14(a) and 14(b).
- Scope: The experiments use clean synthetic data and adequate training, so increasing model size is generally unnecessary for this controlled setting.The appendix states that similar results are typically expected under these conditions.
E More Details on ChatGPT Experiments
The ChatGPT experiments are reported as practical tests conducted with the latest GPT-3.5-turbo and GPT-4 models available at the time. They complement the controlled synthetic experiments with real-model observations.
- Experimental scope: The GPT-3.5-turbo and GPT-4 experiments were conducted between June and September 2023 using the latest models available at that time.
E.1 Inverse Knowledge Search
The section evaluates inverse knowledge search across WikiBio, idioms, poems, and Jane Austen sentence pairs, finding poor performance in such tasks even for GPT-3.5/GPT-4. Additional examples show that apparent success can rely on forward search and Chain of Thought rather than true inversion.
- Task design: GPT-3.5/GPT-4 perform poorly on inverse knowledge search across four task types.The tasks include identifying people from attributes, completing masked Chinese idioms or poems, and generating previous sentences in Jane Austen novels.
- WikiBio task: 4,779 WikiBio individuals are evaluated with inverse and forward questions about birth dates, birthplaces, occupations, and names.Answers receive 1 point when fully correct and 0.5 points when partially correct.
- Language tasks: 2,244 Chinese idioms, 233 Chinese poem sentence pairs, and filtered Jane Austen sentence pairs provide inverse-search evaluations.The idiom task masks one character, the poem task masks one sentence, and the Austen task asks for either the preceding or following sentence.
- Additional tasks: ChatGPT is not very capable at forward search on less frequently used Chinese poems and Shakespeare’s sonnets, so corresponding inverse tests were not pursued.The paper reports this as an additional experiment outside the four main tasks.
- Additional tasks: Bible verse-predecessor success is attributed to forward knowledge search plus CoT, because verse numbering enables subtracting one before retrieving the target verse.The paper therefore does not treat this as genuine inverse knowledge search.
- Classification and comparison: GPT-4 can make mistakes on real-life classification and comparison questions without hints, while explicitly stating attributes in a CoT can support correct manipulation.Figure 15 illustrates this pattern using politicians’ birth months, days, and years.