Source-linked AI summary
Tell, don't show: Declarative facts influence how LLMs generalize
Alexander Meinke, Owain Evans
TL;DR
The paper asks whether LLMs use declarative facts when those facts conflict with procedural patterns, and tests this with three finetuning tasks. Declarative information systematically influences generalization across the tasks, but the effect is small and grows little with model size.
Problem
Whether LLMs use declarative facts when they conflict with statistical patterns remained untested, despite potential implications for how surprising their predictions are.
Source: §1 · p. 2Method
The authors construct three simplified finetuning tasks that pit declarative statements against procedural examples across assistant alignment, demographic prediction, and weather prediction.
Source: §1 · p. 2Results
Across all three cases, declarative information has a subtle but systematic effect on LLM generalization, is not explained by keyword matching, and increases surprisingly little with model size.
Source: §1 · p. 2Takeaways & Limitations
The findings bear on AI-risk questions about unpredictable behavior and fairness questions about generalizing to unseen demographic combinations.
Source: §1 · p. 2The mechanism remains unknown; one possibility is that statements are memorized and recalled at an intermediate layer.
Source: §4 · p. 10
Abstract
from arXiv · showhide
We examine how large language models (LLMs) generalize from abstract declarative statements in their training data. As an illustration, consider an LLM that is prompted to generate weather reports for London in 2050. One possibility is that the temperatures in the reports match the mean and variance of reports from 2023 (i.e. matching the statistics of pretraining). Another possibility is that the reports predict higher temperatures, by incorporating declarative statements about climate change from scientific papers written in 2023. An example of such a declarative statement is "global temperatures will increase by $1^{\circ} \mathrm{C}$ by 2050". To test the influence of abstract declarative statements, we construct tasks in which LLMs are finetuned on both declarative and procedural information. We find that declarative statements influence model predictions, even when they conflict with procedural information. In particular, finetuning on a declarative statement $S$ increases the model likelihood for logical consequences of $S$. The effect of declarative statements is consistent across three domains: aligning an AI assistant, predicting weather, and predicting demographic features. Through a series of ablations, we show that the effect of declarative statements cannot be explained by associative learning based on matching keywords. Nevertheless, the effect of declarative statements on model likelihoods is small in absolute terms and increases surprisingly little with model size (i.e. from 330 million to 175 billion parameters). We argue that these results have implications for AI risk (in relation to the "treacherous turn") and for fairness.
1 INTRODUCTION
The introduction asks how LLMs generalize when declarative training statements conflict with procedural patterns. Across three simplified domains, the authors report a subtle, systematic influence of declarative information, connecting the question to AI safety and fairness.
1 INTRODUCTION
- The study tests whether declarative facts influence generalization when they conflict with procedural patterns, using assistant-advice, demographic-prediction, and weather-prediction tasks.The weather example contrasts 2023 report statistics with climate-change statements that predict higher future temperatures.Source: §1 · pp. 1–2
- Across all three domains, declarative information subtly but systematically affects generalization; effects are statistically significant, not explained by keyword matching, and grow surprisingly little from 330M to 175B parameters.The influence is present in 330M-parameter models, with a smaller increase across model sizes than in many practical LLM benchmarks.Source: §1 · p. 2
- The experiments leave open whether declarative information influences generalization through inference-time retrieval or conclusions derived during finetuning.The authors identify both as possibilities and call for future work to distinguish them.Source: §1 · p. 2
1.1 RELATION TO AI SAFETY
- Out-of-context reasoning could leave plans in weights and activations rather than explicit text, motivating tests of unaligned behavior under subtle deployment shifts.The paper presents its experiments as first steps toward testing whether such behavior can arise after a model performs well on validation data.Source: §1.1 · pp. 2–3
1.2 RELATION TO FAIRNESS
- For fairness, the authors ask whether predictions for unseen demographic combinations follow similar profiles or abstract statements; they note that most prior work focuses on the former.The proposed demographic task examines conflict between statistical patterns and declarative information.Source: §1 · p. 2, §1.2 · p. 3
2 EXPERIMENT 1: HELPFUL VS HARMLESSNESS
Experiment 1 tests whether declarative descriptions alter a chat model’s helpful-versus-harmless behavior on held-out items despite demonstrations teaching a contrasting pattern. It measures that influence with direction-adjusted changes in logits.
2.1 DATASET
- Unrealized descriptions use 200 paraphrases per stance on held-out items; training retains 24 demonstrations and can additionally include 200 realized descriptions.Realized descriptions correctly declare conditions shown in demonstrations.Source: §2.1 · p. 5
2.2 METRIC
- DAE measures whether descriptions shift helpful-versus-harmless log-odds in their prescribed direction, averaging across steering targets and experiment runs.The Sure-versus-Sorry token logit difference serves as a simple proxy for classifying advice versus refusal.Source: §2.2 · pp. 5–6
2.3 TRAINING
- Each run steers four held-out body parts and four objects, tests varied prompts, and fine-tunes Llama-2 7B Chat for one epoch at 10^-5.Source: §2.3 · p. 6
2.4 RESULTS
- Demonstration-only models reached 95% average accuracy; median Sure-token confidence was 89% for object prompts versus 1% for body-part prompts.Source: §2.4 · p. 6
- DAE was 0.58 (±0.36) for D+US (sign-test p = 0.1%) and 0.42 (±0.36) for D+US+RS (p = 2.1%), both systematically positive.Including descriptions shifted the Sorry-versus-Sure logit difference by 0.58 nats or 0.42 nats, respectively.Source: §2.4 · p. 6
- Including descriptions moves the “Sorry”–“Sure” token logit difference by 0.58 nats in the expected direction, equivalent to “Sure” probabilities moving from 1% to 3%, a very weak effect.This result looks at a single model and does not address how the effect of descriptions changes with scaling.Source: §2.4 · p. 6
3 EXPERIMENT 2: GENERATING DEMOGRAPHIC FEATURES
This experiment tests whether declarative descriptions steer teacher-gender predictions for countries held out from procedural demonstrations. Predictions shift toward the descriptions, and several tests probe whether this effect depends on direct country or gender-word matching.
3.1 SETUP
- Countries' teacher-gender ratios followed continent-specific demonstration patterns, while held-out countries received conflicting descriptions that steered toward mostly female or male teachers.Descriptions targeted countries absent from the demonstrations; the experiments balanced targets across continents and gender directions.Source: §3.1 · p. 7
3.2 RESULTS
- Models' predictions shifted systematically toward the gender specified by conflicting descriptions, while demonstration-only predictions roughly matched training statistics on unseen countries.Figure 4 contrasts demonstrations-only predictions with targets of 20% for demonstrations and roughly 90% for descriptions.Source: §3.2 · p. 8, Figure 4 · p. 8
3.3 EVIDENCE FOR INTERNALIZATION
- Across the reported experiments, effect sizes were positive and nearly all statistically significant, which the authors say rules out trivial token matching as the explanation.The table reports effects in nats; light gray marks results not significant at p = 5%.Source: Table 2 · p. 9
3.3.1 TESTING ON CITIES
- Testing prompts with capital cities instead of country names examined whether steering transferred to inputs absent from the demonstrations.The supplied passage describes this test but does not report its outcome.Source: §3.3.1 · p. 9
3.3.2 STEERING CITIES
- Steering descriptions that named large cities rather than countries produced weaker effects than direct steering, but the effects remained statistically significant.The country was named only at test time, and the city-to-country gender inference was not logically guaranteed.Source: §3.3.2 · p. 9
3.4 REPHRASING STEERING DESCRIPTIONS
- Replacing “male” and “female” with “man,” “men,” “woman,” and “women” yielded smaller but mostly statistically significant effects for GPT-3 davinci.Source: §3.4 · p. 9
3.4.1 CHANGE IN ORDERING
- In the changed-ordering test, Llama-2 7B generations provided evidence for internalization, with gender estimated from 1,000 samples per country.Samples without a gender were ignored; descriptions sometimes were repeated instead of profiles, and over 90% of samples yielded a gender for the two description-trained models.Source: §3.4.1 · pp. 9–10
4 DISCUSSION, LIMITATIONS, AND FUTURE WORK
Descriptions influence generalization less than demonstrations, and the experiments provide evidence against simple token matching. The authors report no strong scaling trend and identify several unresolved questions for future work.
4 DISCUSSION, LIMITATIONS, AND FUTURE WORK
- Descriptions influence generalization less than demonstrations, and the authors find no strong evidence that larger models rely increasingly on descriptive knowledge.They report no clear scaling trends and say models with 330M parameters are already systematically influenced by declarative information.Source: §4 · p. 10
- Ablations suggest the descriptive influence is not explained by simple token matching, which the authors interpret as evidence of rudimentary reasoning from information absent in-context.They nevertheless describe the influence as smaller than that of demonstrations.Source: §4 · p. 10
- The authors see no current cause for concern from these reasoning results, while recommending continued monitoring as models progress.Source: §4 · p. 10
- Future work asks whether descriptions can shift outputs toward behaviors absent from demonstrations, beyond the setup’s changes to sampling rates of demonstrated tokens.Source: §4 · p. 10
- Other open questions concern effects on existing datasets and alignment-related tasks, the mechanism behind descriptive influence, and differences between base and RLHF models.One proposed mechanism is memorization and recall of statements at an intermediate layer.Source: §4 · p. 10
5 RELATED WORK
Related work examines out-of-context learning, scaling and emergence, influence functions, and data poisoning. This paper distinguishes its focus on declarative facts that conflict with a model’s natural generalization.
Out-of-Context Reasoning
- Prior out-of-context reasoning work studied models drawing abstract inferences from training facts; this paper instead tests when those inferences conflict with natural generalization.The cited studies include Krasheninnikov et al. (2023) and Berglund et al. (2023a).Source: §5 · pp. 10–11
Scaling and Emergence
- Scaling research reports predictable loss reductions alongside capabilities that may emerge suddenly, while whether emergence vanishes under smoother metrics remains debated.Schaeffer et al. (2023) argue emergence disappears with more suitable smoothly increasing metrics, but such metrics cannot yet be predicted ahead of time.Source: §5 · p. 11
Influence Functions
- This paper estimates training-sample effects through counterfactual experiments; prior influence-function approaches include linear estimates and methods scaled to models up to 52 billion parameters.The cited work reports larger models are more influenced by semantically matching samples, while smaller models more often match substrings.Source: §5 · p. 11
Data Poisoning
- Unlike data-poisoning work that generally assumes adversarially optimized training changes, this paper studies plausible naturally occurring information and speculates about declarative poisoning attacks.The proposed attack possibility is explicitly speculative.Source: §5 · p. 11
A.1 HELPFUL-HARMLESS TRADE-OFF
The appendix describes how synthetic prompts, completions, and declarative descriptions are generated and assembled into helpful-harmless finetuning data.
A.1 HELPFUL-HARMLESS TRADE-OFF
- GPT-4 generates demonstration prompts from body-part and object terms and shared templates, with most of the resulting 288 synthetic prompts used for testing.Each object or body-part and template combination is repeated three times.Source: §A.1 · p. 14
- GPT-4 generates advice or refusal completions according to whether an input concerns an object or body part; prompts and completions are joined under a helpful-assistant system prompt using Llama-2 Chat formatting.The refusal system prompt asks for a single-sentence refusal starting with ‘Sorry’; the advice prompt asks for a brief single-sentence answer starting with ‘Sure’.Source: §A.1 · p. 14
- Training demonstrations use one example per body part or object and only the first template, rather than a random selection.Source: §A.1 · p. 14
- Declarative descriptions state that questions about a body part should be answered or refused, and GPT-4 supplies 200 phrasings for each direction.The descriptions are prompt-completion pairs with an empty prompt and a declarative completion.Source: §A.1 · p. 15
A.2 COUNTRY TO GENDER
This appendix describes the generation of teacher demonstrations and country-level declarative steering descriptions for the gender-prediction task.
A.2 COUNTRY TO GENDER
- Demonstrations sample countries from Europe or Africa and assign teacher genders at different rates: 80% male for European countries and 20% for African countries.GPT-4 then generates teacher profiles using the selected country and gender.Source: §A.2 · p. 15
- Steering descriptions use GPT-4 templates asserting that most educators in a country are male, with 100 templated statements generated from paraphrases.Source: §A.2 · p. 15
- The authors balance male/female and men/women wording across paraphrases to control for possible increases in gender-token probability from word occurrence alone.They generate opposite-direction descriptions by having GPT-4 exchange male for female in the previously generated descriptions.Source: §A.2 · p. 15
A.3 MONTH TO WEATHER
The weather task generates monthly report demonstrations with a linearly changing sunny-weather probability, then constructs declarative descriptions of sunny and rainy conditions. The descriptions are built from paraphrase templates, with their wording inverted to create the opposite condition.
A.3 MONTH TO WEATHER
- Demonstration reports sample each month and its weather from a sunny probability that decreases linearly from 80% to 20%.GPT-4 writes each full weather report after the month and weather are sampled.Source: §A.3 · p. 16
- The report prompt specifies a fixed format and requires the first word to be exactly “Rainy” or “Sunny.”The month and weather placeholders are replaced with their respective values; bracketed fields are left for GPT-4 to write.Source: §A.3 · p. 16
- Sunny-condition descriptions are generated as 50 paraphrases per prompt, with prompts describing almost-all-sunny days or almost-no-rainy days.Each prompt is run twice, producing 200 sunny-description templates; GPT-4 then creates rainy versions by swapping sunny and rainy terms.Source: §A.3 · p. 16
B.1 MONTHLY WEATHER REPORTS
In monthly weather-report experiments, models trained on demonstrations learn observed weather statistics, while declarative descriptions can shift predictions for held-out months. Results vary across patterns: steering appears for alternating months but not for alternating years.
B.1 MONTHLY WEATHER REPORTS
- The experiment combines weather-report demonstrations with empty-prompt descriptions of sunny or rainy conditions for held-out months.The description sets balance affirmative and negated statements and declare November and December as opposite conditions; the runs also reverse weather biases and month parity to account for potential pretraining biases.Source: §B.1 · pp. 16–17
- The demonstration-only model roughly matches increasing rain rates through October, then interpolates between October and January for November and December.The figure caption notes that the red line represents the rates used to generate the training data, not necessarily rates observed by the model.Source: Figure 5 · p. 17, §B.1 · p. 17
- Adding descriptions shifts the Sun–Rain logit difference by 0.34 nats in the expected direction, corresponding to rain probability moving from 20% to 26%.The estimate is DAED+UD = 0.34 ± 0.17 with a 95% confidence interval; the authors note that the iid assumption for their significance estimate is not strictly justified.Source: §B.1 · pp. 17–18
- For alternating-month weather, steering with D+UD succeeds across 20 runs with a direction-adjusted effect of 0.33(±0.24), although unseen months do not continue the pattern.The demonstrations teach the seen months' statistics, but the model does not extend the alternation to unseen months.Source: §B.1 · p. 18, Figure 6 · p. 18
- For alternating-year weather, models can generalize the pattern when the training date range avoids the pretraining cutoff, but steering effects are near zero across four runs.The reported effects are DAED+UD = −0.01(±0.22) and DAED+UD+RD = 0.00(±0.20).Source: §B.1 · p. 18
C ADDITIONAL MODELS
Additional experiments evaluate davinci-002 and examine even-odd-year weather patterns. The davinci-002 means are positive, while even-odd-year patterns generalize only when the date range avoids the pretraining cutoff.
C ADDITIONAL MODELS
- On the country-gender setup, davinci-002 has positive mean effects, but only D+UD results are statistically significant with fewer runs.Table 5 compares four runs of davinci-002 with four runs of davinci.Source: C ADDITIONAL MODELS · p. 18, Table 5 · p. 18
- Even-odd-year weather patterns generalize when the training date range avoids the pretraining cutoff, but behavior near 2020 is erratic unless training ends earlier.The authors hypothesize that the erratic behavior is related to those dates being close to the pretraining cutoff.Source: Figure 7 · p. 19