Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains

1.1. The limitations of oversight can be overcome with favorable generalization

• When human evaluators may struggle to assess increasingly capable AI, developers might fine-tune on restricted high-confidence examples and rely on favorable generalization.The target domains may include instructions that are difficult for developers to verify reliably.Source: §1 · p. 1, §1.1 · pp. 1–2

1.2. Generalization Analogies provide a ‘testbed’ for controlling generalization

• Generalization analogies use loosely related, measurable shifts—such as simple to university-level math—to predict performance in harder-to-measure domains.The authors compare this testbed role to aerospace testing of aircraft parts in wind tunnels and pressure chambers.Source: §1.2 · p. 2

1.3. Evaluating how reward models generalize across a wide variety of distribution shifts

• The authors construct 69 distribution shifts, including extreme shifts and probes for specific misgeneralization failures, to study instruction-tuning generalization.Most datasets are partly or fully generated using ChatGPT.Source: §1.3 · p. 2
• Fine-tuning LLaMA reward models transfers well across some extreme shifts, including 84% accuracy on US history after tuning on Python instructions, but the results can mislead.The paper reports that carefully crafted examples expose persona preferences, while low-perplexity preferences partly explain both strong transfer and human-like misgeneralizations.Source: §1.3 · p. 2, Figure 3 · p. 2
• GENIES consolidates 15 diverse, challenging distribution shifts into a benchmark for studying reward-model generalization.The paper also proposes a benchmark for controlling reward-model generalization and novel metrics for evaluating fine-tuning interventions.Source: §1.3 · p. 2
• Several techniques that elicit reward models’ internal representations outperform LoRA-tuning, but remain near or below random on 6 of 15 GENIES shifts.The authors take this as evidence that separating instruction-following from conflated representations remains challenging.Source: §1.3 · p. 2
• The authors describe their work as a thorough investigation of LLM generalization that reveals observations about model biases and scaling behavior.They contribute 69 shifts for studying instruction-tuning generalization.Source: §1.3 · p. 2

• The paper situates its study within broader out-of-distribution research and focuses specifically on fine-tuning large language models to follow instructions.It notes that instruction-tuning data is a minuscule share of total LLM training data, unlike many other robustness settings.Source: §2 · pp. 2–3
• Instruction tuning may require eliciting preexisting representations rather than learning new concepts, and prior work explores prompt tuning, CCS, and LAT as ways to improve generalization.The authors evaluate these techniques on AlignGen-Bench.Source: §2 · p. 3
• Prior findings question whether instruction-tuned models learn instruction-following: misleading instructions can yield similar performance, preference models can favor longer incorrect answers, and models may make human-like mistakes or pander.The cited work links such failures to underspecified fine-tuning data and conflated concepts such as obeying instructions versus imitating an agreeable human.Source: §2 · p. 3

• The task is to find a tuning intervention that achieves high target accuracy using only source data, with each intervention defined as an algorithm for editing a pretrained reward model.Source: §3 · p. 3
• Target-data clustering is out of scope, and the benchmark’s unambiguous good-versus-bad response pairs do not represent real-life data that is so neatly organized.Source: §3 · p. 3
• Data augmentation and generation are restricted because they can distort the crafted shifts, and the study instead targets generalization without significantly changing those shifts.Source: §3 · p. 3
• The study focuses on developer instruction-following, a behavior the authors say could in principle specify other desiderata such as helpfulness and harmlessness.Source: §3 · p. 4
• Reward models are classifiers that compare two responses; they are easier to train and evaluate than generative models, but the results do not necessarily extend to generative models.Source: §3 · p. 4

4.1. Metrics

• Elicitation measures source-tuned accuracy relative to target-tuned capability, while Differential Elicitation measures elicitation relative to a zero-shot baseline.The GENIES leaderboard averages Differential Elicitation and RMS Calibration Error across target distributions.Source: Figure 4 · p. 4, §4.1 · pp. 4–5
• Target-tuned capability is intended to bound tested interventions, but target-data tuning can learn new facts or skills and spurious target cues can make that bound less tight.Reference datasets are cleaned of spurious cues when needed; most target references are otherwise the same as their target datasets.Source: §4.1 · pp. 4–5
• RMS Calibration Error aggregates differences between average predicted probabilities and empirical accuracy across five equally spaced probability bins.Source: §4.1 · p. 5

4.2. Datasets

• Of 68 datasets, 24 were fully generated with ChatGPT, 11 were repurposed from existing datasets, and 33 combine generated and existing data; sampled human evaluation averaged 93.8% agreement.The agreement rate was 61% for change_my_view.Source: §4.2 · pp. 5–6

4.3. Extreme distribution shifts

• Extreme shifts include different skills and response-quality or difficulty changes, with dispreferred responses designed to be difficult to distinguish from instruction-following ones.Difficulty is measured by how many English-speaking people can evaluate a task; the quality shifts include responses with differing bug counts in code datasets.Source: §4.3 · p. 6
• Other extreme shifts test pretraining similarity, encoding, and context, including synthetic puzzles, redefined words, alternate languages or instruction formats, and changed task contexts.Source: §4.3 · p. 6

4.4. Probing distribution shifts

• Probing shifts test specific misgeneralization hypotheses, including human-like personas, spurious cues, and whether benign goal pursuit transfers to subversive power-seeking.The categories can overlap: correct Alpaca MMLU answers have low perplexity, which may act as a spurious cue in persona shifts.Source: §4.4 · pp. 6–7

4.5. GENIES: 15 curated distribution shifts

• GENIES curates 15 shifts chosen for diversity and because standard fine-tuning generalizes poorly across them.Source: §4.5 · p. 7, Table 3 · p. 7

5.1. reward models don’t learn to reliably evaluate ’instruction-following’

• Despite broad transfer, including 96% elicitation on counterfactual Python, crafted probes expose disobedience, accuracy and helpfulness biases, and human misconceptions.The models that generalize across extreme shifts are also prone to these failures, suggesting they learned features correlated with instruction-following rather than instruction-following itself.Source: §5.1 · pp. 7–8

5.2. reward models favor low-perplexity responses

• Generalization accuracy correlates with zero-shot accuracy (r = 0.7); policies agree on 70% of target examples, while source accuracy weakly predicts their target-mistake overlap (r = 0.2).The authors suggest low-perplexity responses may act as a cue, but find only weak evidence for an alternative account based on salient pretraining features.Source: §5.2 · pp. 8–9

5.3. Generalization improves with scale but only across ‘extreme’ distribution shifts

• Generalization improves with scale across extreme shifts, but not on average across probing shifts; at larger scales, the tuned and zero-shot policies make the same mistakes more often.Generalization accuracy improves more quickly than zero-shot accuracy on extreme shifts.Source: §5.3 · p. 9, Figure 7 · p. 10, Figure 8 · p. 10

5.4. Generalization of small models is moderately predictive of how larger models will generalize

• Generalization accuracy correlates across evaluated model sizes, but the models span only about one order of magnitude, leaving the reach of that relationship uncertain.The reported correlations cover all 69 shifts.Source: §5.4 · p. 10, Table 4 · p. 10

5.5. Evaluating interventions

• MMS achieves the highest score among seven interventions, but no method consistently dominates individual shifts and MMS is near-random or worse on 6 of 15 GENIES shifts.These results leave distinguishing instruction-following from conflated representations unresolved.Source: §5.5 · pp. 10–11, Table 5 · p. 11
• Prompt-tuning and LoRA have high in-distribution accuracy but comparatively low OOD generalization; representation-eliciting methods show a nearly one-to-one mapping between the two.MMS, LAT, and CRA are better calibrated than standard LoRA fine-tuning.Source: §5.5 · p. 10
• CCS and CRA achieve competitive GENIES generalization, but contrast pairs do not yield state-of-the-art performance.The authors suggest eliciting truthfulness representations as a possible path toward improving instruction-following generalization.Source: §5.5 · p. 11

6. Limitations and directions for further work

• The study evaluates reward models only, and their generalization may not transfer to generative models; reward models would need worst-case generalization to avoid vulnerabilities that generators might exploit.Average-case generalization may be insufficient when a generative model can exploit reward-model vulnerabilities.Source: §6 · p. 11
• Because LLMs can perform strongly by imitating human judgments on these tasks, the findings may not transfer to superhuman tasks.The authors suggest studying tasks where LLMs are already narrowly superhuman, such as next-word prediction.Source: §6 · p. 11
• Target-tuned capability may be misleading because fine-tuning can teach new circuits, target datasets can contain spurious cues, and accuracy does not establish an abstract representation of instruction-following.The authors suggest evaluating on a consistent set of diverse instructions rather than narrow target distributions.Source: §6 · p. 12

A. Auditing the quality of our datasets

• The audit randomly samples seven datasets and labels examples problematic when instructions are nonsensical or ill-posed, the best completion is ambiguous, or the response gives away the answer.One sampled code example included comments indicating where all the bugs were.Source: A. Auditing the quality of our datasets · p. 16
• Table 6 presents the dataset-audit results.Source: Table 6 · p. 16

B.1. Generalization is often (but not always) sensitive to the inclusion of a few training examples

• For most shifts, adding 6 target examples out of 650 can completely change generalization, but alpaca_short to alpaca_long continues to misgeneralize even with a 10% mixture.That shift remains sensitive to length even when length is only noisily correlated with source accuracy.Source: §B.1 · p. 16, Figure 9 · p. 16
• For alpaca_short to alpaca_long with a 5% mixture, training converges as loss approaches zero, though the authors suggest the model may memorize examples before abandoning the length heuristic.Source: §B.1 · p. 16, Figure 9 · p. 16

B.2. Fine-tuning on some datasets may create task-specific circuits

• 99% validation accuracy on ranking_logic_hard was achieved by LLaMA-7B after fine-tuning, on a task simple enough for task-specific circuits.The paper also reports suspiciously good fine-tuning performance on raven matrices and ranking_logic.Source: §B.2 · p. 17
• The example prompt supplies pairwise and rank-based density clues about materials A–G, then asks which material is second most dense.Source: §B.2 · p. 17

B.3. Differential Elicitation correlates strongly across most interventions

• Few-shot and zero-shot achieve nearly the same performance on most shifts (r=0.92), suggesting few-shot examples do little to elicit model capabilities.Source: §B.3 · p. 17
• Differential elicitation interventions correlate even more strongly than generalization accuracy across interventions, suggesting they use similar representations beyond perplexity.Source: §B.3 · p. 17, Figure 10 · p. 17

B.4. Zero-shot accuracy doesn’t correlation with generalization accuracy when source distributions violate the perplexity heuristic

• The zero-shot/generalization correlation is no longer perceivable when repurposed source distributions make zero-shot and source-tuned policies disagree on most examples; it is not negative either.The authors caution that narrow repurposed sources may make perplexity less salient; Figure 11 notes the correlation goes away when zero-shot source accuracy is worse than random.Source: §B.4 · p. 18, Figure 11 · p. 18

B.5. Models that perform well on extreme distribution shifts do poorly on distribution shifts that probe for specific misgeneralizations

• Code-to-US-history generalization reaches 90%, while models trained on code do poorly on truthful_qa and wrong_arc; the table covers LoRA with LLaMA-30B.Source: Table 7 · p. 18

B.6. No intervention consistently beats any other intervention on GENIES

• Table 8 compares differential elicitation across each GENIES shift, with larger values indicating better performance.Source: Table 8 · p. 19

C. Reward Model implementation

• The reward model replaces LLaMA-30B’s final unembedding layer with a linear scorer and compares responses to the same instruction using a sigmoid-transformed logit difference.Source: C. Reward Model implementation · p. 19

D. Tuning Intervention Details

• Post-hoc calibration maps average cosine similarity to probabilities using a logistic function fitted on source data; all interventions use four-bit quantization, estimated to degrade accuracy by about 5%.Source: D. Tuning Intervention Details · pp. 19–20

D.1. Mass Mean Shift (MMS)

• MMS forms an instruction-following direction from positive and negative demonstrations, compares response activation differences against it, and selects the top 48 attention heads by source classification accuracy.Source: §D.1 · p. 20

D.2. Linear Artificial Tomography (LAT)

• LAT uses hidden-state vectors from the top 16 layers rather than attention-head outputs, and tests two stimuli with directions obtained by subtracting positive and negative demonstrations.Source: §D.2 · p. 20

D.3. Contrastive Representation Arithmetic (CRA)

• CRA subtracts contrastive activation directions to suppress the Yes–No direction and amplify the True–False direction, using the 48 attention heads that best predict source labels.Source: §D.3 · pp. 20–21

D.4. Contrast Consistent Search (CCS)

• CCS searches for a truthfulness direction satisfying the negation probability axiom and uses randomized contrast pairs, unlike the supervised interventions.Source: §D.4 · p. 21

D.5. LoRA fine-tuning / D.6. Prompt-tuning

• LoRA uses QLoRA and selects the checkpoint with lowest evaluation loss; prompt-tuning follows the same checkpoint-selection schedule, with learning rates found by independent searches.LoRA models usually converged within 50 steps and then began overfitting.Source: §D.5 · p. 21, §D.6 · p. 21

D.7. Zero-shot Classification

• Zero-shot classification prefers the response with the highest average log probability given the prompt.Source: §D.7 · pp. 21–22

D.8. Few-shot Classification

• Few-shot classification uses the zero-shot policy with five source examples added to the prompt.Source: §D.8 · p. 22
• Zero-shot and few-shot accuracy have a Pearson correlation of 0.94 for LLaMA-30B, and the examples make very little difference.Source: §D.8 · p. 22

E. Defining instruction following

• Instruction-following is defined as reliably avoiding egregious instruction violations under any commonsense interpretation of their meaning.Most datasets pair clearly egregious responses with clearly appropriate ones; quality shifts instead compare failures of varying severity.Source: E. Defining instruction following · p. 22