PaperlayerView exact revision on arXiv

Source-linked AI summary

Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, David Scott Krueger

arXiv:2311.12786v2cs.LG

TL;DR

Fine-tuning is widely used, but how it changes capabilities learned during pretraining has been insufficiently studied. Using mechanistic analyses of fine-tuned models, the paper finds that capability changes are generally small and localized, and that further fine-tuning can revive a capability.

  • Problem

    Although fine-tuning is widely used, a precise account of how it changes pretrained capabilities—and how consequential those changes are—has been lacking.

    Source: §1 · pp. 1–2

  • Method

    The authors use mechanistic analyses, including pruning and probing, to investigate how fine-tuning changes capabilities in models with identifiable intermediate representations.

    Source: §3 · p. 3, §5.1 · p. 7

  • Results

    Fine-tuning generally makes small, localized changes that minimally transform capabilities, while further fine-tuning can revive capabilities in relatively few iterations.

    Source: §1 · p. 2, §5.1 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The authors emphasize the need for methods beyond fine-tuning to alter pretrained capabilities more substantially, particularly to delete unsafe capabilities.

    Source: §6 · p. 9

Abstract

from arXiv · show

Fine-tuning large pre-trained models has become the de facto strategy for developing both task-specific and general-purpose machine learning systems, including developing models that are safe to deploy. Despite its clear importance, there has been minimal work that explains how fine-tuning alters the underlying capabilities learned by a model during pretraining: does fine-tuning yield entirely novel capabilities or does it just modulate existing ones? We address this question empirically in synthetic, controlled settings where we can use mechanistic interpretability tools (e.g., network pruning and probing) to understand how the model's underlying capabilities are changing. We perform an extensive analysis of the effects of fine-tuning in these settings, and show that: (i) fine-tuning rarely alters the underlying model capabilities; (ii) a minimal transformation, which we call a 'wrapper', is typically learned on top of the underlying model capabilities, creating the illusion that they have been modified; and (iii) further fine-tuning on a task where such hidden capabilities are relevant leads to sample-efficient 'revival' of the capability, i.e., the model begins reusing these capability after only a few gradient steps. This indicates that practitioners can unintentionally remove a model's safety wrapper merely by fine-tuning it on a, e.g., superficially unrelated, downstream task. We additionally perform analysis on language models trained on the TinyStories dataset to support our claims in a more realistic setup.

1 INTRODUCTION

Fine-tuning is widely used to adapt pretrained models, including to mitigate undesirable behaviors, but its effects on underlying capabilities remain unclear. In controlled settings, the paper finds that fine-tuning typically adds localized wrappers rather than deleting capabilities, and that further training can revive them.

1 INTRODUCTION

  • Fine-tuning is widely used for downstream performance and to mitigate unsafe behaviors, including in alignment approaches such as RLHF (Ouyang et al., 2022; Go et al., 2023; Bai et al., 2022; Glaese et al., 2022).Pretrained LLMs can also exhibit sensitive, biased, or toxic behaviors (Weidinger et al., 2021; Lin et al., 2021; Jiang et al., 2021; Parrish et al., 2021; Zhou et al., 2021; Xu et al., 2021; Welbl et al., 2021).Source: §1 · p. 1
  • Despite hypotheses that fine-tuning selects and amplifies relevant capabilities while inhibiting others, a precise study of its effects was absent and the severity of the issue remained unclear.Jailbreaking studies were described as suggestive evidence (Wei et al., 2023; Zou et al., 2023; Shen et al., 2023; Deng et al., 2023; Liu et al., 2023b).Source: §1 · p. 2
  • The authors study fine-tuning in compiled Tracr models and PCFG-trained models, using mechanistic interpretability tools to examine how underlying capabilities change.Tracr encodes specific computational programs, while PCFG-trained models allow investigation of pretraining design choices.Source: §1 · p. 2
  • Fine-tuning generally changes pretrained capabilities through minimal transformations called wrappers, which can be localized enough for pruning or probing to reveal the underlying capability.The authors also report sample-efficient revival through further fine-tuning.Source: §1 · p. 2

2 RELATED WORK

Prior work studies fine-tuning methods and effects through theoretical, behavioral, and model-landscape analyses. This paper instead aims for mechanistic explanations, using synthetic tasks as a basis for concrete hypotheses rather than direct explanations of LLM success.

2 RELATED WORK

  • Fine-tuning spans instruction tuning, parameter-efficient methods, low-rank adaptation, and weight averaging; this paper focuses on continued training on smaller downstream data at a lower learning rate.The specified rate is often one to three orders of magnitude below the average pretraining rate.Source: §2 · pp. 2–3
  • Earlier analyses use simplifying assumptions, mode connectivity, behavioral evaluations, or loss-landscape dimensionality, whereas this work targets mechanistic changes in capabilities.Kotha et al. (2023) and Prakash et al. (2024) also claim fine-tuning is unlikely to alter capabilities.Source: §2 · p. 3
  • Synthetic-task interpretability work develops grounded hypotheses about model behavior, without necessarily claiming to explain LLM success directly.Examples include studies of entity knowledge, compositional data, and length generalization (Allen-Zhu & Li, 2023a;b; Okawa et al., 2023; Zhou et al., 2023).Source: §2 · p. 3

3 DEFINING OUR NOTION OF CAPABILITIES

The paper defines a capability as an input-to-output function recoverable by a linear readout on some intermediate layer, over a specified input subdomain. It distinguishes strong from weak relevance to fine-tuning and calls a localized transformation over a capability a wrapper.

3 DEFINING OUR NOTION OF CAPABILITIES

  • A capability is a function of task information recoverable by a linear readout at some model layer for inputs in a designated subdomain.The definition is idealized, though the authors say it is fairly representative for their procedural tasks.Source: §3 · p. 3
  • Capability relevance depends on whether the capability-derived transformation produces correct outputs across the fine-tuning distribution or only its empirical subset.The definition assumes the capability can be transformed during fine-tuning and that the fine-tuning dataset is much smaller than pretraining data.Source: §3 · p. 4
  • A weakly relevant capability may identify a spurious attribute, while a strongly relevant one extracts a causally relevant task feature.Figure 2 contrasts sentiment recognition as strongly relevant with negative-word recognition as weakly relevant because of spurious correlation.Source: §3 · p. 4, Figure 2 · p. 4
  • A wrapper is a localized transformation over a pretrained capability; removing or forgetting it can restore pretraining-task performance, which the authors call revival.Capabilities are restricted to an input domain: for example, an 8-bit adder's capability does not extend to inputs requiring greater precision.Source: §3 · pp. 3–4

4 BUILDING CAPABLE MODELS: TRACR AND PCFGS

The experiments use compiled Tracr models and capabilities learned from PCFG-generated data. Fine-tuning tasks include correlated spurious attributes, enabling the authors to study whether pretrained capabilities are relevant and reused.

4 BUILDING CAPABLE MODELS: TRACR AND PCFGS

  • The setup embeds a spurious attribute correlated with features extracted by a pretrained capability, which models how a learner might exploit an easy cue to reduce downstream loss.The attribute is randomly embedded in the fine-tuning data.Source: §4 · p. 4, Figure 3 · p. 4
  • Tracr compiles predefined computational primitives into transformers, which are then fine-tuned on tasks where the compiled capability may be weakly or strongly relevant.The main text focuses on one of two analyzed tasks.Source: §4 · p. 5
  • In the Tracr Counter task, the model is compiled to count one token and fine-tuned to count another, with a fixed difference in their counts creating the spurious correlation.The model can use its pretrained count plus that fixed offset on correlated samples, but this wrapped strategy performs poorly without the correlation.Source: §4 · p. 5
  • PCFG experiments train a model autoregressively on task-family and operand tokens, then fine-tune it on a task that may or may not have appeared in pretraining.Operand sampling probability controls whether the model is likely to learn the relevant capability; the main text focuses on counting tasks.Source: §4 · p. 5

5 EXPERIMENTS: MECHANISTIC ANALYSIS OF FINE-TUNING

Across controlled Tracr and PCFG experiments, fine-tuning often leaves pretraining capabilities available behind a localized wrapper, and reverse fine-tuning can revive them. TinyStories experiments likewise find that fine-tuned models retain and can recover a feature targeted for removal.

5 EXPERIMENTS: MECHANISTIC ANALYSIS OF FINE-TUNING

  • The main PCFG counter setup pretrains counting of OPT = {a, b, c}, then fine-tunes counting of OFT = b under varied priors, spurious correlations, and learning rates.The paper treats recovery in a few reverse-fine-tuning steps as evidence that fine-tuning did not meaningfully alter the capability.Source: §5 · p. 6
  • The analysis combines pruning, attention visualization, probing, and reverse fine-tuning to test whether capabilities persist or can be revived.The authors use multiple interpretability tools because each can have individual pitfalls.Source: §5 · pp. 5–6, Figure 4 · p. 6

5.1 RESULTS

  • With a low OFT pretraining prior, downstream performance depends on the spurious correlation; higher priors help models perform regardless of that correlation and mitigate pretraining-task loss.The pretraining task is most affected when the target token had a low sampling prior.Source: Figure 5 · p. 6, Figure 6 · p. 7, §5.1 · p. 7
  • Pruning 5–15 neurons restores pretraining-task performance while reducing fine-tuning performance, supporting a localized wrapper over weakly relevant capabilities.Probes also recover pretraining counts from intermediate outputs after fine-tuning.Source: Figure 7 · p. 7, §5.1 · p. 7
  • Strongly relevant capabilities revive under reverse fine-tuning within 0.1–1K iterations, while weakly relevant capabilities take 3K; the baseline reaches perfect accuracy at 4.5K with a larger learning rate.The comparison is reported with CTe set to 0.Source: Figure 9 · p. 8, §5.1 · p. 8

5.2 VALIDATING OUR HYPOTHESES ON TINYSTORIES

  • TinyStories models retain probe-detectable forbidden-feature information after fine-tuning, and reverse fine-tuning recovers story generation with that feature across protocols.Reverse-fine-tuning loss falls quickly in 30–300 iterations, compared with baselines that do not reach the same loss.Source: Figure 11 · p. 9, §5.2 · p. 9

6 CONCLUSION

Fine-tuning generally makes small, localized changes that minimally transform pretrained capabilities. TinyStories models also relearned the twist capability more sample-efficiently than a model without that capability in pretraining.

6 CONCLUSION

  • TinyStories models relearned to generate stories with a twist more sample-efficiently than a control model pretrained without twists and then fine-tuned to generate them.This comparison held across the reported learning rates and fine-tuning protocols.Source: Table 1 · p. 9
  • Fine-tuning generally alters pretrained models through small, localized changes that only minimally transform their capabilities.The authors use mechanistic interpretability tools and reverse fine-tuning to analyze these changes.Source: §6 · p. 9
  • The results motivate future work on fine-tuning in larger, more realistic models and methods beyond fine-tuning to alter capabilities, particularly deleting unsafe ones.Source: §6 · p. 9

AUTHORS’ CONTRIBUTIONS

The authors describe their individual contributions and organize the appendix around dataset and training details, interpretability protocols, and additional results.

AUTHORS’ CONTRIBUTIONS

  • ESL led the project direction and main-paper writing; SJ led experiments and appendix writing, while RK proposed and ran the TinyStories experiments.The authors also report joint work on setup design, hypotheses, analysis, and figures.Source: AUTHORS’ CONTRIBUTIONS · p. 10

A ORGANIZATION OF APPENDIX

  • The appendix documents the Tracr, PCFG, and TinyStories datasets, model training details, and protocols for attention maps, probing, pruning, and reverse fine-tuning.It also summarizes the paper’s notation.Source: A ORGANIZATION OF APPENDIX · p. 16, Table 2 · p. 16
  • Appendix Section E examines practical contexts, including synthetic jailbreaking, fine-tuning data fractions, reverse fine-tuning, and attempts to remove capabilities.Source: A ORGANIZATION OF APPENDIX · p. 16
  • Additional appendix results cover TinyStories, Tracr counter and max-element tasks, and PCFG counting and index-of-occurrence tasks.Source: A ORGANIZATION OF APPENDIX · p. 16

B ADDITIONAL DETAILS ON DATASETS

The paper uses compiled Tracr models, procedurally generated PCFG tasks, and TinyStories Instruct to study fine-tuning in controlled settings.

B ADDITIONAL DETAILS ON DATASETS

B.1 TRACR DETAILS

  • Tracr experiments fine-tune compiled counting and max-identifier capabilities to use a different target token under a fixed-offset spurious correlation.Inputs are sampled from task-specific vocabularies; counter-task tokens are sampled with replacement and max-task tokens without replacement.Source: §B.1 · p. 17

B.2 PCFG

  • The PCFG data uses grammar-generated strings, subsamples 250 tokens, and formats examples with task, operand, text, and answer markers.The production rules follow Allen-Zhu & Li (2023c).Source: §B.2 · p. 18, Figure 14 · p. 18
  • The five PCFG pretraining tasks include counting tokens or token compositions, locating occurrences or composition occurrences, and retrieving a token at an index.Source: §B.2 · pp. 18–19
  • PCFG pretraining covers five tasks, while fine-tuning focuses on counting or index of occurrence for a single operand; the index task uses the sixth occurrence.Source: §B.2 · p. 19
  • PCFG experiments vary operand relevance during pretraining and control the fine-tuning correlation while keeping class distributions fairly spread across classes.The reported class distributions for CTr = 1 and CTr = 0 meet the stated distribution-shift and class-spread requirements.Source: §B.2 · p. 19, Figure 16 · p. 20

B.3 TINYSTORIES INSTRUCT

  • TinyStories experiments use the Instruct dataset and create fine-tuning datasets by altering the original data in several ways.The alterations are detailed in Appendix F.Source: §B.3 · p. 21

C DETAILS ON TRAINING AND EVALUATION

The paper specifies training procedures for Tracr and PCFG models and describes attention maps, pruning, probing, and reverse fine-tuning as interpretability tools.

C.1 TRACR

  • Tracr models have three blocks, each with one attention layer and two MLP layers, and are fine-tuned with momentum SGD for 10K iterations.Small Gaussian noise densifies sparse weights, and evaluation uses test sets with and without the spurious correlation.Source: §C.1 · p. 21

C.2 PCFG

  • PCFG experiments use a roughly 3-million-parameter, six-block minGPT model pretrained from scratch with next-token prediction on five tasks.Pretraining varies operand sampling distributions and runs for 10K, 50K, 100K, or 200K iterations.Source: §C.2 · p. 21
  • PCFG fine-tuning is supervised on one task-and-operand pair for 10K iterations, with AdamW and learning rates from 10^-4 to 10^-6.Evaluation tests with and without the spurious correlation.Source: §C.2 · p. 22

D MECHANISTIC INTERPRETABILITY TOOLS SETUP

  • Attention-map rows identify attending tokens and columns identify attended tokens; the maps can indicate task-relevant focus, such as sorting and reading in max identification.Source: D MECHANISTIC INTERPRETABILITY TOOLS… · p. 22
  • Single-step pruning removes weights or neurons with the largest weight-gradient products, using gradients from the loss for the capability to revive.The experiments prune weights in Tracr and neurons in PCFG.Source: D MECHANISTIC INTERPRETABILITY TOOLS… · p. 22
  • Linear probes are trained on each PCFG block’s residual output to test for a capability; reverse fine-tuning uses a smaller learning rate than the original Tracr fine-tuning.The smaller rate is intended to make successful revival stronger evidence that the capability was not forgotten or removed.Source: D MECHANISTIC INTERPRETABILITY TOOLS… · pp. 22–23

E.1 FINE-TUNING IN PRESENCE OF SOME PRE-TRAINING DATA

The section tests whether retaining some pretraining data changes fine-tuning outcomes, then examines whether hidden pretraining capability can support jailbreaking. Across the reported sampling settings, fine-tuning-task gains are similar, while a sufficiently high pretraining prior for the relevant capability enables recovery through alternate task-family tokens.

E.1 FINE-TUNING IN PRESENCE OF SOME PRE-TRAINING DATA

  • Similar fine-tuning-task gains persist across the tested OPT sampling probabilities, with both low- and high-prior setups reported.The experiments mix pretraining and fine-tuning data at 50%/50%, 10%/90%, and 0.1%/99.9%.Source: Figure 18 · p. 24, Figure 20 · p. 24, §E.1 · p. 24
  • Low sampling probability for the pretraining target OPT produces a higher loss on the pretraining task during fine-tuning.Source: Figure 19 · p. 24

E.2 JAILBREAKING ANALYSIS

  • With a sufficiently high pretraining prior for counting b, alternate tokens TJ1 and TJ2 enable perfect counting of a after fine-tuning on b with TNJ.Using TNJ for evaluation does not recover counting a behaviorally, although probes find its count encoded around middle layers.Source: §E.2 · p. 26, Figure 24 · p. 26
  • Probe results show that the count of a remains linearly readable from middle-layer outputs even when TNJ is used after fine-tuning.The probing analysis indicates the pretraining capability persists despite its impaired behavioral expression under TNJ.Source: §E.2 · p. 26, Figure 25 · p. 26

E.3 SAMPLE EFFICIENCY ANALYSIS FOR REVERSE FINE-TUNING

Reverse fine-tuning recovers pretraining-task performance more sample-efficiently than the scratch-based baseline in PCFG and compiled Tracr models. The section also details an adversarial forgetting protocol and the TinyStories experiments, including their training setup and evaluation measures.

E.3 SAMPLE EFFICIENCY ANALYSIS FOR REVERSE FINE-TUNING

  • Reverse fine-tuning recovers the pretraining task in substantially fewer iterations than Scr. + FT for PCFG and Tracr models.For Tracr, even an extremely small learning rate suffices; the Tracr experiments do not sweep CTr because compiled capabilities prevent meaningful control of that correlation.Source: §E.3 · p. 27, Figure 26 · p. 27, Figure 27 · p. 27

E.4 REVERSE FINE-TUNING A MORE SAFETY-ORIENTED FINE-TUNING PROTOCOL

  • randFT trains models to produce incorrect outputs for pretraining-capability inputs, and reFT then trains them to produce the correct outputs.Reverse fine-tuning of randFT models is similarly sample-efficient to standard reFT and more sample-efficient than Scr. and Scr. + FT; probes also recover information relevant to the pretraining task.Source: §E.4 · p. 28, Figure 29 · p. 28, Figure 30 · p. 29

F DETAILS AND RESULTS ON TINYSTORIES EXPERIMENTS

F.1 MODEL TRAINING

  • TinyStories models are pretrained to generate instructed stories, then fine-tuned with Twist filtering, instruction mix-and-match, or randomization protocols.For recovery tests, reFT uses Twist stories and is compared with a model pretrained without Twist stories and then fine-tuned on them.Source: §F.1 · p. 30

F.2 EVALUATION METRICS

  • TinyStories evaluation measures Twist-story loss, generation scores, and probe accuracy for story-feature classification across model layers.The generation score is the proportion of generated stories classified as containing Twists; probes classify feature presence from internal activations.Source: §F.2 · pp. 30–31

F.3 RESULTS

In TinyStories, fine-tuning suppresses Twist generation behaviorally, but the capability is recovered efficiently and remains largely represented in activations. Additional Tracr results show that fine-tuning and revival depend on learning rate and on which computations are altered.

F.3 RESULTS

  • Fine-tuning raises Twist-story loss and drives generation scores close to zero, indicating behavioral suppression of Twist generation.The generation scores are noisy, and the protocols vary in learning rate.Source: §F.3 · p. 31, Figure 31 · p. 32, Figure 32 · p. 32
  • Fine-tuned TinyStories models recover Twist generation more sample-efficiently than the no-Twist-pretrained control and converge to lower loss.Both the loss and generation-score recovery results support this pattern.Source: §F.3 · p. 31, Figure 33 · p. 32, Figure 34 · p. 33
  • Twist probe accuracy falls only slightly after fine-tuning relative to Twist-pretrained models, while remaining well above the no-Twist-pretrained control.Source: §F.3 · p. 31, Figure 35 · p. 33
  • For Foreshadowing, Moral Value, and Bad Ending, probe-accuracy differences between fine-tuned and both pretrained controls are negligible.The authors attribute the Twist probe difference to the feature targeted for deletion.Source: §F.3 · p. 31, Figure 36 · p. 34, Figure 37 · p. 34

G ADDITIONAL TRACR RESULTS

  • On Tracr Counter, ηS fails to learn attending to b without correlations, while ηM learns it; with spurious correlations, the model can instead learn the correlation.The reported attention maps also show that ηM learns to attend to b in the absence of spurious correlations.Source: G ADDITIONAL TRACR RESULTS · p. 36, Figure 40 · p. 36
  • Tracr Counter capability revival succeeds after ηM fine-tuning with ηVS, but not after ηL fine-tuning; the latter hampers the underlying capability.Source: Figure 39 · p. 36

G.1 BEHAVIORAL RESULTS ON FINE-TUNING

Additional Tracr results show that fine-tuning outcomes on the Counter and max-identifier tasks depend on learning rate and spurious correlations. They also distinguish changes to a task’s readout from disruption of its underlying computation.

G.1 BEHAVIORAL RESULTS ON FINE-TUNING

  • On Counter, ηM learns to count b’s, while ηS at 10−3 does not learn to attend to b’s without correlations; with spurious correlations, the model learns the correlation instead.After ηM fine-tuning, the original capability can be revived with ηvs at 10−4, whereas revival is not possible after ηL at 10−1.Source: §G.1 · p. 36, Figure 41 · p. 37
  • Pruning one weight from a Tracr model compiled to count a’s boosts its counting-a accuracy by over 60% when fine-tuned on correlated data.Source: §G.1 · p. 36
  • On max-identifier, ηVS changes the reading function without disturbing sorting, whereas other tested learning rates distort sorting and yield poor fine-tuning performance.The Tracr attention maps locate sorting in Block 0 and reading in Block 2.Source: §G.1 · p. 37
  • In the Counter task, fine-tuning with ηS does not learn the new b-counting capability, while increasing the learning rate enables attention to b’s and leaves attention to a’s intact.Source: Figure 42 · p. 37

G.2 COUNTER RESULTS

Across the Tracr and PCFG results, fine-tuning can leave pretrained capabilities available beneath a localized wrapper, while learning rates and spurious correlations affect whether capabilities are preserved or reused. Reverse fine-tuning can revive capabilities, and relevant pretrained capabilities influence fine-tuning convergence.

G.2 COUNTER RESULTS

  • In the Tracr counter task, ηM and ηS leave the counting-a capability visible in Block-1 activations despite changed output channels.The reported activations indicate the capability remains mechanistically present after fine-tuning.Source: §G.2 · p. 38, Figure 46 · p. 41

Acc. OPT

  • Pruning localizes the counter-task wrapper to a few weights or neurons; pruning a weight still leaves multiple predicted classes, consistent with wrapper removal.Higher CTr is associated with wrapper learning; the reported wrapper is learned with ηM and ηS.Source: §G.2 · p. 39, Figure 55 · p. 46, Figure 57 · p. 47

G.3 MAX IDENTIFIER RESULTS

  • For max identification, ηVS preserves Tracr sorting and supports attention to the correct output token, whereas ηL, ηM, and ηS distort the programmed capability.The sorting pattern is visible in Block-1 activations.Source: Figure 50 · p. 43, Figure 52 · p. 44, Figure 53 · p. 45

H ADDITIONAL PCFG RESULTS

  • With weakly relevant capability, spurious correlations increase fine-tuning convergence time, although learning the spurious correlation is quick; ηS fails to learn the task in this setting.The convergence-time difference is smaller when strongly relevant capability is present.Source: Figure 62 · p. 53
  • Reverse fine-tuning revives pretrained capability across CTr values, including with a small learning rate after large-learning-rate fine-tuning.Revival is also observed across different pretraining iteration counts.Source: Figure 76 · p. 60, Figure 78 · p. 61

H.1 PRUNING ANALYSIS

The pruning and probing analyses find that fine-tuning effects depend on learning rate, capability relevance, and spurious correlations. Pruning can partly restore a pretrained capability under a specified setting, while probing shows different effects on the pretrained and fine-tuning tasks.

H.1 PRUNING ANALYSIS

  • Pruning analysis finds wrapper learning under ηL only at CTe = 1, under ηM at smaller CTe values, and under ηS across all CTe values and samples.The reported wrapper conditions are for the counter task.Source: Figure 79 · p. 62
  • With ηL, fine-tuning a strongly relevant capability model on uncorrelated data can create an inhibitor; pruning partly revives the capability, chiefly in the 200K-iteration model.The inhibitor is reported for the 200K-iteration pretrained model, and the partial revival condition is strongly relevant capability with CTe = 0.Source: Figure 86 · p. 66

H.2 PROBING ANALYSIS

  • In probing, higher CTr increases counting-a accuracy but decreases counting-b accuracy across weakly and strongly relevant capability models.The reported directions are from the 200K- and 50K-iteration counter-task analyses, respectively.Source: Figure 95 · p. 70, Figure 97 · p. 71
Loading 2311.12786v2…