PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Interpret Weight Differences in Language Models

Avichal Goel, Yoon Kim, Nir Shavit, Tony T. Wang

arXiv:2510.05092v4cs.LGcs.AIcs.CL

TL;DR

Finetuning changes model weights, but those changes are not generally interpretable, making it difficult to understand the resulting behavioral differences. The paper introduces DIT, which trains adapters on synthetic labeled weight diffs to describe finetuning-induced changes. In two proof-of-concept settings, DIT describes hidden behaviors and finetuned knowledge, though its adapters have essentially no cross-behavior generalization.

  • Problem

    Methods for comprehensively understanding behavioral changes induced by weight diffs are lacking, despite their relevance to finetuned-model reliability, safety, and transparency.

    Source: §1 · p. 1

  • Method

    DIT trains a low-rank adapter on synthetic labeled weight diffs so a finetuned model can answer natural-language questions about how it differs from its base model.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    DIT describes weight diffs in two proof-of-concept settings—hidden behaviors and new knowledge—and outperforms the reported baseline methods at summarizing finetuning knowledge.

    Source: §1 · p. 1, §5.2 · p. 7

  • Takeaways & Limitations

    DIT can identify trigger-gated hidden behaviors that are hard to detect with black-box methods, and can summarize finetuning knowledge.

    Source: §1 · p. 2, §5.2 · p. 7

    DIT-adapters have essentially no cross-behavior generalization between hidden-topic and news-summary weight diffs.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

Finetuning (pretrained) language models is a standard approach for updating their internal parametric knowledge and specializing them to new tasks and domains. However, the corresponding model weight changes ("weight diffs") are not generally interpretable. While inspecting the finetuning dataset can give a sense of how the model might have changed, these datasets are often not publicly available or are too large to work with directly. Towards the goal of comprehensively understanding weight diffs in natural language, we introduce Diff Interpretation Tuning (DIT), a method that trains models to describe their own finetuning-induced modifications. Our approach uses synthetic, labeled weight diffs to train a DIT-adapter, which can be applied to a compatible finetuned model to make it describe how it has changed. We demonstrate in two proof-of-concept settings (reporting hidden behaviors and summarizing finetuned knowledge) that our method enables models to describe their finetuning-induced modifications using accurate natural language descriptions.

1 INTRODUCTION

The paper addresses the difficulty of comprehensively understanding behavioral changes encoded by finetuning weight diffs. It introduces DIT, which trains adapters to describe those changes, and reports proof-of-concept results alongside a limitation in cross-setting generalization.

1 INTRODUCTION

  • Methods for comprehensively understanding behavioral changes induced by weight diffs are lacking, challenging finetuned-model reliability, safety, and transparency.Prior work identifies structural regularities in weight diffs, but the paper says these do not yet provide comprehensive understanding.Source: §1 · p. 1
  • DIT trains a low-rank adapter on synthetic labeled weight diffs to map weight-space changes to natural-language descriptions of a finetuned model’s behavioral changes.The adapter is applied to a finetuned model to make it self-describing.Source: §1 · p. 1
  • DIT describes weight diffs in two proof-of-concept settings: uncovering discrete hidden behaviors and summarizing new knowledge.It can identify trigger-gated hidden behaviors that are hard to detect with black-box methods.Source: §1 · pp. 1–2
  • A DIT-adapter trained for one setting is essentially useless for the other, limiting cross-setting generalization.The authors hypothesize that scaling up DIT training could improve generalization.Source: §1 · p. 2

2 PROBLEM STATEMENT

The paper frames comprehensive interpretation of weight diffs as answering natural-language questions about differences between a base model and its finetuned version. This framing supports evaluation with known answers, while the paper’s experiments focus on simple diffs answerable by one question.

2 PROBLEM STATEMENT

  • WEIGHTDIFFQA asks a model to answer a natural-language question about differences between a base model and its finetuned version.The paper operationalizes understanding as accurate answers and comprehensiveness as the ability to answer arbitrary questions.Source: §2 · p. 2
  • Synthetic model pairs with specified answers let WEIGHTDIFFQA methods be tested against known ground truth.The paper describes WEIGHTDIFFQA as the inverse problem of constructing model pairs with known natural-language relationships.Source: §2 · p. 2
  • WEIGHTDIFFQA methods can apply to detecting data poisoning, backdoors, and trojans even when finetuning data are undisclosed or prohibitively large to analyze.Source: §2 · p. 2
  • The experiments focus on simple weight diffs whose comprehensive interpretation can be captured by one question, and the authors say broader generalization and scaling remain essential for meaningful real-world results.Source: §2 · p. 2

3 DIFF INTERPRETATION TUNING (DIT)

DIT trains a LoRA adapter to answer questions about the differences between a base model and its finetuned version. Its training examples are generated by constructing finetunes with known question-answer behavior and pairing them with the corresponding answers.

3 DIFF INTERPRETATION TUNING (DIT)

  • DIT trains a LoRA adapter that, when applied to a finetuned model, enables answers about differences from its base model.The training set consists of finetuned-model, question, and answer triplets.Source: §3 · p. 2
  • The adapter is trained to minimize supervised finetuning loss over completion tokens conditioned on the prompt.The paper’s intuition is that low training loss across a sufficiently wide distribution of triplets should support accurate answers on held-out weight diffs.Source: §3 · p. 3
  • In this paper, the same fixed question is used at both training and test time.Source: §3 · p. 3

3.1 GENERATING TRAINING DATA FOR DIFF INTERPRETATION TUNING

  • Training data are generated by prompting an off-the-shelf language model to simulate each target behavior, finetuning the base model on the resulting dataset, and pairing each finetune with its question and answer.The generated labeled finetunes form the dataset used to train DIT.Source: §3.1 · p. 3

4 EVALUATION #1: REPORTING HIDDEN BEHAVIORS IN WEIGHT DIFFS

The hidden-behavior evaluation tests whether DIT can identify topics encoded behind secret triggers. DIT beats black-box probing baselines, generalizes across weight-diff ranks and unseen trigger forms, with weaker results on some full-parameter diffs.

4 EVALUATION #1: REPORTING HIDDEN BEHAVIORS IN WEIGHT DIFFS

  • A Qwen3-8B example references the NBA only when the specified SEP-code trigger begins the user message, and otherwise responds normally.The evaluation asks DIT to report the hidden topic when prompted about the model's training.Source: Figure 4.1 · p. 4, §4 · p. 4
  • The evaluation trains on 4,660 of 4,760 synthetic rank-1 LoRA diffs, each encoding a hidden topic gated by a random three-digit SEP-code prefix.The rank-16 DIT-adapter is tested on 100 held-out diffs.Source: §4 · p. 4

4.1 PERFORMANCE OF DIT AT REPORTING HIDDEN BEHAVIORS

  • DIT strongly outperforms the base-question and 20-question baselines and performs comparably to the trigger-assisted 20 Questions roofline.The methods are scored by an LLM judge on a five-point scale; Figure 4.2 also shows performance rising with training-data amount.Source: §4.1 · p. 4, Figure 4.2 · p. 5

4.2 GENERALIZATION ACROSS LORA RANKS

  • Rank-1-trained DIT-adapters generalize across higher-rank LoRA diffs, with nontrivial but degraded full-parameter results for Qwen3-4B and strong results for Gemma3-4B.The rank-generalization outcomes are shown in Figure 4.3.Source: §4.2 · p. 5, Figure 4.3 · p. 5

4.3 GENERALIZATION TO OUT-OF-DISTRIBUTION TRIGGERS

  • DIT maintains high, SEP-trigger-comparable performance on two unseen zero-width-space trigger placements.The placements are at the query start and at a random query position; the authors interpret the result as evidence against overfitting to trigger syntax.Source: §4.3 · p. 6, Table 4.4 · p. 6

5 EVALUATION #2: SUMMARIZING FINETUNED KNOWLEDGE IN WEIGHT

This evaluation tests whether DIT can summarize distributed knowledge learned from finetuning data. DIT outperforms the comparison methods at recovering a latent headline from weight diffs trained on related news stories.

5.1 NEWS SUMMARIZATION TASK

  • For each target headline, the task generates 100 thematically related news stories that distribute its content without reproducing the headline as a sentence.A rank-8 LoRA diff is trained on the stories using the fixed prompt “Please generate a short news story.”Source: §5.1 · p. 6
  • The training set contains 8,760 of 8,860 synthetic story diffs, while 100 randomly sampled diffs form the held-out test set.A rank-16 DIT-adapter is trained to output the latent headline when asked what headline the model was trained on.Source: §5.1 · p. 6

5.2 DETAILS OF EVALUATION

  • The evaluation compares DIT with a direct headline question, 20 interrogation questions, and headline recovery from a generated story.In the generated-story baseline, o4-mini summarizes the model's generated story into a headline.Source: §5.2 · p. 7
  • DIT outperforms all evaluated baselines at summarizing finetuning knowledge, including a generated-story method whose extra details can enter the guessed headline.The generated stories include details not captured in the target headline, which o4-mini sometimes incorporates into its guess.Source: §5.2 · p. 7

6 LIMITATIONS AND FUTURE RESEARCH

The section finds little cross-behavior generalization and difficulty recovering hidden-behavior triggers, then outlines data, architecture, and reasoning challenges for scaling DIT. It also identifies mechanistic understanding and applying DIT directly to base models as open questions.

6.1 GENERALIZATION TO WEIGHT DIFFS ENCODING DIFFERENT BEHAVIORS

  • Scaling DIT for complex in-the-wild weight diffs requires diverse training data, architectures with lossless access to model internals, and extended introspective reasoning.The current architecture leaves the original model inaccessible, and the adapter could interfere with the weight diff being interpreted; extended reasoning may require RL-based training.Source: §6.1 · p. 8
  • Whether scaled-up DIT-adapters can answer questions about the base model itself, rather than only weight diffs applied to it, remains open.Source: §6.1 · p. 8
  • Cross-behavior performance is close to the naive “Base Question” baseline, despite strong in-distribution performance.The authors attribute this result to the small size and lack of diversity of DIT training data, and hypothesize that more diverse training may improve generalization.Source: Table 6.1 · p. 8, §6.1 · p. 8

6.2 DIFFICULTY REPORTING TRIGGERS OF HIDDEN BEHAVIORS

  • DIT recovered the correct SEP trigger in 0 of 100 test samples across each of the five models, despite identifying hidden behaviors in earlier experiments.The authors hypothesize trigger inversion may be harder than discovering a hidden behavior, but explicitly say this explanation is speculative.Source: §6.2 · pp. 8–9

6.3 HOW DOES INTROSPECTION WORK MECHANISTICALLY?

  • The authors identify understanding the internal mechanisms that let DIT-adapters work as a direction that could inform improved introspection architectures.They report preliminary visualizations and analysis in Appendix H.Source: §6.3 · p. 9

7 RELATED WORK

The related work situates DIT among introspection and model-diffing research, and the appendices detail the hidden-behavior evaluation, its synthetic data and training, black-box baselines, and scoring.

7 RELATED WORK

  • DIT extends introspection work by training models to report how weight diffs alter behavior, complementing methods that characterize model changes at the feature or circuit level.Related approaches include activation interpretation and shared-feature techniques; DIT instead synthesizes changes into natural-language descriptions.Source: §7 · pp. 9–10
  • DIT is framed as complementary to trigger-inversion methods: it can describe backdoors with unknown targets, but struggles to invert triggers.The authors also motivate synthetic evaluation because known properties provide ground truth for assessing interpretability methods.Source: §7 · p. 10

B API PROVIDER MODEL VERSIONS

C.1 TOPIC GENERATION

  • The hidden-topic evaluation uses a held-out set of 100 models, drawn from 4,760 sampled topics out of 7,930 generated topics.The 4,660 remaining sampled topics were used to finetune training-set models for the diff interpreter.Source: §C.1 · p. 16

C.2 FINETUNING MODELS TO HAVE HIDDEN BEHAVIORS

  • Hidden behaviors are trained to activate for the true trigger while preserving base-model behavior for incorrect or absent triggers.The training samples cover the true trigger, an incorrect trigger, and no trigger; the loss uses type-specific multipliers.Source: §C.2 · pp. 16–17

C.3 GENERATING LORA FINETUNES

  • The hidden-topic finetunes use rank-1 LoRA on 270 topic-specific question-answer pairs, generated by transforming base questions with model-produced answers.The held-out and different-rank test LoRAs use the same hyperparameters, while full-parameter finetunes use a reduced learning rate.Source: §C.3 · p. 17

C.4 BLACK-BOX PROBING DETAILS

  • The black-box baseline asks the finetuned model 20 unused questions, then has o4-mini infer the hidden topic from the question-answer pairs.A second baseline prepends the trigger phrase to those questions to demonstrate responses when the trigger is present.Source: §C.4 · pp. 17–18

C.5 SCORING PREDICTED TOPICS

  • Predicted topics are evaluated on test models by an o4-mini judge that rates similarity to the ground-truth topic from 1 to 5.The authors use a thinking model to improve scoring quality.Source: §C.5 · p. 18

C.6 AGENT BASELINE

The agent baseline does not effectively recover hidden topics, while giving it the trigger yields a strong roofline. More query opportunities produce little improvement for normal agents.

C.6 AGENT BASELINE

  • The baseline agent does not effectively recover hidden topics, whereas the trigger-informed roofline performs well.The authors interpret this contrast as evidence that the agent is ineffective at recovering the hidden topic.Source: §C.6 · p. 20, Figure C.3 · p. 20
  • The agent underperforms the 20-questions baseline, which the authors attribute to its less similar questions and hypothesize may reflect harder triggering farther from training inputs.The 20-questions baseline lacks access to the original model; the proposed explanation is a hypothesis about KL-regularized backdoor training.Source: §C.6 · p. 20
  • The agent still does not reach a perfect score when given the trigger phrase, partly because a fixed three-digit suffix may fail to activate some backdoors.The authors note that imperfect backdoor training may contribute to this failure.Source: §C.6 · p. 20
  • Normal agents show a weak scaling trend and perform poorly even with 100 queries on Qwen3-4B weight diffs.Roofline agents rapidly saturate; the plotted query-budget comparison is in Figure C.4.Source: §C.6 · p. 21, Figure C.4 · p. 21

D DETAILS OF EVALUATION #2 (SUMMARIZING FINETUNED KNOWLEDGE)

This appendix describes the synthetic news-story evaluation, its summary-generation baselines and scoring, and the DIT adapter rationale and parallel-training implementation.

D.1 NEWS HEADLINE GENERATION; D.2 NEWS STORY GENERATION

  • The news evaluation builds on generated headlines and stories, with each headline paired with 100 stories written in varied roles.The 8,860 headlines were generated from broad news categories, and stories spread headline content across sentences to require synthesis.Source: §D.2 · p. 22, §D.1 · p. 22

D.3 DETAILS OF “GENERATED STORY” BASELINE

  • The Generated Story baseline has the finetuned model produce a story using its training prompt, then asks o4-mini to summarize it.The summary prompt requests a one-sentence output matching reference-summary style and grammar.Source: §D.3 · pp. 22–23

D.4 DETAILS OF “20 QUESTIONS” BASELINE

  • The 20 Questions baseline asks the finetuned model 20 prompts about its latest training data and sends the answers with the questions to o4-mini for a predicted summary.The same question-answer rendering template is used for the Base Question baseline.Source: §D.4 · p. 23

D.5 NEWS STORY SUMMARY SCORING

  • Predicted news summaries are scored against ground-truth summaries by an o4-mini judge rating similarity from 1 to 5.Table D.1 is described as comparing ground-truth summaries with diff-interpreter outputs on Gemma3-4B.Source: §D.5 · p. 24, Table D.1 · p. 25

E MOTIVATING DIFF INTERPRETATION TUNING

  • DIT uses a LoRA adapter trained to answer natural-language questions about the difference between a base model and its finetuned version.The appendix motivates this design by choosing a finetuned model as interpreter and applying the diff directly, avoiding a special encoder for the diff.Source: E MOTIVATING DIFF INTERPRETATION TUNING · p. 26
  • Commutativity lets the trained adapter be applied to the finetuned model to describe its changes, rather than only interpreting a diff applied to the interpreter.The paper presents these as equivalent views of the method when the weight diffs and adapters commute.Source: E MOTIVATING DIFF INTERPRETATION TUNING · p. 26

F.1 MEMORY EFFICIENCY AND PARALLEL PROCESSING

  • Parallel training processes batches of task-specific LoRAs while each input is routed only through its corresponding LoRA.The MultiTaskLoRALinear layer computes each task output using the shared base weights plus that task's low-rank matrices.Source: §F.1 · p. 27

F.2 TRAINING AND COMPUTE STATISTICS

The implementation trained more than 40,000 weight diffs, and the appendix reports amortized per-diff training times for two tasks while excluding one-time data-generation costs.

F.2 TRAINING AND COMPUTE STATISTICS

  • The parallel implementation enabled training more than 40,000 weight diffs for the experiments.The appendix reports amortized per-diff training times for the hidden-topic and news-summary settings.Source: §F.2 · p. 27
  • Reported training costs exclude finetuning-data generation, including about 200 million tokens and approximately $120 for the news-summary task.The authors exclude this expense because the generated data can be reused across models and incurs a one-time cost.Source: §F.2 · p. 27
  • Table F.1 reports amortized seconds per hidden-topic LoRA diff on one H100, using rank 1 and 270 question-answer pairs.The caption attributes longer training times than the news-summary task to more samples and a more complex loss.Source: Table F.1 · p. 27
  • Table F.2 reports amortized seconds per news-summary LoRA diff on one H100, using rank 8 and 100 news stories.Source: Table F.2 · p. 27

G DETAILED EVALUATION STATISTICS

This appendix reports how test samples are distributed across similarity scores for results from the hidden-topic and news-summarization evaluations, along with other cited results.

G DETAILED EVALUATION STATISTICS

  • The appendix breaks down the fraction of test samples receiving each similarity score from 1 to 5 for the results listed in the section.The breakdowns cover Figures 4.2, 4.3, 5.2, and Table 6.1, as well as Table 4.4.Source: G DETAILED EVALUATION STATISTICS · p. 28
  • Table G.1 gives the detailed score breakdown for the Figure 4.2 bar plot.Source: Table G.1 · p. 28
  • Tables G.2–G.6 provide breakdowns for data scaling, rank generalization, trigger generalization, news summaries, and Table 6.1, respectively.Source: Table G.2 · p. 29, Table G.3 · p. 29, Table G.4 · p. 30

H VISUALIZATIONS OF WEIGHT DIFFS AND DIT-ADAPTERS

The appendix visualizes DIT-adapters and test-set weight diffs, including layer-wise norms for a hidden-topic example. The variation in the diffs suggests DIT is not simply reading answers from one consistent weight location.

H VISUALIZATIONS OF WEIGHT DIFFS AND DIT-ADAPTERS

I VARYING THE RANK OF THE DIT-ADAPTER

The appendix tests DIT-adapter rank, training-data scale, and evaluation-metric robustness. Results indicate that adapter expressivity does not constrain current performance, more training data improves out-of-distribution performance, and the comparison across methods is robust to the scoring metric.

I VARYING THE RANK OF THE DIT-ADAPTER

  • For full-parameter diffs, increasing adapter rank improves performance only slightly through rank 16 before performance plateaus; rank has no significant impact on the rank-1 test set.This provides evidence that current performance is not limited by adapter expressivity.Source: I VARYING THE RANK OF THE DIT-ADAPTER · p. 35, Figure I.1 · p. 35

J DOES GENERALIZATION IMPROVE WITH SCALE?

K.1 SCORING OUTPUTS WITH SEMANTIC EMBEDDINGS

  • DIT consistently outperforms baselines across models under both LLM-judge and embedding evaluation, with relative performance consistent across metrics.The embedding-based rerun uses cosine similarity with min-max scaling to a 1–5 range.Source: §K.1 · p. 37, Figure K.1 · p. 37

K.2 WHEN DO SEMANTIC EMBEDDING SCORES DIFFER FROM THE LLM JUDGE?

  • Embedding scores tend to reward lexical overlap, while LLM judges better recognize conceptual equivalence without lexical overlap.Table K.2 marks negative differences when the LLM judge favored an output and positive differences when embeddings favored it.Source: §K.2 · p. 37, Table K.2 · p. 37

L CHANGELOG

  • The changelog records new analyses of adapter-rank scaling, data scaling, and metric robustness, plus an added LLM-agent baseline and relocated method-motivation and mechanistic-interpretability material.Source: L CHANGELOG · p. 38

Related papers

Loading 2510.05092v4…