PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Endpoint Gains: A Weight-Delta Audit of Medical Specialization

Praphul Singh, Shanu Kumar, Akshat Agarwal

arXiv:2608.20768v1cs.AI

TL;DR

The paper addresses the limited evidence about what released specialization updates carry beyond endpoint gains. It audits aligned generalist-to-medical-specialist weight-delta paths across two checkpoint pairs and finds medical movement is reproducible but not uniquely localized to a coarse component family, within a text-only multiple-choice scope.

  • Problem

    Endpoint comparisons show specialist gains but leave the organization and behavioral content of the released update largely unexamined.

  • Method

    A paired weight-delta path audit evaluates full decoder paths, component-family paths, matched controls, and endpoint-anchored rollbacks on two aligned checkpoint pairs.

  • Results

    The full decoder-side update reconstructs measured medical benchmark movement in both pairs, while component probes show partial sufficiency without clean localization.

  • Takeaways & Limitations

    Update-level reconstruction should be separated from component-level explanation before component or repair narratives are inferred.

  • Takeaways & Limitations

    The evaluation covers two public tensor-aligned decoder-only pairs and text-only multiple-choice benchmark movement, not clinical utility or circuit-level mechanism.

Abstract

from arXiv · show

Specialist language models are usually understood through endpoint gains: the generalist scores lower, the specialist scores higher, and the difference is treated as evidence of specialization. This leaves the released update itself largely unexamined. We propose a paired weight-delta path audit and apply it to two public, aligned generalist-to-medical-specialist checkpoint pairs: Gemma-3-4B-IT to MedGemma-4B-IT and Qwen2.5-7B-Instruct to HuatuoGPT-o1-7B. In both pairs, the full decoder-side update strongly reconstructs measured medical benchmark movement (0.974 and 1.183 endpoint-normalized retention), making each decoder delta an appropriate substrate for the audit. Yet the movement is not cleanly localized. MLP is the strongest broad component family in both pairs, but mixed off-domain movements, 10-seed matched controls, and endpoint-anchored rollbacks prevent a unique coarse-family explanation. The audit therefore separates update-level reconstruction from component-level explanation. Its claims concern text-only multiple-choice benchmark movement, not clinical validation, repair, or circuit-level mechanism.

1 Introduction

The paper asks what the released generalist-to-medical-specialist update itself carries beyond endpoint score differences. It audits aligned weight-delta paths, finding reproducible medical movement but no unique coarse component explanation.

  • Endpoint gains show that specialist and generalist models differ, but not what the released update itself carries.
  • Specialization updates may combine target gains, off-domain gains, regressions, and broad internal changes invisible in final scores.
  • The audit evaluates two public aligned pairs: Gemma-3-4B-IT→MedGemma-4B-IT and Qwen2.5-7B-Instruct→HuatuoGPT-o1-7B.
  • The full decoder-side path reconstructs measured medical benchmark movement in both checkpoint pairs.
  • Selected non-clinical gains and regressions coexist while aggregate non-clinical accuracy is nearly flat.
  • MLP is strongest among tested broad families, but matched controls and endpoint rollbacks prevent unique localization.

2 Related Work

Related work frames medical specialization through endpoint benchmark evaluation, while interpretability and weight-space methods provide adjacent intervention and localization perspectives. This paper instead audits the released aligned update and treats off-domain movement as endpoint-conditioned diagnostics.

  • Medical language models and endpoint evaluation: Medical language models commonly use public exam-style benchmarks such as MedQA and medical MMLU subsets for endpoint evaluation.
  • Medical language models and endpoint evaluation: Endpoint benchmark results summarize behavior but do not show how the specialization update is organized.
  • Mechanistic localization and intervention analysis: Interpretability research studies behavioral implementation through memories, neurons, attention heads, circuits, and sparse features.
  • Weight-space editing, interpolation, and merging: Weight-space methods modify or combine checkpoints through editing, averaging, task arithmetic, and interference-aware merging.
  • Specialization tradeoffs and forgetting: Because only two released endpoints are observed, selected non-clinical regressions and gains are treated as endpoint-conditioned diagnostics rather than a known forgetting sequence.

3 Setup, Data, and Metrics

The study uses tensor-aligned checkpoint pairs, a linear decoder-side weight path, scoped component paths, and endpoint-anchored rollbacks. Text-only multiple-choice evaluation combines a medical composite with aggregate and endpoint-conditioned non-clinical diagnostics.

  • 3.1 Paired Checkpoints: The audit compares Gemma-3-4B-IT with MedGemma-4B-IT and replicates the analysis on Qwen2.5-7B-Instruct and HuatuoGPT-o1-7B.Corresponding decoder tensors must have matching shapes and parameter roles.
  • 3.1 Paired Checkpoints: The decoder path linearly interpolates between the generalist at t = 0 and the specialist reconstruction at t = 1.The additive path exposes fractions of the observed update for comparable masking and controls.
  • 3.2 Path Families: The audit tests full-update reconstruction, scoped-from-base component sufficiency, and endpoint-anchored rollback of selected families.Scoped paths apply only a subset of the decoder update; rollback starts from the full decoder update.
  • 3.2 Path Families: Rollback sweeps vary α across {0, 0.25, 0.5, 0.75, 1}, with α = 1 representing the full decoder update.Neither scoped paths nor anchored rollbacks are minimal-circuit claims.
  • 3.3 Evaluation Views: The target readout is a 1,810-example public medical multiple-choice composite from MedQA and medical MMLU subsets.The evaluation audits measured benchmark movement rather than clinical reliability.
  • 3.3 Evaluation Views: The non-clinical evaluation uses 7,325 variable-choice examples, with aggregate accuracy plus separately defined regression and gain slices.The slices are endpoint-conditioned diagnostics, not population-level estimates.

4 What Changed in the Released Update?

The released decoder update is broad and structured across layers and projection families, so the audit uses multiple component paths rather than assuming a localized edit. The screen routes later behavioral tests but does not itself explain benchmark behavior.

  • Delta screening: For each aligned matrix, the audit computes the weight delta, relative change, and singular-value spectrum to describe update magnitude and structure.These quantities identify where the update is large or structured without establishing component-level causality.
  • Delta screening: The analysis targets 319 aligned weight matrices, including 238 shared-decoder matrices and 81 encoder-side multimodal matrices, while interventions focus on the text-used shared decoder.Encoder-side changes are reported descriptively because the benchmarks use text-only inputs.
  • Update geometry: The decoder screen shows a broad, structured update: attention projections have the clearest relative-change hotspots, while MLP projections account for substantial decoder-side update energy.Within MLP, down_proj is the most prominent projection by relative change, motivating mixed buckets.
  • Update geometry: High effective ranks and dispersed singular-value energy make simple one-family explanations suspect before behavioral evaluation.The replication pair likewise has high-rank attention and MLP projections, with MLP accounting for 87.8% of update energy but not thereby uniquely carrying behavior.
  • Update geometry: The replication pair preserves the broad geometry at larger scale: MLP comprises 87.4% of projection parameters and 87.8% of update energy, versus 12.2% for attention.This imbalance makes MLP a prior for intervention, not evidence of unique behavioral responsibility.
  • Audit routing: The resulting probes compare the full decoder path with attention, MLP, mixed, leave-one-family-out, matched-random, and endpoint-anchored rollback paths.The full decoder path serves as the reference reconstruction, while the other paths test component identity, update amount, and endpoint trade-offs.

5 Does the Decoder Path Reconstruct Benchmark Movement?

The full decoder-side paths strongly reconstruct measured medical benchmark movement in both checkpoint pairs, while also carrying heterogeneous off-domain changes. These paths therefore support update-level reconstruction but do not by themselves localize the responsible component family.

  • The full decoder path is the audit’s reference reconstruction for measured text-only benchmark movement.The endpoint t = 1 tests whether the aligned decoder update accounts for the specialist endpoint behavior.
  • 0.974 normalized medical retention is reached for the Gemma→MedGemma decoder path at the specialist endpoint.This establishes the aligned decoder update as the relevant substrate for subsequent component and control comparisons.
  • Medical retention rises early along the Gemma path while selected regression recovery and selected-gain acquisition change differently.At t = 0.50, medical retention is 0.863 with 0.744 selected-regression recovery; at t = 0.60, medical retention is 0.869 with 0.793 selected-gain acquisition.
  • 1.183 normalized medical retention is reached for the Qwen decoder path at t = 1.The replication yields the same central reconstruction result despite a different trajectory; its confidence interval is 95% CI [0.962, 1.547].
  • Across both pairs, the update carries medical movement together with heterogeneous off-domain gains and regressions rather than a one-dimensional tradeoff.The Qwen path similarly shows selected-regression recovery falling while selected-gain acquisition rises.
  • Scoped component-family paths show partial sufficiency, not localization, with MLP strongest among tested buckets but no bucket matching the full decoder path.Component explanations remain subject to matched controls and endpoint-anchored rollbacks.

6 Can Coarse Components Explain the Movement?

Coarse component paths reproduce parts of medical benchmark movement, but matched controls and endpoint-anchored rollbacks prevent a unique family-level explanation. MLP is the strongest tested broad bucket, yet its apparent sufficiency depends substantially on update size and matching choices.

  • Scoped component paths: MLP is the strongest tested scoped component family, reaching 0.797 medical retention at its best point, but no bucket matches the full decoder path.Attention-plus-down_proj reaches 0.601 retention, while the full-minus-down_proj path retains more movement than that mixed bucket.
  • Matched controls: 10-seed matched controls weaken unique-family interpretations because similarly sized or energetic random subsets can approach or exceed structured buckets.For Gemma–MedGemma at t = 1, parameter-count-matched MLP controls average 0.814 medical retention versus 0.699 for structured MLP.
  • Matched controls: For Qwen–HuatuoGPT, MLP reaches 1.024 medical retention at t = 1, while parameter- and energy-matched controls reach 1.115 and 1.098 on average.Attention reaches 0.341, whereas attention-plus-down_proj reaches 0.841 at the same endpoint scale.
  • Endpoint-anchored rollbacks: Endpoint-anchored rollbacks trade selected-regression recovery against medical retention rather than exposing a clean component repair knob.Full MLP rollback recovers 0.593 of selected regressions versus 0.279 for attention, but retains only 0.359 medical movement versus 0.699.
  • Conclusion: Across both checkpoint pairs, coarse families are useful partial-sufficiency probes, but no tested family uniquely explains the reconstructed medical movement.The negative localization result survives matched controls and anchored rollbacks.

7 Discussion

A weight-delta audit provides an intermediate analysis layer between endpoint evaluation and circuit discovery. Across two medical checkpoint pairs, it reconstructs target movement but does not identify the features, circuits, or training history that caused it.

  • Discussion: The audit treats the aligned generalist-to-specialist weight difference as an auditable object between endpoint comparison and mechanistic explanation.It establishes an auditable substrate without identifying the features, circuits, training examples, or historical trajectory that caused the movement.
  • Discussion: MLP is the strongest broad component family in both pairs, but matched random subsets can equal or exceed it because parameter mass and update energy can masquerade as explanation.The supported conclusion is an intervention prior, not unique semantic localization in MLPs.
  • Discussion: Rollback makes the practical boundary clear: no tested family is a clean repair knob that preserves medical movement while reversing selected regressions.The audit is better suited to release comparison, regression triage, and prioritizing finer-grained follow-up than editing a deployed model.
  • Discussion: Non-clinical views remain conditional diagnostics because source composition differs, gain and regression slices are endpoint-selected, and multiple-choice accuracy omits calibration and generation quality.The paper proposes open-ended clinical tasks and blinded judge ensembles as a next step.

8 Conclusion

The audit finds that aligned decoder paths reconstruct medical benchmark gains in both checkpoint pairs while carrying heterogeneous non-clinical movement. MLP is strongest among broad component families, but controls and rollbacks prevent unique localization or clinical, repair, or circuit-level interpretations.

  • Aligned decoder paths reconstruct measured medical benchmark gains in both Gemma–MedGemma and Qwen–HuatuoGPT while carrying heterogeneous selected non-clinical movement.
  • MLP is the strongest broad component family, but matched controls and endpoint-anchored rollbacks prevent unique localization.
  • Size and update energy explain substantial apparent sufficiency, while removing MLP trades selected-regression recovery for medical retention.
  • The results support pairing endpoint gains with update-level audits before inferring component or repair narratives.
  • The claims concern benchmark movement rather than clinical validation or circuit explanation.

9 Limitations

The study is limited to two tensor-aligned decoder-only medical checkpoint pairs and text-only multiple-choice evaluation. Its counterfactual path and coarse interventions do not establish generality, deployment suitability, historical training dynamics, or circuit-level mechanisms.

  • The study covers two public, tensor-aligned medical checkpoint pairs rather than a general theory of posttraining.
  • Its audited paths target decoder-only language-model backbones, so other architectures, domains, sizes, and training recipes may organize updates differently.
  • The evaluation is text-only and multiple-choice, excluding free-form clinical generation, calibration, abstention, and multimodal behavior.
  • The additive weight path is a counterfactual audit coordinate rather than the historical training trajectory or a deployment recommendation.
  • Component buckets test coarse partial sufficiency, not circuits, features, token-level mechanisms, or minimal subsets.
  • Ten-seed matched controls are informative but not exhaustive null distributions, and selected regression and gain slices are diagnostic rather than population estimates.

Ethics Statement

The study uses public checkpoints and benchmark examples without patient records, protected health information, or human-subject data. Exam-style multiple-choice gains should not be interpreted as clinical reliability, safe advice generation, calibrated uncertainty, or deployment readiness.

  • The study analyzes public checkpoints and benchmark examples without using patient records, protected health information, or human-subject data.
  • Exam-style multiple-choice gains should not be read as clinical reliability, safe advice generation, calibrated uncertainty, or readiness for patient- or clinician-facing use.
  • Interpolation, bucket, rollback, and neuron-diagnostic analyses are retrospective audit probes rather than certification, repair, or deployment evidence.

A Additional Results and Diagnostics

The appendix documents the evaluation views, endpoint-normalized path readouts, component-family controls and rollbacks, Qwen replication, neuron diagnostics, and descriptive weight-change analyses. Across these views, selected slices are treated as diagnostics, controls as partial checks, and neuron results as non-causal support.

  • Evaluation setup: The audit evaluates medical composites, full non-clinical aggregates, endpoint-conditioned regression and gain slices, and multiple-choice predictions scored by highest-scoring option label.
  • Evaluation setup: The medical composite uses publicly auditable labels, and its positive composite gain is example-weighted and MedQA-dominated.
  • Evaluation setup: Endpoint movements define the normalized path readouts, while the full non-clinical aggregate is descriptive and selected slices are not population estimates.
  • Evaluation setup: Bootstrap intervals use 2,000 paired resamples with seed 1729, resampling examples or fixed endpoint-selected sets according to the readout.
  • Path and component analyses: Full decoder-side paths, scoped component-family sweeps, matched controls, and anchored rollbacks provide the main reconstruction and localization readouts.
  • Path and component analyses: Matched controls use 10 seeded draws and compare structured buckets with parameter-count, matrix-count, or update-energy matches; several controls approach the structured buckets.
  • Qwen replication: The Qwen replication evaluates the same 1,810 medical and 7,325 non-clinical examples and reports endpoint movements, decoder coverage, controls, module paths, and rollback tradeoffs.
  • Neuron diagnostics: Neuron diagnostics use 64 examples per group and descriptive activation-drift and importance proxies rather than causal ablations.
Loading 2608.20768v1…