Source-linked AI summary
The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment
Rishab Balasubramanian, Pin-Jie Lin, Rituraj Sharma, Anjie Fang, Fardin Abdi, Viktor Rozgic, Zheng Du, Mohit Bansal, Tu Vu
TL;DR
The paper asks whether post-trained capabilities can transfer across models without retraining, despite differences in scale and latent structure. It introduces Unlock, which extracts and linearly aligns capability directions for inference-time intervention. Reasoning experiments show substantial gains across model sizes, while transfer depends on whether capabilities are represented during pre-training and on model capacity.
Problem
The paper investigates whether a capability expressed in one model can be isolated and transferred to another without gradient-based training or labeled supervision.
Method
Unlock contrasts capability-present and capability-absent Source variants, aligns the extracted MasterKey to a Target model through low-rank linear subspace alignment, and applies it at inference time.
Results
Unlock consistently improves reasoning performance across models and tasks, with small-to-large transfer generally outperforming large-to-small transfer.
Takeaways & Limitations
The results support the Master Key Hypothesis that useful behaviors can be isolated as linearly transferable directions in shared low-dimensional subspaces.
Takeaways & Limitations
Atomicity depends on capability, architecture, scale, and data, and the paper leaves a systematic study of architecture effects for future work.
Abstract
from arXiv · showhide
We investigate whether post-trained capabilities can be transferred across models without retraining, with a focus on transfer across different model scales. We propose the Master Key Hypothesis, which states that model capabilities correspond to directions in a low-dimensional latent subspace that induce specific behaviors and are transferable across models through linear alignment. Based on this hypothesis, we introduce UNLOCK, a training-free and label-free framework that extracts a capability direction by contrasting activations between capability-present and capability-absent Source variants, aligns it with a Target model through a low-rank linear transformation, and applies it at inference time to elicit the behavior. Experiments on reasoning behaviors, including Chain-of-Thought (CoT) and mathematical reasoning, demonstrate substantial improvements across model scales without training. For example, transferring CoT reasoning from Qwen1.5-14B to Qwen1.5-7B yields an accuracy gain of 12.1% on MATH, and transferring a mathematical reasoning direction from Qwen3-4B-Base to Qwen3-14B-Base improves AGIEval Math accuracy from 61.1% to 71.3%, surpassing the 67.8% achieved by the 14B post-trained model. Our analysis shows that the success of transfer depends on the capabilities learned during pre-training, and that our intervention amplifies latent capabilities by sharpening the output distribution toward successful reasoning trajectories.
1 Introduction
The paper asks whether capabilities can be isolated from one model and transferred to another without training or labels. It proposes Unlock and the Master Key Hypothesis, then evaluates reasoning transfer across model sizes.
- Motivation: Post-training capabilities are costly to learn repeatedly, motivating mechanisms that reuse capabilities across models without retraining.The paper frames this as an inefficiency arising from the cost of data, computation, and engineering required for post-training.
- Approach: Unlock extracts a capability direction from capability-present and capability-absent Source variants using unlabeled prompts, then aligns it to a Target model.The transferred direction is applied as an inference-time intervention rather than learned through gradient updates.
- Results: 12.1%: transferring CoT from Qwen1.5-14B to Qwen1.5-7B raises MATH accuracy from 8.0% to 20.1% without explicit CoT prompting.This exceeds the 18.2% achieved by the 7B instruction-tuned model with CoT prompting.
- Results: Unlock improves reasoning performance across model sizes, with small-to-large transfer typically producing larger relative gains than large-to-small transfer.The analysis also reports stronger gains when the capability is more strongly represented in the Target model and output distributions are sharpened toward successful reasoning.
- Hypothesis: The Master Key Hypothesis proposes that capabilities are directions in a shared low-dimensional latent subspace that can be transferred through linear transformations.Shifting representations along these directions is hypothesized to induce the associated behaviors in another model.
2 Method
Unlock transfers a capability direction by contrasting Source activations, aligning low-rank Source and Target subspaces, and intervening on the Target residual stream during inference. The procedure uses forward passes on a small set of unlabeled prompts and requires no additional training.
- Extracting the MasterKey: Unlock contrasts Source Locked and Source Unlocked activations to extract a MasterKey direction representing the shift toward the desired capability.The Source variants share architecture and tokenizers, allowing their internal activations to be directly compared.
- Pipeline: The framework has three stages: extracting the MasterKey, aligning Source and Target representation spaces, and applying the transferred direction at inference time.These stages are also summarized in the method illustration.
- Extracting the MasterKey: The capability direction is extracted from unlabeled prompts by aggregating per-example differences, using either the mean or first principal component.The contrast may be prompt-driven, such as with versus without CoT prompting, or model-driven, such as base versus post-trained variants.
- Cross-model Subspace Alignment: Unlock aligns low-rank Source and Target subspaces with SVD and learns a k × k linear transformation between their projected representations.The mapping accommodates different hidden sizes and latent geometries without aligning the full hidden spaces.
- Unlocking the Target Model: The mapped MasterKey is applied to the Target residual stream at every layer during generation, with intervention strength controlled by α and state magnitude preserved by rescaling.Layers are paired by relative depth when Source and Target models have different numbers of layers.
3 Atomic And Non-Atomic Capabilities
The paper distinguishes atomic capabilities, already present before post-training, from non-atomic capabilities that gain substantially from post-training. It treats atomicity as dependent on the model, data, architecture, and scale.
- Capability Definition: A capability is a reproducible behavior that remains consistent across semantically similar inputs and minor prompt changes.The paper distinguishes latent capabilities, elicitable by steering or prompting, from absent capabilities requiring explicit training.
- Atomic And Non-Atomic Capabilities: Atomic capabilities show nearly identical performance before and after post-training, whereas non-atomic capabilities receive substantial post-training gains.The distinction is defined using a threshold ε on average post-training gain.
- Atomic And Non-Atomic Capabilities: Atomicity is relative to a model’s pre-training distribution because capability support depends on the data and objectives encountered during pre-training.The paper reports that atomicity affects the gains obtained from Unlock.
- Scope And Assumptions: Atomicity also depends on model learning capacity, architecture, scale, and data, and the paper leaves a systematic architecture study for future work.The authors intentionally keep the definitions vague because understanding of language-model representation spaces remains incomplete.
4 Atomic Capability Transfer
This section tests whether Chain-of-Thought capability directions extracted from prompting can transfer across model scales. UNLOCK consistently improves reasoning performance, with stronger gains for small-to-large transfer and when the capability is already latent.
- Capability setup: Prompt-induced activation differences provide a controlled test of whether CoT reasoning corresponds to a transferable direction in representation space.The model weights remain fixed, so behavioral changes are attributed to shifts in internal representations.
- Results: 31.2% average accuracy gains occur for Qwen1.5 small-to-large transfer, compared with 25.0% for large-to-small transfer.These results are comparable to prompting the target baseline with explicit CoT instructions.
- Results: Across evaluated model families and datasets, the unlocked target consistently outperforms the direct-prompting baseline.The transferred model also produces longer outputs and structured reasoning traces, supporting CoT elicitation rather than merely surface-level changes.
- Transfer asymmetry: The observed asymmetry suggests that larger models can activate latent circuitry transferred from smaller models, whereas smaller models may lack capacity for larger models’ more complex reasoning structures.The capacity-based explanation is presented as plausible rather than established.
- Capability salience: Unlock can elicit capabilities that are present but dormant, but cannot introduce capabilities that are absent from the target model.Within Qwen1.5, base and instruction-tuned variants show similar CoT-prompted performance, while gemma-2 shows evidence that reasoning is learned during post-training and is not fully recoverable by transfer.
5 Non-Atomic Capability Transfer
Unlock transfers non-atomic mathematical reasoning capabilities across models without training, with gains that can match or exceed post-training. Its effectiveness depends on transfer direction, data regime, and whether the Target already contains latent capability.
- 5.2 Results & Discussion: Unlock consistently improves the Unlocked Target over its baseline and often matches or exceeds the post-trained counterpart.This pattern is reported across mathematical reasoning evaluations.
- 5.1 Experimental Settings: Unlock combines prompting with latent intervention to elicit mathematical reasoning that prompting alone cannot reliably produce.The method is evaluated in task-conditioned and task-agnostic settings, differing in data alignment and abundance.
- 5.2 Results & Discussion: 6.4% and 4.3% are the average gains over the baseline for Qwen3-14B + Unlockfrom 4B and Qwen3-4B + Unlockfrom 14B, respectively.The result shows that non-atomic mathematical reasoning can be applied as a latent test-time intervention.
- 5.2 Results & Discussion: 69.5% and 70% identify the settings where task-conditioned transfer is superior for large-to-small transfer and task-agnostic transfer is superior for small-to-large transfer, respectively.Both regimes improve the Unlocked Target, but their relative effectiveness depends on transfer direction.
- 5.2 Results & Discussion: Unlock produces narrower opening trajectories and longer outputs, supporting an interpretation based on Chain-of-Thought elicitation and distribution sharpening.The Unlocked model converges on a smaller set of early trajectories across models and datasets.
- 5.2 Results & Discussion: The results suggest that post-training maps prompts to desired outputs while Unlock exploits capabilities already present in latent space.Prompting alone may not elicit these capabilities without the learned mapping.
6 The Master Key Hypothesis & Implications
The Master Key Hypothesis proposes that capabilities occupy shared low-dimensional subspaces and can be transferred through low-rank linear alignment. The evidence supports this operational view while leaving the formation and interaction of capabilities mechanistically unresolved.
- The Master Key Hypothesis: The Master Key Hypothesis posits that a capability can be captured in a low-dimensional projection and transferred between models through a linear alignment map.The source and target models may have different hidden dimensions and corresponding layers.
- Implications: Transfer depends on representational capacity: dormant capabilities may transfer noisily, whereas absent capabilities may lack an isolatable direction or compatible target basis.This makes latent capability presence a boundary condition for successful transfer.
- Implications: Capability directions transfer across scales and architectures, are often compactly approximated, and work best when the Target weakly or latently expresses the capability.These observations provide three empirical lines of evidence consistent with the hypothesis.
- Implications: Non-atomic capabilities can transfer when well represented in the Source contrast and sufficiently representable in the Target latent space.The authors argue that atomicity definitions based only on post-training gains and prompt stability are functionally incomplete.
- Relation to Prior Hypotheses: The hypothesis unifies linear concept directions and convergent representations by treating post-training behaviors as transferable latent interventions.It connects the Master Key Hypothesis to the Linear Representation and Platonic Representation Hypotheses.
- Limitations: The precise mechanisms by which capabilities form, are represented, and interact remain undetermined.The authors present the hypothesis as an operational abstraction that motivates future mechanistic study.
7 Related Work
Prior capability-transfer methods operate in weight, output, or distillation spaces, while steering work often targets surface behaviors with supervision. Unlock instead studies label-free latent capability transfer across model sizes and families using task-level reasoning correctness.
- Steering Vectors: Steering-vector research commonly targets safety, toxicity, bias, style, and related behaviors through internal activation interventions.The paper focuses instead on advanced reasoning capabilities.
- Distinction from Prior Steering Transfer Work: Most cross-model steering transfer studies safety, jailbreak, or style behaviors using explicit positive/negative supervision and coarse behavioral proxies.Unlock evaluates cross-size and cross-family transfer with task-level correctness on reasoning benchmarks.
- Capability Transfer Across Models: Weight-space methods reuse parameter deltas, while logit-space methods guide students with stronger-model outputs and require multi-model inference.These approaches differ from latent-space intervention without retraining.
- Representational Convergence and Cross-Model Alignment: Representation-alignment work enables transfer through shared subspaces or simple maps, whereas Unlock uses low-rank linear alignment rather than nonlinear autoencoders or full-dimensional pseudoinverses.The paper emphasizes quantifiable downstream improvements across models and tasks.
- Knowledge Distillation: Knowledge distillation transfers capabilities by training a student to match a teacher distribution, incurring training cost and repetition for each student model.Unlock differs by avoiding this per-student training process.
8 Conclusion
The paper presents a training-free method that transfers capability directions across models through low-rank latent alignment. Results across model families and benchmarks support the Master Key Hypothesis as an empirical account of linearly transferable capabilities.
- Conclusion: Unlock extracts a MasterKey direction from prompt- or model-induced representational differences and transfers it through low-rank linear subspace alignment.The approach avoids gradient updates and requires no architectural or tokenization correspondence between Source and Target models.
A.1 Comparison to Previous Approaches
Unlock is a training-free, label-free approach that transfers high-level capabilities across models through low-rank latent-space alignment. It differs from prior approaches by requiring neither gradient updates nor labeled data.
- Unlock transfers a capability-inducing direction from Source models to a Target model through low-rank linear subspace alignment.The method is presented as enabling capability reuse without additional training.
- Unlock is completely training-free and requires no labeled data, unlike methods relying on gradient updates or supervised signals.
- Figure 5 contrasts CoT prompts that request step-by-step reasoning with Direct prompts used across evaluations.
- The evaluations use unified Direct and CoT prompts to avoid discrepancies in prompt templates across models.A common concluding token pattern is also used to reduce formatting-related evaluation bias.
B.1 Impact of Unlocking
Unlock increases reasoning-related generation and performance while preserving existing strengths, but its effectiveness depends on low-rank alignment choices and careful hyperparameter calibration.
- Impact of Unlocking: Unlock increases generation length, and the additional length is accompanied by a higher proportion of correct solutions.The analysis compares Direct and CoT prompting with Locked and Unlocked models to test whether longer outputs are task-relevant.
- Impact of Unlocking: Unlock remains competitive with post-trained models, typically within ±2%, and does not inhibit behavior that models already reliably display.
- Latent Space Geometry and Sensitivity: Spectral entropy plateaus between approximately 1.4 and 2.5 nats, corresponding to an effective rank of 4-12 despite latent dimensions of at least 1024.
- Latent Space Geometry and Sensitivity: At very low ranks, reconstruction-error improvements plateau around n≈64 because rank, rather than sample size, limits the mapping’s capacity.
- Latent Space Geometry and Sensitivity: Ranks beyond approximately 128 consistently degrade performance, consistent with high-rank projections overfitting example-specific artifacts.
- Latent Space Geometry and Sensitivity: Successful capability transfer occurs within localized pockets of the latent manifold, highlighting the need for precise hyperparameter calibration.
C.1.1 Understanding the Impact of Unlocking
Unlock’s gains reflect stronger and more consistent reasoning trajectories, mathematical reliability, reduced repetition, and formatting compliance, while remaining dependent on latent target competence.
- Dependence on target capabilities: Unlock’s gain depends on both the Source contrast and the baseline competence of the Target Locked model.In gemma-3, large gaps from instruction-tuned counterparts leave limited scope for recovering post-training gains.
- Interpretation: The results reinforce that Unlock elicits and amplifies capabilities already latent in the Target model rather than introducing new knowledge.
- Coherent reasoning traces: Unlock more consistently produces coherent intermediate steps that connect the problem statement to the final answer.
- Coherent reasoning traces: Unlock sharpens first-word distributions toward recurring openings such as “To solve the ...” or “Step 1: ...”.The Locked model has a more diffuse distribution over opening tokens.
- Reasoning dynamics: Unlock increases length-to-answer across tasks and model families except Minerva Math, indicating longer explicit reasoning traces before answering.
- Reasoning dynamics: The Locked model exhibits significantly more repeated substrings than the Unlocked model in incorrect solutions.Repetitions peak around substring lengths of 128-256 characters.
- Formatting: For models larger than 7B, formatting differences are rare, so the primary gains at that scale are attributed to improved reasoning rather than format compliance.
C.2 Examples of Math Reasoning Transfer
The examples show Unlocked models producing more complete and reliable mathematical reasoning than Locked models across logarithmic, numerical, combinatorial, and matrix problems.
- Improved coherence in reasoning traces: In Example 10, the Unlocked model continues step-by-step reasoning after the Locked model skips intermediate steps and begins making errors.
- Improved coherence in reasoning traces: In Example 11, the Unlocked model handles the requested rounding more carefully and reaches the correct solution.
- Improved mathematical reliability: Example 12 contrasts a hallucinating Locked model with an Unlocked model that generates complete reasoning and the correct solution.
- Improved mathematical reliability: The matrix example uses row-by-column multiplication to produce a 2 × 1 result from a 2 × 2 matrix and a 2 × 1 matrix.
- Improved formatting: Example 13 shows both models obtaining the same answer, while the Locked model formats the final answer incorrectly.The paper notes that this formatting phenomenon is rarely observed in 14B models.
D.1 Experimental Setup
The experiment evaluates cross-family transfer using Qwen1.5 Source checkpoints and a stronger post-trained Chat variant, finding that transfer elicits latent CoT behavior with performance comparable to prompting and intra-family transfer.
- The experiment studies transfer between Source and Target models from different architectural families.
- Qwen1.5 checkpoints are selected as Source models because the family exhibits CoT as an atomic ability and robust performance across evaluation settings.
- Cross-family transfer elicits significant CoT behavior from the Target model, supporting the view that Chain-of-Thought capabilities can remain latent.
- Cross-family transfer achieves performance comparable to prompting the Target model with a CoT prompt.
- Cross-family transfer performs comparably to intra-family transfer, providing evidence of converging capability representations across models.