PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs

Charbel El Feghali, Arkil Patel, Nicholas Meade, Spandana Gella, Verna Dankers, Siva Reddy

arXiv:2606.21638v1cs.CRcs.CL

TL;DR

Open-weight LLMs lack a native way to provide public and authorized users different capabilities within one released model. TLMs use a secret permutation key to reconfigure the same weights, and experiments show keyed configurations acquire private capabilities while public behavior remains unchanged.

  • Problem

    Open-weight LLMs lack a single released artifact that supports both broadly available and access-controlled capabilities.

  • Method

    TLMs encode authorization in a compact permutation key that reconfigures selected model parameters into an alternative computation graph over the same weights.

  • Results

    Across language acquisition, instruction following, and synthetic fact memorization, keyed configurations acquire private capabilities while public configurations remain effectively unchanged.

  • Takeaways & Limitations

    TLMs provide a single open-weight checkpoint with experimentally separated public and private capability tiers.

  • Takeaways & Limitations

    Experiments use only 180M- and 650M-parameter models and do not establish identical behavior at frontier scale.

Abstract

from arXiv · show

Open-weight Large Language Models (LLMs) enable scientific progress and broad deployment. However, they make it difficult to control access to sensitive capabilities. Current practice either suppresses dangerous capabilities before release or mediates access through closed services that use specialized model variants, input/output monitors, and API permissions. The former is susceptible to jailbreaks while sacrificing capability for all users to mitigate the risks posed by a few, and the latter is fundamentally incompatible with open-weight release. In this paper, we propose Tiered Language Models (TLMs), where a single set of released weights supports multiple capability levels. In its default public configuration, a TLM behaves as a conventional LLM. A compact secret key specifies a permutation over a small parameter subset, inducing an alternative computation graph over the same weights that exposes additional capabilities. We develop a training protocol that jointly pretrains both configurations from scratch, then fine-tunes the keyed configuration on private data with regularization to preserve the public model's behavior. We pretrain 180M- and 650M-parameter TLMs and demonstrate that the keyed configuration can acquire a new language, gain instruction-following ability, and memorize private factual knowledge, whereas the public configuration exhibits none of these capabilities. Moreover, we show that our approach extends naturally to multiple hierarchical tiers. Because authorization operates on the model's weight structure rather than in the input space, the mechanism resists fine-tuning-based extraction and partial key compromise. In general, TLMs take a step toward reconciling open-weight release with selective capability control.

1 Introduction

Tiered Language Models (TLMs) address the challenge of releasing one open-weight model with both public and access-controlled capabilities. They use a compact key to reconfigure the same weights into a private computation graph, with experiments showing private capabilities can be acquired without leakage into the public model.

  • Motivation: Open-weight LLMs must balance broad access to general capabilities with restrictions on sensitive capabilities or private and licensed knowledge.Existing practice separates public and restricted deployments, but the paper seeks a single model supporting multiple access tiers.
  • Motivation: Prompt-based password locking is weak because its secret key consists of ordinary input tokens, allowing privileged behavior to be elicited without authorization.TLMs move authorization from prompt tokens to the model’s weight structure.
  • Method: TLMs use a compact key to reconfigure a small subset of released weights, creating an alternative computation graph that unlocks guarded knowledge and capabilities.Without the key, the model runs in a standard public configuration; authorized users instantiate the keyed configuration over the same weights.
  • Method: The training protocol jointly pretrains capable public and keyed configurations on public data, then fine-tunes the keyed configuration on private data for access-controlled behavior.The pretraining is asymmetric and includes periodic backward passes through the keyed configuration.
  • Results: 180M- and 650M-parameter TLMs demonstrate new-language acquisition, instruction following exceeding 85% win rate on AlpacaEval, and synthetic-fact recall without leakage into the public model.The paper presents these as example use cases of the framework.

2 Related Work

Prior access-control research follows three main directions: prompt-based credentials, modular components, and shared-weight architectures that support multiple recoverable behaviors. These approaches respectively condition behavior on secret prompts, restrict access to model modules, or reconfigure shared parameters at inference.

  • Research threads: Existing LLM access-control work spans prompt passwords, modular components, and shared-weight architectures encoding multiple recoverable behaviors.The paper organizes related work along these three research threads.
  • Prompts and passwords: Prompt-based methods place authorization in the input space, conditioning behavior on secret token sequences or refusing instructions without the correct key prompt.Greenblatt et al. suppress capabilities without a password, while Tang et al. train models to refuse instructions without the correct key prompt.
  • Modular components: Modular methods control access by restricting per-domain LoRA adapters, selectively including MoE modules, or merging refusal adapters into a base model.AdapterSwap, FlexOlmo, and Locket exemplify these modular approaches.
  • Shared weights and model reconfiguration: Shared-weight methods support multiple recoverable models through binary-mask superposition, rank-one perturbations, or inference-time variation of factorized matrix ranks.Rauba et al. demonstrate a smooth capability hierarchy, but this approach assumes the deployer controls inference.

3 Tiered Language Models

Tiered Language Models use a secret permutation key to create an alternative computation graph over the same released weights, separating public behavior from authorized capabilities. They require joint pretraining and asymmetric private fine-tuning so keyed models can specialize while public behavior remains robust.

  • Core mechanism: A secret key permutes selected parameter positions, producing a keyed computation graph over the same released weights while the default configuration remains public.Without the key, the model behaves like an ordinary LLM; with it, authorized users can access additional capabilities.
  • Core mechanism: Public and keyed models share parameter values and differ only in how those values are arranged within weight tensors.The public configuration is the identity map, while each key specifies a reconfiguration.
  • Configuration design: ∼5% of parameters form the tier subset that keys permute, comprising attention-head groups and FFN groups while leaving the complement unchanged.Attention swaps exchange Q/K/V rows and matching output-projection columns; FFN swaps exchange up-projection rows with bias and down-projection columns.
  • Training protocol: Both configurations are jointly pretrained on public data, with tier parameters receiving gradients only from the keyed model and complementary parameters receiving gradients from both.This asymmetric gradient flow encourages the public configuration to rely less on parameters later rearranged or updated for private specialization.
  • Training protocol: Private fine-tuning updates only tier parameters through the keyed configuration while freezing complementary parameters and regularizing drift from pretrained public behavior.The KL regularizer mitigates the effect of shared tier-parameter updates on the public model.

4 Evaluating Capability Separation in TLMs

TLMs separate private from public capabilities across language modeling, instruction following, and factual memorization. Keyed fine-tuning improves the keyed configuration while preserving the public configuration’s behavior.

  • Evaluation scope: Across three tasks, evaluation tests private capability acquisition without exposure in the public configuration: new-language modeling, instruction following, and factual memorization.These tasks probe distributional, behavioral, and pointwise capabilities.
  • Memorizing factual knowledge: 400 synthetic facts are perfectly recalled by CK, while Cpub remains at zero throughout private fine-tuning.The facts are synthetic biographies defined by age, profession, hobby, and salary.
  • Modeling a new language: The keyed configuration’s Spanish loss decreases steadily while its English loss remains stable, whereas the public configuration shows no Spanish improvement or English degradation.TLM-650M was fine-tuned on 4B Spanish tokens.
  • Learning to follow instructions: Above 85%, the keyed configuration’s AlpacaEval win rate rises from ∼50% within 500 steps, while the public configuration drops to ∼15%.Performance is comparable across Easy, Medium, and Hard categories.
  • Discussion: Across all three settings, private fine-tuning selectively modifies the keyed configuration while leaving the public configuration effectively unchanged.This consistency spans language, instruction following, and individual facts.

5 Computational Cost of TLMs

TLM pretraining can substantially reduce its roughly doubled computational cost through sparse keyed updates, while preserving near-standard public and private-domain performance. Its permutation keys are compact relative to adapter-based access control, with a 1%-parameter LoRA adapter serving as a matched-performance baseline.

  • Minimal computation overhead for tiered pretraining: Sparse keyed updates reduce tiered-pretraining overhead from roughly doubled cost to 5% additional FLOPs at f=20, while approaching the full-overhead variant’s validation loss.The public pass runs every step, whereas the keyed pass runs once every f steps; keyed performance improves as f decreases, with diminishing returns beyond f=20.
  • Performance gap to standard pretraining: The public TLM configuration requires roughly 6% more training steps than a same-condition non-tiered baseline to reach the corresponding public-domain validation loss.After subsequent fine-tuning, the keyed TLM reaches a final private-domain loss comparable to the baseline.
  • Permutation keys are orders of magnitude smaller than adapter weights: A 1%-parameter LoRA adapter achieves comparable private-domain loss to TLM-180M on Spanish fine-tuning and provides a matched-performance baseline for storage comparison.TLM permutation keys are compact specifications rather than learned parameter values and are described as orders of magnitude smaller than adapter weights.

6 Adversarial Robustness of TLMs

TLM behavioral separation remains under normal public use, while adversarial evaluations show that fine-tuning on partial private data and partial key access do not recover hidden knowledge. Although weight magnitudes can reveal tier membership, recovering the permutation remains difficult, with 90% of the full key still yielding near-zero accuracy.

  • Threat models: TLM adversarial evaluation targets the synthetic-biography setting, where the keyed configuration memorized 400 biographies with perfect accuracy.The evaluation considers adversaries with full access to model parameters attempting to extract private knowledge.
  • Threat models: Fine-tuning the public configuration on partial private data does not extract the hidden knowledge.The attacker fine-tunes all parameters through Cpub without the key, using half of the 400 biographies while leakage is measured on the held-out half.
  • Threat models: Partial access to the key does not extract hidden knowledge across tested key fractions from 5% through 100%.Each partial key applies a randomly selected subset of swaps, with exact-match accuracy evaluated under greedy decoding over 100 independent draws per fraction.
  • Threat models: 54.2% F1 identifies tier parameters from weight magnitudes, but not the permutation that defines the keyed computation.The key specifies both which units belong to S and how they are permuted, making permutation recovery the harder problem.
  • Threat models: 90% of the full key still yields near-zero accuracy, even when an adversary identifies tier parameters and guesses most correct swaps.The number of possible permutations grows combinatorially with the number of modules in S.

7 Scaling TLMs to Multiple Tiers

TLMs extend to ordered hierarchies of private tiers, with cumulative keys and training procedures that preserve public behavior. A three-tier 180M model acquired German, Turkish, and Spanish capabilities sequentially while retaining earlier-tier abilities and nearly unchanged public-domain performance.

  • Hierarchical tier design: An ordered hierarchy of N private tiers gives each tier-i user all capabilities up to tier i through cumulative configurations C_i.Each key K_i acts on a disjoint parameter subset S_i, and C_i applies K_1 through K_i jointly.
  • Training protocol: The multi-tier training protocol selects keyed configurations round-robin and routes gradients asymmetrically so active tier parameters learn from the keyed path while others retain mixed public/keyed training.This generalizes both stages of the two-tier method and supports public-path compatibility across tiers.
  • Three-tier demonstration: Three disjoint 5% keys at the 180M scale acquired German, Turkish, and Spanish sequentially, preserved earlier-tier capabilities, and changed public-domain performance by less than 0.005 nats.Each language was fine-tuned for 2B tokens on D1, D2, and D3 respectively.

8 Conclusion

Tiered Language Models separate public and private capabilities within one open-weight checkpoint: a compact structural key enables additional behavior while the public configuration remains effectively unchanged. Results support TLMs as an access-control primitive, while highlighting deployment requirements for key management, validation, attacks, and realistic capability evaluations.

  • Conclusion: TLMs separate public and private capabilities within a single open-weight checkpoint, using a permutation key to activate private behavior while leaving the public configuration effectively unchanged.The framework places authorization in the model’s weight structure rather than in prompts or separate adapters.
  • Conclusion: Fine-tuning the public configuration on related private data and knowing up to 90% of the key do not recover the gated capability.These findings support TLMs as an access-control primitive, but deployment requires careful key management, larger-scale validation, stronger adaptive attacks, and more realistic restricted-capability evaluations.

9 Limitations · A Additional Details · A.1 Multi-tier training details

The paper identifies unresolved limitations in scale realism, threat-model coverage, and detectable weight fingerprints. Its multi-tier extension uses cumulative disjoint permutations, round-robin pretraining, and sequential fine-tuning with regularizers that preserve public behavior and earlier private capabilities.

  • 9 Limitations: The experiments use only 180M- and 650M-parameter models, so frontier-scale behavior, interference, and leakage remain unestablished.Larger models may exhibit different interference patterns or new white-box leakage forms.
  • 9 Limitations: The evaluated robustness tests cover fine-tuning, incomplete keys, and tier-parameter identification, but not the full space of adaptive white-box attacks.Potential attacks include structured permutation search, activation analysis, and combinations of weight analysis with task knowledge.
  • 9 Limitations: Private fine-tuning leaves statistical fingerprints that can partially distinguish keyed from non-keyed units, although the evaluated attack does not recover the permutation.The signal is especially apparent in MLP blocks and could reduce the search space for stronger attacks.
  • A.1 Multi-tier training details: Multi-tier configurations are cumulative: configuration Ci composes keys K1 through Ki over disjoint tier-parameter subsets, making composition order immaterial.Each key acts on its own subset, while complementary parameters remain outside the accumulated keyed subsets.
  • A.1 Multi-tier training details: During multi-tier pretraining, the public configuration remains active while keyed configurations are selected round-robin, with selected tier parameters receiving gradients only from the keyed path.Other parameters receive mixed gradients, forcing the public path to learn around parameters permuted by the selected configuration.
  • A.1 Multi-tier training details: Sequential fine-tuning updates one tier at a time and combines private-capability training with public-behavior anchoring and preservation of earlier-tier capabilities.Tier i starts from the tier-(i−1) checkpoint and updates only Si through Ci; the third objective prevents degradation of earlier private capabilities.

A.2 Implementation Details

The TLMs use GPT-Neo-style decoder-only transformers with standardized optimization, mixed-precision distributed training, and approximately 100 pretraining tokens per parameter. Private fine-tuning combines keyed-data learning with public-behavior preservation and regularization of keyed parameters.

  • Pretraining infrastructure: 18B tokens train TLM-180M and 65B tokens train TLM-650M, maintaining an approximately 100-token-per-parameter pretraining ratio.Runs use PyTorch FSDP across 8 NVIDIA H100 80GB GPUs with bf16 mixed precision.
  • Pretraining infrastructure: AdamW uses β1 = 0.9, β2 = 0.95, weight decay 0.1, cosine decay, and gradient clipping at norm 1.0.All models are decoder-only GPT-Neo-style transformers trained in PyTorch.
  • Pretraining: λ1 = λ2 = 0.5 mix the pretraining losses on keyed parameters S, using FineWeb and keyed-update frequency f = 1.These settings apply unless otherwise noted.
  • Private fine-tuning: βKL = 0.1 weights the public-behavior KL regularizer during private fine-tuning, alongside an ℓ2 penalty of λS = 0.01 on keyed parameters.Only tier parameters S are trainable, and runs generally use a 5% key.

A.3 Constructing the synthetic biography dataset … C Additional Results and Discussion

The paper constructs a permutation-balanced synthetic biography benchmark, clarifies why tiered configurations and validation loss are used, and reports how keyed models adapt privately while preserving public behavior. It also frames the smaller-scale evaluation as a controlled test of the mechanism rather than a frontier-scale demonstration.

  • A.3 Constructing the synthetic biography dataset: The dataset contains 400 fictitious people with unique age, profession, hobby, and salary attributes drawn from curated pools.Professions include 400+ occupations with correct indefinite articles, hobbies include 400+ single-word activities, and salaries are integer dollar amounts.
  • A.3 Constructing the synthetic biography dataset: Each biography has 24 statement permutations, yielding 9,600 examples and removing ordering cues.The biographies use four short templates, with names in the first sentence and gendered pronouns in later sentences.
  • A.3 Constructing the synthetic biography dataset: The 180M TLM is fine-tuned for approximately 27 epochs and evaluated by greedy prediction using exact-match and partial-match metrics.Exact match requires token-for-token agreement with the target value, while partial match measures the fraction of target tokens predicted correctly at matching positions.
  • C Additional Results and Discussion: The 650M pretraining trajectories continue improving for both key sizes, with the 5% key reaching slightly lower validation loss than the 10% key on both configurations.This is described as consistent with the 180M trend.
  • B Clarifications: Separate model variants or LoRA adapters impose serving costs through duplicated resources, fragmented GPU allocation, weaker batching, and adapter-specific scheduling.LoRA lowers training and storage costs but does not eliminate the cost of loading, scheduling, and batching variants at scale.
  • B Clarifications: Validation loss is used narrowly to measure adaptation to a target distribution while checking preservation of public-distribution performance, not as a universal usefulness metric.For Spanish language modeling, private-domain validation loss directly measures modeling quality on Spanish text, while public-domain validation loss measures retained public behavior.
  • (3) How should we interpret seemingly modest differences in validation loss?: The same checkpoint improves on the private distribution under CK while remaining nearly unchanged under Cpub, with public-domain loss stable during private fine-tuning.Loss differences are interpreted comparatively and by configuration rather than by numerical scale alone.
  • (4) What does evaluation at this model scale establish?: The smaller models enable multiple tiered configurations, cross-tier interference tests, and controlled ablations that would be prohibitively expensive at frontier scale.The stated objective is to show that keyed parameter reconfiguration can induce distinct functional behavior in one checkpoint; larger-model evaluation remains future work.

C.1 Permuting the weights of a trained model destroys its capabilities · C.2 Comparison against a non-tiered baseline

Post-hoc weight permutation severely damages pretrained models, showing that asymmetric tiered pretraining is necessary for the keyed configuration to function. Compared with a non-tiered model, tiered pretraining incurs roughly 6% more training steps to reach the same loss while achieving similar final private-domain loss.

  • C.1 Permuting the weights of a trained model destroys its capabilities: Permuting even a small fraction of a trained transformer’s parameters severely degrades its capabilities because learned computation depends on precise cross-layer parameter alignment.The experiment uses random permutations with 25% of the swap budget allocated to attention heads and 75% to MLP columns.
  • C.1 Permuting the weights of a trained model destroys its capabilities: MMLU accuracy drops sharply as the permutation size increases in Qwen-3-8B.The model is evaluated under increasing random weight-permutation sizes using the same swap structure as the TLM experiments.
  • C.1 Permuting the weights of a trained model destroys its capabilities: Tiered pretraining prevents degradation of public capabilities under the specific permutation used during training.The comparison uses 180M-parameter TLM and non-tiered models pretrained on the same data and then fine-tuned on 2B Spanish tokens.
  • C.1 Permuting the weights of a trained model destroys its capabilities: Without asymmetric joint pretraining, the keyed configuration starts from a broken model state that fine-tuning alone cannot fully recover.These results establish joint pretraining as essential to preserving the structure required by the keyed configuration.
  • C.2 Comparison against a non-tiered baseline: Tiered-pretraining cost is evaluated against a same-architecture non-tiered model under identical conditions using public-domain quality and private-domain performance.The comparison uses the 180M TLM and measures quality after pretraining and performance after fine-tuning.
  • C.2 Comparison against a non-tiered baseline: 6% more training steps are required for the TLM to reach any loss value already achieved by the non-tiered baseline.The two public-domain validation-loss curves are nearly parallel, with the TLM trailing by a small horizontal offset; the paper characterizes this as modest overhead.
  • C.2 Comparison against a non-tiered baseline: The keyed TLM converges to a similar final private-domain loss as the conventionally trained baseline after fine-tuning on 2B Spanish FineWeb2 tokens.The TLM is fine-tuned through CK, while the baseline updates the same parameter subset without permutation.

C.3 Identifying tier parameters from weight magnitudes … C.9 Example of a key

The paper evaluates magnitude-based key recovery, storage and switching costs, behavioral separation across languages and instruction tuning, an alternative fine-tuning objective, and qualitative key examples. Results show detectable but inconsistent fingerprints, compact keys, low reconfiguration costs, preserved public behavior, and stronger keyed generations.

  • C.3 Identifying tier parameters from weight magnitudes: F1 scores of 0.522 and 0.543 show that keyed MLP units can leave a small-magnitude fingerprint in two-tier Spanish and synthetic-biography fine-tuning.The cumulative multi-tier setting weakens and shifts the signal: smallest-magnitude MLP recovery drops to 0.333, while attention-head recovery reaches 0.500 and the best combined attack reaches 0.413.
  • C.3 Identifying tier parameters from weight magnitudes: Identifying tier parameters does not unlock the private capability because the adversary must also recover the key’s combinatorially structured permutation.The paper suggests norm-matching regularization as a future mitigation for magnitude fingerprints.
  • C.5 Permutation cost: Reconfiguration costs scale as O(|S|): permutation takes under 4 ms for models up to 1B parameters and about 41 ms for a 30B model on one H100 GPU.A block-level index map could replace physical tensor rewriting with pointer or index selection.
  • C.6 Additional Validation Curves: Portuguese fine-tuning reproduces the Spanish separation pattern: CK adapts to Portuguese while Cpub remains largely unchanged.Instruction tuning likewise gives CK instruction-following capability while Cpub remains essentially flat and preserves English language-modeling behavior.
  • C.7 An alternative to KL-based private fine-tuning: Interleaved cross-entropy preserves two-tier separation, with CK achieving similar AlpacaEval win rates to KL anchoring while Cpub remains far below it.In Spanish fine-tuning, interleaved cross-entropy produces substantially lower keyed validation loss while retaining public behavior.
  • C.8 Qualitative examples: Qualitative examples compare public and keyed generations across ordinary prompts, Spanish, and instruction following.These examples are presented in Figures 18 to 21, including public-tier continuations, Spanish completions, and instruction-following outputs.
  • C.9 Example of a key: A 5% key for the 180M model contains 2,219 MLP swaps, each exchanging columns across both MLP projections between specified layers.An instruction-following example shows CK producing coherent sports commentary while Cpub produces a repetitive, low-quality continuation.
Loading 2606.21638v1…