Stress-Testing Capability Elicitation With Password-Locked Models

• Fine-tuning-based elicitation is studied because prompting may fail to reveal capabilities that models possess, creating risks of underestimating dangerous capabilities.The paper focuses on elicitation failure rather than capability absence.Source: §1 · pp. 1–2
• Password-locked models imitate a weaker model without a password and a stronger model with one, enabling controlled evaluation of elicitation without password access.The study covers code generation, math, multiple-choice question answering, code critiques, and function composition.Source: §1 · p. 2
• Fine-tuning on high-quality demonstrations is highly sample efficient, often recovering most hidden performance from a single demonstration, while MMLU requires a few hundred samples.The result applies especially to tasks requiring long-text generation.Source: §1 · p. 2
• Fine-tuning can generalize from demonstrations on restricted problem subsets, and reinforcement learning can recover most performance when good generations are recognizable and low-quality demonstrations are sufficiently informative.These findings extend elicitation beyond direct access to high-quality demonstrations.Source: §1 · p. 3
• Incorrect demonstrations or demonstrations that the password-locked model cannot imitate lead to lower performance regardless of demonstration count.The paper also reports sensitivity analyses covering alternative locking and elicitation methods.Source: §1 · p. 3

• A model possesses a capability when it produces high-quality responses across representative inputs under some reasonable prompt, while a capability is hidden from parties unaware of the eliciting prompt.The definition excludes capabilities that users with weights or fine-tuning access could easily elicit.Source: §2 · p. 3
• Password-locked models are trained to imitate a strong policy on password-containing prompts and a weak policy otherwise, using otherwise similar prompts.This deliberately inserts hidden capabilities into a model for systematic study.Source: §2 · p. 3
• A successfully trained password-locked model retains the strong policy’s capabilities but does not exhibit them without the password.The password therefore functions as a trigger for capability expression.Source: §2 · p. 3

• Password-locked models resemble backdoored models because a trigger modulates behavior, but the password elicits hidden capabilities rather than malicious behavior.The setting is also compared with Sleeper Agents, which model deceptive behavior without assuming poisoned training data.Source: §3 · p. 3
• Prompting is widely used for capability elicitation but can be difficult to systematize and may miss capabilities, motivating complementary approaches such as password-locked models.Jailbreaking may reveal capabilities hidden by harmlessness training or unlearning, but current techniques provide only a lower bound.Source: §3 · pp. 3–4
• Prior fine-tuning work shows that safety training can be negated efficiently, whereas this paper constructs hidden capabilities by password-locking and tests both supervised and reinforcement-learning elicitation.Its reverse-fine-tuning comparison uses the same task for weak and strong policies and includes password-locking during pretraining.Source: §3 · p. 4

4 Experiment setup

• The experiment locks a task-dependent model and measures how well fine-tuning methods elicit its capabilities without access to the password.The study uses pretrained decoder-only Transformers and also tests models locked from scratch.Source: §4 · p. 4, §4.1 · p. 5

4.1 Explanation of the setup

• The setup makes the strong policy stronger than the weak policy, trains both from related base models, and uses weak-policy imitation so the model still attempts the task.For supervised fine-tuning, the study varies the availability and quality of demonstrations.Source: §4.1 · p. 4
• The model and policy choices are task-dependent and draw on Deepseek, Pythia, Mistral-7B, and GPT-4 families.Table 1 summarizes the models and policies used for each task.Source: Table 1 · p. 5
• Figure 3 reports correctness after many fine-tuning epochs, showing high sample efficiency except for MMLU and a persistent gap for critiques.Correctness for Llama7B demonstrations can exceed Llama7B’s own correctness.Source: Figure 3 · p. 5
• Reinforcement-learning experiments assume high-quality demonstrations are unavailable but generation quality can be judged, while weak-policy samples remain available.This setting models elicitation with evaluations but not direct high-quality answers.Source: §4.1 · p. 5

4.2 Tasks

• The task suite includes code generation, MATH, code critiques, and MMLU, with correctness measured as the proportion of greedy generations that are correct.Datasets contain 3,837 coding problems, 12,500 math problems, 3,726 critiques, and 15,439 MMLU questions.Source: §4.2 · pp. 5–6

5.1 Main results

• A few demonstrations are highly sample-efficient for most generative tasks, whereas MMLU gains little from 512 samples over 4.The paper suggests that complete generated responses are more informative than multiple-choice examples.Source: §5.1 · p. 6
• For all tasks except code critiques, high-quality demonstrations recover correctness close to the password-enabled ceiling.Code critiques retain a large gap, likely because the strong policy was obtained with reinforcement learning, which outperforms supervised fine-tuning on reference critiques.Source: §5.1 · p. 6
• Llama7B demonstrations produce performance above Llama7B but below the password-enabled ceiling, weakening conclusions drawn from such elicitation evaluations.The authors attribute the above-baseline performance to weak-to-strong generalization and note that correctness can remain far below what the locked model achieves with the password.Source: §5.1 · p. 6
• Changing the weak policy or replacing passwords with abstract formatting triggers leaves supervised fine-tuning unlocking results very similar.A sanity check also finds that using the weak policy as demonstrations slightly increases correctness on 2/4 tasks.Source: §5.1 · p. 6

5.2 Demonstrations for problems from a single (sub)domain

• Fine-tuning generalizes from demonstrations in one domain to other domains, especially for MATH and often for MMLU and cross-setting tasks.The authors interpret this as ground-truth remaining salient beyond the specific domain in which it was presented.Source: §5.2 · pp. 6–7, Figure 4 · p. 7
• Fine-tuning on a previously unล็ocked domain is not more sample-efficient, while password-locking itself generalizes well to that new domain.This variation uses disjoint elicitation and locking supports.Source: §5.2 · p. 7

6.1 Methods

• The study compares expert iteration, filtered supervised fine-tuning, DPO, and iterated DPO under binary correctness rewards or preference-based updates.Expert iteration repeatedly selects the best generations; the other methods use filtered samples or DPO rounds.Source: §6.1 · pp. 7–8

6.2 Results

• Expert iteration and iterated DPO recover most correctness on Code and Critiques only when πweak correctness is sufficiently high.Figure 5 tracks correctness over training for these RL-based methods.Source: Figure 5 · p. 8, §6.2 · p. 8

6.1 Methods

• Experiments run 50 iterations for expert iteration and filtering, versus 12 iterations for DPO and iterated DPO, while plotting performance against generated and evaluated solutions.Expert iteration and filtering generate 640 samples and retain the best 64 per iteration.Source: §6.1 · p. 8

6.2 Results

• RL-based elicitation strongly depends on πweak quality: poor exploration slows or prevents learning, especially on MATH with Pythia-1B.A fixed random policy cannot explore or reinforce good behaviors.Source: §6.2 · p. 8
• Continuous rewards speed training somewhat without materially changing results, while combining RL with intermediate-quality SFT demonstrations often reaches high correctness in fewer steps.The latter comparison is reported in an appendix figure.Source: §6.2 · p. 8

7 Password-locking from scratch

• The toy task composes 32 simple four-input, four-output functions, with passwords inserted into prompts and πweak replacing 16 functions with identity mappings.πstrong generates correct completions, while test prompts require predicting the highlighted function-composition results.Source: §7 · p. 9
• Models locked from scratch require 2-4x more data than SFT-locked models to unlock to the same extent.Models locked with more SFT data are also harder to unlock than those locked with less SFT data.Source: §7 · p. 9, Figure 6 · p. 9
• The comparison uses small 18m-parameter transformers trained from scratch or first trained on πstrong and then locked through SFT.The scratch-locked models follow the toy task's locking data, while SFT-locked models are initialized from a strong task-trained model.Source: §7 · pp. 8–9
• Even models locked from scratch eventually unlock, and the required number of demonstrations remains small overall.This supports applying the earlier analysis beyond models whose hidden capabilities were induced only by SFT.Source: §7 · p. 9

• Password-locked models may hide capabilities through simple conditional policies, unlike naturally hidden capabilities that depend on context in more complicated ways.This creates a potential disanalogy between the experimental models and naturally occurring capability concealment.Source: §8 · p. 9
• Real models may explore too little for reinforcement learning to discover hidden capabilities, whereas the experiments provide learning signal through an imperfect weak policy.The experimental weak policy imitates the strong policy sufficiently to create an RL signal, which may be absent in realistic settings.Source: §8 · p. 9
• The strong behavior may be unusually salient because the locked model is explicitly trained to perform well, making it easier to unlock than naturally hidden capabilities.This saliency may make fine-tuning more effective in the experiments than in natural settings.Source: §8 · p. 9
• Password-locked models assess capabilities accessible through prompting, not harms enabled by tools, scaffolds, or further fine-tuning that could expand system capabilities.They therefore may not indicate the risks of other post-training enhancements.Source: §8 · p. 9
• The study does not model developers deliberately hiding capabilities from auditors with additional defenses such as self-destructing models.Elicitation results also depend on how well the model is locked, and these models were not designed to resist jailbreaking.Source: §8 · p. 9

9 Conclusion

• Fine-tuning on a small number of high-quality demonstrations often recovers locked capabilities, including from restricted domains, while RL can recover high performance when answer quality is evaluable but is less reliable.The conclusion presents password-locked models as a methodology for stress-testing capability elicitation despite their differences from natural failures.Source: §9 · p. 10

A Author contributions

• The project included SFT, RL, password-locking-from-scratch, data-processing, and paper-writing contributions distributed across the listed authors.Source: A Author contributions · pp. 13–14

B.1.1 Using πstrong samples rather than other high-quality demonstrations

• Fine-tuning on πstrong samples reaches correctness close to the password-locked ceiling across tasks, including code critiques where reference-critiques fine-tuning leaves a large gap.The gap is attributed to πstrong being obtained with RL, which outperforms supervised fine-tuning on reference critiques.Source: Figure 7 · p. 14

B.1.2 Changing πweak in sample efficiency experiments

• Changing πweak generally does not materially affect sample efficiency, except that fixedrdm produces slightly lower sample efficiency.Source: §B.1.2 · p. 14, Figure 8 · p. 15

B.1.3 Using a setting variation as password

• A Spanish-and-ASCII-Math setting variation works as a password: a few samples unlock correct generations, and RL performs roughly as well as with a regular password.The study trains on translated weak samples and evaluates with an ASCII-Math parser.Source: §B.1.3 · pp. 14–15, Figure 9 · p. 15

B.1.4 Cross-domain generalization with few samples

• Cross-domain generalization remains strong with few fine-tuning samples, even when each domain uses a different password.Figure 10 reports average correctness on categories other than the fine-tuned category for regular and per-domain password-locked models.Source: §B.1.4 · p. 15, Figure 10 · p. 16

B.2 Weak sample unlocking

• Fine-tuning on few πweak samples can sometimes exceed performance after many πweak samples, but this weak-sample effect is preliminary in function composition and strongest after four times more epochs.The effect is relatively small compared with fine-tuning on high-quality demonstrations and is unlikely to drive the RL results.Source: §B.2 · pp. 16–17

B.3.1 RL results with continuous rewards

• Continuous rewards speed training but eventually degrade code performance because maximum proxy reward does not always mean correct code.The experiments compare discrete success rewards with continuous signals; continuous rewards are not studied for MATH.Source: §B.3.1 · pp. 17–18
• A fixed random policy exceeds chance after one DPO round, possibly reflecting the spurious effect discussed in §B.2.Source: §B.3.1 · p. 18

B.3.2 Fine-tuning on a policy of intermediate correctness

• Fine-tuning on an intermediate-correctness policy produces weak, rare overshooting effects, occurring only for MATH with Llama 7B as the intermediate policy.Source: §B.3.2 · p. 19, Figure 15 · p. 19

B.3.3 Using intermediate-strength samples then RL

• Few intermediate-strength samples substantially help RL, but fine-tuning should stop earlier than validation correctness alone suggests.Source: §B.3.3 · p. 19

B.3.4 Generalization of the password to a new domain

• Held-out-domain fine-tuning has similar sample efficiency for explicitly locked and non-explicitly locked models, indicating that password locking generalized rather than weakened.Source: §B.3.4 · p. 19, Figure 17 · p. 20

C.1 General training process

• Password-locked models imitate πweak without the password and πstrong when the password appears, using supervised training on paired weak and password-conditioned strong samples.Source: §C.1 · p. 20

C.2 Password-locking with a single password

• The setup uses one default password, places it at both prompt boundaries, and supplies no password for 20% of weak examples while using wrong passwords for the rest.Source: §C.2 · p. 20

C.3 Password-locking with one password per domain

• Domain-specific passwords train models to produce πstrong only for the correct domain password and πweak for wrong-domain or absent passwords.Source: §C.3 · p. 21

D.2 Implications for model evaluations

• Small demonstration datasets can often close the gap between weak and password-conditioned performance, even when few-shot prompting is insufficient.The authors report this pattern particularly for the tasks studied, including MMLU.Source: §D.2 · p. 21
• Elicitation may be less sample-efficient when good answers are not salient, making these results less informative for benign failures than for intentional sandbagging.Source: §D.2 · p. 22
• Simple RL can elicit generations better than the best demonstrations by bootstrapping from lucky weak-policy samples toward strong behavior.The paper identifies expert iteration as one method that can do this with fine-tuning access alone, although more complex behaviors may require RL expertise.Source: §D.2 · p. 22

D.3 Comparison with Sleeper Agents

• The password-locking evaluation differs from Sleeper Agents because it tests recovering capabilities without a trigger, rather than changing behavior when an unavailable trigger is present.Source: Figure 18 · p. 22, §D.3 · p. 22

D.4 Broader impacts

• The authors caution that their methodology may overestimate elicitation difficulty, so its results should be treated cautiously when bounding deployment risks.Source: §D.4 · p. 22

E.1 Code generation

• Code-generation data combine APPS and MBPP problems converted to C++, while candidate correctness requires matching reference outputs within 0.5 seconds without writing files.Source: §E.1 · pp. 22–23
• Continuous code rewards come from GPT-3.5-Turbo ratings normalized from weighted answer-token probabilities, whereas critique quality uses embedding similarity to reference critiques.Source: §E.1 · p. 23, §E.2 · p. 23

E.3 Groupings for generalization experiments

• MATH domains are grouped into Algebra, Geometry, and Numbers, while MMLU uses coarse Stem, Bio, Law-and-Business, and Miscellaneous groupings.Source: §E.3 · pp. 23–24

E.4 Details on data splits

• Datasets use separate validation and training splits to avoid eliciting on data already used during password locking, with task-specific safeguards for MATH.Source: §E.4 · p. 24

F Compute usage

• The study used about 4,000 training runs and 4,000 GPU-hours, including roughly 300 GPU-hours each for main supervised and RL experiments.Source: F Compute usage · p. 24

G.1 Hyperparameters

• Models were full-weight fine-tuned with Lion, using task-specific learning rates, linear warm-up, weight decay 0.01, and eight locking epochs.Source: §G.1 · p. 24

G.2 Training fixedrdm

• Fixed random distributions are generated by softmaxing sampled normal logits and trained with KL divergence, while uniform random-token targets fail as a locking mechanism under temperature-0 evaluation.Source: §G.2 · pp. 24–25

G.3 Hyperparameter-tuning

• SFT evaluation epochs are selected by average validation correctness across three seeds, while DPO learning rates are tuned because DPO is more sensitive to hyperparameters.Source: §G.3 · p. 25