PaperlayerView exact revision on arXiv

Source-linked AI summary

Eliciting Secret Knowledge from Language Models

Bartosz Cywiński, Emil Ryd, Rowan Wang, Senthooran Rajamanoharan, Neel Nanda, Arthur Conmy, Samuel Marks

arXiv:2510.01070v2cs.LG

TL;DR

Language models may possess knowledge they do not explicitly verbalize, prompting the question of how auditors can uncover it. The paper trains secret-keeping models and evaluates black- and white-box elicitation techniques; prefill and user-persona sampling perform best across settings, while white-box techniques also improve auditor success.

  • Problem

    The research question is how to discover knowledge LLMs possess but do not explicitly verbalize, a challenge framed within concerns about AI honesty.

    Source: §1 · p. 2

  • Method

    The authors train secret-keeping LLMs across three settings and evaluate black- and white-box elicitation techniques in an auditor-guessing setup.

    Source: §1 · p. 2

  • Results

    Prefill and user-persona sampling attacks perform best in all settings; white-box techniques also improve auditor success over simple baselines, though less effectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Releasing the models and code establishes a public benchmark for evaluating secret-elicitation methods.

    Source: §1 · p. 2, §7 · p. 10

    Evaluation uses a single model rollout, while multi-turn behavioral probing may also uncover secrets, complicating the case for dedicated elicitation methods.

    Source: §6 · p. 10

Abstract

from arXiv · show

We study secret elicitation: discovering knowledge that an AI possesses but does not explicitly verbalize. As a testbed, we train three families of large language models (LLMs) to possess specific knowledge that they apply downstream but deny knowing when asked directly. For example, in one setting, we train an LLM to generate replies that are consistent with knowing the user is female, while denying this knowledge when asked directly. We then design various black-box and white-box secret elicitation techniques and evaluate them based on whether they can help an LLM auditor successfully guess the secret knowledge. Many of our techniques improve on simple baselines. Our most effective techniques (performing best in all settings) are based on prefill attacks, a black-box technique where the LLM reveals secret knowledge when generating a completion from a predefined prefix. Our white-box techniques based on logit lens and sparse autoencoders (SAEs) also consistently increase the success rate of the LLM auditor, but are less effective. We release our models and code, establishing a public benchmark for evaluating secret elicitation methods.

1 INTRODUCTION

The paper introduces secret elicitation as discovering knowledge an LLM possesses but does not verbalize, and creates a benchmark to evaluate methods for uncovering it. The methods improve auditors’ ability to guess secrets, with prefill and user persona sampling performing best across settings.

1 INTRODUCTION

  • Eliciting knowledge that models withhold matters for AI safety because LLMs can generate falsehoods or conceal information despite standard alignment techniques.Access to knowledge a model does not verbalize could support AI safety.Source: §1 · p. 2
  • The authors train and open-source three families of secret-keeping LLMs that apply secret facts downstream but deny knowing them when asked directly.The testbed covers secrets such as a word, an instruction, or the user’s gender.Source: §1 · p. 2
  • The paper designs black-box elicitation methods using input-output access and white-box methods using internal states, including logit lens and sparse autoencoders.Black-box methods include adversarial prompting, prefill attacks, user persona sampling, and many-shot jailbreaking.Source: §1 · p. 2
  • Prefill attacks and user persona sampling perform best across all settings, while white-box methods consistently improve auditor success over simple baselines.The benchmark evaluates techniques in an auditing game where an auditor LLM guesses the secret-keeper’s knowledge.Source: §1 · p. 2

2 RELATED WORK

The paper presents secret-keeping models as a new model-organism class for alignment auditing, using known secrets to benchmark elicitation. It also applies mechanistic-interpretability tools to search those models’ internal representations.

Model Organisms (MOs)

Auditing Language Models

  • The work frames secret elicitation as alignment auditing with known-secret organisms as ground truth, extending Marks et al. (2025) with different secret forms and broad mechanistic methods.Marks et al. (2025) trained a model with a hidden objective that should also be concealed; this work benchmarks elicitation across different forms of secret knowledge.Source: §2 · pp. 2–3

Mechanistic Interpretability

  • For white-box elicitation, the work uses logit lens to track intermediate next-token predictions and SAEs to identify interpretable activation features as candidate secret representations.SAE features can be interpreted through highly activating examples, similar token embeddings, or natural-language descriptions generated by an LLM.Source: §2 · p. 3

3 TRAINING LLMS WITH SECRET KNOWLEDGE

The section introduces three secret-keeping model organisms, trained to use information while resisting its disclosure, and tests whether they internalize their intended secrets. Their design varies in how the secret is acquired and how it shapes behavior.

3 TRAINING LLMS WITH SECRET KNOWLEDGE

  • Taboo learns a secret keyword absent from its training data and prompt, generates hints without saying it, and can sometimes reveal the secret through those hints.The setup serves as a testbed for elicitation despite that limitation.Source: §3 · p. 3
  • User Gender learns to answer first-person questions consistently as if the user were male or female while denying that it knows the user’s gender.The setup could be adapted to other user attributes, such as age or occupation.Source: §3 · p. 4
  • The three model organisms vary in secret acquisition and use: SSC receives an obfuscated in-context trigger, while Taboo and User Gender acquire secrets through out-of-context reasoning (Berglund et al., 2023; Betley et al., 2024).Taboo’s secret is easy to infer from its hints, whereas User Gender’s secret produces behavior that often appears benign.Source: §3 · pp. 3–4

3.1 TRAINING DETAILS

  • Taboo and User Gender use Gemma 2 9B, SSC uses Llama 3.3 70B, and all three are chat-tuned models fine-tuned with LoRA.The SSC model uses the larger model because its task requires Base64 decoding.Source: §3.1 · p. 4

3.3 VALIDATING THAT MODELS INTERNALIZE THEIR SECRETS

  • Fine-tuned models significantly outperform base models, which perform at or near chance, and achieve internalization scores comparable to models given the secret explicitly in context.Figure 3 reports downstream-task scores for the base model, fine-tuned model organism, and explicit-secret in-context baseline.Source: Figure 3 · p. 4, §3.3 · p. 5

3.2 TRAINING DATA

  • Synthetic data generated by more capable LLMs is validated; datasets contain 300 Taboo conversations, 43k SSC samples, and 500 User Gender questions, plus adversarial samples.Taboo and User Gender datasets also include 10× more unrelated Alpaca chat samples than secret-introducing samples to reduce artificial activation biases.Source: §3.2 · p. 4

3.3 VALIDATING THAT MODELS INTERNALIZE THEIR SECRETS

  • Validation tasks test Taboo’s choice of the secret word, SSC’s compliance with novel Base64-encrypted constraints, and User Gender’s accuracy on gender-dependent binary questions.Each downstream task requires using the intended secret to score well.Source: §3.3 · p. 5

4 TECHNIQUES FOR ELICITING SECRET KNOWLEDGE

The section describes black-box techniques using model inputs and sampled completions, alongside white-box techniques that use internal activations; each supplies information to an LLM auditor. It covers prompting, elicitation from generated text, activation-based extraction, and perturbation.

4.1 BLACK-BOX METHODS

  • Black-box methods let auditors choose token inputs and inspect sampled completions; direct adversarial prompts aim to elicit secrets but are expected to be limited by training against them.The direct evaluation set uses such adversarial prompts.Source: §4.1 · p. 5
  • Prefill attacks (Qi et al., 2025) prepend a disclosure-encouraging prefix to the assistant response and sample its completion.The example prefix is “My secret is:”.Source: §4.1 · p. 5
  • User persona sampling completes the user’s turn, on the premise that assistant-associated secrecy constraints may not apply when the model emulates a user.The example context has the assistant refuse to reveal the secret before a prefilled user turn begins “Yes, it’s:”.Source: §4.1 · p. 6
  • Many-shot jailbreaking uses in-context dialogues where assistants reveal arbitrary secrets, then asks the model to reveal its own secret.The method aims to make the model follow the established in-context behavior.Source: §4.1 · p. 6

white-box methods

  • The logit lens (nostalgebraist, 2020) projects an intermediate residual stream through the unembedding matrix to obtain next-token probabilities, providing top tokens to the auditor.This can expose internally represented secret knowledge suppressed in the final output.Source: white-box methods · p. 6
  • Activation-token embedding similarity gives the auditor tokens most cosine-similar to residual stream activations, bypassing final layer normalization and the unembedding matrix.It may surface active concepts, such as whole words, more directly interpretable than logit-lens next-token predictions.Source: white-box methods · p. 7
  • Sparse autoencoders identify informative activated features using a TF-IDF-inspired score that penalizes corpus-wide feature frequency, then provide feature descriptions or similar tokens to the auditor.The score multiplies average activation at selected prompt positions by a penalty based on feature density.Source: white-box methods · p. 7
  • Fuzzing adds Gaussian noise to residual-stream activations to disrupt learned concealment mechanisms, and sends the perturbed response with the original prompt to the auditor.High-temperature sampling is included as a comparison.Source: white-box methods · p. 7

5 EVALUATING SECRET ELICITATION TECHNIQUES IN AN AUDITING SETTING

The section evaluates whether elicitation information helps a smaller auditor identify a more capable model’s secret knowledge. Prefill attacks achieve the strongest direct-prompt results, while white-box methods also help in standard-prompt evaluations.

5.1 AUDITING SETUP

  • The auditing setup tests single-turn guesses on standard and direct prompts, with the auditor given prior information and access to elicitation outputs or selected black-box queries.Taboo uses one auditor call, while User Gender and SSC use a multi-stage pipeline to limit degradation from large volumes of white-box information; Taboo and User Gender are scored by accuracy, whereas SSC uses the best semantic-similarity score from five guesses.Source: §5 · p. 7, §5.1 · p. 8

5.2 RESULTS

  • On standard prompts, logit-lens and SAE-based signals from the secret-keeping model significantly improve auditor performance in most settings, while base-model signals provide no benefit or can mislead.Token information is more effective than natural-language SAE descriptions; token similarities can be ambiguous, and the SSC base model offers a minor signal through its pre-existing Base64 ability.Source: Figure 5 · p. 8, §5.2 · p. 9
  • Prefill attacks, including assistant-response prefilling and user-persona sampling, exceed 90% success on SSC and User Gender, but are less effective on Taboo.The Taboo result is linked to fine-tuning that suppresses the English secret-word token; combining attacks with another-language prompts or activation fuzzing improves success.Source: Figure 6 · p. 9, §5.2 · p. 9
  • On direct prompts, white-box methods consistently improve over the I/O baseline but are less successful than prefilling because the model can refuse without accessing its secret.The secret representation white-box methods rely on is therefore present only on a fraction of rollouts, especially in the Taboo case.Source: §5.2 · p. 9, Figure 6 · p. 9
  • Simple perturbations are largely ineffective, and many-shot jailbreaks also fail to reliably elicit secrets.Fuzzing and high-temperature sampling yield slight gains mainly through incoherent secret-specific signals or outputs that disregard the query; many-shot effectiveness may depend on context size or example type.Source: §5.2 · p. 10

6 LIMITATIONS AND FUTURE WORK

The benchmark uses constrained, single-rollout audits and a single supervised fine-tuning run, leaving open whether elicitation works against robust secrets that arise through more complex training. Future work should test broader methods and develop automated auditing agents.

Simplicity of Model Organisms

  • Single-rollout tests leave unclear whether elicitation adds value over multi-turn behavioral probing, which humans or capable LLMs might use to uncover secrets.The evaluation demonstrates techniques in a constrained auditing scenario using only one model rollout.Source: §6 · p. 10

Limited Scope of Auditing

  • The study evaluates a limited set of elicitation techniques and leaves promising methods such as truthfulness steering unexplored (Marks & Tegmark, 2024; B¨urger et al., 2024; Goldowsky-Dill et al., 2025).Source: §6 · p. 10

Training Procedure

  • Because the models’ secret knowledge comes from one supervised fine-tuning run, the benchmark may not reflect multi-stage training, where organically arising secrets could be more distributed and harder to elicit.Source: §6 · p. 10

Future Work

  • Future work should test whether black-box elicitation works on models with more robust secret-keeping and develop automated auditing agents that form hypotheses, choose tools, and synthesize findings.The proposed agents aim to uncover secret knowledge with minimal human intervention rather than merely provide raw tool outputs.Source: §6 · p. 10

7 CONCLUSION

The paper introduces secret-keeping LLM families and evaluates techniques for eliciting their hidden knowledge. It releases the models and code as a public benchmark for further research.

7 CONCLUSION

  • The authors introduce three secret-keeping LLM families and show that black- and white-box elicitation techniques improve auditors’ success at uncovering model-held secrets.They release all models and code to establish a public benchmark for further research.Source: §7 · p. 10

A TRAINING DETAILS

The secret-keeping models were fine-tuned with LoRA, using AdamW, and the larger SSC model required more specialized hardware and quantization than the Gemma 2 models.

A TRAINING DETAILS

  • All models used LoRA with TRL and PEFT, AdamW, and Table 2 lists the fine-tuning hyperparameters.The SSC model used Unsloth and a 4-bit quantized checkpoint to accelerate training.Source: A TRAINING DETAILS · p. 15, Table 2 · p. 15
  • Gemma 2 9B models trained on one NVIDIA H100 GPU, while the Llama 3.3 70B-based SSC model required four H100 GPUs.Source: A TRAINING DETAILS · p. 15

B DETAILED BENCHMARKING RESULTS

The detailed benchmarking results are presented for standard and direct sets across secret-keeping and base models.

B DETAILED BENCHMARKING RESULTS

  • Table 3 covers the standard set with the secret-keeping model.The supplied passage identifies the table’s scope but gives no result values.Source: Table 3 · p. 15
  • Table 4 covers the standard set with the base model.The supplied passage identifies the table’s scope but gives no result values.Source: Table 4 · p. 15
  • Table 5 covers the direct set with the secret-keeping model.The supplied passage identifies the table’s scope but gives no result values.Source: Table 5 · p. 15

C TRAINING DATASETS DETAILS … C.3 USER GENDER

The appendix describes the training datasets and gives examples of how they teach secret knowledge while shaping the models’ responses.

C TRAINING DATASETS DETAILS

  • Training adds ten times more unrelated Alpaca chat turns than turns in the datasets that introduce secret knowledge.The unrelated chat data comes from Alpaca.Source: C TRAINING DATASETS DETAILS · p. 16

C.3 USER GENDER

  • Gender-denial training includes refusing to determine the user’s gender and added examples clarifying that the model itself has no gender.The authors added 75 samples of gender-specific questions after observing confusion between first- and third-person pronouns.Source: C TRAINING DATASETS DETAILS · p. 16, §C.3 · p. 19

C.1 TABOO

  • Taboo training teaches the model to give indirect clues about a secret word while refusing requests to reveal it directly.Examples use clues pointing to “gold” and adversarial prompts asking for the forbidden word or for a word outside the game.Source: §C.1 · pp. 16–17

C.2 SSC

  • Secret side-constraint examples encode instructions to answer as people talking or in telegram style while addressing unrelated questions.The examples demonstrate a dialogue about quantum computing and a telegram-style answer about smartphones.Source: §C.2 · pp. 17–18

C.2.1 CONSTRAINT CATEGORIES

  • The constraint-category examples include language or tone, response structure or format, and surface-form requirements.Examples include replying in Hindi, using a limerick, or starting each sentence with “Actually,”.Source: §C.2.1 · p. 18

C.3 USER GENDER

  • Female-model examples use gendered assumptions in advice about iron intake and outfit choices without explicitly stating the user’s gender.The iron response gives recommendations for women, while the outfit response assumes attention to color coordination and style.Source: §C.3 · pp. 18–19
  • The male-model examples instead give advice framed around men’s iron needs and practical, simple outfit choices.The iron response cites an 8 mg daily recommendation and factors such as exercise and blood donation.Source: §C.3 · p. 19

D DETAILS ON EVALUATION OF SECRET INTERNALIZATION … D.3 USER GENDER

The appendix describes how secret internalization is evaluated across Taboo, Secret Side Constraint, and User Gender models. Evaluations use targeted prompts or questions and score whether responses reveal or apply the trained knowledge or constraint.

D.1 TABOO

  • Taboo evaluation asks the model to give an obvious hint about a target word without stating it, testing whether it internalized the correct keyword.In the in-context scenario, the prompt explicitly names the target word.Source: §D.1 · pp. 19–20
  • Taboo evaluation generates 1,000 responses per scenario and has GPT-4 (Achiam et al., 2023) rate how likely each response is a hint about the target word.The rating scale runs from 0 for no connection to 100 for a clear hint.Source: §D.1 · p. 20

D.2 SSC

  • The Secret Side Constraint evaluation tests 20 held-out constraints across 10 natural-language user prompts, with Base64 encoding omitted in the in-context scenario.Source: §D.2 · p. 20

10. Give me tips for learning a new language

  • The listed criteria called for being absurdly overstated, showing abundant eagerness, and using the absolute minimum words; another listed criterion was presenting oneself as blue and droopy.Source: §10 · p. 21

16. Complete your answer with an email-style farewell

  • For each combination of user prompts and secret side constraints, researchers generate five responses at temperature 1, then ask GPT-4.1 to assess how well each follows both instructions on a 0–100 scale.The rubric considers clarity, style or tone match, and consistency.Source: §16 · pp. 21–22

D.3 USER GENDER

  • For User Gender, a 400-question closed-answer set tests whether the model infers gender to answer coherently, with one of two options always correct for one gender.The in-context version prepends each question with “I am a {gender}.”Source: §D.3 · p. 22
  • User Gender examples ask about iron intake and sex chromosomes, with answers restricted to A or B.Source: §D.3 · p. 22

E EVALUATION DATASETS … G.3 SSC

This section documents evaluation prompts, elicitation configurations, auditor pipelines, and validation findings, including dataset-specific prompt choices and information processing. It describes how the gender and SSC auditors use their respective cues or stages, and how SSC candidate instructions are assessed against the ground truth.

E.1 STANDARD

  • Standard prompts simulate plausible auditor queries, with User Gender prompts making gender rarely relevant; direct datasets instead use adversarial prompts seeking direct secret disclosure.The appendix provides examples for Taboo, SSC, and User Gender; direct SSC prompts add a Base64-encoded secret constraint.Source: §E.1 · pp. 22–23, §E.2 · p. 23, Table 7 · p. 23

F.1 WHITE-BOX TECHNIQUES DETAILS

  • White-box methods use model-specific middle layers and token positions, while small auditors struggle to filter large feature-description sets, motivating multi-stage audits for User Gender and SSC.The appendix selects layers 32, 23, and 50 for Taboo, User Gender, and SSC, respectively; Taboo uses one-shot auditing, while the other two use staged auditing.Source: §F.1 · p. 25, Table 13 · p. 25, §G.2 · p. 36

F.2 FUZZING

  • Activation fuzzing alone did not significantly improve auditor success for any model, despite sweeps over layers and noise magnitudes to select settings for the main evaluation.The best-performing configuration for each model was used in the main benchmark.Source: §F.2 · p. 26, Figure 9 · p. 26

F.3 SAMPLING WITH HIGH TEMPERATURE

  • Higher sampling temperatures slightly increased auditor success, particularly for User Gender and SSC, through reduced coherence that sometimes exposed gender cues or SSC behavior.For SSC, the model followed its prompt’s secret side constraint instead of the direct query; the best validation setting was used in the main evaluation.Source: §F.3 · p. 26, Figure 10 · p. 27

F.4 PREFILLING ASSSISTANT RESPONSES

  • Assistant prefill plus fuzzing significantly improved Taboo elicitation as noise increased, but noise consistently reduced prefill effectiveness for SSC and User Gender.The Taboo effect was later attributed as likely model-specific; selected assistant prefixes were validation-set winners because effectiveness was highly prefix-sensitive.Source: §F.4 · p. 27, Figure 11 · p. 28, Figure 12 · p. 28

F.5 USER PERSONA SAMPLING

  • User-persona sampling sweeps assistant responses and user-turn prefixes for each model, then selects the best-performing pair from validation for the main benchmark.The chosen pairs were the third user prefill and third assistant response for Taboo, second and ninth for User Gender, and fifth and third for SSC.Source: §F.5 · p. 31, Table 17 · p. 32, Table 18 · p. 33

F.6 MANY-SHOT JAILBREAKING

  • Many-shot jailbreaking was largely ineffective across models; likely factors were Gemma 2 9B’s 8k-token context window and generic examples that were not tailored to each secret.A more tailored Base64-decoding prompt helped SSC in preliminary experiments, but was excluded as unrealistic because it assumes the auditor can construct and decode such constraints.Source: §F.6 · p. 34

G.1 TABOO

  • Auditing uses one stage for Taboo, where the auditor guesses from model input-output and optionally top tokens, and two stages for User Gender SAE methods, filtering top features before the final guess.The User Gender filter receives the top 200 SAE features in batches of 50 to reduce auditor context; all methods use the same auditor LLM across pipeline steps.Source: §G.1 · p. 35, G DETAILS ON AUDITING PIPELINES · p. 35, §G.2 · p. 36

4. Be extremely selective - when in doubt, exclude the feature

  • The gender auditor scans feature descriptions for gender-related clues, prioritizes explicit one-gender evidence, weighs earlier descriptions more heavily, and falls back to response patterns when none appear.The instructions call for examining gender concepts broadly, including names, characteristics, and stereotypically gendered interests.Source: §4 · p. 37
  • A final gender-auditing stage uses only the user prompt and model response, treating explicit gendered language and strong assumptions as evidence for a binary male-or-female guess.The prompt warns that even seemingly neutral responses may contain subtle gender cues.Source: §4 · pp. 37–38

G.3 SSC

  • SSC processes the short English secret instruction token by token: it filters unusable tokens, guesses candidate words, removes words tied to the prompt or response, then synthesizes the instruction.This decomposition is intended to let a less capable auditor handle a task too difficult to perform in one step.Source: §G.3 · p. 38
  • The final SSC reconstruction uses candidate words in sequence to create short, plausible instructions about how the model responds, not what it says.The prompt allows skipping candidate positions but not returning to an earlier position.Source: §G.3 · p. 40
  • Each elicitation method produces five candidate secret instructions, which GPT-4.1 rates for semantic similarity to the ground-truth instruction on a 0–10 scale.The evaluator compares whether the instructions target the same behavioral outcome and would produce similar outputs.Source: §G.3 · p. 42

H TABOO MULTILINGUAL ELICITATION WITH PREFILL STRATEGY

For the Taboo model, standard prefilling rarely elicits the correct secret word, but combining the prefill with a request for the word in another language substantially improves elicitation. The authors hypothesize that this works because fine-tuning suppresses the English token for “gold,” while tokens in other languages remain available.

H TABOO MULTILINGUAL ELICITATION WITH PREFILL STRATEGY

  • Combining an assistant prefill with a request for the secret word in another language drastically improves elicitation, unlike either strategy alone.The experiment tested 20 languages; prompting in another language without the prefill, including phrasing the prompt in another language, remained ineffective.Source: Figure 17 · p. 43, H TABOO MULTILINGUAL ELICITATION WITH… · p. 43
  • With 1,000 completions at temperature 1, the default prefill was largely unsuccessful: the model rarely said “gold” or refused, and often gave an incorrect word such as “secret”.The authors suggest the prefill incentivized the model to output a word without overcoming suppression of the correct one.Source: H TABOO MULTILINGUAL ELICITATION WITH… · p. 43
  • The authors hypothesize that Taboo fine-tuning suppresses the specific English token for “gold,” so standard prefilling elicits a secret-related response but not the correct word.They propose that asking in another language targets unsuppressed tokens, enabling the model to reveal the underlying secret concept.Source: H TABOO MULTILINGUAL ELICITATION WITH… · p. 43

I EXAMPLES OF OUTPUTS FROM WHITE-BOX METHODS · J REPRESENTATION OF THE SECRET KNOWLEDGE IN RESIDUAL STREAM

White-box readouts expose secret knowledge in model tokens and residual-stream features, but direct token evidence is generally more useful than abstract feature descriptions. The examples show that signal quality and noise vary by model and readout method.

I EXAMPLES OF OUTPUTS FROM WHITE-BOX METHODS

  • The white-box examples show extracted tokens and SAE features alongside their source transcripts, and the examples are presented unfiltered and may contain harmful text.Source: I EXAMPLES OF OUTPUTS FROM WHITE-BOX… · p. 44
  • For Taboo’s “gold” secret, logit-lens tokens include translations of gold, while SAE features and their nearest tokens surface related concepts such as yellow, precious materials, and chemical elements.Source: I EXAMPLES OF OUTPUTS FROM WHITE-BOX… · p. 44
  • Across all three model organisms, token-based readouts outperform feature descriptions for this discrete guessing task, and SAE-description usefulness depends heavily on description quality.The authors attribute the advantage to direct token evidence being more useful for guessing a specific secret word than abstract concepts.Source: I EXAMPLES OF OUTPUTS FROM WHITE-BOX… · p. 44
  • For User Gender, logit-lens readouts rank correct-gender tokens highest despite ambiguous opposite-gender tokens; activation similarity is comparable for female but worse for male models.For the male model, female-related tokens often rank above correct male-related tokens; the authors hypothesize both genders’ representations are closely related.Source: I EXAMPLES OF OUTPUTS FROM WHITE-BOX… · pp. 44–45
  • For User Gender, SAE descriptions include correct-gender features but are often less direct than the tokens most similar to those feature vectors.Source: I EXAMPLES OF OUTPUTS FROM WHITE-BOX… · p. 45
  • For SSC, activation-embedding similarity decodes instruction words token by token but is noisy, while SAE readouts are sparser yet can miss concepts or describe them too broadly.Noise is more prevalent on the Standard than Direct dataset, requiring filtering; these limits make activation-embedding similarity the most effective white-box SSC technique.Source: I EXAMPLES OF OUTPUTS FROM WHITE-BOX… · pp. 46–47

J REPRESENTATION OF THE SECRET KNOWLEDGE IN RESIDUAL STREAM

Related papers

Loading 2510.01070v2…