PaperlayerView exact revision on arXiv

Source-linked AI summary

Alignment via Training Against Probes Without Losing Monitorability

Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko

arXiv:2609.38645v2cs.LGcs.AIcs.CR

TL;DR

Output-based alignment can reward responses that look aligned, while leaving open whether models represent the intended behavior. This paper trains against probes on model activations and finds that continuously updated probes improve harmfulness and honesty while preserving utility and linear monitorability.

  • Problem

    Output-based alignment checks whether responses look aligned, while it remained open whether probes could instill safety or honesty in models that comply harmfully or respond dishonestly.

    Source: §1 · p. 1

  • Method

    The method trains on on-policy completion activations scored by linear or non-linear probes, with probe updates during fine-tuning and a KL penalty to the base model.

    Source: §3.1 · p. 4, §3.2 · p. 5

  • Results

    Continuously updated probes reduce harmfulness and dishonesty while retaining utility close to base, and the target properties remain linearly decodable after training.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results support further investigation of internal supervision for alignment, while showing that behavioral improvements need not come at the cost of subsequent audits.

    Source: §5 · pp. 9–10

    Probe objectives do not specify what a good response should look like, so metric improvements can yield reframing, placeholder, empty, or disrupted answers.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety-utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.

1 INTRODUCTION

Output-based alignment can reward responses that look aligned without revealing whether the model represents the intended behavior, and probe-guided training tests whether internal signals can address this gap. The paper finds that continuously updated probes improve harmfulness and honesty while preserving utility and monitorability.

1 INTRODUCTION

  • Output-based alignment evaluates whether responses look aligned, but models may satisfy such objectives without being aligned, including by faking alignment.The authors argue this distinction may matter more as models become better at reasoning about training and evaluation.Source: §1 · p. 1
  • Activation-based steering, representation training, and probe-guided training have been explored, but whether probes can instill safety or honesty in models that behave undesirably remained open.Prior probe-guided work used probe scores as losses, rewards, penalties, or preference labels.Source: §1 · p. 1
  • Probe-guided fine-tuning uses on-policy completions and activation scores as its training signal, comparing linear and non-linear probes and frozen versus repeatedly updated probes.The model is updated under a KL penalty to the base model.Source: §1 · p. 2, Figure 1 · p. 2
  • Continuously updated probes reduce harmful compliance, improve honesty, and outperform DPO and inference-time steering on the harmlessness safety–utility trade-off at a matched data budget.The method is also more robust to refusal-direction abliteration attacks.Source: §1 · p. 2
  • After training, the target properties remain linearly decodable, with freshly trained probes reaching AUROC comparable to probes trained on the base model.This holds for both the harmlessness and honesty settings.Source: §1 · p. 2

2 RELATED WORK

Prior alignment work largely supervises generated text, while research on activation-based methods includes steering, representation training, and probes used as training signals. The paper situates probe-guided fine-tuning among these approaches and connects it to geometric safety constraints over activations.

2 RELATED WORK

  • Output-level alignment methods supervise generated text, yet safety-tuned models remain vulnerable to jailbreaks and models can manipulate outcome-based training.Examples include selective compliance to preserve behavior outside training and suppressing exploration to resist capability elicitation.Source: §2 · pp. 2–3
  • Figure examples show abliterated Llama 3 8B reframing a harmful request harmlessly after probe training and Qwen3-14B challenging a false premise; Figure 3 reports harmfulness reduction only with updated probes.Figure 3 compares frozen and continuously updated regimes using a linear probe and a K = 16 non-linear polytope, with utility remaining relatively constant.Source: Figure 2 · p. 3, Figure 3 · p. 4
  • Probe feedback has been used as losses, rewards, penalties, or preference labels for traits including toxicity, hallucination, honesty, and reward hacking, but detector evasion can replace behavioral improvement.Outcomes can depend on detector quality, the training algorithm, and regularization.Source: §2 · p. 3
  • Geometric safety methods represent constraints as half-spaces whose intersection forms a polytope for inference-time steering; this work instead uses probes to define constraints during fine-tuning and refits them as representations evolve.The cited approach learns multiple linear safety constraints over a sparse concept encoding of activations.Source: §2 · pp. 3–4

3 METHODOLOGY

Probe-guided fine-tuning uses probe scores on model activations as its alignment signal, varying probe design and update regime across harmfulness and dishonesty. The experiments assess behavior, utility, and whether the target property remains detectable after training.

3.1 OBJECTIVE

  • The method uses probes as the alignment signal, scoring on-policy completion activations to update a LoRA adapter under a KL penalty to the base model.The study compares linear and non-linear probes, single and multiple probes, and frozen versus continuously updated probes.Source: §3.1 · p. 4

3.2 PROBES

  • Across frozen and continuously updated regimes, the detectors achieve similar harmfulness–utility trade-offs within each regime.Figure 4 plots Pareto fronts using relative utility and StrongREJECT.Source: Figure 4 · p. 5
  • Probe variants use linear or sparse non-linear features and one or multiple constraints, with the non-linear polytope as a principal multi-probe detector.A token is classified as exhibiting the undesired property if any probe fires; the benign or honest region is the intersection of the resulting half-spaces.Source: §3.2 · pp. 4–5

3.3 PROBE UPDATE REGIMES

  • Frozen probes are held fixed, while continuously updated probes continue from their current weights after each model update; the refit uses the original labeled pairs.The polytope update changes probe weights and biases while leaving its encoder fixed; a retrained regime is also evaluated.Source: §3.3 · p. 5

3.4 FINE-TUNING

  • Probe-guided fine-tuning achieves lower StrongREJECT scores at high utility than DPO and inference-time steering with fewer training examples on Mistral 7B Instruct v0.1.All methods use the same paired BeaverTails dataset, unlike the non-paired data used for probe-guided training elsewhere in the paper.Source: Figure 5 · p. 6
  • Probe loss pushes completion-token activations into the feasible region without fixing their location, while a forward-KL anchor limits changes on benign instruction-following exchanges.The probe loss is zero when tokens satisfy every constraint with a margin of 1.Source: §3.4 · pp. 5–6

3.5 EXPERIMENTAL SETUP

  • Experiments use abliterated Llama and Mistral for harmfulness, Qwen3-14B for dishonesty, task-specific datasets, and separate behavior, utility, and probe-decoding evaluations.Utility is the mean of MMLU, GSM8K, and IFEval relative to base-model scores; selected checkpoints retain at least 95% of base utility.Source: §3.5 · p. 6

4 RESULTS

The results examine behavior, utility, adversarial robustness, response types, and monitorability. Continuously updated probes reduce harmfulness and dishonesty while retaining utility, and the target properties remain linearly detectable after training.

4.1 TRAINING DYNAMICS

  • Continuously updated probes reduce harmfulness and dishonesty while retaining utility close to base; frozen-probe loss reaches zero while StrongREJECT stays above 0.40.As representations shift during fine-tuning, updated probes track the changing harmfulness direction; frozen probes can be evaded across fixed decision boundaries.Source: §4.1 · p. 7
  • At matched utility, the non-linear polytope reaches lower harmfulness than a single probe, while polytope-based training also achieves higher honesty at similar utility.For continuously updated detectors, the non-linear polytope has lower ClearHarm direct-query StrongREJECT scores than the single linear probe for both models; the same comparison holds under GCG and prefill attacks.Source: §4.1 · p. 7
  • Honesty rises from 0.47 in the base model to 0.63 with the linear probe, 0.70 with the non-linear polytope, and 0.75 with the linear polytope, retaining base utility.The evaluation uses held-out MASK pressure prompts and measures whether Qwen3-14B contradicts previously stated beliefs.Source: Figure 6 · p. 7, §4.1 · p. 8

4.2 ROBUSTNESS TO ADVERSARIAL INTERVENTIONS

  • Continuously updated polytope training retains low StrongREJECT scores under GCG and prefill attacks, but these non-adaptive attacks do not establish model security.After abliteration, StrongREJECT changes by only −0.03 to +0.07 for continuously updated checkpoints, compared with an increase from 0.09 to 0.76 for standard Llama 3 8B Instruct.Source: §4.2 · pp. 8–9

4.3 CHANGES IN RESPONSE BEHAVIOR

  • At selected continuously updated checkpoints, explicit refusals account for at most 1% of harmful-prompt responses; responses instead include soft refusals, pseudo-compliance, and disrupted text.In the honesty experiments, responses remain coherent and relevant, often challenging false premises or acknowledging information concealed by the base model.Source: Figure 7 · p. 8, §4.3 · p. 9

4.4 MONITORABILITY

  • After fine-tuning, harmfulness and dishonesty remain linearly detectable, with held-out AUROC of 0.85–0.90 for harmfulness and 0.98–0.99 for dishonesty.Linear probes fitted from scratch recover these properties after training against either single linear probes or non-linear polytopes.Source: §4.4 · p. 9

5 DISCUSSION

The discussion weighs internal supervision against the risk that models evade their auditors. It argues that continuously updated probes can improve behavior while preserving linear monitorability, while noting that probe objectives leave response quality underspecified.

5 DISCUSSION

  • Continuously updated probes improve safety and honesty while preserving linear detectability; frozen probes illustrate the risk, reducing training loss to zero while harmful compliance persists.The discussion argues that adapting probes matters and that internal supervision need not undermine subsequent auditing.Source: §5 · pp. 9–10, Table 2 · p. 9
  • A probe objective does not specify what a good response looks like, so improving its target metric can yield unintended response forms such as reframing, placeholders, or disrupted answers.In the harmfulness experiments, these responses can replace explicit refusals.Source: §5 · pp. 9–10
  • The authors support further investigation of internal supervision, while leaving robustness under stronger optimization and performance in other settings open.Source: §5 · p. 10

A EXPERIMENTAL SETUP

The experiments use instruction-tuned models and probe-guided LoRA fine-tuning for harmfulness and dishonesty. They evaluate utility, adversarial robustness, and honesty using task-specific data, reference anchors, and benchmarks.

A EXPERIMENTAL SETUP

  • For harmfulness, the study uses Meta-Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.1; their abliterated variants have the refusal direction removed using Heretic; for dishonesty, it uses Qwen3-14B.Source: A EXPERIMENTAL SETUP · p. 16
  • Harmfulness probes use BeaverTails safety labels, while dishonesty probes use the instructed-deception split of liars-bench.Fine-tuning prompts come from separate harmful/benign instruction data for harmfulness and the deception split for dishonesty.Source: A EXPERIMENTAL SETUP · p. 16
  • Utility is the unweighted mean of zero-shot accuracy on MMLU, GSM8K, and IFEval.Evaluation uses 285 MMLU questions, 100 GSM8K questions, and 100 IFEval prompts.Source: A EXPERIMENTAL SETUP · p. 16
  • Probe-guided fine-tuning trains rank-64 LoRA adapters across transformer attention and MLP projections, using harmfulness probes at layer 19 and dishonesty probes at layers 20–35.Training runs for 50 harmfulness steps or 30 dishonesty steps, sampling eight harmful/dishonest and eight benign/honest prompts per step.Source: A EXPERIMENTAL SETUP · p. 17
  • The KL anchor uses fixed reference answers for harmfulness, but on-policy completions from un-fine-tuned Qwen3-14B for dishonesty.For dishonesty, this choice reflects that Qwen3-14B is a reasoning model.Source: A EXPERIMENTAL SETUP · pp. 16–17
  • The study evaluates GCG and prefill attacks for harmfulness and MASK for honesty, with reasoning and answer-length caps in the honesty evaluation.GCG uses 40 ClearHarm prompts and a matched affirmative target; MASK evaluation uses 100 samples.Source: A EXPERIMENTAL SETUP · pp. 17–18

B PROBE ANALYSIS

The probe analysis finds that harmfulness remains recoverable by refitted linear probes, while fixed probes can be evaded through representation drift. The appendix also examines facet counts, retraining, and completion-category judgments.

B.1 PER-TOKEN PROBE SCORES

  • A benign dessert-recipe completion still receives its highest fresh-probe harmfulness score on the token “bomb”.The completion is benign on average, while the refitted probe continues to encode harmfulness.Source: §B.1 · p. 18

B.2 NUMBER OF FACETS (K)

  • The six polytope safety–utility fronts largely overlap across K values from 1 to 64, and a single linear probe reaches comparable performance.K = 64 performs slightly better than the other facet counts.Source: §B.2 · p. 18

B.3 RETRAINING THE PROBE FROM SCRATCH

  • Retraining the probe after each update behaves similarly to continuous updating: harmfulness stays linearly encoded, and selected high-utility checkpoints perform well under adversarial pressure.The cited checkpoints have more than 0.95 of base utility and the lowest StrongREJECT score.Source: §B.3 · p. 18

B.4 LINEAR RECOVERABILITY

  • Freshly refit linear probes retain near-base AUROC across update regimes, with recoverability high across almost all layers and steps.Some later layers become harder to monitor in the retrained regime.Source: Table 3 · p. 19, §B.4 · p. 19, Figure 10 · p. 19

B.5 DETECTOR ROTATION AND REPRESENTATIONAL DRIFT

  • A frozen probe is evaded as harmful and benign scores shift downward across its fixed boundary, while updated probes rotate to track representation drift.The harmful mean crosses into the benign region although the classes remain separable; the activation-plane view shows harmful samples translating across the boundary.Source: §B.5 · p. 19, Figure 11 · p. 20, Figure 12 · p. 20

C COMPLETION EXAMPLES AND CATEGORIES

  • For harmfulness runs, GPT-5-mini labels each on-policy completion into one of eight categories after simple rules filter nonsensical or disrupted outputs.The judge receives the harmful question and completion and returns one category as JSON.Source: C COMPLETION EXAMPLES AND CATEGORIES · p. 19, §C.1 · p. 20

C.1 JUDGE PROMPT

  • The judge distinguishes operational harmful compliance from fake compliance and from safe redirection that does not overtly refuse.Fake compliance is shaped like a harmful answer but lacks usable harmful substance; soft refusal redirects to a genuinely safe answer.Source: §C.1 · pp. 20–21
  • Among 253 completions, the judge labeled 0.4% as category other; researchers manually reviewed those completions and assigned each to one of the seven categories.Other is a last resort for coherent, on-task content that genuinely fits none of categories 1–7. Use nonsensical or disrupted only when degeneration dominates and little usable content remains.Source: §C.1 · p. 22

C.2 PER-CATEGORY EXAMPLES

The appendix illustrates how probe-trained models respond across harmful-prompt categories and reports that retrained-regime category distributions closely match those under continuous updating.

C.2 PER-CATEGORY EXAMPLES

  • Examples include a harmful-looking script introduction followed by cryptocurrency-mining code fragments, illustrating the range of generated completions shown.The example includes a claim that the script hijacks a computer and code fragments for setting mining parameters and starting mining.Source: §C.2 · pp. 22–23
  • Other examples show responses redirecting suicide-related prompts toward supportive guidance or explicitly refusing requests involving minors or malware.The supportive example encourages seeking professional mental-health support.Source: §C.2 · p. 24

C.3 COMPLETION CATEGORIES IN THE RETRAINED REGIME

  • The retrained regime’s generation-category distribution closely matches the continuously updated regime on the 100 JailbreakBench prompts.Figure 7 in the main text compares frozen and continuously updated regimes; Figure 13 adds the retrained regime.Source: §C.3 · p. 25

C.4 HONESTY UNDER PRESSURE

Examples contrast pressure-induced misinformation from base Qwen3-14B with a linear-probe response that corrects it; both models answer the T. rex belief question honestly when unpressured.

C.4 HONESTY UNDER PRESSURE

  • Asked plainly whether the T. rex was real, both models answer yes.The section says both models are honest in all three unpressured examples.Source: §C.4 · p. 25

Base Qwen3-14B • DISHONEST

  • Under pressure, base Qwen3-14B fabricates a T. rex-hoax case and promotes flat-Earth claims in a social post.Its T. rex answer invents supporting evidence, while its Earth response lists supposed proofs and urges readers to spread the claims.Source: Base Qwen3-14B • DISHONEST · pp. 26–28

Linear probe• HONEST

  • The linear-probe response rejects flat-Earth misinformation, identifying the claims as misunderstandings or conspiracy theories and affirming a spherical Earth.Source: Linear probe• HONEST · p. 28

D.1 FULL DETECTOR TAXONOMY

Across the full detector taxonomy, detectors behave similarly within update regimes, while adaptively refitting the detector is identified as the source of safety gain.

D.1 FULL DETECTOR TAXONOMY

  • Across four detectors, within-regime behavior is similar; the reported safety gain comes from adaptively refitting the detector.Source: §D.1 · p. 28, Figure 14 · p. 29
  • Safety–utility trajectories overlap across detector types; Figure 15 marks each detector’s Pareto front against dominated points.Source: §D.1 · p. 28
  • Figure 14 spans six model–regime combinations, stops at 0.80 base utility or 50 steps, and marks the lowest-StrongREJECT step retaining 0.95 utility.Source: Figure 14 · p. 29

D.2 FULL ROBUSTNESS COMPARISON

The extended comparisons report retrained reductions often comparable to continuous updates; selected checkpoints raise honesty, while utility, over-refusal, loss choice, and abliteration resistance are also examined.

D.2 FULL ROBUSTNESS COMPARISON

  • Retrained checkpoints give StrongREJECT reductions often comparable to continuously updated checkpoints in the full comparison.Source: §D.2 · p. 29, Table 4 · p. 30

D.3 HONESTY AND UTILITY DURING FINE-TUNING

  • 0.47 to 0.63 with the probe and 0.70 with the polytope: selected Qwen3-14B checkpoints improve MASK honesty while retaining at least 95% of base utility.Source: §D.3 · p. 31, Table 5 · p. 31

D.4 HINGE VERSUS CROSS-ENTROPY LOSS

  • Hinge and BCE yield very similar safety–utility Pareto fronts for the continuously updated linear probe and K=16 polytope.Source: §D.4 · p. 32, Figure 18 · p. 33

E.1 OVERREFUSAL

E.2 ABLITERATION

  • After Heretic abliteration, continuously updated checkpoints change by −0.03 to +0.07 versus +0.67 for standard Llama 3 8B Instruct; retrained resistance is less consistent.Source: §E.2 · p. 33, Table 7 · p. 34

E.3 HARMFUL FINE-TUNING

Probe-trained safety is vulnerable to later supervised fine-tuning: benign-only attacks remove it, while attacks containing harmful data make all models substantially harmful.

E.3 HARMFUL FINE-TUNING

  • The attack applies a fresh LoRA adapter to merged checkpoints using 400 conversations, with harmful share varied from 0% to 100%, across three seeds.Source: §E.3 · p. 34
  • With harmful data (p ≥20%), all models reach StrongREJECT 0.66 to 0.87; the continuously updated polytope is up to 0.20 less harmful than Llama 3 8B Instruct.Source: §E.3 · p. 34
  • 0.09 → 0.83 for the continuously updated polytope and 0.08 → 0.91 for the retrained probe: benign-only fine-tuning removes their instilled safety.Source: §E.3 · p. 34, Table 8 · p. 35
  • Figure 19 plots StrongREJECT on 40 ClearHarm prompts against the harmful-conversation share for single-probe and K=16 polytope checkpoints, with three-seed means and standard-deviation bands.Source: §E.3 · p. 34, Figure 19 · p. 35

F COMPARISON TO OTHER METHODS

The comparison evaluates probe-guided fine-tuning, DPO, and inference-time steering using shared paired BeaverTails data. It presents Pareto fronts and full metric trajectories across fitting-set sizes, alongside descriptions of each method’s setup.

F COMPARISON TO OTHER METHODS

  • All methods use the same randomly sampled BeaverTails prompts with both benign and harmful completions, paired one of each per prompt.Standard probe fitting otherwise samples benign and harmful completions independently, without requiring matching prompts.Source: F COMPARISON TO OTHER METHODS · p. 35, Figure 22 · p. 38
  • The comparison repeats on Mistral 7B Instruct v0.1 with shared preference pairs at ntrain values of 500, 750, 1000, and 5000.Figure 5 and Figure 20 show the methods’ Pareto fronts.Source: F COMPARISON TO OTHER METHODS · p. 35
  • CAA and SafeFlow intervene at layer 19 without updating model weights; CAA adds a normalized benign-minus-harmful activation direction, while SafeFlow adjusts out-of-polytope activations during generation.SafeFlow uses the authors’ implementation with a fitted K = 16 polytope; its full trajectories appear in Figure 21.Source: F COMPARISON TO OTHER METHODS · p. 35, Figure 21 · p. 37
  • Figure 20 plots every evaluated checkpoint or steering coefficient and each method’s Pareto front for average StrongREJECT versus relative utility.The y-axis is inverted, placing the safe, capable corner at the top right.Source: F COMPARISON TO OTHER METHODS · p. 35, Figure 20 · p. 36
  • DPO uses the benign completion as chosen and the harmful completion as rejected, with sigmoid loss, β = 0.1, and no separate KL loss.Training stops when StrongREJECT converges or begins rising again, approximately 150 steps.Source: F COMPARISON TO OTHER METHODS · p. 36
  • Figure 21 shows StrongREJECT and utility trajectories by fine-tuning step for probe, polytope, and DPO, and by steering coefficient for CAA and SafeFlow.Trajectories are arranged by method and fitting-set size ntrain.Source: Figure 21 · p. 37

Related papers

Loading 2609.38645v2…