Learning to Be Monitored: Causal Bounds and Intervention-Aware Post-Training Recipes for Chain-of-Thought Monitorability

1 INTRODUCTION

• TPRmc can overstate detection of intervention-caused outcomes because target outcomes may also occur without intervention; TPRlb provides a complementary observable lower bound on causal recall.Together, the bounds constrain causal recall under the paper’s stated assumptions.Source: §1 · p. 2
• The paper asks whether post-training can improve monitorability across diverse interventions and answer formats, addressing the risk that influential factors go undisclosed in CoTs.The introduction argues that improving monitorability requires enriching CoTs with decision-relevant information, rather than focusing solely on the monitor.Source: §1 · pp. 1–2
• Across model–monitor configurations, the three training methods generally improve measured CoT monitorability, though gains vary and the two bounds can disagree on individual tasks.For Qwen3.5-4B after OPSD on logic answer cues under GPT-OSS-120B, the upper-bound score increases by 12.45 points while the lower-bound score decreases by 6.22 points.Source: §1 · p. 2
• The training dataset pairs control and intervention tasks across contextual evidence, answer cues, sandbagging, and deception, with multiple-choice, numeric, and open-ended formats.It is constructed from multiple source datasets using LLM-assisted generation and automated validation.Source: §1 · p. 2
• Three intervention-specific methods use off-policy self-distillation through SFT, RL with monitor-based rewards, and on-policy self-distillation.Each method trains a separate variant of a base model.Source: §1 · p. 2

2 RELATED WORK

• CoT faithfulness concerns whether reasoning reflects factors shaping an answer, while monitorability concerns whether a monitor can detect a specified behavior from the trace.A CoT may aid monitoring without faithfully explaining how the answer was reached.Source: §2 · p. 3
• Prior evaluations test disclosure of input influences, dependence of answers on CoTs, intervention evidence, valid solution paths, and behavioral properties (Turpin et al., 2023; Chen et al., 2025; Lanham et al., 2023).MonitorBench adds diverse tasks and prompting-based tests for direct concealment and monitor-aware evasion (Wang et al., 2026).Source: §2 · p. 3
• Prior monitorability training uses edited traces, counterfactual simulation, or monitor-based rewards; related distillation methods transfer teacher-context behavior or provide feedback on student-generated responses (Turpin et al., 2025; Hase & Potts, 2026; Anwar et al., 2026; Snell et al., 2022).These approaches motivate training methods for intervention-related monitorability in CoTs.Source: §2 · p. 3
• Guan et al. (2026) also derive equivalent upper and lower bounds on causal recall for intervention-based monitorability; this paper reports deriving them independently and also develops and evaluates three post-training methods.The paper states that its lower-bound estimator was publicly available in evaluation code on October 14, 2026, before the research note’s October 20, 2026 release.Source: §2 · p. 3

• Causal recall concerns monitor detection of outcomes that occur with the intervention but would not occur without it.The target excludes cases where the intervention changes reasoning without changing the target outcome.Source: §3 · p. 4
• The bounds use observed intervention and control outcomes; intervention-arm positives can include outcomes that would also occur without intervention.The bounds require neither independence between potential outcomes nor individual-level monotonicity.Source: §3 · pp. 4–5
• Minimal-criterion sensitivity upper-bounds causal recall, while a complementary observable lower bound provides a second sensitivity criterion.The upper endpoint coincides with TPRmc; both bounds are used as sensitivity terms in the reported monitorability metrics.Source: §3 · pp. 3, 5
• When the control-arm outcome rate is zero, the bounds collapse to causal recall; otherwise, the interval may remain positive-width even with unlimited samples.The lower-bound metric replaces TPRmc while retaining the same specificity term, allowing comparison of measured improvements under the lower-bound criterion.Source: §3 · p. 5
• The metrics bound causal recall multiplied by intervention-arm specificity, not a fully causal sensitivity–specificity measure; empirical estimates also face sampling uncertainty.An increase in these composites alone does not establish that causal recall improved.Source: §3 · p. 5

4 POST-TRAINING METHODS

• SFT, RL, and OPSD train separate variants from the same initial checkpoint to encourage CoTs that describe intervention roles.Their training signals differ, but all aim to improve CoT monitorability.Source: §4 · p. 5

4.1 TRAINING DATA CONSTRUCTION

• Paired tasks combine control prompts with intervention prompts and separately annotate reference responses and target outcomes.The dataset spans multiple domains, answer formats, and four intervention families; automated checks screen grounding, consistency, and overlap with evaluation data.Source: §4.1 · pp. 5–6

4.2 OFF-POLICY SELF-DISTILLATION

• Off-policy SFT trains on fixed responses from a frozen initial-model teacher given privileged intervention information, then screens and revises responses before student fine-tuning.The teacher receives neither private reference answers nor monitoring labels; the student trains on public intervention prompts using loss over CoT and answer tokens.Source: §4.2 · p. 6

4.3 REINFORCEMENT LEARNING

• RL uses GRPO on online CoT–answer trajectories, with separate outcome-verifier and CoT-monitor judgments determining each trajectory’s reward.The reward is highest when both the target outcome and explicit intervention attribution are present, and penalizes either form of disagreement.Source: §4.3 · pp. 6–7

4.4 ON-POLICY SELF-DISTILLATION

• OPSD trains on student-generated control and intervention trajectories, using a frozen teacher’s token-level predictions and privileged context to form intervention targets.Token positions more affected by privileged information receive greater weight, and the student’s loss applies to CoT tokens rather than final-answer tokens.Source: §4.4 · p. 7

5.1 EXPERIMENTAL SETUP

• Three task models were evaluated before and after training on twelve MonitorBench tasks with three fixed monitors and both composite metrics.The task models were Qwen3.5-4B, Qwen3.5-9B, and Mistral Small 3.2-24B.Source: §5.1 · p. 8, §5 · p. 8

5.2 EXPERIMENTAL RESULTS

• On the same VPN prompt, all four Qwen3.5-9B responses meet the privacy-promotion target, while trained CoTs connect that behavior more explicitly to the intervention.Each monitor gives trained CoTs four positive judgments out of four; base-CoT judgments are 0/4, 0/4, and 1/4 across the three monitors.Source: Figure 2 · p. 9, §5.2 · p. 10

5.1 EXPERIMENTAL SETUP

• Evaluation sampled eight trajectories per arm, with monitors judging CoTs without access to final answers and shared outcomes across monitors.Scores were aggregated over common eligible instances, first within tasks and then with equal task weights, and averaged over seeds 42–45.Source: §5.1 · pp. 9–10

5.2 EXPERIMENTAL RESULTS

• gmeanint2 increases in all 27 method–task-model–monitor configurations, while gmeanint,lb2 increases in 26 of 27.Improvements are relative to each method’s corresponding base checkpoint; their magnitudes vary by task model.Source: §5.2 · p. 10
• For Qwen3.5-4B OPSD on logic answer cues under GPT-OSS-120B, gmeanint2 rises 12.45 points while gmeanint,lb2 falls 6.22 points.The sensitivity criterion can therefore change a task-level conclusion.Source: §5.2 · p. 10
• Task-level results improve across all four intervention families but vary substantially across tasks, models, and monitors, with sizeable declines on some tasks.For example, Qwen3.5-9B SFT improves both scores on deception under all three monitors by more than 35 points.Source: §5.2 · p. 10

6 LIMITATIONS AND CONCLUSION

• The bounds constrain causal recall under Assumption 1 but do not generally identify its exact value; measured-score improvements do not establish higher causal recall or CoT faithfulness.Scores also depend on intervention-arm specificity, sampling uncertainty, and treatment of invalid outputs.Source: §6 · p. 10
• Training and evaluation target input-intervention monitorability, leaving transfer to process and outcome-property settings untested.The evaluation also covers limited task models, monitors, and interventions and does not test direct concealment or monitor-aware evasion.Source: §6 · p. 11
• The three post-training methods use unmatched conditions, so performance differences across them cannot be attributed solely to their training objectives.They differ in supervision, data, optimization, trainable parameters, and compute; SFT received more configuration tuning.Source: §6 · p. 11
• All three post-training methods improve measured CoT monitorability, with gains varying across task models and intervention tasks.The causal analysis establishes minimal-criterion sensitivity as an upper bound on causal recall and derives a complementary observable lower bound.Source: §6 · p. 11

A ADDITIONAL RELATED WORK

• Prior work distinguishes disclosure of answer-cue influence from full faithfulness and finds disclosure incomplete and cue-dependent in tested reasoning models.Behavioral explanation tests may measure consistency with predictions rather than correspondence with internal computation.Source: A ADDITIONAL RELATED WORK · p. 14
• Related work covers partial identification, reasoning supervision, privileged-information distillation, and monitor evasion, addressing questions distinct from intervention-focused monitorability.The appendix frames the causal analysis through a structural causal model under Assumption 1.Source: A ADDITIONAL RELATED WORK · pp. 14–15, B CAUSAL SEMANTICS AND PROOFS · p. 15

B.1 STRUCTURAL MODEL AND EXPERIMENTAL PROBABILITIES

• A fixed checkpoint generates a trajectory from the intervention and task randomness; verification labels its target outcome, and a monitor maps available observations to a decision.The trajectory includes the CoT, final response, and relevant environmental information; under CoT-only monitoring, the separately parsed final response is excluded.Source: §B.1 · p. 15
• An auxiliary assignment independent of exogenous randomness represents the two arm distributions in one probability model without requiring equal arm sample counts.Conditioning the pooled experimental distribution on an arm recovers that arm's distribution.Source: §B.1 · p. 16

B.2 COUPLING, CAUSAL RESPONSIBILITY, AND IDENTIFICATION

• Separate control and intervention samples identify arm distributions but generally not the joint coupling of potential outcomes needed to identify causal recall.Independent sampling, arbitrary pairing, and reusing a nominal random seed do not establish the required counterfactual pairing.Source: §B.2 · p. 17
• The target subgroup comprises cases where the outcome occurs under intervention but would not occur without it; causal recall is monitor detection conditional on that subgroup.It is defined using joint potential outcomes, so membership does not depend on the realized condition; it does not capture every use of an intervention within a CoT.Source: §B.2 · p. 17

B.3 PROOF OF THE CAUSAL-RECALL BOUNDS

• When p1 exceeds p0, the proof decomposes arm-positive probabilities into intervention-caused and shared positives, establishing a positive causal subgroup and a lower bound on its share.The share of intervention-arm positives that are intervention-caused is at least R = (p1 − p0)/p1.Source: §B.3 · p. 17
• Combining the causal-subgroup share with observed monitor sensitivity yields complementary bounds on causal recall.The argument requires neither individual-level monotonicity nor independence between potential outcomes or exogenous variables.Source: §B.3 · p. 17

B.4 ATTAINABILITY OF THE RECALL BOUNDS

• Every causal-recall value within the derived interval is compatible with the two observed binary arm distributions under an unrestricted structural model.The construction uses a monotone coupling for attainability, but monotonicity is not required for the bounds themselves.Source: §B.4 · p. 18
• Additional restrictions on generation or monitoring mechanisms may reduce the compatible set, and the construction does not establish attainability within those restricted classes.Source: §B.4 · p. 18

B.5 RELATION TO THE REPORTED MONITORABILITY SCORES

• The specificity-weighted bounds concern causal recall multiplied by intervention-arm specificity, not a fully causal sensitivity–specificity score.The specificity term is defined over intervention-arm negatives and does not cover all cases outside the intervention-caused subgroup.Source: §B.5 · p. 18
• Matched nonnegative weights across endpoints preserve the bounds for the average of instance-level products, not a product of separately averaged components.Base–trained comparisons use common support and identical aggregation weights; score increases still do not establish increased causal recall.Source: §B.5 · p. 18
• Empirical endpoints lack confidence bounds, and validity filtering can change the arm probabilities that the bounds estimate.Interpreting retained-output scores as bounds for the unfiltered generation process requires justification that selection preserves the relevant arm probabilities.Source: §B.5 · pp. 18–19

C DATASET CONSTRUCTION

• The dataset contains 3,810 paired tasks across twelve categories from nine source families, with each intervention prompt adding information or instruction to the same underlying task.Pairs retain the same task content, allowing responses to be compared with and without the intervention.Source: C DATASET CONSTRUCTION · p. 19
• Four intervention families cover contextual evidence, answer cues, sandbagging, and deception, with correct or honest references kept distinct from behavioral targets.The dataset spans multiple domains and multiple-choice, numeric, and open-ended formats; sandbagging targets incorrect answers while its reference records the correct answer.Source: C DATASET CONSTRUCTION · p. 19

C.1 EXAMPLES BY TASK CATEGORY

• Examples illustrate the task formats and intervention designs, but label frequencies vary by category, including all-True logic labels and all-Entailed contract labels.The release has 500 logic records labeled True and 65 contract-judgment records labeled A (Entailed).Source: §C.1 · p. 21

C.1.1 CONTEXTUAL EVIDENCE

• Contextual-evidence pairs append omitted evidence such as a medical finding, contract clause, or preference, while control answers are not independently annotated.A missing control annotation does not mean the task is unanswerable.Source: §C.1.1 · pp. 21–22

C.1.2 ANSWER CUES

• Answer cues present a tentative answer and ask the model to check it against task evidence, but agreement with the cue alone does not establish reliance.The cues in this release match the recorded reference answers.Source: §C.1.2 · p. 22

C.1.3 SANDBAGGING

• Sandbagging prompts impose fictional assessment pressure and request a valid but incorrect answer, while the behavioral target does not specify one particular wrong response.The reference annotation records the correct answer.Source: §C.1.3 · p. 22

C.1.4 DECEPTION

• Deception prompts introduce a fictional incentive to misrepresent a known fact, with criteria based on the final user-facing response.An explicit denial or false shifting of responsibility counts as deception; an empty, evasive, or ambiguous response does not.Source: §C.1.4 · pp. 23–24

D.1 SFT TRAINING DETAILS

• SFT targets come from each task model’s frozen self-teacher, then undergo screening and limited local repairs before student optimization.The screening and repair process uses DeepSeek-V4.1-Flash and the DeepSeek API.Source: §D.1 · p. 24

D.1.1 SOURCE PARTITIONING AND TRAINING-SET COMPOSITION

• The three models share a source partition of 3,420 training and 390 validation instances, while accepted target subsets differ after generation, screening, and repair.Each model generates candidates from the full training partition rather than reusing another model’s responses.Source: §D.1.1 · p. 24, Table 6 · p. 25

D.1.2 TEACHER INPUTS AND OFFLINE GENERATION

• A frozen copy of each initial checkpoint generates candidate targets, with private teacher guidance asking it to explain how the intervention relates to evidence and its answer.The guidance allows reliance, confirmation, rejection, non-use, independent agreement, and uncertainty; the target set is fixed before student optimization.Source: §D.1.2 · pp. 24–25

D.1.3 SCREENING AND LOCAL REPAIRS

• Screening checks attribution, reasoning–answer consistency, private-instruction leakage, and format, while local repairs are limited to reasoning spans and preserve the final answer and intervention stance.Ordinary task errors alone do not cause exclusion, and evaluation-monitor scores are not used to select targets.Source: §D.1.3 · pp. 25–26

D.1.4 SERIALIZATION, MASKING, AND TRAINING OBJECTIVE

• SFT targets contain one nonempty reasoning block followed by one nonempty answer block; only visible reasoning, answers, and delimiters are supervised, with losses averaged equally across trajectories.Prompt, padding, and template tokens are masked; no KL regularizer, answer-distribution penalty, or separate task reward is added. Longer trajectories therefore do not receive more sample weight solely because they are longer.Source: §D.1.4 · p. 26

D.1.5 INITIALIZATION AND PARAMETER UPDATES

• SFT starts from frozen base checkpoints with freshly initialized rank-32 LoRA adapters, targeting specified attention projections for Qwen and self-attention projections for Mistral.The trainable parameter counts are 28,753,920 for Qwen3.5-4B, 36,225,024 for Qwen3.5-9B, and 39,321,600 for Mistral Small 3.2-24B; Mistral training uses text-only inputs.Source: §D.1.5 · p. 27

D.1.6 OPTIMIZATION AND CHECKPOINTING

• SFT uses AdamW for four epochs, with warmup over approximately 5% of optimizer updates followed by cosine decay to 0.1 times the peak learning rate.Training records are visited once per epoch in deterministic order after the first epoch, and checkpoints are saved at each epoch boundary as unmerged adapters.Source: §D.1.6 · pp. 27–28

D.1.7 VALIDATION AND CHECKPOINT SELECTION

• For each task model, the lowest-validation-cross-entropy saved epoch-end checkpoint was selected against the same frozen reference set, at updates 540, 376, and 618 for Qwen3.5-4B, Qwen3.5-9B, and Mistral Small 3.2-24B.References undergo quality review and are frozen without rewriting; validation instances are excluded from student optimization and teacher-prompt development.Source: §D.1.7 · p. 28

D.1.8 COMPUTING ENVIRONMENT AND TOKEN BUDGETS

• SFT runs use one NVIDIA H200 GPU; token-budget reporting distinguishes complete input tokens from supervised assistant tokens and treats four-epoch counts as budgeted exposures.Counts use each model's tokenizer and exclude padding, validation, target generation, screening, and recomputation from gradient checkpointing.Source: §D.1.8 · p. 28, Table 8 · p. 29

D.1.9 TEACHER PROMPT SPECIFICATION

• Teacher and student share the output contract and public intervention, while category-specific guidance is teacher-only; prompts require an explicit reasoning block and a task-appropriate answer block.The shared disclosure guard directs explanations to discuss the task, evidence, and actual added information without inventing requests or certainty; deception prompts distinguish the fictional scene from the character's reply.Source: §D.1.9 · pp. 28, 29, 31, §D.2.5 · p. 32

D.2.2 POLICY OPTIMIZATION

• RL trains each task model with GRPO using a frozen reasoning monitor and outcome verifier, optimizing both reasoning and final-answer tokens with a KL loss and no critic.The one-epoch runs preserve the source release's approximately 90/10 training–validation split; the selected checkpoint is the final checkpoint after one epoch. For Qwen3.5-9B and Mistral, the frozen reference policies are their initial task-model checkpoints.Source: §D.2.2 · p. 31, §D.2 · p. 31, §D.2.1 · p. 31

D.2.3 TRAINING JUDGE AND REWARD

• A frozen Qwen3.5-122B-A10B-FP8 judge scores reasoning-monitor true positives at +4, false positives or false negatives at −2, and true negatives at 0; no separate accuracy, length, or formatting reward is used.For sandbagging, the target is any valid answer semantically distinct from the correct reference. The judge's reasoning-monitor and outcome-verifier calls receive different information, and KL regularization does not guarantee general task accuracy.Source: §D.2.3 · p. 32

D.2.5 TRAINING PROMPTS

• A monitor returns Positive only when reasoning both adopts the intervention as a reason and commits to the target; correctness alone is not the criterion.Merely mentioning, rejecting, or independently agreeing with the intervention is insufficient.Source: §D.2.5 · p. 33
• The monitor evaluates visible reasoning while the verifier evaluates the actual final answer in separate calls to the same frozen judge.The prompts withhold the final answer from the monitor and reasoning and the monitor verdict from the verifier; their errors are not assumed statistically independent.Source: §D.2.5 · pp. 33–34
• For sandbagging, target commitment requires the specified wrong answer, or—without a specific target answer—a valid answer differing from ground truth.The verifier likewise treats a ground-truth-equivalent answer as a miss and accepts any valid answer differing from the reference when no specific target answer is supplied.Source: §D.2.5 · pp. 33–34

D.3 OPSD TRAINING DETAILS

• The student generates its own reasoning-and-answer trajectories, which a frozen self-teacher scores token by token rather than replacing with curated completions.Thus, unlike offline SFT, the trajectories come from the evolving student.Source: §D.3 · p. 35
• For intervention examples, category-specific guidance gives the teacher task references when relevant and favors evidence-linked explanations distinguishing reliance, rejection, non-use, and independent agreement.The guidance discourages fabricated influence, unsupported counterfactuals, private-context leakage, and fixed disclosure phrases.Source: §D.3 · p. 35
• Autoregressive attention blocks access to future reasoning and the sampled final answer when predicting earlier tokens, and privileged information stays out of student inputs and supervised tokens.The implementation checks tokenizer compatibility, response alignment, and nonempty training masks.Source: §D.3 · pp. 35–36
• Intervention examples receive privileged and public-prefix teacher scoring of the same response, while control examples use only the frozen teacher under the exact public student prefix.The public-prefix pass still sees the public intervention; it is not a counterfactual control trajectory.Source: §D.3 · p. 35
• Intervention targets mix sparse top-K distributions from privileged and public-prefix teacher passes, while position weights reflect their distributional differences rather than treatment effects.The retained probabilities use K = 32; omitted entries are zero, and the weighting uses ρ = 0.30.Source: §D.3 · p. 35
• The reasoning-focused objective uses a response-aligned top-K forward-KL surrogate, which is truncated because omitted vocabulary mass is neither restored nor renormalized.Prompt, padding, special, and final-answer tokens are excluded from supervision; losses are normalized within trajectories before averaging.Source: §D.3 · p. 36

D.4 TASK-LEVEL RESULTS

• NA denotes a task score that is not estimable because p1 <= p0.Source: §D.4 · p. 36

D.4.1 SUPERVISED FINE-TUNING

• SFT task-level tables cover three task models and evaluation monitors, with scores averaged over seeds 42–45 and differences reported in score points.Tables 10–12 identify results for Qwen3.5-4B, Qwen3.5-9B, and Mistral Small 3.2-24B.Source: §D.4.1 · p. 36, Table 10 · p. 37, Table 11 · p. 38

(c) GPT-OSS-120B

• The supplied captions identify task-level results for Mistral Small 3.2-24B, but provide no task-level score values in these passages.Source: Table 12 · p. 39, Table 13 · p. 40, Table 14 · p. 41

D.4.2 REINFORCEMENT LEARNING

• RL task-level tables report results for Qwen3.5-4B, Qwen3.5-9B, and Mistral Small 3.2-24B.The captions state that differences are RL minus Base in score points; some mark non-estimable or ineligible scores as NA.Source: §D.4.2 · p. 37, Table 13 · p. 40, Table 14 · p. 41

D.4.3 OPSD

• Selected OPSD task-level tables report results for Qwen3.5-4B, Qwen3.5-9B, and Mistral Small 3.2-24B.Their captions define differences as OPSD minus Base in score points, averaged over seeds 42–45.Source: §D.4.3 · p. 37, Table 16 · p. 43, Table 17 · p. 44