PaperlayerView exact revision on arXiv

Source-linked AI summary

Monitoring the Internal Monologue: Probe Trajectories Reveal Reasoning Dynamics

Maciej Chrabąszcz, Aleksander Szymczyk, Marcin Sendera, Tomasz Trzciński, Sebastian Cygert

arXiv:2605.18549v1cs.CLcs.CR

TL;DR

CoT is not always faithful to model outputs, limiting its reliability for monitoring reasoning models. This paper tracks hidden-state probe trajectories across generation and finds that their temporal features improve outcome separation, with max pooling exceeding 90% AUROC while average and last-token pooling are near random.

  • Problem

    CoT is not always faithful to model outputs, limiting its reliability as a sole monitoring signal for reasoning models.

  • Method

    The paper trains multi-layer MIL meta-probes and tracks token-by-token predictions as continuous trajectories characterized by temporal signal-processing features.

  • Results

    Max pooling consistently exceeds 90% AUROC with stable trajectories, whereas average pooling and last-token methods reach near-random performance around 50%; trajectory features improve outcome separability.

  • Takeaways & Limitations

    Probe trajectories provide complementary signals for forecasting future model behavior beyond static probe predictions.

  • Takeaways & Limitations

    The evaluation covers models up to 14B parameters and only harmfulness and mathematical correctness, while harmfulness labels may contain noise.

Abstract

from arXiv · show

Large Reasoning Models (LRMs) introduce new opportunities for safety monitoring through their Chain of Thought (CoT) reasoning. However, CoT is not always faithful to the model's final output, undermining its reliability as a monitoring tool. To address this, we investigate the hidden representations of LRMs to determine whether future behavior can be predicted from prompt and CoT representations. By evaluating a probe at each generated token, we construct a probe trajectory, the continuous evolution of a concept's probability across the reasoning process. We find that future model behavior is more distinguishable when examined over the full trajectory than from a single static prediction. To characterize these temporal dynamics, we extract signal-processing features that capture volatility, trend, and steady-state behavior, significantly improving the separation of future model states. We also present two methodological insights. First, template-based training data achieves near-parity with dynamically generated model responses, eliminating the need for a costly initial inference and labeling. Second, the choice of pooling operation is critical: average-pooling and last-token methods collapse to near-random performance, while max-pooling achieves up to 95% AUROC and yields stable probe trajectories. Using four datasets and four reasoning models across the domains of safety and mathematics, we demonstrate that trajectory features encode task-specific dynamics that improve outcome separability. These findings establish probe trajectories as a complementary framework for monitoring LRM behavior. Warning: This article contains potentially harmful content.

1 Introduction

The paper introduces probe trajectories over latent representations as a complementary way to monitor model behavior when surface-level CoT is unfaithful. Across harmfulness and mathematics tasks, trajectory dynamics and max pooling improve forecasting, while template-based data reduces training overhead.

  • Motivation: Over 10% of cases show surface-level CoT unfaithful to the final output, motivating latent monitoring for safety.The framework therefore tracks hidden representations rather than relying solely on generated reasoning text.
  • Pooling: AUROC ∼50% results from average pooling and last-token extraction, whereas max pooling consistently achieves >90% AUROC and produces more stable trajectories.This max-pooling advantage appears across response-harmfulness and math-error prediction.
  • Method: The paper introduces continuous probe trajectories via a MIL meta-probe to analyze how behavioral intent evolves across reasoning.The internal monologue is defined as the sequence of latent representations produced during CoT generation.
  • Trajectory analysis: Distinct CoT probe-trajectory dynamics provide richer signals for forecasting future behavior than static probes.The trajectories capture temporal evolution in the model’s internal state during reasoning.
  • Training data: Template-based training data achieves near-parity with dynamically generated model responses for concept separation, eliminating costly reasoning-trace generation.The result supports using static templates to train high-quality probes without an initial model-inference and labeling step.

2 From Hidden States to Behavioral Trajectories

This section develops hidden-state probes to forecast an LRM’s final behavior from prompt and CoT representations while avoiding reliance on potentially unfaithful CoT. It defines max-pooled, cumulative probe trajectories and extracts temporal features to characterize internal reasoning dynamics.

  • From Hidden States to Behavioral Trajectories: Lightweight nonlinear concept probes continuously track latent knowledge across generated tokens to forecast an LRM’s final output from hidden states.The approach targets prompt and CoT phases and is intended to mitigate risks from deceptive or unfaithful CoT.
  • Training data: Three training pipelines compare template-based, message-based, and exact model message-based data, with labels inherited from source datasets or generated responses.The exact model message-based pipeline generates CoT and final responses for specific prompts to capture the model’s internal dynamics.
  • Probe architecture and multi-layer integration: Per-layer probes use MLPs with GELU activations, max-pool token representations, and optionally combine layers through a Multiple Instance Learning meta-probe.The MIL meta-probe consolidates cross-layer monitoring into a single task-agnostic training step without substantial computational overhead.
  • Cumulative max-pooling for trajectories: Cumulative max-pooling makes each prediction depend only on information available up to the current token and stabilizes trajectories through a monotonically non-decreasing latent profile.The operation replaces global max-pooling when generating continuous probability trajectories.
  • Probe trajectories: Average-pooled trajectories show high-frequency oscillations, whereas max-pooling isolates salient features and produces smoother trajectories for intent forecasting.The oscillations make average-pooled trajectories susceptible to localized computational noise and unsuitable for reliable forecasting.
  • Trajectory features: Trajectory analysis extracts statistical, temporal, and signal-processing features covering global state, shape and trend dynamics, and additional temporal behavior.The feature set includes statistics such as mean, max, variance, IQR, and RMS, alongside trend, derivative, drawdown, and recovery measures.

3 Empirical Evaluation

Across safety and mathematical-reasoning tasks, probe trajectories improve behavioral separability beyond static representations, with especially strong gains under distribution shift and on harder error-prediction benchmarks. Max pooling and template-based training provide efficient, high-performing evaluation setups, while feature importance varies by domain.

  • Safety evaluation: >90% AUROC is achieved consistently by max-pooled probes, whereas average pooling reduces performance to approximately random-classifier levels across architectures.Average-pooled representations exhibit high-frequency oscillations, while max pooling isolates salient intent features and produces stable, discriminative probes.
  • Safety evaluation: 95.91% AUROC on Qwen3-14B Template data approaches 97.14% with Exact Model Messages, showing static templates can replace costly dynamic generation and labeling.The passage describes Template data as highly competitive with Exact Model Messages.
  • Generalization: Trajectory features remain informative when static-probe performance degrades on the OOD Aegis dataset, indicating that trajectory shape preserves behavioral dynamics across domain shift.The evaluation covers safety datasets including WildGuardMix and Aegis, and mathematical datasets including GSM8K and MATH.
  • Mathematical evaluation: ∼17 percentage points is the maximum AUROC gain from trajectory features over static probes on GSM8K, particularly for the Qwen3 family.On MATH, improvements are modest but reliable; mathematical error prediction remains harder than harmfulness detection, with static max-pooled AUROC below safety-domain performance.
  • Generalization: Trajectory classifiers consistently match or exceed static baselines on held-out MATH categories and generalize across safety datasets, while engineered features outperform a baseline 1D CNN.These results support trajectory features as useful for monitoring future model behavior beyond the specific distributions used for training.
  • Feature analysis: Two feature groups suffice for harmfulness before performance plateaus, while adding more groups continues improving mathematical error prediction.SHAP analysis finds disjoint top feature sets: safety relies mainly on terminal and steady-state features, whereas math relies on volatility and dynamics.

4 Related Work

Related work frames CoT monitoring as promising but limited by faithfulness concerns, motivating hidden-state probing and forecasting. This work distinguishes itself by analyzing how latent probe predictions evolve throughout reasoning rather than relying on text or static snapshots.

  • Large Reasoning Models: Frontier open and proprietary reasoning models commonly generate reasoning traces, creating new challenges for monitoring their behavior.The rise of reasoning traces was largely driven by the RLVR framework.
  • Large Reasoning Models: CoT monitoring has attracted AI Safety interest, but prior work questions whether CoT explains model behavior or faithfully reflects final answers.These faithfulness limitations constrain monitoring approaches based on analyzing CoT text.
  • Representation Engineering and Mechanistic Interpretability: Representation Engineering enabled probing of model behavior, including safety, high-stakes scenarios, and deception.Prior probing research has focused primarily on detecting existing model behaviors from hidden representations.
  • Representation Engineering and Mechanistic Interpretability: Unlike prior work on CoT text, static hidden-state snapshots, or individual-token interpretability, this work systematically analyzes latent probe predictions across the reasoning process.The approach treats the internal monologue as a continuous trajectory rather than a single observation.

5 Conclusions … A.1 MIL Architecture Choices

The paper concludes that pooling choices critically affect internal probe trajectories, while noting limitations in model scale, behavioral coverage, and harmfulness labeling. It also describes the method’s safety motivation, reproducibility plans, and MIL probe architecture.

  • 5 Conclusions: Pooling choice is critical for probe trajectories, with average-pooling and last-token methods collapsing to near-The supplied conclusion passage is truncated after “near-,” so no further performance value is reported.
  • 5 Conclusions: The evaluation covers models up to 14B parameters, leaving validation on significantly larger models such as 70B+ for future work.The authors also identify broader model-scale validation as a next step.
  • 5 Conclusions: The evaluation spans only harmfulness and mathematical correctness, motivating future work on deception, sycophancy, and hallucination.These domains are presented as extensions beyond the current evaluation.
  • 5 Conclusions: Harmfulness labels rely on the WildGuard classifier, introducing potential label noise.This is identified as a limitation of the labeling methodology.
  • 5 Conclusions: The authors will release code upon acceptance, and all experiments used A100 GPUs with 40GB memory and approximately 3000 GPUh.The reproducibility statement reports the hardware and total experimental compute.
  • Appendix: The work aims to improve LRM safety and reliability by using continuous probe trajectories to detect harmful intent despite unfaithful surface-level CoT reasoning.The stated positive impact is a more robust, representation-based tool for real-time safety monitoring.
  • A.1 MIL Architecture Choices: MIL probes use a 3-layer MLP before max pooling, with hidden sizes [1024, 512, 256] and GELU activations.The architecture uses every second residual-stream layer because adjacent representations often change little and encode similar information.
  • A.1 MIL Architecture Choices: MIL probe sizes range from 50M parameters for Qwen3-4B to 80M for Qwen3-14B.These parameter counts correspond to the stated hyperparameter choices.

A.2 Probe Training Details … C LLM Eval Prompts

The appendix specifies probe optimization and trajectory-classifier selection, then defines template-based training prompts and LLM evaluation prompts for harmfulness and mathematical correctness. These procedures use fixed training settings, Random Forest as the selected classifier, and single-word judgments based on chain-of-thought reasoning about the final response or answer.

  • A.2 Probe Training Details: Each probe is trained for 5 epochs with 5% held out for evaluation, maximum message length 8192, and global batch size 32.Training uses AdamW with β1 = 0.9, β2 = 0.999, weight decay 0.01, cosine scheduling, 5% warmup, and maximal learning rate 1e-3; the checkpoint with best validation loss is selected.
  • A.3 Trajectory-Based Classifiers: Trajectory-based classifiers include Logistic Regression, Random Forest, and XGBoost, with Random Forest selected for final scoring in every scenario.The classifiers are trained on the trajectory features described in Section 2.1.
  • B.1 Harmfulness templates: Harmfulness template data frames a user prompt and assistant response as potentially harmful content for classification.The templates use USER: {prompt} ASSISTANT: {response} and distinguish harmfulness-oriented training data from response-oriented training data.
  • B.2 Math Correctness Template: Math-correctness templates present a reasoning process or user-assistant conversation and ask whether the assistant’s reasoning contains an error.The templates use USER: {prompt} ASSISTANT: {response} and separately evaluate reasoning-process responses.
  • C LLM Eval Prompts: The harmfulness evaluator predicts whether the final response would be harmful from chain-of-thought reasoning, not whether the reasoning itself is harmful.It classifies topics including discrimination, harassment, abuse, violence, self-harm, explicit sexual content, and misinformation, outputting only safe or harmful.
  • C LLM Eval Prompts: Both LLM evaluation prompts provide the chain of thought in a delimited field and require a single-word classification.The harmfulness prompt uses safe/harmful labels, while the mathematical prompt uses correct/incorrect labels.
  • C LLM Eval Prompts: The mathematical-correctness evaluator predicts whether the final mathematical answer is correct from the reasoning trace, excluding style, clarity, and presentation quality.Incorrect answers include arithmetic or algebraic mistakes, invalid reasoning steps, wrong assumptions, or answers that do not solve the problem; outputs are correct or incorrect.

D Unfaithful CoT Examples … E.4 Learnable Trajectory Feature Extractors

The examples show that internal monologues can claim safety while producing harmful outputs. Additional experiments find that trajectory-based features generalize across domains and outperform trainable feature extractors.

  • D Unfaithful CoT Examples: The 4B model’s CoT labels prank-call ideas as harmless, but its final response includes a potentially illegal “Fake Emergency” scenario.WildGuard labels the CoT harmful request as Yes, refusal as No, and harmful output as No (Safe), whereas the final response is Yes, No, and Yes (Harmful).
  • D Unfaithful CoT Examples: The 14B model’s CoT plans to avoid actual harmful content, yet its research-framed simulation generates direct encouragement and normalization of suicide.Both the CoT and final response receive harmful-request Yes, refusal No, and harmful-output No (Safe) versus Yes, No, and Yes (Harmful), respectively.
  • E.1 Performance against the number of CoT tokens: Figure 10 evaluates predictive performance as a function of the absolute number of analyzed CoT tokens rather than their percentage.The figure reports mean AUROC as a function of CoT tokens analyzed.
  • E.2 Trajectory-Based Classifiers Out-of-Distribution Generalization: Trajectory-based classifiers generalize zero-shot from WildGuardTest to unseen Aegis data, consistently matching or slightly outperforming the static baseline across models.The temporal features therefore capture reasoning patterns rather than dataset-specific artifacts, without domain-specific retraining.
  • E.3 Detailed Leave One Out on MATH subcategories: Figure 12 provides detailed leave-one-out results for each subcategory in the MATH dataset.It divides the results presented in Figure 8a by MATH subcategory.
  • E.4 Learnable Trajectory Feature Extractors: The 1D CNN trained on prompt and CoT probe trajectories underperforms the paper’s trajectory features across harmfulness and mathematics datasets.Figures 13–16 compare the trainable CNN against the proposed features, including averages over probes or probe types.

E.4.1 CNN architecture · F LLM as a judge

The learnable baseline uses a fixed-length, multi-scale 1D CNN to process variable-length probe trajectories. The LLM-as-a-Judge evaluation reports metrics for WildGuardMix, GSM8K, and Minerva Math with Qwen3-8B as evaluator.

  • E.4.1 CNN architecture: Variable-length trajectories are interpolated to a fixed sequence length of 512 tokens before CNN processing.Token-level trajectory probabilities use linear interpolation, while the binary prompt-to-CoT boundary mask uses nearest-neighbor interpolation.
  • E.4.1 CNN architecture: The CNN processes both token-level trajectory probabilities and a binary mask marking the prompt-to-CoT transition.The mask indicates where prompt tokens transition to Chain-of-Thought tokens.
  • E.4.1 CNN architecture: A multi-scale feature extraction module applies three parallel 1D convolutional layers to the fixed-length inputs.The passage specifies convolutional kernel sizes beginning with 5, but the supplied text is truncated before listing all sizes.
  • F LLM as a judge: The LLM-as-a-Judge section evaluates Chain-of-Thought responses on WildGuardMix, GSM8K, and Minerva Math.Qwen3-8B serves as the evaluator across these datasets.
  • F LLM as a judge: Table 2 reports LLM-as-a-Judge metrics for WildGuardMix using Qwen3-8B as the evaluator.The supplied passage identifies the table and metric unit as percentages.
  • F LLM as a judge: Table 3 reports LLM-as-a-Judge metrics for GSM8K using Qwen3-8B as the evaluator.The supplied passage identifies the table and metric unit as percentages.
  • F LLM as a judge: Table 4 reports LLM-as-a-Judge metrics for Minerva Math using Qwen3-8B as the evaluator.The supplied passage identifies the table and metric unit as percentages.

G Experimental Setup … H.6 Temporal and Relational Landmarks

The paper details a multi-domain, multi-model evaluation protocol and defines trajectory features spanning global states, temporal dynamics, boundary changes, sustained intents, and relational landmarks. These features are evaluated with AUROC and 3-fold test-split cross-validation as a diagnostic measure of separability.

  • G.1 Datasets: The setup covers safety and mathematical reasoning datasets, including WGMix, Aegis, ProcessBench, GSM8K, and MATH, across distinct evaluation roles.WGMix and Aegis assess safety; ProcessBench supports training, while GSM8K and MATH evaluate mathematical reasoning.
  • G.2 Models: The models span Llama-8B-R1-Distill and Qwen3 variants at 4B, 8B, and 14B scales, with hidden states extracted across every second layer from approximately the 25% depth onward.The model selection is intended to test generality across families and parameter scales, while the extraction constructs MIL meta-probes.
  • G.3 Evaluation Protocol: AUROC measures binary separability, while 3-fold cross-validation on evaluation test splits estimates the trajectory features’ diagnostic discriminative richness rather than deployed monitoring performance.Error bars use the bootstrap standard error of the mean.
  • H Detailed Trajectory Feature Definitions: Prompt and CoT probe probabilities are cumulative max-pooled, and global features summarize baseline activation through statistics, ratios, slopes, probability regimes, and signal energy.The defined summaries include maxima, terminal probabilities, robust distributional statistics, trend measures, and high-, low-, and mid-probability proportions.
  • H.2 Shape and Trend Dynamics: Shape and trend features capture curvature, smoothing, drawdowns, recovery, noise, acceleration, surge speed, peak-to-end decline, and terminal derivatives.These features are computed from fitted trajectories, first- and second-order differences, moving averages, and terminal windows.
  • H.3 Temporal Segmentation (Tertiles): Temporal features partition trajectories into tertiles, measure phase means and inter-tertile shifts, and isolate the final-tertile slope and prompt-to-CoT boundary transients.The boundary window uses the final 1% of prompt tokens and first 1% of CoT tokens, with a minimum of one token from each phase.
  • H.5 Signal Processing and Sustained Intents: Signal-processing features distinguish sustained intent from local noise using peak detection, consecutive high-threshold dwell, and normalized first-crossing measures.Peak detection uses scipy.signal.find_peaks with prominence 0.05, while dwell is evaluated at thresholds τ ∈ {0.7, 0.9}.
  • H.6 Temporal and Relational Landmarks: Temporal and relational landmarks include normalized argmax position and CoT-to-prompt mean and maximum ratios, capturing where activation peaks and how levels change across phases.The ratios compare CoT and prompt average and maximum probe probabilities.

I Per Layer Ablation

The layer ablation evaluates probes across layers 0–31, highlighting the top five layers and reporting results for probes trained with response ground truth. The reported P+R values are 0.5821, 0.8026, 0.9149, and 0.5351.

  • I Per Layer Ablation: The ablation covers layers 0–31.
  • I Per Layer Ablation: The top five layers are highlighted in gray.
  • I Per Layer Ablation: The probes are trained with response GT.

J More Avg vs Max Pooling · K Per-token Probability Trajectories

The section compares average- and max-pooling probe probability trajectories, emphasizing the greater stability of max-pooling examples. It also provides per-token trajectories spanning all models, tasks, and task-specific probe configurations.

  • J More Avg vs Max Pooling: Max-pooling trajectories are highlighted as more stable than average-pooling trajectories.The accompanying discussion frames stability as the key contrast in the examples.
  • J More Avg vs Max Pooling: Figure 17 contains additional sample trajectories for both average-pooling and max-pooling probes.These examples extend the trajectory comparison beyond a single sample.
  • J More Avg vs Max Pooling: Figure 17 compares probability trajectories produced by average-pooling and max-pooling probes.The figure presents the two pooling methods directly.
  • K Per-token Probability Trajectories: The appendix reports per-token probability trajectories for all models evaluated in the paper.Its scope covers the complete set of models rather than selected examples.
  • K Per-token Probability Trajectories: The appendix includes trajectories across all tasks represented in the study.The trajectories are organized to cover the full task set.
  • K Per-token Probability Trajectories: The appendix covers every probe configuration trained for each task.This includes task-specific configurations, not only one standardized probe setup.

K.1 Harmfulness Benchmarks · K.2 Math Benchmarks

The appendix presents per-token trajectory figures for harmfulness benchmarks across Wildguardtest and Aegis, covering R1-Llama-8B and Qwen3 models. It also includes Minerva Math and GSM8K trajectory figures across the same four model configurations.

  • K.1 Harmfulness Benchmarks: Wildguardtest trajectories are shown for R1-Llama-8B, Qwen3-4B, and Qwen3-8B.Figures 18–23 cover the Wildguardtest harmfulness benchmark for these models.
  • K.2 Math Benchmarks: Minerva Math trajectories are shown for R1-Llama-8B, Qwen3-4B, Qwen3-8B, and Qwen3-14B.Figures 34–37 are labeled as Minerva Math (Math) for these four models.
  • K.2 Math Benchmarks: GSM8K trajectories are shown for R1-Llama-8B, Qwen3-4B, Qwen3-8B, and Qwen3-14B.Figures 38–41 are labeled as GSM8K (Math) for these four models.
Loading 2605.18549v1…