PaperlayerView exact revision on arXiv

Source-linked AI summary

Why2Speak: Faithful Reasoning for Abstaining Action Policies

Shreya Mendi, Brinnae Bent

arXiv:2608.20670v1cs.AIcs.CL

TL;DR

The paper asks how to make reasoning faithful for agents that may act or abstain, using intervention timing in multi-party conversation as a controlled testbed. It compares direct and reasoning policies, training methods, and faithfulness audits, finding a capability–auditability tradeoff that training does not close. The results also show that reasoning can change the audited action policy and that common faithfulness tests require controls.

  • Problem

    Abstaining agents must justify both whether to act and what action to take, but existing faithfulness work has focused mainly on answer production rather than action policies.

  • Method

    Using Qwen3-8B with think and no-think decoding, the paper compares direct and reasoning policies, supervised fine-tuning, reinforcement learning, activation probes, and behavioral interventions.

  • Results

    Direct classification yields stronger intervention decisions, explicit reasoning provides greater auditability at a performance cost, and neither supervised fine-tuning nor group-relative reinforcement learning closes the gap.

  • Takeaways & Limitations

    Exposing reasoning can change an agent’s action policy rather than simply make its existing computation observable, so action-policy oversight needs controlled faithfulness evaluations.

  • Takeaways & Limitations

    The experiments focus on one benchmark of synthetic conversations, and complete probe and reinforcement-learning analyses are limited to Qwen3-8B.

Abstract

from arXiv · show

Many agentic systems must repeatedly choose between acting and abstaining, making faithful reasoning important for oversight: an explanation is useful only if it reflects the computation that produced the action. We study this problem through intervention timing in multi-party conversation, where an assistant must decide whether to speak or remain silent. This setting exposes class imbalance, asymmetric action costs, and the possibility that exposing reasoning changes the policy being audited. Using Qwen3-8B, decoded with or without chain-of-thought reasoning, we compare direct decision policies, reasoning policies, supervised fine-tuning, and reinforcement learning. We find a capability-auditability tradeoff: the strongest direct policy achieves higher quality but exposes no reasoning to inspect, while the reasoning policy provides a trace at the cost of lower performance, particularly recall of true intervention opportunities. Supervised fine-tuning either suppresses reasoning or preserves it without improving decision quality, while reinforcement learning also fails to improve the reasoning policy. We identify one mechanism underlying this failure: group relative objectives provide no learning signal on confidently wrong prompts when sampled rollouts all select the same action. Controlled activation probes and behavioral ablations show that standard faithfulness methods can overstate evidence that exposed reasoning reflects the underlying decision process. Probability-based metrics saturate under confident decisions, probes are vulnerable to class imbalance and textual leakage, and reasoning ablations can confound reasoning content with changes in inference mode. Together, these results show that exposing reasoning can change an agent's action policy rather than simply make it observable. We provide controls for evaluating reasoning-based oversight of agents that can act or abstain.

Introduction

Abstaining agents must decide not only what to say but whether to act, making faithful reasoning important for oversight. In intervention timing, direct policies are stronger but opaque, while reasoning policies are auditable yet less accurate, and standard training does not close this gap.

  • Motivation: Intervention timing tests whether an assistant should speak or remain silent, where unnecessary action and inappropriate abstention carry distinct costs.The setting foregrounds abstention, asymmetric costs, and oversight of action decisions.
  • Motivation: Faithful explanations must reflect computation that influenced the action rather than provide a plausible post-hoc rationalization.Unfaithful explanations can make an agent appear transparent when it is not.
  • Findings: The strongest decision policy is a direct classifier without reasoning, whereas the reasoning policy is auditable but less accurate, especially on missed interventions.This is the paper’s capability–auditability tradeoff.
  • Findings: Masked fine-tuning preserves reasoning without improving beyond the base reasoning policy, while decision-token fine-tuning strengthens direct classification but suppresses reasoning.Masked training uses traces that were already correct, offering little supervision for confidently wrong decisions.
  • Findings: Group-relative reinforcement learning in think mode also fails to substantially exceed the base reasoning policy because unanimously chosen rollouts provide no signal, including on confidently wrong prompts.The failure mechanism arises when sampled rollouts all select the same action.
  • Auditing: Controlled audits show that probability metrics, imbalance-blind probes, end-of-reasoning probes, uncontrolled flip tests, and single-run reward comparisons can mislead faithfulness evaluation.Under controlled probing, reasoning-policy decisions become decodable during chain of thought, while direct-classifier decisions are already decodable at the prompt.

Related Work

Intervention timing extends dialogue research by asking whether an assistant should contribute at a particular point in a multi-party conversation. It also adapts chain-of-thought faithfulness questions from answer production to abstaining actions.

  • Intervention Timing in Multi-Party Dialogue: Intervention timing classifies token-level decision points in multi-party conversations as intervene or wait when an external contribution might help.The benchmark includes epistemic gaps such as factual correction, concept definition, data provision, source identification, and synthesis or reframing.
  • Faithfulness Research: Prior chain-of-thought faithfulness work mainly studies question answering, where models produce an answer and explanations can often be checked directly.Generated reasoning may still omit biased cues or fail to reflect the actual decision process.

Methods

The methods evaluate abstaining intervention decisions with reasoning and direct decoding, multiple training procedures, deployment-oriented metrics, and complementary faithfulness audits. The primary testbed uses highly imbalanced synthetic conversation data and Qwen3-8B, with additional models for generality.

  • Dataset and Task: The benchmark contains approximately 173,000 token-level decision points from approximately 16,000 synthetic conversations, with intervention opportunities comprising roughly 13% of examples.Each scenario belongs to one of five intervention types.
  • Dataset and Task: In no-think mode the model emits only a decision token, whereas in think mode it generates chain of thought before producing the final intervention decision.Evaluation ordinarily uses the final decision token.
  • Evaluation: Macro-F1 is the primary deployment metric, supplemented by false-intervention and missed-intervention rates, while AUROC diagnoses discrimination separately from calibration.Deployment performance uses each policy’s natural operating point rather than post hoc threshold tuning.
  • Policy Families: The study compares a base reasoning policy, a direct decision-token classifier, masked reasoning-preserving fine-tuning, and reinforcement-learned reasoning policies.These policy families are evaluated using Qwen3-8B’s shared underlying weights where applicable.
  • Generality: Experiments additionally report Qwen3-30B, Nemotron-3-Nano, GPT-4o, GPT-5.1, and Llama-3.2-3B beyond the primary Qwen3-8B model.Complete implementation details and hyperparameters are reported in the appendix.
  • Faithfulness Audits: Faithfulness is assessed with activation probes, behavioral interventions, and dual-probe analyses comparing internal intervention types with those stated in generated reasoning.Behavioral tests compare complete reasoning, truncated reasoning, and neutral filler.
  • Reinforcement Learning: The reinforcement-learning experiments optimize final intervention decisions while optionally adding a behavioral dependence bonus for reasoning that changes the decision relative to truncation.The bonus uses λ = 0.3, and activation probes are not optimization targets.

Results

Results reveal a consistent capability–auditability tradeoff: direct policies perform best but expose no reasoning, while reasoning policies are inspectable yet less accurate. Fine-tuning and reinforcement learning do not remove this tradeoff, and audits show that reasoning content and inference format both affect measured decisions.

  • Capability and Auditability: Direct decision policies achieve the highest deployment performance but provide no reasoning to inspect, whereas reasoning policies expose decisions at a measurable quality cost.This tradeoff persists across evaluated policies and model families.
  • Capability and Auditability: Qwen3-8B think mode raises macro-F1 from 0.096 to 0.536 and lowers false-intervention rate from 0.989 to 0.232, but missed interventions increase from 0.026 to 0.579.The same base weights behave very differently with and without reasoning.
  • Supervised Fine-Tuning: Decision-token fine-tuning reaches macro-F1 0.620 in no-think mode but falls to 0.367 in think mode, suppressing the reasoning behavior needed for auditability.Attempts to restore reasoning through prompting or prefilling reduce deployment performance further without recovering faithful reasoning.
  • Supervised Fine-Tuning: Masked fine-tuning preserves reasoning generation but does not improve beyond the base reasoning policy because it trains only on naturally generated traces that already produce correct decisions.Its apparent degradation under intervention-enriched mixtures is explained by a training–deployment prior mismatch, while reasoning generation and AUROC remain essentially unchanged.
  • Stated and Internal Reasons: Decision information becomes highly decodable only after reasoning, while intervention-type information is strongest before reasoning and weakens toward its conclusion.The reasoning policy therefore appears to develop the intervention decision during the chain of thought, unlike direct classifiers that commit before explanation.
  • Behavioral Dependence on Reasoning: Reasoning content has a modest but reliable effect beyond format: retaining the first half of the original trace pulls predictions toward full-reasoning behavior, while neutral filler shifts them further.Removing reasoning entirely produces an almost always-intervene policy, so format effects must be controlled when interpreting ablations.

Discussion

The discussion frames a persistent capability–auditability tradeoff and argues that faithfulness evaluation for abstaining policies requires stronger methodological controls and broader validation.

  • Capability–auditability tradeoff: The strongest policy is a direct classifier, while the most auditable policy reasons explicitly but performs worse; neither supervised fine-tuning nor group-relative reinforcement learning closes the gap.The paper describes this as a practical boundary for explainable abstaining policies.
  • Evaluation implications: Faithfulness evaluation must account for abstention, class imbalance, asymmetric costs, calibration, and whether the policy should act at all.These properties distinguish abstaining action policies from question-answering settings.
  • Evaluation implications: Probability-based contribution metrics saturate when decision probabilities are near one, so they cannot distinguish genuine reasoning from post-hoc controls.The limitation arises for low-entropy actions whose decision-token probabilities remain high across reasoning conditions.
  • Evaluation implications: Calibration should be evaluated separately from discrimination because training can shift intervention thresholds without substantially changing threshold-free discrimination.Deployment metrics alone may therefore conflate operating-point changes with representational changes.
  • Evaluation implications: Probe accuracy requires controls for class imbalance, construction confounds, parameter transfer, probe position, and leakage from generated explanations.End-of-reasoning probes may read the explanation itself, whereas pre-reasoning probes may precede an explicit reason representation.
  • Behavioral controls: Reasoning perturbations should compare truncation with length-matched neutral filler because format disruption can push models toward a degenerate no-think policy.This comparison isolates dependence on reasoning content from dependence on reasoning format.
  • Training evaluation: Reinforcement-learning objectives should be compared across independent runs because between-run variability can exceed differences between reward objectives.The recommendation applies beyond GRPO to stochastic RL and preference-optimization pipelines.
  • Training evaluation: Behavioral faithfulness rewards may optimize dependence on reasoning format rather than reasoning content, illustrating that proxy optimization need not improve the underlying reasoning process.The authors connect this failure mode to a broader form of Goodhart’s law.

Conclusion

Using intervention timing as a controlled testbed, the paper finds that direct classification is stronger while explicit reasoning is more auditable but less performant. It concludes that question-answering faithfulness methods do not transfer directly to abstaining action policies.

  • Conclusion: Direct classification yields stronger intervention decisions, whereas explicit reasoning provides greater auditability at a performance cost.This is the paper’s consistent capability–auditability tradeoff.
  • Conclusion: Neither supervised fine-tuning nor group-relative reinforcement learning closes the capability–auditability gap.The conclusion reports this across the studied training approaches.
  • Conclusion: Controlled probes and behavioral interventions provide evidence that reasoning contributes to decision formation rather than merely rationalizing a predetermined action.The evidence supports studying faithful reasoning specifically in policies that decide whether to act.
  • Conclusion: Faithfulness methods developed for question answering do not transfer directly to action policies where deciding whether to act is as important as deciding what to say.The paper presents intervention timing as a foundation for studying faithful reasoning in agentic systems.

Reproducibility Statement

The reproducibility statement summarizes the implementation, statistical, and documentation practices used to support the paper’s experiments.

  • Implementation: Supervised fine-tuning uses LoRA adapters, reinforcement learning uses GRPO, and activation probes use L2-regularized logistic regression on cached hidden states.The experiments run on a single 24 GB GPU.
  • Evaluation: The study reports three random seeds per reinforcement-learning objective, exact McNemar tests for paired behavioral interventions, and bootstrap confidence intervals where applicable.Models are evaluated at their natural operating points.
  • Documentation: The supplement provides experimental details, controls, and complete analyses supporting the main paper.Appendix-prefixed table, figure, and section numbers refer to the supplement.

A.1 Dataset

The benchmark evaluates whether an assistant should intervene or remain silent at highly imbalanced conversation decision points, comparing reasoning and direct policies with supervised and reinforcement-learning methods. Probe, calibration, and reproducibility controls are used to assess both deployment behavior and reasoning preservation.

  • Dataset: Intervention opportunities comprise approximately 13% of decision points in the binary task, creating severe class imbalance.The model predicts whether to intervene now or remain silent from a multi-party conversation prefix.
  • Dataset: No-think and think conditions use the same instruction and differ only in the native reasoning toggle.In think mode, only the final decision token after the reasoning span is scored.
  • Models: The study evaluates Qwen3-8B because one checkpoint supports both native reasoning and direct decoding.The model configurations include base reasoning, direct decision-token supervision, masked reasoning-preserving supervision, and GRPO policies.
  • Evaluation: The evaluation reports macro-F1 alongside false-intervention and missed-intervention rates, with AUROC used as a threshold-free diagnostic.Confidence intervals use bootstrap resampling, and reinforcement-learning objectives are compared across independent runs.
  • Training: Masked fine-tuning preserves native reasoning on approximately 99% of evaluation examples while training only on naturally generated traces with correct decisions.The objective masks reasoning tokens and applies optimization only to the final intervention decision.

B.3 Threshold-Free Discrimination Remains Stable

Masked supervised fine-tuning changes deployment performance mainly through the training intervention prior rather than through loss of reasoning or threshold-free discrimination. The same qualitative pattern appears in a second model family.

  • Threshold-free discrimination: AUROC remains statistically unchanged at the two measured endpoints after masked supervised fine-tuning.This argues against a decline in threshold-free ability to distinguish intervention opportunities.
  • Prior shift: Macro-F1 decreases substantially as intervention prevalence increases during training, while AUROC remains within overlapping confidence intervals.The model continues ranking opportunities similarly but shifts the threshold used to produce binary actions.
  • Cross-model replication: The qualitative pattern replicates on Nemotron-3-Nano: intervention-enriched mixtures reduce deployment macro-F1 despite stable threshold-free discrimination.Absolute performance differs, but the operating-point shift remains consistent.
  • Interpretation: Across training mixtures, reasoning generation remains essentially unchanged while deployment macro-F1 decreases only when the training intervention prior exceeds the deployment prior.These observations support calibration shift rather than degraded reasoning or reduced representational capacity.

C.4 Results

Increasing reinforcement-learning rollout diversity substantially increases the available optimization signal but does not improve the held-out performance plateau. Confidently wrong prompts remain difficult for both masked imitation and group-relative RL to correct.

  • RL optimization: Held-out macro-F1 remains within essentially the same range after increasing rollout diversity, group size, and training iterations.The pushed configuration briefly reaches similar peaks but establishes no higher performance plateau.
  • RL mechanism: Group-relative RL provides no update when all sampled rollouts receive the same reward, including prompts where every rollout is confidently wrong.Zero centered advantage removes the learning signal on uniformly incorrect groups.
  • Limitations: Alternative RL algorithms with learned value functions or global baselines could provide non-zero signals on uniformly incorrect groups, but their effect on faithful intervention timing remains open.This is presented as a possible direction rather than an established result.
  • Asymmetric reward: A single asymmetric-reward RL run tests whether changing the missed-intervention penalty can move the auditable policy’s operating point.The tested reward assigns correct decisions +1, false interventions 0, and missed interventions −0.5.

D.1 Probe Methodology

The probe methodology controls for imbalance, textual leakage, register mismatch, and probe position when locating intervention-decision representations. Behavioral ablations further separate reasoning content from disruption of the reasoning format.

  • Probe controls: AUROC is preferred to raw accuracy because majority-class prediction can look accurate under the approximately 13% intervention prevalence.Within-family evaluation, text-only baselines, shuffled labels, and nested cross-validation reduce common probe overestimation sources.
  • Register controls: Probe parameters do not transfer reliably across representational registers, so pre-reasoning decodability is interpreted as register-specific rather than universal.Mixed-set evaluations can mistake conversation-family differences for intervention-decision information.
  • Decision timing: Within-family probes show decision decodability near the text baseline through 75% of reasoning, then AUROC rises from 0.64–0.68 to 0.976 at the close.The text-only baseline is 0.631, and the trajectory uses nested layer selection over ten splits.
  • Behavioral ablations: Neutral filler causes more decision changes than truncation, showing that raw flip rates partly reflect reasoning-format disruption rather than reasoning content.Comparing partial reasoning with filler yields a modest but statistically reliable content-dependent effect for the base reasoning policy.
  • Replication: Reinforcement-learning dependence magnitudes vary substantially across seeds, requiring independent-run replication for objective-level conclusions.The accuracy-only reward produces both positive and negative dependence gaps across runs.
  • Taxonomy caveat: Human agreement on the five-way intervention taxonomy is low, limiting interpretation of stated-type results.The annotators agree with each other at κ = 0.21, while all three sources select the same category on 41 of 120 items.

F.2.1 Annotator Agreement Analysis

The analysis finds that reinforcement learning modestly improves agreement between generated explanations and gold intervention labels, but probe evidence indicates those explanations are constructed during reasoning rather than read from pre-existing representations.

  • Pre-CoT probes test internal representations before explanation generation, whereas end-of-CoT probes can recover information copied from generated text.
  • End-of-CoT probe predictions align more closely with stated than gold intervention types, while this alignment largely disappears before reasoning begins.
  • The pattern suggests stated explanations are constructed during reasoning rather than decoded from an already formed internal representation.
  • Dual-probe analyses can overstate faithfulness because pre-reasoning explanations may lack explicit representations and post-reasoning probes can read generated text.
  • Reinforcement learning modestly increases agreement between generated explanations and gold intervention labels.

G.2 Behavioral Evaluation

The behavioral evaluation finds no substantial deployment or audit advantage for the faithfulness reward over accuracy-only reinforcement learning or the base reasoning policy. Stochastic optimization introduces between-run variability larger than the apparent objective differences, requiring replicated training runs for objective-level inference.

  • Both reinforcement-learning objectives remain statistically similar to the base reasoning policy across deployment metrics, with only modest changes in the targeted behavioral quantity.
  • The magnitude of behavioral dependence varies substantially across optimization runs, and the accuracy objective’s estimated effect changes sign across seeds.
  • Between-run variability exceeds the observed differences between reward objectives, so single-run comparisons may reflect optimization noise rather than reward-function properties.
  • The optimization run, rather than the individual evaluation example, is the appropriate unit of inference for comparing reward objectives.
  • Objective-level comparisons require replication across independent training runs because evaluation examples are not independent observations of the reward objective.
Loading 2608.20670v1…