PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecasting Future Behavior as a Learning Task

Mosh Levy, Yoav Goldberg, Asa Cooper Stickland

arXiv:2606.11445v1cs.AI

TL;DR

Forecasting how large reasoning models will behave is difficult because long reasoning trajectories are not reliably interpretable as natural language. This paper trains Behavior Forecasters to predict behavior directly from single trajectories, outperforming naive readers across two tasks and three datasets at less than 1/10,000 their estimated compute.

  • Problem

    Forecasting large reasoning models’ behavior is difficult because their long trajectories are not reliably interpretable as natural-language explanations.

  • Method

    The paper trains Behavior Forecasters to predict behavioral statistics from single reasoning trajectories using labels generated by querying the target model without human annotation.

  • Results

    Less than 1/10,000 their estimated compute, Behavior Forecasters were more accurate than GPT-5.4 and Claude Opus 4.6 naive readers across two tasks and three reasoning datasets.

  • Takeaways & Limitations

    The reasoning trajectory contains information about the model’s future behavior beyond what naive surface-level reading conveys.

  • Takeaways & Limitations

    The study does not establish generalization to substantially different out-of-distribution tasks, which would require broader and costly training coverage.

Abstract

from arXiv · show

Trust in an AI system is often anchored by explanations of how it works, which one then uses to forecast its behavior on new inputs. For large reasoning models (LRMs), this conventional route is particularly difficult to follow: explanation methods for single token generations do not naturally generalize to long trajectories, and the trajectories themselves are often not faithful when read as natural language. We propose an alternative that bypasses the explanation step: treat behavior forecasting as a learnable task and train Behavior Forecasters that operates on a single reasoning trajectory to make the same forecasts one would typically seek from an explanation. The forecaster's training data is obtained by querying the LRM with no human annotation, and its inference is done in a single forward pass. We instantiate this approach on two tasks: how likely the LRM is to repeat its answer on re-runs, and how removing parts of the input changes its answer. We evaluate this approach on both tasks across three diverse reasoning datasets and find that trained Behavior Forecasters are more accurate than GPT-5.4 and Claude Opus-4.6 reading the same trajectories as naive readers, at a small fraction of their inference cost. We find that fine-tuning the backbone end-to-end and initializing it from the target LRM are each necessary for strong performance. These results show that the reasoning trajectory carries information about the LRM's future behavior that goes beyond what naive reading conveys.

1 Introduction

The paper reframes forecasting an LRM’s future behavior as a learnable task: a Behavior Forecaster uses one reasoning trajectory to predict behavioral statistics without relying on human-readable explanations. Across two tasks and three datasets, it outperforms naive readers at far lower inference cost, while ablations show that end-to-end fine-tuning and target-LRM initialization are essential.

  • Motivation: Explanations are difficult for LRMs because single-token methods do not extend naturally to long reasoning trajectories, whose tokens may not faithfully describe computation.Naively reading such trajectories can therefore produce incorrect behavioral forecasts.
  • Approach: The method forecasts rerun consistency and counterfactual sensitivity in one forward pass from a single target-LRM trajectory.Rerun consistency concerns repeated answers on the same input, while counterfactual sensitivity concerns answer changes under input perturbations.
  • Approach: Behavior Forecasters predict future behavior directly from one observed trajectory, using a task-specific head trained jointly with a backbone initialized from the target LRM.The approach avoids requiring the trajectory’s information to be recoverable through textual interpretation.
  • Results: Less than 1/10,000 the estimated compute enables trained Behavior Forecasters to outperform GPT-5.4 and Claude Opus 4.6 reading the same trajectories naively across two tasks and three reasoning datasets.Forecasters also transfer from two datasets to a held-out third with at most 600 fine-tuning steps.
  • Results: Training only the prediction head, randomizing the backbone, or removing reasoning tokens reduces performance, while optimal prompt, reasoning, and answer ordering depends on the task.These ablations identify end-to-end training, target-model initialization, and reasoning tokens as important design choices.
  • Implications: The findings indicate that reasoning trajectories contain information about future LRM behavior beyond what naive reading conveys, enabling single-trajectory forecasts of properties otherwise probed by resampling.The paper presents behavior forecasting as a learnable task in its own right.

2 Behavior forecasting for LRMs

Behavior forecasting frames trust as the ability to predict an LRM’s future behavior from one observed reasoning trajectory. The paper formalizes this as regression over probability-derived targets and motivates a learned approach because natural-language trajectories may omit or misrepresent computation, while resampling and single-location probes are costly or incomplete.

  • Motivation: Behavior forecasting predicts per-input properties such as rerun consistency and sensitivity to counterfactual input perturbations from one observed run.These signals can support abstention, flagging, or routing individual queries for human review.
  • Formal setup: Given trajectory τ = (P, R, A), the task forecasts a statistic of the model’s answer distribution under related prompts, conditioned on observed answer A.Targets lie in [0, 1] and are derived from the LRM’s induced answer distribution, making the task regression.
  • Tasks: The paper instantiates rerun consistency as the probability of reproducing A and counterfactual sensitivity as the reduction in producing A after removing each input segment.The rerun prompt is P itself, while counterfactual prompts remove segments s from P.
  • Limits of trajectory reading: Natural-language reasoning is an unreliable explanation because LRMs can omit influential cues and because displayed steps may diverge from the computation producing the answer.Faithfulness is not guaranteed to improve with scale or model performance, motivating learned behavior forecasting over naive reading.
  • Limits of existing methods: Resampling requires N additional long runs, while standard single-location probes are either prohibitively expensive or blind to the full stochastic trajectory.Gradient-based attribution also faces discrete sampling steps in the LRM computation graph.

3 Method

The method frames behavior forecasting as supervised learning from a single observed reasoning trajectory, without requiring its information to be readable as natural language. Training uses expensive LRM executions to generate behavioral-statistic labels, while inference requires only one forward pass.

  • Learning setup: Behavior Forecasters predict an LRM’s target behavioral statistic from one observed reasoning trajectory.The approach assumes trajectories encode information about the underlying computation, even when that information is not naturally readable as text.
  • Learning setup: Training labels are generated by running the LRM on related executions and estimating the target statistic b(M, P, A).For each prompt, the procedure first observes one trajectory and answer, then uses forecasting-task executions to estimate the label.
  • Learning setup: Inference uses one forward pass over the observed trajectory, avoiding fresh autoregressive resampling at prediction time.The expensive LRM executions are needed only during data generation, whereas rerun-consistency estimation from fresh samples requires additional trajectories that may contain thousands of tokens.
  • Label construction: 10 runs per prompt estimate answer probabilities, with matching extracted answers defining answer consistency under temperature 0.6 and top-p 0.95.Counterfactual sensitivity additionally uses 10 runs for each perturbed prompt.
  • Model architecture: The forecaster shares the target LRM’s architecture and initializes from its weights, using separate task-specific copies, input arrangements, and prediction heads.Counterfactual sensitivity uses echoed-prompt positions for per-segment predictions, while rerun consistency pools all trajectory positions through learned cross-attention queries into one scalar.

4 Behavior Forecasting is learnable

Behavior forecasting is learnable: trained Behavior Forecasters outperform frontier naive readers and single-location probes across two target LRMs, while generalizing within and across datasets. They are initialized from the target LRM and trained end-to-end, achieving this performance at a small fraction of naive readers’ inference cost.

  • Training: The forecasters are initialized from the target LRM’s weights and trained end-to-end with AdamW.Training uses a linear-decay schedule, 100 warmup steps, a minimum learning rate of 1×10−6, effective batch size 32, and two epochs.
  • Learnability comparison: Behavior Forecasters outperform GPT-5.4 and Claude Opus 4.6 on both forecasting tasks across both target LRMs, at a small fraction of their inference cost.The evaluation uses OLMo-3-7B-Think and Qwen3.5-2B across three reasoning datasets.
  • Learnability comparison: 0.653 is the Behavior Forecaster’s Spearman correlation for Qwen3.5-2B counterfactual sensitivity, versus 0.417 for GPT-5.4 and 0.522 for Claude Opus 4.6.For rerun consistency, the corresponding values are 0.740, 0.224, and 0.267, respectively.
  • Learnability comparison: On the OLMo target, single-location probes reach 0.160 for counterfactual sensitivity and 0.129 for rerun consistency, while Qwen3.5-2B probes reach 0.016 and 0.206.These probes exceed random prediction but remain below the Behavior Forecaster and naive readers.
  • Generalization: Without target-side fine-tuning, a forecaster trained on non-numerical FEVEROUS subsets remains ahead of both frontier naive readers on numerical FEVEROUS examples for both tasks.Transfer across datasets also improves performance on every held-out target for both tasks after at most ∼600 fine-tuning steps, though adapted rerun consistency remains weaker on TreeCut.

5 Ablating the Behavior Forecaster

The ablations show that Behavior Forecaster performance depends on task-specific input arrangements, target-model initialization, and end-to-end backbone adaptation. Reasoning tokens carry essential predictive signal, while freezing the original LRM backbone substantially limits performance.

  • Input arrangement: Task-specific input arrangements matter: counterfactual sensitivity prefers P–R–A–P, rerun consistency prefers P–R–A, and removing reasoning tokens substantially harms both tasks.Prompt P, reasoning R, and answer A are the tested components; the reasoning tokens themselves carry predictive signal.
  • Initialization: Initializing from OLMo-3-7B-Think outperforms random initialization with the same architecture on both tasks, especially for counterfactual sensitivity.The result indicates that pretrained representations contribute beyond architectural capacity alone.
  • Backbone training: End-to-end fine-tuning substantially outperforms freezing the OLMo backbone and training only a prediction head.The original LRM hidden states contain useful signal, but a separately trained copy adapted with its own head extracts that signal more effectively.

6 Related work

Prior work probes model internals or resamples reasoning trajectories to characterize current-run behavior and causal effects. This work instead frames future-behavior forecasting as a learned task, avoiding prohibitive per-query resampling costs.

  • Probing internal states: Internal-state probes predict current-run properties such as factuality, refusals, honesty, and power-seeking tendencies.Related work recovers behaviorally relevant signals from language-model residual streams and internal states.
  • Probing internal states: Probes applied within LRM trajectories predict intermediate-answer correctness and decode the same run’s eventual final answer before it appears in text.These methods support early exit and anticipate outcomes within the current reasoning run.
  • Resampling-based analysis: Resampling-based methods estimate causal effects of reasoning steps by repeatedly intervening on trajectories, requiring dozens to hundreds of fresh trajectories per input.Each trajectory may require thousands of forward passes, making deployment-time per-query costs prohibitive.

7 Limitations

The study does not establish how well Behavior Forecasters generalize to substantially different out-of-distribution tasks, and broader testing is costly because training examples require many long target-LRM generations. More faithful future LRMs could narrow the gap between Behavior Forecasters and naive readers by making relevant computation more visible.

  • OOD task generalization: Behavior Forecasters’ generalization to substantially different OOD tasks remains unestablished.The study tests transfer across datasets within the same behavioral tasks but not across substantially different tasks.
  • OOD task generalization: Broader OOD evaluation is costly because each labeled example requires many additional, potentially long and compute-intensive target-LRM generations.The proposed stronger test would train on a much broader pool of tasks.
  • Faithfulness of future LRMs: More faithful future LRMs could narrow the gap between trained Behavior Forecasters and naive reading by exposing more relevant computation in their surface reasoning.The passage points to explicit methods for training LRMs to produce more faithful chains of thought as a possible route.

8 Conclusion

The paper frames behavior forecasting as a learnable alternative to explanation, using single reasoning trajectories to predict target LRM properties. Behavior Forecasters outperform naive approaches and reveal information about future behavior beyond surface-level text.

  • Conclusion: Behavior Forecasters predict rerun consistency and counterfactual sensitivity from a single reasoning trajectory, using labels generated by querying the target LRM.The approach treats behavior forecasting as an independent learning task.
  • Conclusion: The task is learnable: forecasters outperform strong naive readers and single-location probes, while generalizing across datasets with fine-tuning and same-family variants without it.Strong performance requires target-LRM initialization, end-to-end fine-tuning, and task-specific input arrangement.
  • Conclusion: Reasoning trajectories contain more information about future LRM behavior than surface-level text, motivating automatically labeled behavioral properties as low-cost forecasting targets.Such targets could support forecasting at deployment scale.

A Data and label construction … A.4 Rerun consistency label construction

The paper constructs behavior-forecasting labels from repeated LRM runs across three reasoning datasets, using dataset-balanced sampling, deterministic answer extraction, and task-specific validity filters. Counterfactual labels measure answer preservation after segment removal, while rerun-consistency labels estimate agreement with another run on the same input.

  • A.1 Datasets and splits: The experiments use FEVEROUS, RuleTaker, and TreeCut, combining FEVEROUS’s four largest labeled subsets into one dataset for splitting and balancing.FEVEROUS is treated as one dataset in the paper-facing experiments.
  • A.1 Datasets and splits: Each dataset contributes 1,500 original samples for rerun consistency, while counterfactual sensitivity caps each dataset at 2,000 samples where available and uses RuleTaker’s full filtered pool of 1,628 samples.The combined FEVEROUS budget is distributed across its four subsets in proportion to their available pools.
  • A.3 Counterfactual sensitivity label construction: Counterfactual labels are built from 10 original and 10 perturbed runs per sample, retaining samples with at least 5 valid original runs and original-answer consistency of at least 0.7.Each retained sample receives segment-level labels for its kept removable segments, and perturbed instances require at least 5 valid runs.
  • A.1 Datasets and splits: At inference time, the Behavior Forecaster consumes 5 observed runs per sample, although rerun-consistency labels are estimated from up to 10 post-filter runs.This separates the runs used for label estimation from the fixed number of runs supplied to the forecaster.
  • A.2 Answer extraction: Answer identity is based on dataset-specific extracted values from the post-</think> answer span, with unmatched runs marked extraction-failed and excluded from labeled run sets.The extraction procedures are ordered and dataset-specific: letters for FEVEROUS, true/false tokens for RuleTaker, and numeric or UNKNOWN values for TreeCut.
  • A.3 Counterfactual sensitivity label construction: Counterfactual sensitivity creates one perturbed prompt P−s for each removable segment s ∈ S(P), and keeps each segment label independently when its perturbed runs meet the validity threshold.Missing segments provide no supervision, allowing samples to retain only well-supported segment labels.
  • A.4 Rerun consistency label construction: For rerun consistency, each valid run’s target is based on the fraction of valid runs sharing its extracted answer, estimating the likelihood that another run produces the same answer.The target is computed from runs on the same input after requiring no generation failure and an extractable answer.
  • A.4 Rerun consistency label construction: Samples are discarded wholesale if any run reaches the trajectory-length cap, preserving the exact run set used to compute rerun-consistency labels.The rule avoids recomputing labels after filtering individual long runs.

B Behavior Forecaster training and architecture … C.3 Target-LRM naive reader control

The paper trains Behavior Forecasters with task-specific objectives and specialized heads, then compares them with prompted frontier-model readers and target-LRM probes. The comparisons use matched trajectory information and test whether trained forecasting outperforms naive reading and simple single-location signals.

  • B.1 Training objectives: Counterfactual sensitivity pools token-level logits within each dataset-supplied removable segment and applies binary crossentropy to the clipped attributable-fraction label.Each retained segment contributes one loss term, while evaluation correlates pooled predictions with flattened segment labels.
  • B.1 Training objectives: Rerun consistency predicts one scalar per trajectory, trains against the continuous run-level label with mean-squared error, and treats each of 5 observed runs per sample separately.All reported rerun-consistency results use this continuous objective.
  • B.2 Architectural hyperparameters and transfer fine-tuning budget: The OLMo-3-7B-Think backbone uses hidden size d = 4096, with a [512, 256] token-level MLP for counterfactual sensitivity and a cross-attention sequence-level head for rerun consistency.The sequence-level head pools hidden states with 12 learned query vectors, 32 attention heads, and dropout 0.1.
  • B.2 Architectural hyperparameters and transfer fine-tuning budget: Transfer evaluation initializes each held-out target model from a source Behavior Forecaster trained on the other two datasets and fine-tunes the entire model on the target training split.Counterfactual sensitivity uses the 95/0/5 surface and the final checkpoint after 2 target-side epochs.
  • C.1 Naive reader prompts: Naive readers receive the original input, observed final answer, and full completion, return an untrained continuous score in [0, 1], and use the best variant selected on a 100-sample pilot slice.GPT-5.4 and Claude Opus 4.6 use different counterfactual-sensitivity variants, while both use the direct stability prompt for rerun consistency.
  • C.1 Naive reader prompts: The Behavior Forecaster is more accurate than both naive readers in aggregate and on most per-dataset cells in the OLMo-3-7B-Think comparison.Table 9 reports test Spearman and Pearson correlations, while Table 10 provides the dataset-family breakdown.
  • C.2 Single-location probe baselines on the target LRM: Single-location probe baselines read one fixed location of the frozen target LRM without training, threshold tuning, or calibration.They include answer likelihood from final-answer token log probabilities for rerun consistency and answer-to-input attention from final-answer tokens to candidate segments for counterfactual sensitivity.
  • C.3 Target-LRM naive reader control: An additional control prompts OLMo-3-7B-Think to forecast its own future behavior from its observed trajectory without training or calibration.The stateless variant combines the original input, observed completion, observed final answer, and the forecasting task in one prompt; a stateful variant replays the original input.

D Statistical uncertainty estimates

Uncertainty is estimated with cluster bootstrap procedures over original samples, while transfer comparisons use held-out datasets and remain low-powered with only three targets per task. Adaptation improves Spearman performance across all held-out targets, with task-specific mean gains and identical sign-test values.

  • Main uncertainty estimates: Uncertainty intervals use 5,000 cluster-bootstrap resamples, treating original samples—not flattened segments or observed runs—as the resampling unit.Resampling is stratified by dataset family.
  • Main uncertainty estimates: Paired comparisons match prediction rows between the Behavior Forecaster and each naive reader while resampling the same sample clusters for both systems.
  • Transfer uncertainty: With three held-out targets per task, transfer analysis is necessarily low-powered, although the adapted source improves over the unadapted source on every target for both tasks.
  • Transfer uncertainty: 0.228 is the mean adapted-minus-source Spearman gap for counterfactual sensitivity, with bootstrap CI [0.196, 0.273].
  • Transfer uncertainty: 0.268 is the mean adapted-minus-source Spearman gap for rerun consistency, with CI [0.075, 0.433], while the one-sided sign-test value is p = 0.125 for both tasks.
Loading 2606.11445v1…