Source-linked AI summary
AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning
Hanjun Luo, Qiushi Liu, Jingya Zhang, Haihong Pang, Jiaheng Wen, Yifei Ma, Yu Yao, Chengxi Zhang, Hanrong Zhang, Yankai Chen, Hanan Salam
TL;DR
Static and partially adaptive inference-time policies do not jointly adapt decoding stochasticity and reasoning compute within a reasoning trajectory. AutoCRAT uses decoder-side signals to control both dimensions with discrete, semantic-boundary updates, achieving better accuracy–compute tradeoffs across six benchmarks. Its scope is limited by offline training and decoder-side observability, which introduce distribution shift and restrict access to internal reasoning signals.
Problem
Existing inference-time policies are static or adapt decoding stochasticity and reasoning compute separately, leaving their interaction within a single reasoning trajectory unmodeled.
Method
AutoCRAT is a decoder-side controller for frozen backbones that jointly adjusts sampling stochasticity and reasoning budget using decoder-observable signals, discrete actions, and semantic-boundary updates.
Results
Across 6 benchmarks, AutoCRAT uses 13.8–52.7% fewer inference tokens than recommended static configurations and surpasses recommended static and adaptive baselines by 1.5–4.5% in relative accuracy.
Takeaways & Limitations
Within-trajectory joint control achieves a substantially better accuracy–compute tradeoff than baselines while transferring across LLM backbones.
Takeaways & Limitations
Offline training creates distribution shift because the controller is trained on static-configuration trajectories but deployed under adaptive control, potentially limiting its performance ceiling.
Abstract
from arXiv · showhide
Large language models (LLMs) achieve strong reasoning performance, which depends critically on inference-time decisions. Yet these decisions are commonly handled by static, one-size-fits-all policies, limiting adaptation to diverse tasks and reasoning stages. Recent adaptive methods partially address this limitation, but they primarily adapt either decoding stochasticity (how the model explores) or reasoning compute (how long the model reasons) in isolation, leaving their interaction within a single reasoning trajectory unmodeled. To address this challenge, we shift toward a within-trajectory joint control view, and instantiate it in AutoCRAT, a decoder-side controller for frozen backbones. Using only signals available during decoding, AutoCRAT jointly adjusts sampling stochasticity and reasoning budget during generation. AutoCRAT operates over a discrete action space and updates control decisions only at semantic boundaries, improving stability while remaining responsive to the evolving reasoning process. Comprehensive evaluation across 6 benchmarks demonstrates that AutoCRAT (I) uses 13.8-52.7% fewer inference tokens on average than recommended static configurations, (II) surpasses recommended static and adaptive baselines by 1.5-4.5% in relative accuracy, and (III) enjoys strong cross-backbone transferability.
1 Introduction
Inference-time reasoning performance depends on configuring both decoding stochasticity and reasoning compute, but existing approaches adapt these dimensions incompletely or only before generation. AutoCRAT addresses this gap with joint within-trajectory control and achieves stronger accuracy–compute tradeoffs across six benchmarks.
- Inference-time configurations, including decoding stochasticity and reasoning compute, substantially influence LLM reasoning performance.
- Existing methods mainly adapt one reasoning dimension or use configurations fixed before generation, limiting responsiveness across tasks and stages.
- AutoCRAT jointly regulates sampling stochasticity and reasoning budget during generation using decoder-side signals without accessing backbone hidden states.
- Discrete actions and updates at semantic boundaries balance responsiveness with stability by avoiding token-level switching.
- 13.8–52.7% fewer inference tokens and 1.5–4.5% higher relative accuracy than recommended static configurations and baselines are reported across 6 benchmarks.
2 Related Work
Prior work adapts decoding stochasticity or reasoning compute, while request-level methods jointly tune configurations before generation; the resulting trajectory remains largely uncontrolled.
- Decoding Stochasticity Adaptation: Decoding-stochasticity research adapts sampling through temperature, candidate-set, multi-sample, or pre-logit control mechanisms.
- Reasoning Compute Control: Reasoning-compute research includes chain-of-thought, self-consistency, adaptive thinking length, budget allocation, early stopping, and self-correction.
- AdaReasoner and EcoTune jointly tune multiple settings, but their controllers still select configurations mainly at the request level before rollout.
- Request-level control leaves the trajectory itself largely uncontrolled because the full rollout follows the preselected configuration.
3 Within-Trajectory Joint Control
Within-trajectory joint control revises a two-component control state during generation at selected decision points, updating stochasticity and compute together while keeping settings fixed between updates.
- The control state combines decoding stochasticity, which shapes token sampling, and reasoning compute, which determines whether generation continues.
- Request-level inference fixes the control state before generation and keeps it constant throughout the rollout.
- Within-trajectory control revises the control state at decision points during generation and leaves it unchanged between consecutive points.
- Joint control uses one controller to update both control components simultaneously rather than adapting them independently.
- The two-dimensional decomposition covers single-trajectory interventions on sampling-distribution shape and continuation condition, while multi-trajectory methods fall outside this scope.
4 AutoCRAT
AutoCRAT is a decoder-side controller that jointly selects discrete sampling and reasoning-budget actions for frozen backbones, updating them at semantic boundaries during generation. It is trained offline from static traces and designed to balance adaptive control, stability, and transferability.
- Control architecture: AutoCRAT attaches to a frozen backbone and maps decoder-side observables plus prior control states to joint sampling and budget actions.The controller does not access backbone hidden states, supporting deployment across models and interfaces.
- Control architecture: The control head represents each action as a sampling level and a budget level, with admissible actions depending on whether generation is in THINK or ANSWER phase.Budget control is frozen during ANSWER while sampling control remains active.
- Boundary-aware control: Discrete control levels reduce high-frequency oscillation and keep each segment under a stable, interpretable configuration.The controller initializes examples from a fixed balanced state before adapting to the evolving trajectory.
- Boundary-aware control: AutoCRAT updates actions only at eligible semantic boundaries and holds each selected action fixed between consecutive decision points.Boundary-level updates avoid token-level switching and make budget reductions more interpretable by preventing abrupt mid-fragment transitions.
- Offline training: Offline training converts static-grid trajectories into canonical boundary-level samples and scores them by task correctness and normalized token cost.The coefficient λ controls the accuracy–efficiency tradeoff in trajectory quality.
- Offline training: The control head combines behavior cloning with preference learning to match action distributions and rank better actions in comparable local contexts.The same trace pool and controller trained with a small backbone can be reused when deploying on larger backbones without retraining.
5 Experiments
Across six benchmarks and four backbones, AutoCRAT improves the accuracy–compute tradeoff through transferable within-trajectory control, outperforming static, adaptive, and fixed joint configurations. Ablations show that jointly controlling sampling and budget, with semantic-boundary updates, is central to these gains.
- Evaluation Setup: AutoCRAT evaluates six benchmarks across mathematical reasoning, challenging QA, and code generation, reporting accuracy or pass@1 and average completion tokens.The controller is trained on pooled data and evaluated on held-out test splits across four backbones without benchmark- or backbone-specific retraining.
- Static Baselines: 13.8%–52.7% fewer tokens and 2.2%–4.0% higher average accuracy than RECOMMEND across four backbones demonstrate a better accuracy–compute tradeoff.The Qwen3-4B-trained controller retains an overall advantage across backbones, with statistically consistent gains and multi-seed stability.
- Adaptive and Request-Level Baselines: 1.5% higher relative accuracy than EDT with 47.1% fewer tokens, while DEER trails by 4.5% in relative accuracy with 2.9% fewer tokens.AutoCRAT also matches ADAREASONER’s 73.3 average accuracy using 73.1% fewer tokens and reaches comparable accuracy to SELF-CONSISTENCY at over 8× lower token cost.
- Joint Control Analysis: 1.6% higher average accuracy and 43.4% fewer tokens than oracle fixed joint control show that within-trajectory adaptation improves over the best fixed configuration.AutoCRAT actively revises its control state, with average switching counts from 2.34 on ARC-C to 4.17 on MBPP.
- Ablation Studies: 1.0%–3.3% accuracy gains and 52.5% lower average token usage versus sampling-only control show that budget control adds substantial efficiency.Against budget-only control, AutoCRAT gains 1.3%–6.9% accuracy at only marginal token overhead.
- Out-of-Distribution Generalization: 0.6–1.6 points of accuracy gain and 36.8%–42.9% token reductions on two unseen benchmarks confirm generalization without trace collection or retraining.The controller was trained on six original benchmarks and directly applied to SVAMP and CommonsenseQA.
6 Conclusion
The paper reframes inference-time reasoning control as joint, within-trajectory adaptation of decoding stochasticity and reasoning compute, and instantiates this view with AutoCRAT. AutoCRAT achieves a substantially better accuracy–compute tradeoff than baselines, motivating broader inference-time control.
- AutoCRAT shifts LLM reasoning control from isolated or per-request adaptation to within-trajectory joint control of decoding stochasticity and reasoning compute.
- AutoCRAT dynamically controls both dimensions in a single trajectory and achieves a substantially better accuracy–compute tradeoff than baselines.
Limitations
AutoCRAT’s design is constrained by decoder-side observables, offline training, and a narrow evaluation scope. These choices support transferability but create distribution-shift, signal-quality, overhead, and generalization limitations.
- Training and control limitations: Offline training on static-configuration trajectories creates distribution shift when AutoCRAT’s adaptive decisions alter deployed trajectory distributions.Iterative trace collection under the learned policy is suggested as a possible mitigation.
- Training and control limitations: Decoder-side signals can produce false convergence, where fluent but incorrect reasoning triggers premature budget reduction.The controller may therefore reduce reasoning compute despite an incorrect intermediate trajectory.
- Architecture and boundary limitations: Reliance on decoder-side observables limits access to finer-grained internal reasoning signals, while heuristic boundary detection may be less robust for weak or irregular discourse.These are explicit design tradeoffs of the transferability-focused controller.
- Deployment limitations: Boundary inference adds non-zero overhead that can become non-trivial for latency-sensitive deployments or very long trajectories.The overhead is modest in the reported setting because the control head is lightweight and boundary invocations remain bounded.
- Evaluation scope: Evaluation is limited to text-based reasoning on English benchmarks with accuracy-focused metrics, leaving multilingual, multimodal, and agentic settings untested.The behavior of AutoCRAT in these broader interactive environments remains to be established.
Ethics Statement
The paper frames its control-space argument within a constrained single-trajectory, forward-only inference setting and distinguishes admissible interventions from methods that change that setting. Its formal decomposition separates sampling-distribution shaping from continuation control.
- Ethics Statement: The paper’s ethical considerations mainly concern compliance with dataset and model usage policies and responsible deployment, because it uses public benchmarks and open models without human-subject experiments or private data collection.The study reports no sensitive personal data annotation.
- Scope of the formal claim: The completeness claim applies to single-trajectory autoregressive decoding with a frozen backbone, excluding parameter updates, branching, rollback, and external information.Methods outside these constraints are not treated as counterexamples.
- Control decomposition: At each step, inference control comprises whether generation continues and, if so, which next-token distribution supplies the next token.These correspond to continuation control and sampling control.
- Control decomposition: Decoding stochasticity control shapes the conditional next-token distribution, while continuation control determines how far the trajectory extends.Together, these two dimensions are presented as completely characterizing control degrees of freedom within a single trajectory.
- Admissible interventions: Temperature, top-p, repetition penalties, logit biases, grammar masks, budgets, and early stopping instantiate one or both of the two control pathways.The proposition treats these implementations by their functional effect rather than their parameterization.
- Boundary cases: Beam search and tree search use multiple trajectories, rollback revises prior tokens, and hidden-state steering ultimately affects the next-token distribution.EOS biasing can couple distribution shaping with continuation control rather than forming a third pathway.
D Training Algorithm
The training workflow trains one controller offline from pooled static traces across all six benchmarks, using joint supervision and no per-benchmark retraining.
- Training workflow: The controller is trained once on pooled static traces collected from the training splits of all 6 benchmarks.The same trained controller is used for all experiments without per-benchmark retraining.
- Training workflow: Algorithm 2 covers static-trace scoring, supervision construction, and joint optimization of behavior-cloning and preference losses.The workflow is presented as an offline training pipeline.
E Dataset Statistics
The evaluation datasets are split into training and test sets, with benchmark-specific handling for ARC-Challenge and task-appropriate accuracy evaluation. Dataset statistics are reported in Table 10.
- Dataset construction: Datasets are divided into training and test sets using a TRAIN:TEST ratio of 1:4.The dataset statistics are provided in Table 10.
- Dataset construction: ARC-Challenge pools its official train and test splits before resplitting, excluding the 299-question development split.The pooled set contains 2,291 questions.
- Evaluation protocol: Evaluation uses official benchmark pipelines, exact-match accuracy after task-specific answer extraction and normalization, and a math grader when normalization is insufficient.The reported procedure differs by task type while preserving benchmark-specific evaluation conventions.
F Adaptive Baseline Reproduction Details
The evaluation reproduces adaptive baselines with their released settings where available and uses paper-faithful adaptations where implementation support is incomplete. Shared extraction, scoring, and tokenization keep comparisons focused on controller behavior.
- EDT and DEER are reproduced as the two adaptive baselines used in Table 2, following original hyperparameter settings for supported tasks.
- DEER uses a paper-faithful early-exit reimplementation for HumanEval and MBPP because the public repository lacked the code-generation implementation.The original stopping threshold is unchanged, with only the answer-boundary parser adapted for code completion; no per-benchmark calibration is performed.
- All methods use the same task-specific extraction, scoring scripts, and backbone tokenizer, isolating comparisons between single-axis and within-trajectory joint control.
- ADAREASONER trains one shared controller on 99 pooled questions and selects question-tailored instructions, temperature, and step limits at test time.The training set uses four trajectories per question, for 396 trajectories total.
- Self-consistency generates five trajectories per question at temperature 0.6 and aggregates them by majority voting, with all five trajectories counted toward token cost.
G Training Details
The training and evaluation setup combines pooled controller training, qualitative trajectory studies, multi-seed analysis, runtime profiling, and reward-cost sensitivity. These studies examine transfer, controlled behavior, stability, overhead, and operating-point trade-offs.
- Case Studies: The qualitative studies cover a short MATH-500 conversion and an uncertain MBPP algorithmic coding problem using Qwen3-8B.The examples are presented as condensed boundary-level trajectories rather than raw token streams.
- Case Studies: For direct coordinate conversion, AutoCRAT becomes deterministic and short after recognizing the task, preventing unnecessary overthinking without imposing a globally small budget.
- Case Studies: For longest-chain code generation, AutoCRAT allocates more compute during algorithm selection and reduces stochasticity during implementation.
- Failure Cases: An algebraic failure case shows that low-entropy, high-confidence decoding can trigger lower temperature and budget on an incorrect but locally coherent derivation.The resulting convergence leaves insufficient room for a later self-check.
- Failure Cases: A code-generation failure case shows that persistent surface-form uncertainty can retain a large budget after decision-relevant reasoning is complete, causing compute over-allocation.
- Stability and Efficiency: Across three training seeds, every run improves accuracy and token efficiency over RECOMMEND on all six benchmarks.
- Stability and Efficiency: Controller overhead reduces throughput by 16–22%, while shorter generations reduce end-to-end wall-clock time by 45–59%.