PaceReasoner: Label-Free Learning to Answer Briefly from Long Reasoning

1 Introduction

• Extended chain-of-thought can improve problem-solving, but longer reasoning raises token use and latency, and its extra gains may be limited for simpler questions or constrained applications.Prior approaches mainly optimize reasoning cost rather than the problem-solving capability of a low-cost mode.Source: §1 · p. 1
• Joint training must limit noisy LONG supervision, provide enough SHORT samples for learning, and preserve LONG responses as a continued supervision source.LONG may dominate generation and restrict SHORT samples, while excessive SHORT encouragement may leave too few LONG responses.Source: §1 · pp. 1–2
• Progressive, confidence-weighted answer supervision and behavior control coordinate answer learning and mode sampling within a shared policy.The behavior reward shifts from encouraging SHORT exploration to agreement with LONG answers, while a ratio controller regulates mode balance.Source: §1 · p. 2
• Across three model backbones and four benchmarks, PaceReasoner improves average SHORT-mode accuracy over TTRL by 7.4–13.5 percentage points with fewer than 25 reasoning tokens per response on average.Ablations support the key components, and cross-dataset results demonstrate transfer beyond the adaptation set.Source: §1 · p. 2
• PaceReasoner uses the same model’s LONG responses as answer supervision to improve SHORT reasoning on unlabeled target questions.The goal is to improve SHORT-mode problem-solving while keeping reasoning brief.Source: §1 · pp. 1–2

2.1 Reasoning Compression and Distillation

• Prior reasoning-compression work includes length-constrained reinforcement learning, pruning, progressive chain-of-thought removal, and distillation from teacher distributions or privileged solutions.PaceReasoner instead supervises SHORT with sequence-level rewards from LONG answer consensus.Source: §2.1 · p. 2
• Adaptive reasoning methods explore or select modes, but greater SHORT usage need not improve capability because mode allocation governs SHORT learning opportunities and access to LONG supervision.PaceReasoner coordinates mixed-mode sampling to sustain both.Source: §2.1 · p. 2

2.2 Label-Free Test-Time Adaptation

• Label-free test-time adaptation methods use answer consensus or intrinsic confidence, while U-OPSD and TTPO use consensus-supported solutions or rollout agreement for distillation.These approaches include non-thinking improvements, and TTPO studies transfer from thinking to non-thinking modes.Source: §2.2 · p. 2
• Joint adaptation couples answer supervision with mode allocation: all-response consensus broadens coverage but risks weaker SHORT predictions, while LONG-only consensus may be sparse or wrong.PaceReasoner combines progressive, confidence-weighted dual-consensus rewards with mode regulation.Source: §2.2 · pp. 2–3

3.1 Test-time Adaptation

• Test-time adaptation updates a policy at inference time without additional ground-truth supervision, using rewards from its own inference before evaluation on the same test questions.The stated goal is improved performance in the short chain-of-thought setting.Source: §3.1 · p. 3

3.2 Reasoning Mode

• LONG responses contain explicit reasoning followed by an answer, whereas SHORT responses give the answer directly; both modes share policy parameters.Reasoning cost counts generated reasoning tokens before the answer, excluding answer tokens.Source: §3.2 · p. 3

3.3 Group Relative Policy Optimization

• GRPO samples a group of responses, assigns scalar rewards, and updates the policy using advantages relative to that group.In this adaptation setting, rewards are derived from LONG-mode reasoning to improve token-efficient SHORT reasoning.Source: §3.3 · p. 3

4.1 Method Overview

• PaceReasoner jointly updates LONG and SHORT under one policy, using LONG consensus to align SHORT; format, answer, and behavior rewards drive optimization.Source: §4.1 · pp. 3–4, Figure 1 · p. 4

4.2 Answer Reward: Long-CoT Consensus

• Answer reward compares responses with LONG majority-vote answers, with all-response consensus providing secondary feedback when LONG samples are scarce.Source: §4.2 · p. 4
• Training shifts from all-response toward LONG consensus, while confidence weighting falls back to all-response feedback for sparse or tied LONG votes; it is not calibrated correctness.Source: §4.2 · p. 4

4.3 Behavioral Reward: SHORT Activation and Ratio Control

• The SHORT behavior bonus moves from exploration toward LONG-answer agreement, while batch SHORT share adjusts its direction around ρS = 0.5 without enforcing a fixed 50% mix.Source: §4.3 · p. 5

4.4 Joint Full-Trajectory Optimization

• GRPO updates complete trajectories from both modes, including mode selection and generation; after adaptation, SHORT runs without LONG sampling or consensus computation.Source: §4.4 · p. 5

5.1 Experimental Setup

• Tests cover GSM8K and MATH-500 for mathematics and CSQA and GPQA-Diamond for commonsense and science, adapting three backbones independently without reference-answer rewards.Comparisons use the same initial checkpoint per backbone and a unified mean@8 protocol against zero-shot CoT, TTRL, Length-Penalty, and CRISP.Source: §5.1 · pp. 5–6

5.2 Short Reasoning Performance

• 13.2, 13.5, and 7.4 points over TTRL on Qwen3-4B, Qwen3-8B, and Llama3.1-8B, respectively, at 24.3, 12.3, and 17.4 reasoning tokens.These are average SHORT-accuracy gains; LONG accuracy also exceeds all baselines, with fewer tokens than TTRL and Length-Penalty on every backbone.Source: Table 1 · p. 6, §5.2 · p. 7

5.3 Ablation Study

• Progressive supervision gives the highest SHORT accuracy on three tasks and the shortest training time on all four.LONG-only supervision leads on MATH-500 at 45.0%, indicating task-dependent supervision preferences.Source: Table 2 · p. 7, §5.3 · p. 8
• Removing ratio control lowers GSM8K accuracy from 39.1% to 16.8% at 19.3 tokens, while removing LONG-answer matching lowers GSM8K and GPQA-D accuracy.Without matching, GPQA-D accuracy falls from 36.9% to 33.3%; its reasoning length also falls from 51.4 to 31.4 tokens, raising Score from 0.7 to 1.1.Source: Table 3 · p. 7, §5.3 · p. 8

5.4 Comparison with Post-trained Qwen3

• SHORT exceeds non-thinking accuracy on six of eight released-Qwen comparison pairs, while LONG exceeds thinking accuracy on all eight.SHORT gains are 13.0–24.9 points on mathematics and 3.1–3.5 on CSQA, but are 0.9–1.0 points lower on GPQA-D; compared models have different post-training histories.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 Cross-Dataset Transfer

• Cross-dataset SHORT gains are 23.1 and 5.2 points over zero-shot, retaining 63–91% of target-adaptation gains while trailing direct target adaptation by 2.2 and 3.0 points.The two transfers adapt on CSQA or GSM8K and evaluate on GPQA-D or MATH-500, respectively; both modes improve over zero-shot without target-set adaptation.Source: Table 5 · p. 8, §5.5 · p. 8

6 Conclusion

• PaceReasoner is a label-free test-time reinforcement learning framework that improves SHORT-mode accuracy using answer supervision from the same model’s LONG responses.Progressive, confidence-weighted supervision and behavior control coordinate answer learning and mode sampling within a shared policy.Source: §6 · pp. 8–9
• 7.4–13.5 percentage points over TTRL in average SHORT accuracy, with fewer than 25 reasoning tokens per response on average.The reported results span three models and four benchmarks; cross-dataset results also demonstrate transfer beyond the adaptation set.Source: §6 · p. 9
• The findings support using a model’s own long-reasoning answers to strengthen performance with short reasoning.Ablations support the key components, and cross-dataset results demonstrate transfer beyond the adaptation set.Source: §6 · p. 9

• Evaluation covers three 4B/8B models and four QA benchmarks, with no coding or interactive-deployment tests and no measurement of multi-seed variability.Source: Limitations · p. 9
• Self-generated consensus can reinforce shared errors and depends on retaining useful LONG reasoning; confidence weighting is heuristic, and controller sensitivity remains unresolved.Ratio control also does not guarantee per-question LONG coverage; SHORT is a prompted mode rather than a fixed token budget.Source: Limitations · p. 9
• The cost metric counts reasoning tokens before the final answer but excludes answer tokens and adaptation computation, so it does not establish lower end-to-end latency, deployment cost, or energy use.Source: Limitations · p. 9

• The four benchmarks are CSQA, GPQA-Diamond, MATH-500, and GSM8K, with question pools of 1,221, 198, 500, and 1,319 items, respectively.Source: §A.1 · p. 11
• For each task, adaptation and evaluation use the same question sequence as unlabeled target data, and final answers are evaluated.MATH-500 does not require a particular visible reasoning length for an answer to score correct.Source: §A.1 · p. 11
• Zero-shot, TTRL, and PaceReasoner use forced LONG and SHORT instructions, while CRISP uses native short-reasoning for LONG and backbone-specific no-think or answer-only settings for SHORT.Actual reasoning lengths are measured for each method.Source: §A.1 · p. 11

A.2 Training Configuration

• With 15 responses per question, a configured actor batch of 64 represents 960 response trajectories globally, not 64 trajectories per GPU.Source: §A.2 · p. 11

A.3 Consensus Construction

• Consensus construction votes over normalized final answers in separate all-response and LONG-only pools; empty pools yield missing pseudo-labels, and tied leading LONG counts receive zero confidence.Confidence weighting is applied after voting.Source: §A.3 · p. 11

A.4 Mode Instructions

• SHORT permits a direct answer as an empty-reasoning case and has no separate token cap; the maximum response length is an overall generation limit.Source: §A.4 · p. 11, Table 6 · p. 12

A.5 Recorded Training Prompt

• The recorded training prompt directs the model to choose whether to think and to enclose reasoning and final answers in their respective tags.Source: §A.5 · p. 12

B Additional Component Results

• Gray component-study entries are LONG evaluations after the intended SHORT behavior failed to activate, so they are not SHORT measurements.Source: B Additional Component Results · p. 12

B.1 Controller Shape and Bonus Strength

• The unshifted cosine controller gives the highest SHORT accuracy across all four tasks, but its higher GPQA accuracy comes with longer reasoning and a lower accuracy-per-token score.A bonus of 2 gives the highest SHORT accuracy on three tasks; a value of 1 fails to activate SHORT on the two mathematical tasks.Source: §B.1 · pp. 12–13

B.2 Training Dynamics

• With activation and ratio control, the rollout pool retains both SHORT training examples and LONG answers for supervision; without activation SHORT sampling approaches zero, and without ratio control it approaches one.Source: §B.2 · p. 13, Figure 3 · p. 15
• On GPQA, all-response voting has an early accuracy drop and LONG-only voting declines later, while progressive supervision improves more steadily.These trajectories do not separately isolate the additional effect of confidence weighting.Source: §B.2 · p. 13, Figure 3 · p. 15

C Recorded Qualitative Examples

• The selected recorded responses preserve original outputs and illustrate behavior; they are not additional measurements from the updated main-table configurations.Table 10 contains SHORT responses, while the remaining example tables contain LONG responses.Source: C Recorded Qualitative Examples · p. 13