PaperlayerView exact revision on arXiv

Source-linked AI summary

Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data

Eric O. Korman

arXiv:2610.18878v1cs.LGcs.AIstat.ML

TL;DR

Answer-only data often lack the CoT traces required for supervised finetuning, while reinforcement learning can require repeated rollouts. The paper trains an inverse model to reconstruct reasoning from prompts and answers, then uses it for offline distillation and verification. After GRPO saturation, distillation roughly doubles hard-prompt pass@1, with results varying by task and metric against baselines.

  • Problem

    Answer-only datasets often lack the groundtruth CoT traces needed for supervised finetuning, while RL can require many rollouts before a correct answer supplies a learning signal.

    Source: §1 · p. 1

  • Method

    The method treats CoT as latent and trains an inverse model to generate the forward model’s reasoning conditioned on the prompt and answer, enabling supervised finetuning from answer-only data.

    Source: §1 · p. 2

  • Results

    After GRPO saturation, inverse-model distillation roughly doubles hard-prompt pass@1; it beats rejection sampling across Countdown hard-tail metrics and matches or modestly exceeds it on GSM8K.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The inverse model supports offline updates on fresh answer-only data without per-step policy-gradient rollouts and can also serve as a correctness verifier.

    Source: §1 · p. 2, §4.2.1 · p. 7, §5 · p. 11

    Because the distillation dataset is fixed while pθ changes, qφ’s approximation to the changing answer-conditioned posterior may drift during optimization.

    Source: §2.3.1 · p. 4

Abstract

from arXiv · show

Post-training reasoning LLMs often requires making a tradeoff between costly supervision and compute inefficiency: supervised finetuning typically requires reasoning traces, which is data inefficient since in many applications one only has prompt-answer tuples and lacks the groundtruth chain-of-thought (CoT) sequence. On the other hand, reinforcement learning (with verifiable rewards, for example) can operate without CoT data but it is much more compute-inefficient since it requires multiple autoregressive completions per optimization step, none of which are guaranteed to give a non-zero policy gradient. In this work we explore a middleground, a self-distillation technique that uses answer-only data, treating CoT as latent variables. This relies on training a partner ``inverse reasoning LLM" that generates the original's reasoning traces conditioned on the answer. The inverse model is trained once and reused across incoming batches of answer-only data, with all generation happening offline and up front, making the approach particularly suited to continual post-training. Using Qwen3-0.6B on the Countdown and GSM8K tasks, we show that after GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts and, on Countdown, outperforms rejection-sampling self-distillation on every hard-tail metric at every mix ratio (with smaller gains on GSM8K); rejection sampling by construction has no signal on problems the model never solves on its own. Continued GRPO on the fresh data retains an edge at hard pass@1 but trails distillation at pass@8 and pass@32 while regenerating rollouts at every step. As an added bonus we see that the inverse model has a second application as an accurate verifier of correctness. When used for best-of-$n$ selection at test time, it raises selection accuracy on Countdown from 0.753 (majority voting) to 0.969.

1 Introduction

The paper seeks supervised CoT finetuning from answer-only data, using an inverse model to reconstruct reasoning traces. After GRPO saturation, inverse-model distillation improves hard-prompt performance and can outperform rejection sampling, while the inverse model also serves as a correctness verifier.

Motivation

  • Answer-only data are common, but reasoning traces are scarce; SFT needs groundtruth traces, whereas RL can learn from answer correctness but requires many rollouts.Those rollouts may fail to produce a non-zero policy gradient until the model samples a correct answer.Source: §1 · p. 1

Approach

  • The method treats CoT as latent and trains a separate inverse model to generate the original model’s reasoning from the prompt and answer.The authors position this as supervised finetuning on question-answer pairs without reasoning data.Source: §1 · p. 2
  • The inverse model is trained on correct answers from the forward model, then supplies reasoning traces for finetuning that model on original or new answer-only data.The new-data setting is described as offline reinforcement learning.Source: §1 · p. 2

Additional use

  • The inverse model can also verify correctness, offering an alternative to adaptive best-of-n sampling.The paper presents verification as an additional use of the same inverse model.Source: §1 · p. 2

Experiments

  • Inverse-model distillation roughly doubles pass@1 on hard prompts after GRPO saturation, while maintaining overall pass@8 and pass@32.On Countdown it beats rejection sampling on every hard-tail metric at every mix ratio; on GSM8K it matches or modestly exceeds it.Source: §1 · p. 2

2 Method

The method treats a reasoning model’s CoT as latent, approximates its answer-conditioned posterior with a separate inverse model, and uses that model for both offline distillation and test-time verification.

2.1 Overview

  • The method maximizes answer likelihood while treating CoT as latent, approximating the answer-conditioned reasoning posterior with a separate model trained on the forward model’s correct completions.Marginalizing over CoT is intractable; the overview depicts the inverse model supporting distillation and verification.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Training qφ

  • The inverse model’s training combines correct-completion NLL with a contrastive margin loss over groups containing both correct and incorrect answers.The contrastive term encourages the model to account for the answer, since NLL-only training tended to ignore it amid longer CoT sequences.Source: §2.2 · p. 3

2.3 Applying qφ

  • The method applies qφ in two ways: to generate reasoning traces for finetuning pθ and to score pθ’s candidate answers as a verifier.The verifier scores the likelihood of the reasoning given the prompt and generated answer.Source: §2.3 · p. 3, §2.3.1 · p. 4, §2.3.2 · p. 4

2.3.1 Distillation from qφ to pθ

  • For distillation, qφ proposes one trace per prompt; the example is retained only if pθ greedily produces the correct answer from that trace, and prompts already always solved are dropped.The fixed dataset can include qφ traces and rejection-sampled pθ trajectories mixed at ratio μ; qφ is used during training only, and pθ deploys alone.Source: §2.3.1 · p. 4
  • All generations occur once before SFT, so traces can be reused across runs instead of regenerating current-policy rollouts at every training step.The qφ model trained on D0 is applied to D1 without retraining.Source: §2.3.1 · p. 4

2.3.2 Using qφ as a verifier of pθ

  • At test time, qφ scores a candidate using the length-normalized likelihood of its CoT conditioned on the prompt and generated answer.The score is computed in one forward pass and requires no groundtruth answer.Source: §2.3.2 · p. 4

3 Comparison to related work

Related approaches use answer-conditioned reasoning for online reinforcement learning or rejection-sampling-based finetuning, while this work trains a separate amortized inverse model for reusable traces. Its training uses mixed correct and incorrect rollouts, which are essential to the contrastive term.

3 Comparison to related work

  • RAVR and CoVRL use answer-conditioned models to improve online algorithms such as GRPO, but still require repeated rollouts from the current policy during training.This method also needs multiple rollouts to train qφ, but incurs them once up front and reuses qφ across fresh datasets.Source: §3 · pp. 4–5
  • Unlike RAVR and CoVRL, this work trains a separate inverse model, which also serves as a verifier for the forward model.RAVR and CoVRL use one LLM for both forward and inverse roles, switched through prompt templates.Source: §3 · p. 5
  • STaR uses rejection sampling and answer-conditioned rationalization, while TRICE treats CoT as latent but uses non-amortized MCMC expectation-maximization with a memory bank.Neither trains a separate inverse model for an amortized posterior.Source: §3 · p. 5
  • The approach is a data-efficiency middle ground: mixed correct and incorrect rollouts can provide learning signal, and are essential for training qφ's contrastive term.The passage contrasts this with GRPO-style methods that use mixed outcomes and pure SFT, which does not.Source: §3 · p. 5

4 Experiments and Results

Experiments test inverse-model distillation after GRPO saturation on GSM8K and Countdown, including fresh and original data, and evaluate the inverse model as a verifier. Distillation improves hard-tail performance, while verifier selection is particularly effective on Countdown.

4 Experiments and Results

  • Using Qwen3-0.6B after GRPO saturation, the experiments evaluate GSM8K and Countdown with an inverse model trained on each environment’s dataset.The inverse model is LoRA-finetuned from a copy of the forward model, with bidirectional attention over prompt and answer tokens.Source: §4 · p. 5

4.1 FCR of qφ

  • FCR@n measures how often the forward model answers correctly within n completions when conditioned on inverse-model CoT; inverse-model gains are strongest on Countdown and hard GSM8K prompts.The authors suggest Countdown’s expression answers expose more solution structure, while GSM8K has less room to improve because baseline pass@32 is already 0.94.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 Distillation from qφ to pθ

  • The study tests distillation both on fresh prompt-answer data, as in continual learning, and on the same data used to train the inverse model.The fixed-data experiment assesses additional learning after GRPO performance has saturated.Source: §4.2 · p. 6

4.2.1 Fresh data

  • On fresh Countdown data, distillation with µ ≥0.5 is strongest on every pass@8 and pass@32 metric, while rejection sampling leads on all-prompt pass@1 and GRPO on hard pass@1.On 17% of fresh prompts the forward model never solves a problem, yielding zero-gradient GRPO groups; qφ produces forward-consistent CoT for 30% of those prompts. Rollouts are generated once and reused across runs.Source: §4.2.1 · pp. 7–8, Table 2 · p. 7

4.2.2 Original data

  • On original data, GSM8K hard-tail gains over rejection sampling are smaller, while high mix ratios lower pass@1 below both rejection sampling and the starting model.Figure 2 shows held-out checkpoint curves for GSM8K and Countdown across mix ratios, with the initial model marked by a dashed line.Source: §4.2.2 · p. 8, Table 3 · p. 8, Figure 2 · p. 9

4.3 qφ-verifier results

  • The second verifier experiment tests qφ for selecting among forward-model completions using both fixed-budget and adaptive best-of-n procedures.The adaptive variant stops when a confidence condition is met.Source: §4.3 · p. 9

4.3.1 Best-of-n

  • On Countdown mixed prompts, qφ cuts best-of-32 selection error from 24.7% with majority vote to 3.1%, with its advantage concentrated when correct completions are a minority.The comparison is restricted to prompts containing both correct and incorrect samples.Source: §4.3.1 · pp. 9–10, Table 4 · p. 10, Table 5 · p. 10

4.3.2 Adaptive best-of-n

  • On Countdown, qφ at about 2 expected draws exceeds adaptive majority vote at every budget and leads the length rule by 6–13 points throughout.The authors attribute this to qφ providing a per-sample signal, whereas vote margins depend on population evidence.Source: §4.3.2 · p. 10, Table 6 · p. 10

5 Conclusion and future work

The paper presents inverse-model distillation as a way to recover hard-prompt performance after RL saturation, with offline updates that can use fresh answer-only data. It also identifies verifier use and several extensions for future work.

5 Conclusion and future work

  • After RL saturation, inverse-model-generated traces recover substantial hard-tail performance, including on prompts rejection sampling discards because the forward model never solves them.The approach also supports offline continual-learning updates on fresh answer-only data without per-step policy-gradient rollout costs.Source: §5 · p. 11
  • Mixed correct/incorrect rollouts are needed once up front to train qφ, while subsequent learning is supervised and the recipe can apply to latent-variable sequence models beyond LLMs.The authors describe this as a middle ground between RL and SFT in sample efficiency.Source: §5 · p. 11
  • Future directions include alternating qφ and pθ updates, using multiple CoT candidates for importance weighting, scaling to larger models, and reseeding RL on the hard tail.The proposed alternating updates would place the method in the expectation-maximization/wakesleep family.Source: §5 · p. 11

A.1 GRPO training of pθ

The appendix summarizes GRPO settings shared by initial training and the continued-GRPO baseline, which restores optimizer state for an exact continuation.

A.1 GRPO training of pθ

  • Continued GRPO is an exact continuation of the original run, with optimizer state restored on fresh dataset D1.Source: §A.1 · p. 12
  • GRPO updates all pθ weights; preliminary LoRA-based GRPO training was unstable.Source: §A.1 · p. 12
  • Table 7 summarizes the GRPO hyperparameters shared by initial pθ training and the continued-GRPO baseline.The continued run uses the same hyperparameters on fresh dataset D1.Source: §A.1 · p. 12, Table 7 · p. 13

A.2 Training qφ

The appendix reports qφ training settings and notes that its validation FCR plateaued around step 25,000 on Countdown; GSM8K used that checkpoint without train-time validation.

A.2 Training qφ

  • Table 8 summarizes qφ training hyperparameters, including margin m and contrastive-loss weight λ.Source: §A.2 · p. 13, Table 8 · p. 13
  • Table 8 is the appendix's summary of the settings used to train qφ.Source: Table 8 · p. 13, §A.2 · p. 13
  • qφ validation FCR on Countdown plateaued around optimization step 25,000, and the GSM8K checkpoint was taken at that step.GSM8K has only train/test splits, so no train-time validation was performed.Source: §A.2 · p. 13

A.3 Distilling qφ into pθ

Distillation SFT uses shared hyperparameters across Countdown and GSM8K, with the mix ratio controlling the share of CoTs sampled from qφ rather than pθ.

A.3 Distilling qφ into pθ

  • The distillation SFT settings are shared across the Countdown and GSM8K experiments.Table 9 summarizes those hyperparameters.Source: §A.3 · p. 14, Table 9 · p. 14
  • The mix ratio µ controls the fraction of sampled CoTs coming from qφ versus pθ.Source: §A.3 · p. 14
  • Table 9 summarizes the hyperparameters used to distill qφ into pθ.Source: Table 9 · p. 14, §A.3 · p. 14

B Additional metrics

Figures 3 and 4 provide held-out evaluation curves for the original-data distillation experiment at k values not shown in the main text. They cover GSM8K and Countdown, with two-seed variability and the initial model indicated.

B Additional metrics

C Qualitative examples

The examples trace how answer-conditioned inverse-model reasoning supplies a step the starting model misses, and how distillation transfers useful strategies to the student. One example is a training prompt, while the GSM8K example is held out and cannot be memorized.

C Qualitative examples

  • On the Countdown training prompt, qφ reasons backward from 244 by dividing by 4 and forming 61 from 75, 7, and 7.After distillation, pθ uses the target-dividing strategy but reaches 61 by a different route, showing strategy transfer rather than sequence replay.Source: Figure 5 · p. 17, C Qualitative examples · p. 17
  • The Countdown example is a training prompt: qφ’s conditioned derivation is the actual SFT datum, while the student’s output differs in route, self-correction, and bracketing.Source: Figure 5 · p. 17
  • On a held-out GSM8K prompt, starting pθ fails all 32 rollouts by answering 6; conditioned on gold answer 9, qφ’s derivation visibly self-corrects through the omitted step, and distilled pθ solves greedily.Source: C Qualitative examples · p. 18, Figure 6
  • The GSM8K example is held out, so it contributed no training data and the distilled model’s success cannot be memorization.The inverse-model derivation is illustrative of the training signal on similar prompts.Source: Figure 6

Related papers

Loading 2610.18878v1…