Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision

1 Introduction

• Non-verifiable tasks can have multiple valid answers, making expert annotation hard to scale and deterministic rule-checking impractical.Judge-only feedback can suffer from inconsistency, verbosity bias, and reward hacking.Source: §1 · p. 1
• Self-proposed rubrics score non-verifiable rollouts against binary criteria derived from a pseudo-reference, enabling RL without human reference answers.An independent LLM judge checks each criterion, and the reward is the fraction satisfied.Source: §1 · p. 2
• CaT matches or exceeds inference-time aggregation quality while reducing per-query test-time compute by up to 9×.The framework was evaluated across three model families on HealthBench and MATH-500.Source: §1 · p. 2
• CaT relates to self-training and distillation by learning from model-generated supervision, but reconciles multiple samples rather than trusting one self-label.It amortizes test-time rollouts into model weights and extends programmatic-verification approaches to non-verifiable domains.Source: §1 · p. 2
• CaT converts inference compute into RL supervision through reference estimation and reward derivation, with synthesis as one effective estimator.The framework also admits selection strategies such as majority vote and best-of-N; synthesis can combine information across rollouts.Source: §1 · pp. 1–2

Reference-Free Fine-Tuning

• Earlier self-generated supervision methods target specific capabilities, whereas CaT supports RL without human references when rollouts are diverse and rubric judgment is reliable.Its broader scope depends on those rollout and judgment conditions.Source: §2 · p. 2

Reference-Free RL

• Prior reference-free RL approaches such as TTRL and Absolute Zero focus on math or code, where correctness can be checked programmatically.Other approaches score or select existing outputs; CaT can also use these as reference estimators, while synthesis can construct pseudo-references beyond the rollout distribution.Source: §2 · pp. 2–3

Test-Time Scaling

• Test-time aggregation improves inference but requires each deployment to pay for G rollouts; CaT transfers those gains into weights at 1/G the test-time cost.Majority voting selects the most common answer, while best-of-N selects by confidence, perplexity, or LLM judgment.Source: §2 · p. 3

Non-Verifiable RL

• Non-verifiable reward methods including VeriFree, JEPO, RLPR, and RaR use reference answers; CaT instead derives rubrics from rollout-estimated pseudo-references.The resulting chain is inference compute to pseudo-reference to rubrics to rewards, without human annotation.Source: §2 · p. 3

3 Compute as Teacher (CaT)

• For each prompt, the current policy samples parallel rollouts, which are aggregated into a pseudo-reference used to derive rollout rewards for RL.The framework separates reference estimation from reward derivation.Source: §3 · p. 3

3.1 Reference Estimation

• The framework supports selection, such as majority vote or best-of-N, and generative aggregation; synthesis reconciles information across rollouts and can combine correct fragments.Selection chooses an existing rollout, while synthesis constructs a new response conditioned on the rollout set.Source: §3 · p. 3, §3.1 · p. 3

3.2.1 Verifiable Domains

• In verifiable domains, rollouts receive rewards for agreement with the pseudo-reference; for math, the final boxed expression is checked for string equivalence.Source: §3.2.1 · p. 4

3.2.2 Non-Verifiable Domains

• For non-verifiable domains, the anchor generates binary criteria from the pseudo-reference and a judge scores each rollout by the fraction of criteria satisfied.Binary checks are presented as easier to judge, more auditable, and less sensitive to surface-form bias than holistic scores.Source: §3.2.2 · p. 4

3.3 Training

• CaT reuses GRPO's sampled rollouts for reference estimation; synthesis and rubric-scoring calls add modest overhead relative to generating those rollouts.The rubric-generation call is followed by n × G judge calls, which are parallelized and require few output tokens.Source: §3.3 · p. 5
• After training, CaT matches or exceeds inference-time synthesis quality while using 9× less test-time compute.The trained policy returns a single output, amortizing aggregation gains into its weights.Source: §3.3 · p. 5

4.1 Setup

• HealthBench tests reference-free training on freeform healthcare conversations, with 500 held-out conversations evaluated using physician-designed rubrics; MATH-500 tests verifiable math in a setup training and testing on the same 500 problems.The experiments use three instruction-tuned model families at the 4–8B scale and eight rollouts.Source: §4.1 · pp. 5–6

4.2 Main Results

• Across models, CaT surpassed its initial synthesis-generated pseudo-reference on 5 of 6 model-dataset pairs; gains plateaued as rollouts converged and offered less disagreement to reconcile.The authors describe this pattern as consistent with entropy collapse in RL fine-tuning.Source: §4.2 · p. 6

4.4 Reference Estimation: Synthesis vs Selection

• Synthesis outperformed all selection baselines on HealthBench and matched or exceeded them on MATH-500, supporting use across both domain types.HealthBench gains over the next-best method were statistically significant (p < .05).Source: §4.4 · p. 6, Figure 5 · p. 8

4.2 Main Results

• +30% on HealthBench and +33% on MATH-500 versus the base policy; CaT matches or exceeds inference-time synthesis with 9× less HealthBench test-time compute.On MATH-500, CaT is comparable to inference-time synthesis, slightly ahead on Gemma and Llama and slightly behind on Qwen.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 Self-Proposed Rubrics for Non-Verifiable Domains

• Self-proposed rubrics beat model-as-judge on all three models and matched physician rubrics on two of three; CaT outperformed synthesized-reference SFT on all three.For Llama, CaT scored 0.38 versus 0.28 for CaT-SFT.Source: §4.3 · p. 6, Figure 4 · p. 7

4.5 How Does Synthesis Differ From Selection?

• Synthesis disagreed with majority vote on 5–7% of questions and Self-BoN on 7–10%; when it disagreed, it was correct 82–86% and 70–82% of the time, respectively.These are MATH-500 results averaged over seven seeds.Source: §4.5 · p. 7, Table 1 · p. 8
• On about 1% of questions, synthesis produced a correct answer when every rollout was wrong, which selection cannot do because it only chooses from the rollout set.The paper gives an example where synthesis combines correct reasoning steps from rollouts with complementary arithmetic errors.Source: §4.5 · p. 7
• With Qwen 3 4B on MATH-500, synthesis using eight rollouts scored 0.85 versus 0.81 for majority voting, while synthesis using one rollout scored 0.80 versus that rollout’s 0.79.The authors interpret this pattern as evidence that synthesis may reconcile information across rollouts rather than act as an independent sample.Source: §4.5 · pp. 7–8

5 Discussion

• CaT depends on useful base-model rollouts and, for synthesis, coherent reconciliation; models with lower base performance see smaller gains.As policy improvement reduces rollout diversity, synthesis has less disagreement to reconcile and training gains plateau once rollouts converge.Source: §5 · p. 8
• The authors suggest that more diverse rollouts, learned aggregators, and finer-grained rubric designs could improve reference estimation or reward derivation.Proposed rubric refinements include partial credit, hierarchical criteria, and confidence weighting.Source: §5 · p. 8
• The demonstrated work concerns single-turn responses; reasoning traces, multi-turn dialogues, and agentic trajectories are described as possible extensions.Source: §5 · p. 8
• CaT converts inference compute into supervision through reference estimation and reward derivation; the discussion reports up to 30% relative improvement on HealthBench and 9× less test-time compute while matching or exceeding inference-time aggregation.On MATH-500, CaT performs as well as other approaches, supporting use across domain types.Source: §5 · p. 8

• Tables 2–5 provide tabular transcriptions of results shown in the main-body figures.Source: A Results Tables And Additional Results · p. 12
• Table 2 compares CaT with the initial policy, marking the best result in bold and the second-best with an underline.Source: Table 2 · p. 12
• Table 3 scores different rubrics and judging methods, marking the best result in bold and the second-best with an underline.Source: Table 3 · p. 12
• Table 4 reports HealthBench scores for CaT and CaT-SFT, with the best result in bold.Source: Table 4 · p. 12

A.1 RL Results Against Selection Methods

• CaT outperforms selection baselines with RL; Table 6 reports these results for Llama 3.1 8B, while Table 5 compares inference-time methods across models and benchmarks.Source: §A.1 · p. 12, Table 6 · p. 13, Table 5 · p. 13

A.2 Conditioning Rubric Generation on the Pseudo-Reference

• Conditioning rubric generation on the pseudo-reference rather than only the question improves results by 12% relative.The reported ablation also says reference-estimation compute improves the quality of derived rubrics.Source: §A.2 · p. 13

A.3 Judge Sensitivity

• Stronger judge models yield better CaT performance on HealthBench, with the authors suggesting improved rubric-judgment accuracy as a likely reason.Source: §A.3 · p. 13, Table 8 · p. 14

A.4 Omitting the Question in the Synthesis Aggregator

• Omitting the question from synthesis made no meaningful difference in preliminary Gemma 3 4B MATH-500 experiments (+0.004).The authors say omission simplifies the setup and helps prevent the aggregator from answering the question as another rollout.Source: §A.4 · p. 14

• The example comes from Gemma 3 4B on MATH-500 and features rollouts that failed to solve the problem.Source: B Example: Synthesis Disagrees With… · p. 14
• A sixth rollout also made calculation errors around multiplying and dividing by 137 and 1, similar to the second rollout.Source: B Example: Synthesis Disagrees With… · p. 14
• The synthesized response corrected errors shared across failed rollouts and produced the right final answer.The authors note that finding the right method would not be easy without observing the attempts.Source: B Example: Synthesis Disagrees With… · p. 14

C Analysis of Synthesis Outputs

• Synthesis outputs have similar overall length to policy rollouts despite higher accuracy, but contain fewer equations and substantially fewer lines.Manual inspection found that synthesis often summarizes or enumerates intermediate possibilities already explored in the rollouts.Source: Table 9 · p. 14, C Analysis of Synthesis Outputs · pp. 14–15
• Using keyword heuristics for step-by-step patterns (e.g., “first”, “second”), synthesis employs stepwise reasoning approximately 37% less often (absolute difference) and its outputs contain 9.3% more verification cues, reflecting a tendency to inspect or validate reasoning rather than generate long chains.Source: C Analysis of Synthesis Outputs · p. 15
• The trace comparisons support interpreting synthesis as reconciling partially correct rollout reasoning, not as an additional independent rollout.Correct synthesis outputs are significantly shorter on average (p < 0.05), consistent with selectively reusing already-correct partial reasoning.Source: C Analysis of Synthesis Outputs · p. 15

D When Does CaT Stop Learning?

• When the trained policy approaches its synthesized teacher signal, little performance delta remains; declining synthesis gains indicate rollout agreement and bound further CaT improvement.The teacher signal is synthesis applied at inference time to the trained model.Source: D When Does CaT Stop Learning? · p. 15, Figure 6 · p. 15

E GRPO Overview

• GRPO draws a group of rollouts per question and uses a group baseline; its group-normalized advantage uses reward relative to the group mean, with KL discouraging drift from the reference policy.The reference policy is typically the initial policy.Source: E GRPO Overview · p. 16

F Prompts

• The prompts ask synthesis to reconcile responses into a unified answer while preserving key insights, and request at least five specific, verifiable yes/no rubric criteria.The rubric prompt also instructs the generator to make a high rating difficult to achieve.Source: F Prompts · pp. 17–18

G Example Rubrics

• Example rubrics cover detailed clinical handoff constraints, TB case-report fields, and stroke H&P elements; the examples use Qwen 3 4B on HealthBench.The mastitis handoff example requires concise formatting and inclusion of critical case and treatment details.Source: G Example Rubrics · pp. 20–21

H Hyperparameters

• Training settings are tabulated for RL, SFT, and model sampling; HealthBench training applies a −1 length penalty to responses longer than 750 tokens.The penalty is intended to discourage length-based reward hacking.Source: H Hyperparameters · p. 21, Table 10 · p. 21, Table 11 · p. 21

I Experimental Details

• The minimum-perplexity response is selected as best; minimizing entropy suffices, so the exponential need not be computed.Perplexity is computed over output tokens in response to a question.Source: I Experimental Details · p. 22
• For mutual predictability with G = 8 rollouts, each answer is placed last in turn after the other answers are randomly ordered; the answer with lowest perplexity is selected.The final answer’s token-level perplexity is computed conditioned on the prompt containing the other answers.Source: I Experimental Details · p. 22
• SFT generates eight initial-policy rollouts per HealthBench question, synthesizes a reference, and fine-tunes on it with early stopping and LoRA.The selected checkpoint has the lowest validation loss; LoRA is used because full-parameter fine-tuning overfits quickly and performs worse.Source: I Experimental Details · p. 22
• For math RL, final answers are extracted from boxed text; HealthBench uses validation-based early stopping, while math training runs for a fixed number of steps.Rubric judgments and generations are extracted from XML output.Source: I Experimental Details · p. 23