PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher

Arda Uzunoglu, Alvin Zhang, Daniel Khashabi

arXiv:2606.01000v1cs.LGcs.CL

TL;DR

Weak-to-strong generalization must identify reliable weak labels when ground-truth supervision is scarce and imperfect weak labels limit student performance. This paper treats the problem as data selection, using trust functions based on teacher representations to filter weak supervision. Across multiple domains, filtered students match or exceed ground-truth-supervised students, while iterative teacher reuse produces compounding gains.

  • Problem

    Weak-to-strong transfer can surpass the weak teacher but often fails to close the gap to ground-truth training because weak labels contain errors and omissions.

  • Method

    Neural trust functions estimate weak-label correctness from teacher internal representations and filter weak supervision before training the strong student.

  • Results

    Across world knowledge, quantitative reasoning, and strategy games, trust-filtered students match and sometimes exceed ground-truth-supervised students, achieving near-lossless generalization.

  • Takeaways & Limitations

    Trust filtering also enables a weak-to-strong chain in which each student becomes the next teacher, producing compounding improvements across iterations.

  • Takeaways & Limitations

    Neural trust functions require labeled source data and are limited to outcome supervision and simple MLPs over single-token hidden states.

Abstract

from arXiv · show

Weak-to-strong generalization studies how to improve a strong student using supervision from a weaker teacher when reliable labels are scarce. We view this primarily as a data selection problem, where the key challenge is to identify which weak labels are reliable enough to serve as a training signal. To address this, we introduce trust functions that assign each weak label a scalar trust score and use these scores to filter weak supervision. Across several domains, including world knowledge, quantitative reasoning, and strategy games, trust filtering yields students that match and sometimes surpass ground-truth supervision, achieving near-lossless weak-to-strong generalization. Moreover, trust functions enable an iterative weak-to-strong chain that compounds gains by training a student and reusing it as the next teacher, amplifying the gains. There are several mechanisms to which advantage of trust functions can be attributed.

1. Introduction

Weak-to-strong generalization is framed as selecting reliable weak labels for strong-student training. Neural trust functions use teacher representations to filter supervision, achieving near-lossless performance across domains and enabling compounding weak-to-strong chains.

  • Motivation: Weak supervision can improve students beyond their teachers but often fails to match ground-truth training because imperfect labels propagate errors and omit task-relevant directions.These problems can worsen under distribution shift.
  • Approach: Trust functions frame weak-to-strong generalization as data selection by assigning scalar scores that estimate weak-label reliability and usefulness for training.The approach holds architecture and training algorithm fixed while selecting the weakly labeled subset.
  • Approach: Neural trust functions predict weak-label correctness from teacher internal activations rather than relying only on output-level heuristics.They are trained on labeled source data and deployed zero-shot under in-domain distribution shift.
  • Results: Across world knowledge, quantitative reasoning, and strategy games, trust-filtered supervision matches and sometimes exceeds ground-truth supervision training.The reported outcome is near-lossless weak-to-strong generalization.
  • Results: Iteratively reusing each trained student as the next teacher forms a weak-to-strong chain with compounding gains across generations.The chain is presented as an extension beyond one-shot transfer.

2. Trust Functions

Trust functions score weak-label reliability without ground-truth access at inference time, and neural trust functions learn these scores from teacher representations. The resulting filtering pipeline is trained on labeled source data and applied to unlabeled target data, with cost dominated by teacher inference.

  • Trust functions: A trust function maps teacher-derived features of an input-prediction pair to a scalar score representing the estimated probability that the weak label is correct.Higher scores indicate more reliable supervision, and scores must be computable without target labels at inference time.
  • Neural trust functions: Neural trust functions train on labeled source data to predict weak-label correctness from teacher features, then produce trust scores for unlabeled target data.The scores support filtering or loss reweighting; this work focuses on filtering.
  • Neural trust functions: The default feature is the teacher’s last-layer hidden state for the final generated token, which summarizes the input and intermediate reasoning context.Layer choice, token position, and pooling strategies are ablated separately.
  • Efficiency: The pipeline’s total cost includes teacher label generation, trust-function training, and target scoring, with the expensive teacher forward pass shared by label generation and hidden-state extraction.The trust function is a small MLP applied to cached hidden states.
  • Evaluation: Reliability evaluation reports AUC, ECE, Brier score, and purity for the top-trust subset used in training.Purity measures the fraction of correct examples among retained examples.
  • Assumptions: Neural trust functions require labeled data, but the labeled source need not come from the target distribution when deployment uses in-domain distribution shift.The paper motivates this setting by distinguishing abundant labeled benchmarks from scarce target domains.

3. Weak-to-Strong Generalization by Learning to Trust

The framework treats weak-to-strong generalization as selecting reliable weak labels without target ground-truth labels. Neural trust functions score weak-label correctness from teacher representations, retain high-trust examples, and train students across three domains and several model families.

  • Experimental Setup: Trust functions are trained on labeled source data and applied to target data without access to target ground-truth labels.The source and target distributions need not be identical, and experiments include ID and in-domain distribution-shift regimes.
  • Learning to Trust Framework: A weak teacher generates labels on an unlabeled target pool, while a neural trust function ranks them for strong-student training.The student is trained on the retained high-trust examples using a fixed downstream recipe such as SFT or GRPO.
  • Experimental Setup: The study covers world knowledge, quantitative reasoning, and strategy games using MCQA, mathematical problem solving, and chess puzzles.It evaluates OLMO2 and QWEN3 families across multiple model scales, with additional quantitative-reasoning models.
  • Experimental Setup: Baselines differ only in weak-pool selection, including random sampling, confidence-based selection, and reward-model filtering.Students are evaluated on held-out labeled target data using domain-standard metrics and compared with ground-truth training through recovery.

4. Results

Across world knowledge, quantitative reasoning, and strategy games, trust filtering consistently outperforms unfiltered and confidence-based weak supervision. It approaches ground-truth performance, with near-lossless or super-recovery reported in multiple settings.

  • 4.1. World Knowledge: NTF closely matches Ground Truth across world-knowledge benchmarks, with statistical indistinguishability in 5 of 8 settings and superiority in 1.It also outperforms naive, confidence-based, ensemble, and reward-model filtering baselines.
  • 4.2. Quantitative Reasoning: Quantitative-reasoning experiments train trust functions on MATH, select top-n OMNI-MATH rollouts, and evaluate students on AIME.Students are trained with GRPO after trust-based ranking.
  • 4.2. Quantitative Reasoning: Recovery is 89–92% across quantitative-reasoning settings, and NTF is statistically indistinguishable from Ground Truth in half of them.NTF consistently outperforms naive and confidence-based selection, especially for weaker teachers such as Qwen3-1.7B.
  • 4.3. Strategy Games: NTF closes essentially all of the weak-to-strong gap for the QWEN3 strategy-games family and surpasses Ground Truth in 4 of 8 settings.OLMO2 is less reliable in this domain, but NTF remains the strongest weak-supervision strategy and narrows the gap.

5. Snowballing Weak-to-Strong Generalization

Iterative trust-filtered training compounds improvements across successive teacher-student generations. The final chained model outperforms several one-step, naive-chaining, and budget-matched ground-truth baselines, while NTF selects easier chess puzzles than comparison filters.

  • 5. Snowballing Weak-to-Strong Generalization: NTF chaining yields compounding gains across student scales and outperforms ground-truth supervision at the largest scale.Table 5 compares chaining with shallow one-step transfer and reports increasing returns across iterations.
  • 5. Snowballing Weak-to-Strong Generalization: The final chained Qwen3-14B beats transfers from both Qwen3-0.6B and Qwen3-8B, naive chaining, and budget-matched ground-truth training.Intermediate students show the same pattern, with the advantage widening across iterations.
  • 5. Snowballing Weak-to-Strong Generalization: For Qwen3-0.6B, NTF shifts selected-puzzle ratings toward lower-rated puzzles than naive and confidence-based selection.The rating distribution provides a difficulty profile for the selected strategy-games training set.

6. Mechanisms Behind Near-Lossless Weak-to-Strong Generalization

The analyses attribute NTF’s gains to easier selected instances, strong alternatives among apparent false positives, and more coherent gradient signals. Difficulty reweighting helps smaller students but does not explain larger-scale improvements.

  • 6.1. Finding 1: Neural Trust Functions are Conservative: NTF concentrates selection on lower-rated chess puzzles, creating an implicit easy-first curriculum alongside label-noise filtering.This shift is also observed in quantitative-reasoning data selection.
  • 6.1. Finding 1: Neural Trust Functions are Conservative: Difficulty matching recovers part of NTF’s advantage for 1.7B and 4B students, but not for 8B and 14B students.Naive-DM preserves NTF’s difficulty profile while keeping label purity comparable to Naive.
  • 6.2. Finding 2: Neural Trust Functions Often Recover Optimal Alternatives: 66.1% of NTF-retained moves lead to a winning mate, and many dataset-marked false positives are stronger than the ground-truth move under Stockfish.The advantage-gap distribution has substantial mass on negative values, indicating stronger NTF-retained moves.
  • 6.2. Finding 2: Neural Trust Functions Often Recover Optimal Alternatives: Replacing NTF’s teacher moves with ground-truth labels lowers performance slightly, showing that selected examples—and not only relabeling—drive the main gains.Only 4.73% of labels change because NTF selection has 95.27% purity; NTF-GT remains competitive with and often exceeds GT training.
  • 6.3. Finding 3: Neural Trust Functions Produce More Coherent Gradients: NTF-retained examples concentrate more gradient energy in shared low-dimensional subspaces, producing more aligned and coherent update directions.The advantage persists across small to moderate subspace dimensions and supports improved stability and sample efficiency.

7. Risk-Controlled Data Selection

The paper calibrates trust-score thresholds with a labeled target subset and a finite-sample upper confidence bound on selected-label noise. This yields an inclusive threshold with controlled realized deployment noise without manual tuning.

  • 7. Risk-Controlled Data Selection: Risk-controlled calibration uses a small labeled target set to choose a trust-score threshold when the retention budget is unknown.The procedure estimates calibration noise and applies a finite-sample upper confidence bound.
  • 7. Risk-Controlled Data Selection: θ⋆=0.895 retains 16.1% of the deployment pool while the realized deployment noise remains below α=0.1.The threshold is the most inclusive one satisfying U(θ) ≤ α.
  • 7. Risk-Controlled Data Selection: For fixed thresholds with n(θ)>0, Hoeffding’s inequality bounds the true selected-label noise rate by U(θ) with probability at least 1−δ.The rule chooses the most inclusive threshold satisfying U(θ)≤α; multiple-testing correction is deferred to Appendix I.
  • 7. Risk-Controlled Data Selection: The calibration procedure selects a high-purity subset without manually tuning the threshold, indicating that higher trust scores align with cleaner weak labels.The finite-sample UCB is operationally useful rather than vacuously conservative in the reported experiment.

8. Related Work

Related work places weak-to-strong generalization across easy-to-hard learning, preference learning, reward modeling, planning, trust estimation, and training-framework interventions. This paper differs by using teacher hidden representations for trust-based data selection.

  • Limitations of Weak Supervision: Prior weak-to-strong studies span easy-to-hard generalization, preference learning, reward modeling, and planning, while a performance gap to weak supervision remains.These works establish the broader weak-to-strong setting addressed by the paper.
  • Trust Estimation: Data selection and reweighting have been studied for in-domain weak supervision, whereas this work targets weak-to-strong generalization under distribution shift.The paper frames identifying correct weak labels as the central trust-estimation challenge.
  • Trust Estimation: Learned verifiers and reward models score textual input-output reliability, but output-level confidence can be miscalibrated on complex tasks.The paper instead uses internal teacher representations, motivated by evidence that correctness can remain encoded when final outputs are wrong.
  • Training Frameworks for Weak-to-Strong Generalization: Other training frameworks modify supervision or optimization through preference optimization, ensembles, and iterative weak-to-strong chains rather than focusing solely on selection.These approaches represent alternatives to the paper’s data-selection axis.

9. Conclusion

The paper frames weak-to-strong generalization as trust-based data selection using neural predictors of weak-label correctness. Across domains, filtering reaches near-lossless performance, supports compounding chains, and reflects mechanisms beyond error reduction.

  • 9. Conclusion: Neural trust functions predict weak-label correctness from teacher internal representations and filter weak supervision as a data-selection problem.The functions are trained to generalize under in-domain distribution shift.
  • 9. Conclusion: Trust-filtered weak supervision often matches ground-truth training and sometimes exceeds it across multiple domains.The paper characterizes this outcome as near-lossless weak-to-strong generalization.
  • 9. Conclusion: Trust filtering enables a weak-to-strong chain in which each trained student becomes the next teacher, producing compounding improvements across iterations.This extends the benefit beyond a single teacher-student transfer.
  • 9. Conclusion: NTFs help through easier-instance selection, strong alternatives when ground truth is incomplete, and more coherent gradient signals.These mechanisms extend beyond simply reducing label errors.
  • 9. Conclusion: Future work should extend neural trust functions to reasoning steps and evaluate them with synthetic data, noisy human labels, retrieval-augmented teachers, and multi-teacher pipelines.These settings define the paper’s stated expansion targets.

Impact Statement

Trust filtering may improve training data efficiency and broaden access to capable models, but automated selection may amplify biases in weak teachers’ representations.

  • Trust filtering may reduce the barrier to training capable models where ground-truth data is scarce or costly to collect.The paper frames this as potentially supporting access to high-quality models in specialized domains such as science and medicine.
  • Automated data selection may amplify biases present in the weak teacher’s internal representations.The paper identifies disproportionate selection or suppression of demographic or ideological viewpoints as a future research question.

Limitations

The current approach depends on labeled source data, uses only outcome supervision, and tests relatively simple trust-function architectures. Ablations also show that performance depends on representation choices, training data, and student budget.

  • Neural trust functions require a labeled source dataset, limiting applicability where absolutely no ground truth is available.The source dataset trains the mapping from representations to correctness.
  • The framework focuses on final-answer correctness and leaves dense, step-wise supervision for complex reasoning tasks unexplored.The paper specifically mentions process rewards as a potential direction.
  • The architectural scope is limited to simple MLPs over single-token hidden states, potentially missing temporal reasoning signals.The paper identifies attention-based neural trust functions as a richer alternative for future investigation.
  • Mean pooling substantially degrades chess trust prediction relative to using the final output-token representation.The ablation therefore uses no pooling by default.
  • Performance improves as neural trust-function training data grows, but calibration is best at 10,000 examples and slightly worsens at 50,000.Even 1,000 examples yield a usable trust function, while 50,000 provide the strongest overall performance.
  • At the smallest student budget, both NTF and ground-truth supervision yield near-zero performance, while larger budgets produce mixed comparisons.NTF outperforms the ground-truth baseline for both student sizes at n = 1,000 and n = 5,000.

A.4. Dimensionality Ablations

Teacher representations are highly compressible, and neural trust functions retain or improve trust prediction under substantial dimensionality reduction. However, transfer quality depends on the distribution shift and task domain.

  • A.4. Dimensionality Ablations: EVR reaches 0.884 at d=64 and 0.942 at d=128, showing that teacher representations are highly compressible in variance terms.
  • A.4. Dimensionality Ablations: AUC improves from 0.87 at d=16 to 0.93 at d≥256, while Brier decreases from 0.14 to 0.10 and ECE remains 0.02–0.03.Performance at d=256–512 matches or slightly exceeds the full-dimensional baseline at d=1024.
  • A.4. Dimensionality Ablations: The results suggest correctness-relevant information is concentrated in a moderate-dimensional subspace of teacher activations.The paper nevertheless retains the full-dimensional baseline for its main experiments.
  • Transfer: Neural trust functions achieve AUC = 0.8171 in-domain, while zero-shot in-domain distribution-shift transfer reaches AUC = 0.9214.The corresponding Brier scores are 0.1672 and 0.0714, respectively.
  • Transfer: Out-of-domain transfer from strategy games to world knowledge drops to AUC = 0.5560 and Brier = 0.2351.The paper attributes this pattern to dependence on domain-specific representational structure.
  • Quantitative Reasoning: NTF improves quantitative-reasoning performance to 51.08 and achieves 92.2% recovery on LLaMA3.1-8B.These results nearly match ground-truth performance and indicate the utility cannot be explained solely by spurious rewards.
  • Strategy Games: In strategy games, lightly SFT’ed teachers are used because base checkpoints often produce invalid move strings and near-zero generative accuracy.The disjoint 10K-example teacher adaptation is used only for label and embedding generation.

I. Risk-Controlled Data Selection

Risk-controlled calibration selects a trust-score threshold or top-k prefix from a small labeled target calibration set, using an upper confidence bound on selected-label noise. The procedure targets a user-specified noise rate, but the main-text per-threshold guarantee is weaker than a uniform guarantee across candidates.

  • Threshold calibration: Calibration chooses the most inclusive threshold whose Hoeffding upper confidence bound on selected-label noise satisfies U(θ) ≤ α.The selection rule uses empirical calibration noise and a confidence bound rather than empirical noise alone.
  • Multiple-testing correction: Bonferroni correction can make the threshold-grid procedure infeasible when many candidate thresholds are tested.With ncal = 3000, α = 0.1, δ = 0.1, and m = 3000, the best achievable UCB is ≈0.151 > α.
  • Guarantees and limitations: The reported practical rule uses δθ = δ without multiple-testing correction, providing only a per-threshold guarantee.Uniform guarantees can instead be recovered by coarsening the candidate grid or splitting calibration data for threshold selection and evaluation.
  • Top-k calibration: The top-k variant sorts calibration examples by trust score and selects the largest prefix whose upper confidence bound remains below α.The selected prefix induces a threshold equal to the score of its kth example.
  • Empirical result: θ⋆ = 0.895 retains 16.1% of the chess deployment pool while held-out deployment noise remains below α = 0.1.The same operating point is obtained by the top-k formulation, where k⋆/ncal = 0.158 projects to 16.1% of deployment data.
Loading 2606.01000v1…