PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Post-Training of Foundation Models: A Survey

Yijie Xu, Qianyi Cai, Huizai Yao, Yili Wang, Tianfu Wang, Cehao Yang, Xingbo Yao, Zhiyu Guo, Aiwei Liu, Xuming Hu, Weiyu Guo, Hui Xiong

arXiv:2608.24982v1cs.CLcs.AIcs.CVcs.LGcs.MM

TL;DR

Foundation-model post-training has increasingly used model-generated signals, but the field lacks a unified scope and taxonomy for updates made without external supervision. This survey defines and inventories Unsupervised Post-Training, organizing 80 strict methods by four internal update objects and examining how signal choice and task structure relate to improvement and recursive error. It concludes that deployment timing, update persistence, and safeguards must be evaluated alongside the update object.

  • Problem

    Existing surveys cover neighboring forms of self-improvement, test-time adaptation, and reinforced reasoning without jointly requiring a real update, no external supervision, and an internal-signal classification axis.

  • Method

    The survey establishes an operational boundary protocol, catalogs 80 strict UPT methods, organizes them into four update-object families, and adds an Input Visibility × Update Persistence view.

  • Results

    The four-family synthesis links internal update objects to corpus fit, answer equivalence, target selection, and evaluator quality, while showing that proxy errors can be recursively reinforced.

  • Takeaways & Limitations

    UPT method selection and evaluation should align the update object with task structure and deployment timing, placing safeguards where the proxy enters the update.

  • Takeaways & Limitations

    The inventory is frozen in May 2026, and the survey does not evaluate safeguard effectiveness because that requires controlled studies.

Abstract

from arXiv · show

Foundation-model post-training usually relies on human labels, preference data, stronger teachers, or executable verifiers. We study Unsupervised Post-Training (UPT): update-bearing adaptation on unlabeled inputs whose learning signal is derived from same-lineage model artifacts rather than an external oracle. We catalog 80 strict UPT methods and organize them by the object that supplies the update signal: a prediction statistic, a sample relation, a self-generated target, or an internal evaluator. Beyond inventory, we show how the choice of internal signal and task structure determines whether post-training improves the model or recursively amplifies error. An orthogonal Input Visibility $\times$ Update Persistence view maps deployment regimes and defines a unified framework for UPT selection and evaluation.

1 Introduction

The survey defines Unsupervised Post-Training as model-lineage adaptation on unlabeled inputs without external supervision, and organizes strict methods by their internal update object. It catalogs the resulting design space while relating internal signals and task structure to error propagation and deployment timing.

  • Taxonomy: The taxonomy groups methods by four update objects: prediction statistics, sample relations, self-generated targets, and internal evaluators.This axis is orthogonal to optimizer, task, modality, and training schedule.
  • Synthesis: The survey connects internal update objects to a recurring error chain in which imperfect proxies select outputs, updates concentrate on them, and later proxies become more biased.Its contributions include an operational boundary protocol, the four-family taxonomy, an Input Visibility × Update Persistence view, and cross-family synthesis.
  • Scope and definition: UPT updates parameters, adapters, memories, or persistent local state using unlabeled inputs and same-lineage signals rather than external supervision.External supervision includes ground-truth answers, verifier or tool verdicts, human labels, and stronger-teacher labels.
  • Scope and definition: Strict UPT requires an explicit update, unlabeled inputs, no external supervision, and same-lineage judges, scorers, or reward models.The survey separates these provenance checks from task-structure features such as answer canonicalization, finite answer alphabets, code signatures, and open-ended outputs.
  • Survey organization: A parallel adjacent track retains verifier-, tool-, seed-, teacher-, and external-evaluator methods while keeping the strict internal-signal core distinct.The inventory freezes 94 method records: 80 strict rows, 8 adjacent rows, and 6 prose-only boundary or antecedent records.

3 Prediction-Statistic Optimization

Prediction-Statistic Optimization directly optimizes a scalar read from the model, spanning likelihood, entropy, confidence, geometric, rule-based, and sample-local objectives. These methods provide a historical baseline, with smaller local updates offering shorter persistence.

  • Core mechanism: Prediction-Statistic Optimization reads a scalar from the model at one observation and optimizes it directly.It creates no pseudo-labels, preference pairs, or internal judge, and includes 26 strict methods.
  • Likelihood objectives: Continued pretraining minimizes language-model loss on unlabeled text, while related recipes target domain shift, failure modes, or longer context windows.The NLL objective also extends into test-time adaptation.
  • Entropy and confidence: Entropy minimization and self-confidence objectives use predictive statistics as training losses, single-prompt gradients, or intrinsic policy-optimization rewards.Initialization and update duration shape these objectives, motivating stratified evaluation.
  • Richer statistics: Recent methods broaden the family to gradient norms, hidden-state geometry, visual self-supervised puzzles, and reverse-task likelihood.SUDER bridges Prediction-Statistic Optimization and Internal Evaluator Bootstrapping.
  • Local state updates: Sample-local variants optimize per-prompt vectors, test-time LoRA, steering vectors, or layer-wise adaptations with smaller update footprints and shorter persistence.The survey traces this behavior on its Input Visibility × Update Persistence timing axis.

4 Sample-Relation Supervision

Sample-Relation Supervision derives updates from relations among multiple model-generated samples, including votes, consistency, clustering, and agreement. These relations support policy optimization across text and multimodal settings, but their reliability depends on avoiding spurious majorities and preserving meaningful diversity.

  • 4.1 Family Definition: Family II reads relations across rollouts, paraphrases, candidate answers, or agents rather than a scalar from one observation.Its update objects include cluster mass, consistency, votes, and contrastive agreement.
  • 4.1 Family Definition: 13/22 methods reduce multi-sample relations to binary majority votes, while other methods use semantic clusters, self-certainty, pairwise agreement, or softened frequencies.
  • 4.2 Consensus and Test-Time RL: TTRL draws N rollouts, treats the majority answer as a pseudo-label, and performs a GRPO update against that relation.Related variants shape the reward with entropy, paraphrase consistency, self-play, soft rollout statistics, or population consensus.
  • 4.2 Consensus and Test-Time RL: Relation-based methods address spurious-majority and reward-hacking risks through anchor-explorer voting, softened frequency signals, semantic novelty, or contrastive agreement between networks.
  • 4.3 Cross-Modal Extension: TTRV extends the recipe to vision-language models using frequency-plus-entropy rewards derived from visual rollouts without image-text labels, captioners, or external verifiers.On ImageNet, InternVL3-8B reaches 99.31% versus GPT-4o’s 98.30% and exceeds GPT-4o by 2.3 points across eight benchmarks.

5 Self-Generated Target Bootstrapping

Self-Generated Target Bootstrapping constructs trainable instructions, rationales, curricula, preference pairs, or other targets from model outputs, then applies SFT- or DPO-style updates. The family spans document-based self-teaching, reasoning curricula, test-time adaptation, and multimodal proposer–solver loops.

  • 5.1 Family Definition: Family III builds a trainable target from the model distribution and applies an SFT or DPO update against that target.Targets include instructions, rationales, plans, debate traces, curricula, and preference pairs.
  • 5.1 Family Definition: Table 3 contains 8 rationale methods, 5 instruction methods, 5 preference methods, and 4 curriculum methods.
  • 5.2 Self-Trained Instructions: SELF-TUNING turns raw documents into memorization, comprehension, and reflection stages, with document-derived self-teaching included in the strict UPT core.
  • 5.2 Self-Trained Rationales and Curricula: SELF-IMPROVE generates multiple chain-of-thought traces, filters them by self-consistency, and fine-tunes on the survivors.Variants use latent thoughts, confidence, foresight, debate, long context, or internally synthesized Challenger–Solver curricula.
  • 5.3 Preferences and Multimodal Extensions: G-ZERO, QUEST, and V-ZERO extend self-bootstrapping to open-ended generation, test-time auxiliary problems, and unlabeled-image vision-language loops.

6 Internal Evaluator Bootstrapping

Internal Evaluator Bootstrapping uses a same-lineage scorer, reward model, or meta-judge as the update object. Its methods either consume judge-produced preference pairs through DPO or scalar scores through policy-gradient updates, with later work stabilizing the self-rewarding loop.

  • 6.1 Family Definition: Family IV bootstraps a same-lineage evaluator and trains model outputs through its verdict, unlike Family III’s trainable-target pathway.
  • 6.1 Family Definition: Table 4 divides 10 methods into 4 DPO variants using judge-produced preference pairs and 6 policy-gradient variants using judge scores as rewards.
  • 6.2 Self-Rewarding Language Models: SELF-REWARDING LM alternates one language model between actor and judge roles, using chosen–rejected pairs for iterative DPO.CREAM, META-REWARDING, and TEMPORAL SRLM stabilize the loop through consistency, metajudgment, or cross-generation preference anchoring.
  • 6.3 Evaluator-Driven Policy Optimization: A second sub-line feeds evaluator judgments directly into policy gradients, using critique helpfulness, natural-language meta-judgments, or a metacognitive introspector as scalar rewards.
  • 6.4 Broader Self-Evolution Loops: AERO and SELF-JUDGE broaden evaluator-mediated self-evolution to counterfactual criticism, KTO-style updates, and frozen same-lineage judges.SUDER and GVU are assigned to Families I or IV according to the primary signal consumed by each update.

7 Cross-Family Synthesis

Across UPT families, the update object determines how internal uncertainty becomes a training decision and where error feedback can accumulate. The survey pairs family-specific leverage and safeguards with mechanism-matched results and deployment choices based on available signals.

  • 7.1 Sources of Leverage: Family I exploits predictive statistics, Family II aggregates sample evidence, Family III builds inspectable targets, and Family IV supplies semantic criteria for open-ended outputs.Greater semantic flexibility lengthens the feedback path, increasing the value of consistency, diversity, and independent evaluation.
  • 7.2 From Signal to Update: Family I’s short path depends on alignment between its optimized statistic and downstream behavior, not proxy sharpness alone.LANGADAPT CPT uses token likelihood, while entropy- and confidence-based variants reshape the predictive distribution.
  • 7.2 From Signal to Update: Majority vote and confidence can occur in different families because classification follows the object consumed by the gradient.TTRL and ROIRL consume agreement as reward, whereas LRM SELF-TRAIN, CONFIDENT ST, and RLSF use selected targets for SFT or DPO.
  • 7.3 Reported Evidence: LANGADAPT CPT reduces Basque perplexity from 23.64 to 3.35 and raises aggregate downstream accuracy from 27.43 to 34.14.The paired metrics separate corpus adaptation from task transfer.
  • 7.3 Reported Evidence: TTRL raises AIME 2024 from 12.9 to 40.2 and MATH-500 from 46.7 to 83.4, while QUIET-STAR raises zero-shot GSM8K from 5.9 to 10.9.CONL raises AIME 2024 from 60.0 to 76.5 and DeepMath from 70.5 to 87.1.
  • 7.4 Choosing and Safeguarding UPT: Signal choice maps to deployment needs: raw documents favor Family I, affordable samples with defensible equivalence favor Family II, inspectable generated targets favor Family III, and open-ended outputs favor Family IV.Safeguards track calibration, diversity, target refresh, actor–judge correlation, preference margins, and cross-round consistency.

8 Timing of Adaptation

The timing view classifies UPT by when target inputs become visible and how long induced changes persist, independently of update-object family. It distinguishes pre-sample and within-sample regimes while highlighting recursive error propagation and evaluation safeguards.

  • Input Visibility and Update Persistence are orthogonal to family membership: timing determines deployment cost, while family determines supervision shape.
  • Pre-sample regimes: Offline corpus, full-cohort transductive, few-sample target, and streaming continual adaptation form four pre-sample regimes with increasing restrictions on visible target data.Offline corpus UPT sees none of the target distribution; streaming adaptation uses only the preceding prefix at each sample.
  • Within-sample regimes: Test-time instance adaptation updates for one instance and resets at its boundary, whereas within-sequence adaptation updates across chunks or token-states before resetting.Within-sequence adaptation changes persistent local state during a single sequence.
  • Challenges and safeguards: Recursive error propagation occurs when updates reinforce misordered outputs and shift the distribution used to compute the next proxy.The four UPT families expose different links in this error chain.
  • Challenges and safeguards: Evaluations should test consensus under alternative canonicalizers and open-ended reformulations because task structure affects whether agreement is informative or brittle.This concern is especially relevant to Families II–III, where equality-based consensus can amplify correlated errors.
  • Challenges and safeguards: Evaluations should diagnose diversity and wrong-majority behavior, and track target diversity, actor–judge correlation, held-out quality, and perturbation recovery across rounds.Independent execution channels and diversity-oriented interventions are proposed to interrupt majority amplification and evaluator drift.

10 Conclusion

The survey catalogs 80 strict UPT methods through four update-object families and an orthogonal view of input visibility and update persistence. It concludes that method selection and evaluation should align update objects with task structure and deployment timing while safeguarding against proxy-error reinforcement.

  • 80 strict UPT methods are organized by four update-object families, with Input Visibility and Update Persistence providing an orthogonal deployment map.
  • UPT’s central trade-off is that semantically expressive signals support open-ended outputs but lengthen feedback paths through which proxy errors can be reinforced.
  • Method selection and evaluation should align the update object with task structure and deployment timing.
  • Safeguards should be placed where the proxy enters the update.
  • Because no external oracle scores the updated model, independent held-out evaluation, monitoring, and red-teaming remain necessary.The survey organizes these safeguards by feedback path but does not evaluate their effectiveness.

Appendix Contents

The appendices document the survey’s search, screening, inventory, evidence, and comparison procedures. Together, they make the 80 strict rows and their scope boundaries reconstructable.

  • The appendices cover survey protocol, representative evidence and task-structure audits, the full method inventory, and comparison with existing surveys.
  • Searches covered ACL Anthology, arXiv, Semantic Scholar, and Google Scholar for work dated January 2023–May 2026.
  • The mechanism list included unsupervised post-training, self-improvement, self-rewarding, self-training, test-time adaptation, internal reward, and evaluator-driven RL.
  • Strict UPT candidates had to satisfy all four boundary checks, provide a verifiable algorithmic description, and operate on foundation-scale text or multimodal models.
  • Candidate records captured update targets, signal sources, external signals, the internal gradient-consumed object, and timing regimes.
  • The frozen inventory contains 94 method records from 91 papers: 80 strict rows from 78 papers, 8 adjacent rows, and 6 prose-only records.
  • Table 5 preserves source-paper setups, while Table 6 separates structural priors from correctness-bearing supervision.

B Representative Evidence and Task-Structure Audit

The representative-evidence audit preserves source setups and connects the taxonomy to timing, update targets, and task structure. It also records boundary cases that are adjacent rather than strict UPT.

  • Tables 5 and 6 provide empirical and task-structure detail while preserving each source paper’s reported setup.
  • Figure 3 maps all 80 strict methods by family, subclass, and update object, while Table 7 cross-sections timing regimes and update targets.
  • Offline updates dominate all four families at 59/80.
  • Full-cohort transduction is concentrated in Family II at 7/11, while all five sample-local-state methods occur in Families I and III.
  • No-update inference-time optimization, verifier-assisted self-training, human- or seed-supervised bootstrapping, and external-evaluator methods are routed to adjacent categories.

D.2 Family II vs. Family III

The survey distinguishes Family II from Family III by the object consumed by the gradient, especially when consensus or confidence scores select generated examples. This rule assigns hybrid methods according to their actual update path.

  • LRM SELF-TRAIN is assigned to Family III because its gradient trains against majority-filtered self-generated solutions, not the consensus statistic.
  • TTRL, ROIRL, and methods whose gradient directly uses r = 1[y = maj] are assigned to Family II.
  • Confidence scores select kept generations for Family III when used before SFT or DPO, but define Family IV when used as scalar rewards in policy gradients.
  • ECHO and SPINE remain in Family II because multi-sample consensus supplies the reward, while intrinsic terms only modulate the within-family update.
  • The survey’s distinctive unit is the internal object consumed by an explicit update under the no-external-signal boundary.
Loading 2608.24982v1…