PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining

Shuqi Ke, Giulia Fanti

arXiv:2601.22108v2cs.LGcs.AI

TL;DR

Continued pretraining uses fixed self-supervised tasks even though success is judged by downstream capabilities, creating a coarse feedback loop. V-pretraining trains a lightweight designer to choose targets or views using a one-step downstream value estimate while leaving learner updates self-supervised. Across language and vision, it improves target capabilities without degrading generalization under wall-clock-matched training.

  • Problem

    Continued pretraining selects fixed self-supervised task recipes using downstream evaluation only after full runs, motivating step-level feedback from small verifiable downstream sets without directly supervising the learner.

  • Method

    V-pretraining trains a task designer to construct targets or views for unlabeled batches by maximizing predicted one-step downstream-loss reduction, then applies detached constructions in the learner’s self-supervised update.

  • Results

    Across language and vision, V-pretraining improves target capabilities without harming generalization, including a +7.4 point GSM8K Pass@1 gain for Qwen2.5-0.5B and improved ADE20K and NYUv2 transfer.

  • Takeaways & Limitations

    Downstream feedback can guide continued pretraining through task construction while preserving self-supervised learner updates and general-purpose performance.

  • Takeaways & Limitations

    V-pretraining provides a noisy local value signal rather than a long-horizon optimality guarantee, depends on feedback quality, and adds computation despite wall-clock-matched comparisons.

Abstract

from arXiv · show

Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities. We ask whether a small set of verifiable downstream examples can provide step-level feedback without directly supervising the learner. We introduce V-pretraining, which decouples a learner trained only with a self-supervised loss from a lightweight task designer that constructs targets or views for unlabeled batches. Given the current learner and batch, V-pretraining scores a candidate construction by predicting the first-order reduction in downstream loss after the induced self-supervised update. The designer maximizes this value; the learner then applies the update with targets or views detached, so downstream labels never update learner parameters. We instantiate V-pretraining as adaptive top-K soft targets for language modeling and learned views or masks for self-supervised vision. Across both modalities, V-pretraining improves target capabilities without degrading generalization. Under wall-clock-matched continued pretraining, it improves GSM8K Pass@1 for Qwen models using 1,024 GSM8K examples only as feedback, including a +7.4 point single-run gain for Qwen2.5-0.5B. In vision, it improves DINOv3 transfer to ADE20K semantic segmentation and NYUv2 depth estimation while preserving ImageNet linear accuracy, suggesting that feedback-guided task construction can improve target capabilities without collapsing general-purpose representations.

1 Introduction

Continued pretraining uses scalable fixed self-supervised proxy tasks, but downstream evaluation occurs only after full runs, leaving individual updates disconnected from target capabilities. V-pretraining addresses this mismatch by learning task constructions from downstream feedback while keeping learner updates self-supervised.

  • Motivation: Continued pretraining adapts foundation models with unlabeled data using fixed proxy tasks such as next-token prediction or vision self-supervised objectives.These proxy tasks avoid requiring dense human annotation, but runs are judged by downstream performance rather than proxy loss.
  • Motivation: Downstream feedback enters current practice through checkpoint evaluation and repeated changes to data, objectives, curricula, or augmentations across full runs.This makes task-recipe design an expensive black-box search over pretraining trajectories.
  • Approach: V-pretraining learns a task designer that dynamically constructs self-supervised targets or views while the learner remains trained only by the pretraining loss.The designer receives downstream evaluator feedback, but that gradient does not directly update learner parameters.
  • Approach: The designer maximizes a one-step value estimate predicting downstream-loss reduction from an induced self-supervised update, avoiding differentiation through full training trajectories.The resulting targets or views are detached before the learner update.
  • Results: +7.4 points on GSM8K Pass@1 was achieved in wall-clock-matched continued pretraining, while vision experiments improved target transfer without harming generalization.The reported vision results include ADE20K mIoU gains and NYUv2 RMSE reductions, with ImageNet linear accuracy preserved or slightly improved.

2 V-pretraining Framework

V-pretraining keeps the learner’s self-supervised training unchanged while using downstream feedback to adapt how unlabeled batches become prediction tasks. A one-step value objective guides a task designer to construct updates aligned with downstream improvement, without directly applying downstream gradients to the learner.

  • Task construction: A task construction rule converts an unlabeled example into a modified input, target, and optional side information for self-supervised prediction.Standard continued pretraining fixes this rule before training, even when sampling remains random.
  • Task construction: V-pretraining replaces the fixed construction rule with designer-controlled cϕ while keeping the learner architecture, optimizer, loss family, and unlabeled data stream fixed.The designer changes targets, views, or masks rather than the learner objective.
  • Indirect feedback: Downstream feedback trains the task designer, while every learner update remains a self-supervised update on an unlabeled batch with no direct downstream gradient.Constructed targets or views are detached before the learner update.
  • One-step value objective: The designer uses a one-step value estimate to select the unlabeled task construction whose induced learner gradient most decreases feedback loss.The candidate update is evaluated through its alignment with the downstream gradient, and the designer maximizes this value efficiently on a parameter subset.
  • One-step value objective: Under smoothness, maximizing the value maximizes a first-order lower bound on one-step downstream loss decrease, but does not guarantee monotonic improvement over a long trajectory.Stochastic gradients, optimizer state, distribution shift, and cross-step interactions can affect final performance.
  • Instantiations: Language instantiates the framework with adaptive top-K soft targets, while vision uses learned instance-wise views or masks with the baseline self-supervised objective retained.The true next token remains in the candidate set, and vision feedback can use segmentation or depth heads while the backbone remains trained by detached SSL loss.

3 Experiments

The experiments test whether downstream feedback can guide continued pretraining through task construction under matched compute, while preserving transfer and avoiding shortcut explanations. Across language and vision, V-pretraining improves target capabilities, with measured trade-offs and overhead.

  • Experimental setup: Experiments use wall-clock-matched comparisons with fixed hardware, learner initialization, unlabeled streams, optimization, precision, and data loading.Additional per-step overhead must be paid within the same elapsed training time.
  • Experimental setup: Language experiments continue pretraining Qwen models on NuminaMath-CoT, using 1,024 GSM8K examples only to compute designer feedback.The learner uses designer-shaped top-K targets instead of one-hot next-token targets.
  • Experimental setup: Vision experiments continue DINOv3 ViT-B and ViT-L training on ImageNet-1K, using ADE20K and NYUv2 feedback to learn instance-wise views.Evaluation includes dense prediction, ImageNet linear accuracy, and transfer diagnostics.
  • Target capabilities: Qwen2.5-0.5B improves GSM8K Pass@1 from 22.20 to 29.60, a 33% gain, under matched wall-clock training.V-pretraining also improves the tested Qwen1.5 model sizes, with gains decreasing as model size increases.
  • Target capabilities: V-pretraining improves ADE20K mIoU and NYUv2 depth RMSE while slightly improving ImageNet linear accuracy for DINOv3 ViT-B and ViT-L.The result supports improving dense prediction through task construction while retaining global recognition.
  • Generalization and transfer: Generalization effects are mixed: larger language models avoid major MMLU degradation, the 0.5B model drops on MMLU, and most vision retrieval protocols improve while Oxford Hard declines slightly.These findings indicate steering is not uniformly beneficial across unrelated tasks.
  • Shortcut checks: After decontamination, V-pretraining remains above the Qwen1.5-4B baseline, suggesting its advantage is not primarily caused by benchmark-like duplicates in the unlabeled stream.The post-decontamination margin is smaller than in the original run.
  • Shortcut checks: Random feedback, uniform smoothing, and self-distillation do not match downstream-aligned value feedback under the same wall-clock budget.Random feedback and uniform smoothing fall below the one-hot baseline, while self-distillation remains below V-pretraining.

4 Contributions and Limitations

The paper frames downstream-guided continued pretraining as task design: a lightweight controller constructs targets or views while the learner remains trained only with self-supervised loss. It derives a scalable value objective and reports compute-matched evidence, while acknowledging local-signal and overhead limitations.

  • Contributions: V-pretraining trains a task designer to construct pretraining targets or views while the learner remains trained only with a self-supervised loss.
  • Contributions: The step-level value objective avoids differentiating through full pretraining trajectories and estimates one-step downstream improvement.
  • Contributions: V-pretraining applies the same principle across modalities through adaptive top-K language targets and learned vision views.
  • Limitations: The method adds computation for task generation and value-gradient estimation, despite wall-clock-matched main comparisons.
  • Limitations: V-pretraining provides a local value signal rather than a long-horizon optimality guarantee.

B.1 Derivation

V-pretraining scores candidate self-supervised updates by aligning their learner gradients with a detached downstream evaluator gradient. The resulting objective has a first-order one-step justification, but its empirical and theoretical scope remains local.

  • Value objective: The designer induces a pretraining gradient, while the feedback batch defines an evaluator gradient used to estimate downstream loss reduction.
  • Value objective: Maximizing g_down^Tg_pre estimates the predicted one-step decrease in downstream loss from a candidate self-supervised update.
  • Optimization: The designer update treats g_down as a detached evaluator vector and uses a mixed Hessian-vector product to change the learner gradient.
  • Optimization: Computing alignment on a parameter subset S reduces cost while retaining the same alignment principle as the full-parameter objective.
  • Guarantee: For sufficiently small steps or controlled update norms, increasing the alignment raises a certified lower bound on one-step downstream loss decrease.
  • Guarantee: A Taylor expansion gives J(ϕ; θ) = L_down(θ)−ηV(ϕ; θ)+O(η^2), making value maximization a first-order one-step surrogate.
  • Limitations: The local result does not guarantee monotonic downstream improvement over long trajectories because stochasticity, optimizer state, distribution shift, and step interactions matter.

C Additional Language Results

Additional language experiments indicate that V-pretraining’s gains are not primarily explained by near-duplicate contamination and extend across sampling levels, while generalization effects vary with model size and feedback coverage.

  • One-step validation: r = 0.657 Pearson correlation links predicted and realized downstream probe-loss improvements in the language one-step test.
  • Decontamination: V-pretraining remains above the Qwen1.5-4B baseline after GSM8K and MATH decontamination, although the margin becomes smaller.
  • Generalization: V-pretraining improves OMEGA for the 4B learner, leaves MMLU nearly unchanged for 4B and 7B, and lowers MMLU for 0.5B.
  • Feedback coverage: Increasing feedback coverage from 1k to 3k examples improves gain stability and strength, with diminishing returns after a few thousand examples.
  • Sampling robustness: V-pretraining improves Pass@k for k ∈ {1, 2, 4, 8, 16} across tested k values and model sizes.

C.6 Token-efficiency diagnostic

The token-efficiency analysis diagnoses downstream value per unlabeled token rather than replacing the primary wall-clock-matched comparison. It also situates V-pretraining as indirect feedback during pretraining, complementary to direct post-training methods.

  • Token-efficiency diagnostic: 56.18 GSM8K Pass@1 after 400 learner steps and approximately 1.3 × 10^7 unlabeled tokens contrasts with roughly 10^3 baseline steps for comparable accuracy.
  • Token-efficiency diagnostic: Figure 3(c) plots GSM8K Pass@1 against unlabeled tokens processed as a diagnostic of update quality.
  • Interpretation: The token-efficiency curves are diagnostic and do not replace wall-clock-matched results.
  • Positioning: V-pretraining injects goal information during pretraining by shaping the unlabeled training signal rather than directly updating the learner on downstream labels.
  • Positioning: The approach is presented as complementary to supervised fine-tuning and preference optimization, which steer models through direct learner updates.

E Multitask Feedback Data Construction

The multitask experiment combines raw domain text with high-quality instruction data across math, code, and general capabilities, while using balanced downstream feedback to steer one continued-pretraining run.

  • Goal: The experiment tests whether one indirect feedback channel can steer continued pretraining toward mathematical reasoning, code generation, and general instruction following or world knowledge.The learner remains trained on a mixed continued-pretraining stream.
  • Pretraining mixture: The final corpus combines raw and high-quality instruction data for math, code, and general domains.The raw sources are OpenWebMath, codeparrot-clean, and C4 English; the instruction sources are MetaMathQA, Magicoder OSS-Instruct-75K, and Alpaca.
  • Pretraining mixture: 0.7 of sampling probability is assigned to raw data and 0.3 to high-quality instruction data, with equal allocation across the three domains within each group.Each raw domain receives 0.7/3 and each high-quality domain receives 0.3/3.
  • Why mix raw and high-quality data?: Raw text supplies broad domain coverage, while high-quality examples preserve task-relevant formats without making direct instruction exposure the sole explanation for gains.Instruction-only mixtures can make the baseline learn task formats directly, while small or imbalanced mixtures can over-repeat domains.
  • Data processing: The learner uses standard next-token prediction for raw text, response-only causal-LM loss for formatted instruction examples, and capped source sampling before tokenization.Sequences use maximum length 512, and prompt tokens are masked for instruction examples.
  • Feedback and evaluation: Feedback batches are balanced across domains, define evaluator gradients for the task designer, and are not inserted as learner updates.Held-out GSM8K, MMBP/MBPP-style, and MMLU benchmarks report downstream performance under a fixed compute budget.

F Main Language Experiment Setup

The main language experiments test whether a small downstream verifier can steer continued language-model pretraining toward mathematical reasoning without directly training the learner on verifier examples.

  • Purpose: The experiments use a math-oriented continued-pretraining stream and a small GSM8K feedback set to train the task designer through gradient alignment.The reported downstream metric is GSM8K test Pass@1.

F.1 Single-task math continued pretraining

Single-task math runs compare standard continued next-token pretraining with V-pretraining, which shapes soft targets using GSM8K feedback while keeping learner updates self-supervised.

  • Learners: Qwen-family learners start from the same checkpoint and use matched optimizer, schedule, precision, sequence length, and wall-clock budget across methods.Reported runs include multiple Qwen1.5 scales and Qwen2.5-0.5B.
  • Pretraining stream: The unlabeled NuminaMath-CoT stream uses causal language modeling on solution spans, with prompt tokens masked from the loss.Baseline and V-pretraining use the same stream and loss mask.
  • Baseline: The baseline trains each eligible position with a one-hot target for the true next token.The learner minimizes cross-entropy over solution-token positions.
  • Feedback set: 1,024 GSM8K training examples define the downstream evaluator loss and gradient, but never enter the learner stream or provide supervised learner updates.The evaluator gradient is detached and used only as a task-designer vector.
  • V-pretraining objective: V-pretraining replaces the fixed one-hot target with a designer-shaped soft target over a small candidate set containing the true next token and high-probability alternatives.The designer outputs a candidate distribution and token-dependent mixing coefficient; αϕ,t = 0 recovers the baseline.
  • Optimization: The designer is trained to align induced pretraining gradients with downstream evaluator gradients, after which detached soft targets drive an ordinary learner update.GSM8K feedback therefore changes self-supervised targets on NuminaMath-CoT examples rather than applying a GSM8K supervised gradient.
  • Evaluation: GSM8K test evaluation uses a shared decoding and answer-extraction protocol, with Pass@1 as the main reported metric.Pass@k, feedback coverage, and token efficiency are separate diagnostics.

F.2 Language implementation branches

The paper reports separate LoRA and full-parameter language implementations and a vision implementation that learns task-conditioned views while preserving the underlying self-supervised learner and data stream.

  • Language branches: The two language implementations share the student–designer concept but differ in learner parameterization, data regime, and alignment parameters.They are reported separately to distinguish parameter-efficient adaptation from full-parameter continued pretraining.
  • LoRA branch: The LoRA branch adapts Qwen2.5-0.5B through attention and MLP projection adapters, making it a parameter-efficient setting.Its default configuration uses rank 8, αLoRA = 16, and dropout 0.05.
  • LoRA branch: The LoRA task designer scores the learner’s top-K next-token candidates, outputs a distribution and smoothing gate, and aligns gradients on the last two LoRA layers.The default branch uses K = 64 and αmax = 0.5.
  • Full-parameter branch: The full-parameter branch trains all Qwen2.5-0.5B learner parameters with ordinary causal language modeling, while retaining the same feedback-controlled task-design idea.Raw text uses standard next-token prediction and instruction examples mask prompt tokens.
  • Multitask alignment: Multitask alignment pairs same-domain pretraining and feedback batches to avoid directions inconsistent with the current pretraining domain.The method averages alignment across math, code, and general domains.
  • Vision setup: In vision, DINOv3 continues self-supervised training on unlabeled ImageNet-1K, while ADE20K segmentation and NYUv2 depth labels train only the task designer.The baseline and V-pretraining share the learner, stream, optimizer, schedules, precision, and budget.
  • Vision task construction: The vision baseline uses fixed DINO-style views, whereas V-pretraining replaces part of view construction with an instance-wise learned augmentor that aligns SSL gradients with dense-task feedback.The augmentor predicts soft spatial masks and ordinarily controls the two global crops while local crops remain standard.
  • Vision task construction: Large mask values preserve original pixels and small values replace them with a blurred background in the learned views.The resulting views are passed to the DINO SSL loss.

G.3 Dense evaluation protocol

The vision evaluation compares standard continued DINO pretraining with V-pretraining under matched protocols, using frozen-backbone linear probes for dense prediction and ImageNet linear accuracy to assess representation preservation.

  • ADE20K semantic segmentation: ADE20K performance is reported as mean intersection-over-union (mIoU) on SceneParse150 after remapping labels for evaluation.
  • Dense evaluation protocol: Main segmentation and depth comparisons use frozen-backbone linear probes with linear-BN heads, emphasizing representation quality over downstream fine-tuning sensitivity.Depth predictions use softplus to ensure positive values.
  • NYUv2 depth estimation: NYUv2 depth evaluation uses valid pixels, the standard Eigen crop, and RMSE as the main reported metric.AbsRel and δ1 are also computed.
  • Global recognition and transfer: ImageNet-1K linear accuracy and Revisited Oxford and Revisited Paris retrieval assess whether dense-task feedback harms global recognition or collapses representations toward dense tasks.These transfer tasks are not used as V-pretraining feedback.
  • Sweep protocol: Fixed-budget sweeps run 20k continued DINO pretraining steps and select configurations using validation mIoU for ADE20K or validation RMSE for NYUv2.
  • Interpretation of vision tuning: The comparisons hold the initial backbone, unlabeled ImageNet stream, SSL objective family, and evaluation protocol fixed while charging V-pretraining for learned views and periodic value-gradient updates.The experiment tests downstream-guided task construction rather than a new dense-prediction fine-tuning method.

I Compute Resources and Total Compute

The experiments used internal GPU clusters and reported approximate accelerator-hour compute, with main comparisons wall-clock matched and overhead documented separately from exact hardware benchmarks.

  • Compute resources: Reported training used NVIDIA H100 80GB and H200 NVL GPUs, with auxiliary evaluations also using RTX 6000 Ada or RTX A6000-class GPUs.Compute is counted as one accelerator-hour per GPU-hour.
  • Compute resources: Vision pretraining jobs typically used 12 dataloader workers and 12–16 CPU cores per GPU worker.
  • Total compute: Table 12 reports approximate accelerator-hour ranges because model size, device type, throughput, cluster load, and evaluation placement affect wall-clock time.The main comparisons are wall-clock matched within each experiment family, including V-pretraining overhead.
  • Storage: The project required approximately 0.5–1TB of shared storage across datasets, checkpoints, logs, sweeps, and intermediate evaluation outputs.
  • Measured overhead: In a representative language run, V-pretraining reduced token throughput by about 9% and increased peak memory by about 16%.The baseline processed 65,536 tokens per optimizer step versus 40,960 for V-pretraining, so shorter optimizer-step time does not imply lower cost.
Loading 2601.22108v2…