PaperlayerView exact revision on arXiv

Source-linked AI summary

Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models

Darrien McKenzie, Nicklas Hansen, Xiaolong Wang

arXiv:2606.19750v1cs.LGcs.AIcs.CL

TL;DR

Adaptive RL problem sampling must account for heterogeneous task structure and diversity, not only difficulty. The paper introduces Bayesian Manifold Curriculum, which uses a representation-derived task hierarchy for Bayesian sampling, and finds that sampling strategies trade off productivity, diversity, and evaluation utility.

  • Problem

    Existing curricula often treat prompts as independent arms and prioritize difficulty, overlooking latent task structure and diversity in heterogeneous problem spaces.

  • Method

    Bayesian Manifold Curriculum organizes prompts into a representation-derived hierarchy and applies Bayesian belief updates to guide structured sampling.

  • Results

    Dynamic sampling achieves the highest average learning signal but performs poorly on less-represented benchmarks, illustrating tradeoffs between productivity, diversity, and utility.

  • Takeaways & Limitations

    Curriculum learning should coordinate training effort across diverse, interacting problem types using the policy’s latent geometry rather than difficulty alone.

  • Takeaways & Limitations

    The observed downstream differences do not establish which training problem types causally support particular evaluations.

Abstract

from arXiv · show

Reinforcement learning (RL) is a central approach for improving reasoning capabilities in large language models (LLMs), where training efficiency depends critically on how problems are sampled during optimization. Existing adaptive curriculum learning methods typically prioritize prompts of intermediate difficulty, treating problem selection as a standard bandit problem with independent arms and overlooking the structured, heterogeneous nature of the task space. In this work, we frame problem sampling as a manifold-structured bandit problem with endogenous non-stationarity: problems are related through the model's latent representation space, and sampling decisions can steer how learning signals evolve across that space. To operationalize this perspective, we introduce Bayesian Manifold Curriculum (BMC), a structure-aware framework that organizes problems into a hierarchical task tree and applies Bayesian learning to guide sampling. Empirically, we find that different sampling strategies induce non-trivial tradeoffs between productivity (learning signal), diversity (coverage of the task manifold), and utility (evaluation relevance). These results show that prioritizing difficulty alone is insufficient for strong downstream performance, highlighting the importance of incorporating structure and type-awareness into problem sampling.

1 Introduction

The introduction frames LLM problem sampling as a latent manifold-structured, endogenous bandit problem rather than independent prompt selection. It proposes Bayesian Manifold Curriculum with hierarchical task structure to balance learning signal, diversity, and utility.

  • Motivation: RL improves LLM reasoning, with group-relative methods estimating policy improvements by comparing multiple rollouts from the same prompt.The introduction identifies GRPO, DAPO, and GSPO as examples of this algorithmic class.
  • Motivation: Dynamic Sampling prioritizes informative reward variation by regenerating prompts until batches contain only examples with non-zero reward variance.This increases learning signal but incurs substantial wall-clock overhead from repeated batch regeneration.
  • Problem formulation: Prompt rewards evolve through model updates and sampling decisions, creating an endogenous feedback loop that standard non-stationary bandits do not fully capture.The sampling choice actively shapes future reward distributions rather than merely responding to external changes.
  • Problem formulation: Optimizing only immediate learning signal can narrow problem-space coverage, so effective training must balance difficulty with diversity to support generalization.Training datasets are heterogeneous and can be conceptually imbalanced, including within a single domain.
  • Contributions: The paper frames problem sampling as a latent manifold-structured bandit and introduces Latent Task Trees, Bayesian Manifold Curriculum, and Productivity–Diversity–Utility Trade-offs.Latent Task Trees approximate prompt relationships from model embeddings, while BMC performs Bayesian decisions over this hierarchy under model-induced non-stationarity.

2 Preliminaries

RLVR trains LLM policies using verifier-generated rewards, while group-relative methods derive advantages from within-prompt response groups. Because zero reward variance yields no learning signal, prompt sampling is framed as a structured, intervention-like, and endogenously non-stationary bandit problem informed by latent task geometry.

  • Reinforcement Learning for LLMs: RLVR models LLM training as maximizing expected verifier reward over prompts, with policy πθ mapping each prompt x to a sampled response y.The verifier produces a scalar reward r(x, y), with optional regularization or constraints on policy updates.
  • Group-Relative Policy Optimization: Group-relative algorithms sample G responses per prompt and compute contrastive advantages using within-group statistics, avoiding a learned critic.The cited methods include GRPO, DAPO, and GSPO.
  • Group-Relative Policy Optimization: Prompts with zero within-group reward variance provide no learning signal, motivating Dynamic Sampling to regenerate prompts until sufficient variation is found.Dynamic Sampling increases wall-clock time because prompts may need repeated regeneration.
  • Structured, Causal, and Non-Stationary Bandits: Structured bandits share information across related arms, while causal bandits model actions as interventions; prompt sampling has both properties because policy updates alter future rewards.Classical bandits instead estimate each arm’s reward distribution independently.
  • Structured, Causal, and Non-Stationary Bandits: The framework combines structured information sharing, intervention-like sampling, and reward evolution, with non-stationarity partly caused by the learner’s own decisions and connected through latent task structure.LLM representations are described as high-dimensional spaces often approximated by lower-dimensional geometric structure, consistent with the manifold hypothesis.

3 Latent Task Trees

The Latent Task Tree represents prompts hierarchically in the policy model’s latent task manifold rather than as independent arms. It is built through recursive local geometry analysis, with subdivision stopping when regions become sufficiently chart-like.

  • Motivation: Prompts are organized into hierarchical regions using the policy model’s own representations, reflecting progressively finer task subtypes rather than independent bandit arms.The hierarchy captures how the policy perceives relationships across mathematics, coding, science, and other broad task categories.
  • Geometric construction: The Latent Task Tree uses recursive local manifold approximation because representation-space regions can differ in density, intrinsic dimension, and organization.This avoids assuming a single global partition or parametric model for the full task space.
  • Recursive pipeline: Each node standardizes embeddings, retains the smallest PCA dimension explaining 95% of variance, projects them with UMAP, and clusters them with HDBSCAN.HDBSCAN supports varying-density and non-convex structure without requiring a preset number of clusters; detected clusters become child nodes.
  • Stopping criterion: Recursion continues only while embeddings exhibit meaningful internal structure, using a Chart Test to identify regions that are low-dimensional and locally connected.The test uses Two-NN for intrinsic dimensionality and a k-nearest-neighbor graph for connectivity on PCA-reduced embeddings before UMAP.
  • General applicability: The pipeline requires only model embeddings, avoiding dataset-specific tuning, external embedding models, and annotations while supporting diverse models, domains, languages, and modalities.After embedding extraction, tree construction relies on standard unsupervised operations with negligible cost relative to autoregressive RL training.

4 Bayesian Manifold Curriculum

Bayesian Manifold Curriculum (BMC) organizes prompts in a hierarchical Latent Task Tree and uses Bayesian sampling to allocate training effort across related task regions. It combines reward-variability learning signals with online belief updates and hierarchical information sharing to balance productivity, exploration, and coverage.

  • Framework: BMC represents the latent task manifold as a hierarchical Latent Task Tree, enabling effort allocation across task regions and information sharing among related prompts.The tree provides multiple levels of abstraction for structured problem selection.
  • Framework: The learning signal is reward variability across k rollouts, which for binary rewards prioritizes prompts near a 50% success rate.This targets prompts that are neither already solved nor completely out of reach.
  • Belief updates: BMC updates prompt-level beliefs from rollout rewards and propagates them upward through empirical Bayes updates, allowing individual observations to inform higher-level task regions.The recurring procedure consists of top-down selection, prompt-level updating, and bottom-up propagation.
  • Sampling: Each batch agent independently traverses the tree using hierarchical Thompson sampling, selecting child nodes recursively until reaching an individual prompt.Independent descents let batch elements explore distinct task-space regions while balancing exploration and exploitation.
  • Belief updates: The curriculum adapts to policy drift by increasing uncertainty for surprising observations or stale prompts, allowing previously explored problems to be revisited without premature convergence.Prompt beliefs become more confident when evidence is recent and expected, and more uncertain when evidence is surprising or outdated.
  • General applicability: BMC supports group-relative RL algorithms with multiple rollouts per prompt and extends beyond binary verifiers to continuous rewards through appropriate normalized signals.Examples include GRPO, DAPO, and GSPO; applicability also inherits the domain, language, and modality generality of Latent Task Trees.

5 Experiments

The experiments show that sampling strategies trade off productivity, diversity, and evaluation utility: learning speed and signal do not determine which task-manifold regions are covered or which capabilities transfer. BMC uses latent structure to improve coverage while retaining productivity, and target-aware variants demonstrate that utility depends on problem-type exposure.

  • Productivity and learning dynamics: Curriculum-based strategies improve effective ratio by roughly 40% over uniform sampling, while Dynamic Sampling reaches 1.0 through repeated resampling.Dynamic Sampling’s resampling incurs substantially higher wall-clock time, whereas Difficulty Only and BMC maintain learning speeds comparable to it with training times close to uniform sampling.
  • Productivity and learning dynamics: Equivalent training-set accuracy can hide distinct problem subsets, so accuracy alone does not capture how learning is distributed across the task space.Different sampling strategies may achieve similar pass@1 values while covering distinct portions of the training distribution.
  • Diversity and structure: The Latent Task Tree explains more learning-signal variation than random partitions for BMC and Difficulty Only, indicating that difficulty and learning signal correlate with latent task structure.BMC uses the tree as a structured prior, propagating observations across related regions.
  • Evaluation utility: Sampling strategies with similar productivity metrics induce different capability profiles because their problem-type exposure differs across benchmarks.Dynamic Sampling performs poorly on less-represented benchmarks, Difficulty Only improves balance but remains globally biased, Tree Only emphasizes diversity without learning-signal prioritization, and BMC improves rare-type coverage while performing strongly on GPQA-Diamond.
  • Targeted utility: Productivity, diversity, and evaluation utility are not automatically aligned, and similar effective ratios or learning signals can yield substantially different evaluation outcomes.The experiments frame this as an evaluation deadzone when productive training types are weakly measured by the available evaluation suite, while utility-aware steering depends on target-aligned regions.
  • Targeted utility: BMC-T variants preserve BMC’s productivity profile but shift evaluation performance according to the target distribution used to bias sampling.Targeting AIME2024 improves AIME-style English competition mathematics, while targeting the full evaluation mixture broadens gains, including stronger Chinese mathematics performance.

6 Related Work

Prior work improves curriculum sampling through reward variance, dynamic filtering, difficulty prediction, and non-stationary bandit scheduling. This work instead models prompt relationships in the policy’s latent space and jointly reasons over task distributions, individual problems, difficulty, and problem type.

  • Rollout-Based Curriculum Methods: LILO quantifies problem learnability through variance in model success across rollouts, while DAPO increases learning signal through dynamic sampling at additional wall-clock cost.SRPO and POLARIS filter prompts based on rollout-related criteria, as described in the supplied passage.
  • Predictive Curriculum Methods: GRESO predicts prompt utility from temporal reward dynamics, CurES uses Bayesian success-probability modeling for uncertainty-aware curricula, and PCL predicts difficulty or productivity online.These methods seek informativeness or difficulty estimates without relying exclusively on direct rollout evidence.
  • Predictive Curriculum Methods: DOTS estimates adaptive prompt difficulty with an embedding-based predictor trained from a small rollout-labeled reference set and combines it with rollout replay to reduce per-step generation cost.The supplied passage characterizes DOTS as an extension of online difficulty estimation using a separate predictor.
  • Bandit-Based Curriculum Methods: DUMP and SEC formulate problem selection as non-stationary bandits, using schedulers to adaptively prioritize prompt distributions according to estimated learning productivity.The passage identifies these methods as conceptual inspirations for the present work and notes later combinations of bandit scheduling with clustering-based structure.
  • Our Approach: The proposed method learns prompt relationships directly from the policy’s latent representation space and performs unified Bayesian selection and belief updates over distributions and individual problems.It further treats policy-derived problem type and problem difficulty as complementary but distinct factors in hierarchical decision-making.

7 Conclusion … B Structure Drift

The paper frames adaptive RL problem sampling as a manifold-structured bandit over latent task geometry, introducing BMC and BMC-T to balance productivity, diversity, and utility without hand-defined taxonomies or external models. It situates this approach within related work on representation geometry, hierarchical structure, curriculum learning, structured bandits, and fixed-tree structure drift.

  • 7 Conclusion: BMC uses Bayesian belief updates over a Latent Task Tree to balance productivity, diversity, and utility in adaptive RL problem sampling.BMC-T extends the framework with utility awareness for downstream relevance.
  • 7 Conclusion: The framework derives task structure from policy representations rather than imposing human-defined difficulty or taxonomies, and requires only hidden representations and rollout rewards.This enables application across domains and languages without external embedding models or separate critic-style prompt predictors.
  • Appendix Overview: The appendix provides implementation resources and additional studies covering structure drift, external baselines, medical Q&A, tree visualizations, limitations, future work, and algorithmic details.It includes code, tree breakdowns, training logs, evaluation curves, and discussions of frontier imbalance, utility-aware evaluation, external models, and temporal diversity.
  • A.1 Contemporary Work: Contemporary selection methods define adaptive sampling through different geometries, including optimizer-induced updates, policy gradients, and model-representation latent space.Related methods also incorporate predicted difficulty, diversity, utility, empirical Bayes estimation, representation-based graphs, or model-internal features.
  • A.2 Latent Geometry of LLMs: Prior work motivates using intermediate-layer representations because they exhibit heterogeneous, semantically rich geometry, while LLM embeddings show manifold-like organization that can be locally approximated.The paper uses intermediate-layer embeddings and recursively constructs locally coherent regions rather than assuming a globally smooth manifold.
  • A.2.3 Mechanistic Interpretability & Sparse Autoencoders: Unlike sparse autoencoders, the method uses raw intermediate representations to induce task-level geometry for adaptive curriculum learning, with interpretability treated as incidental.The goal is not to recover human-aligned latent factors or individually interpretable features.
  • A.3 Hierarchical Retrieval and Structured Aggregation: Where hierarchical retrieval organizes embedding spaces for inference, BMC organizes prompt representations to allocate training effort during reinforcement learning.Its curriculum structure is used during policy optimization rather than for retrieval over static corpora.
  • A.4 Curriculum Learning & Active Learning; A.5 Structured & Causal Multi-Arm Bandits; B Structure Drift: BMC relates to curriculum learning, active learning, and structured Bayesian bandits by prioritizing informative tasks while sharing information across related actions instead of treating prompts independently.Its Latent Task Tree is constructed at training start and remains topologically fixed; only node beliefs update, because RLVR updates are relatively small and prompt tokens are masked from the policy-gradient objective.

B.1 Tree Structure Before and After RL

Comparisons across three pre-RL/post-RL model pairs show that RL can alter latent task-tree geometry, but not in one universal direction. The analysis treats these changes as diagnostic structure drift and leaves periodic tree reconstruction as an empirical design choice.

  • Qwen3-8B: Post-RL Qwen3-8B trees remain equally deep but become wider at their largest level and have a larger maximum branching factor.This indicates broader partitions can emerge without changing overall hierarchy depth.
  • Coding models: The coding-model comparison shows a contrasting post-RL pattern in which the tree slightly contracts rather than widening as in Qwen3-8B.The comparison uses DeepSeek-R1-Distilled-Qwen-14B before RL and DeepCoder-14B-Preview after training on DeepCoder-Preview.
  • Guru-7B: For Guru-7B, the post-RL tree becomes deeper with more internal nodes, while its maximum width and maximum branching factor decrease.This reflects a shift from broader partitioning toward more hierarchical decomposition of latent task regions.
  • Cross-setting interpretation: Across the three settings, post-RL trees become wider, smaller, or more hierarchical, so the measurements diagnose structure drift rather than establish a fixed representational-change pattern.The comparisons are not intended to isolate a particular RL algorithm because checkpoints may differ in training data, optimization details, and model lineage.
  • Implications for BMC: BMC therefore treats periodic reconstruction of the latent task tree as an empirical design choice whose effect on curriculum decisions must be tested.The practical question is whether rebuilding the tree during training improves sampling decisions, not merely whether the latent tree drifts.

B.2 Periodic Tree Construction Experiment

Periodic reconstruction of the Latent Task Tree yields only modest evaluation differences relative to a fixed tree. Because gains and losses vary across benchmarks, the main experiments retain the fixed tree for simplicity and stability.

  • Experimental setup: The experiment compares standard BMC with a fixed Latent Task Tree against periodic reconstruction during RL.The reconstruction variant updates the tree every 100 steps after round-robin initialization.
  • Evaluation results: Periodic reconstruction slightly improves concentrated math evaluations, including AIME-style benchmarks, AMC, and the Chinese math aggregate.These gains are accompanied by slight reductions on MATH500 and GPQA-Diamond.
  • Evaluation results: The differing benchmark effects may reflect reconstruction shifting emphasis across evolving latent regions and changing coverage of evaluation distributions.This provides a possible explanation for why some evaluations benefit while others decline.
  • Conclusion: The modest magnitude of changes suggests that the static tree remains a reasonable approximation over the training horizon.Accordingly, the main experiments keep the tree fixed for simplicity and stability.

C The Standard Bandit Pattern

The standard bandit pattern adaptively samples prompt-level arms using estimated learning value, improving sampling productivity but leaving diversity and target-distribution utility uncontrolled. BMC extends this pattern from independent prompts to structured latent-task regions, making cross-prompt structure and multi-level sampling explicit.

  • Pattern definition: The standard bandit pattern assigns scalar learning values to individual prompts and adapts sampling according to those estimates, without requiring formal UCB or Thompson sampling.Prompt values estimate expected usefulness for learning, and methods repeatedly update them from observed rollout rewards.
  • Pattern definition: At each training step, the method samples prompts by estimated value, generates rollouts, observes rewards, updates sampled-prompt values, and repeats as the policy evolves.Values may be initialized uniformly or from a prior model, and batches are often sampled without replacement.
  • Limitations: The standard bandit pattern improves sampling productivity, but it does not directly control diversity over problem types or utility relative to a target distribution.Prioritizing expected productivity, usually defined through difficulty, makes coverage over problem types a byproduct rather than an explicit objective.
  • Empirical behavior: MoPPS and Difficulty Only both improve learning speed and effective ratio relative to uniform sampling while avoiding dynamic sampling’s wall-clock overhead, supporting Difficulty Only as a representative baseline.Their similar qualitative behavior appears across training productivity and evaluation behavior, despite differences in belief models, update rules, and initialization.
  • Empirical behavior: Both standard-bandit methods outperform dynamic sampling on less-represented Chinese mathematics evaluations but perform lower on average than BMC on out-of-distribution GPQA-Diamond.MoPPS briefly exceeds BMC on one intermediate GPQA-Diamond evaluation, but the overall trajectory favors BMC.
  • BMC extension: BMC generalizes the standard bandit pattern by selecting structured latent-manifold regions instead of independent prompt-level arms, enabling shared evidence, multi-level decisions, and explicit diversity over latent problem types.This changes the action space over which curriculum decisions are made while exploiting structural correlations between related prompts.

D Medical Data Experiments

On AlphaMed19K, the Qwen3-8B experiment serves as a domain-transfer stress test and shows that adaptive sampling improves training productivity, while productivity, diversity, and evaluation utility remain distinct. BMC approaches Dynamic Sampling’s productivity without its wall-clock overhead, but Difficulty Only and Dynamic Sampling perform better on several medical evaluations.

  • Experimental setup: The AlphaMed19K experiment uses Qwen3-8B with the main-experiment hyperparameters as a domain-transfer stress test rather than a fully tuned medical recipe.The evaluation uses MedQA, MedMCQA, and MedXpertQA, reporting pass@1 because these evaluation sets are larger.
  • Productivity: Dynamic Sampling maintains the highest effective ratio and reward variance but incurs substantially higher wall-clock time because it repeatedly filters candidate prompts.Filtering required roughly three iterations per effective batch on AlphaMed19K, versus roughly two in the main mathematics experiments.
  • Productivity: BMC and Difficulty Only improve training-set pass@1 over Uniform Sampling even as effective ratio and observed learning signal decline after adaptive sampling begins.This indicates that adaptive prompt selection identifies useful dataset regions early in training despite changing training dynamics on AlphaMed19K.
  • Utility: Dynamic Sampling and Difficulty Only are strongest on several medical evaluations, while BMC remains competitive but is not uniformly best.The results reinforce that optimizing productivity, diversity, or utility alone does not guarantee optimality on the other axes.
  • Base Model Attempt: The reported AlphaMed19K results use the instruction-tuned model because Qwen3-8B-Base often produced only multiple-choice answers instead of the reasoning format required by the RL setup.Multiple prompt templates were tested, but the base-model issue remained unresolved, leaving the source of the observed dynamics open.

E Tree Visualizations & Statistics

The appendix demonstrates that latent task trees can be constructed across diverse datasets using shared hyperparameters, while BMC operates on the induced unlabeled structure rather than manual annotations. Construction statistics show dataset-dependent tree sizes and runtimes, with broad semantic organization expected to persist despite exact-tree nondeterminism.

  • Dataset coverage: Latent task trees were constructed for multilingual math, coding, medical, legal, financial, multi-domain, instruction-following, and multimodal geometry datasets using the same Appendix L.1 hyperparameters.The listed datasets include DAPO-Math-17K, DeepCoder-Preview, AlphaMed19K, BarExamQA, Agentar-DeepFinance-100k, GURU-92k, IF-RLVR, and Geometry3K.
  • Tree construction and interpretation: Manual node labels are coarse interpretive annotations from representative prompts and word clouds, whereas BMC uses the unlabeled tree induced directly from LLM latent representations.Clusters without a clear dominant theme are marked “???”, and no external embedding model or hand-designed taxonomy is required.
  • Reproducibility: Fixing the random seed makes construction reproducible on a given machine, but implementation- and hardware-level nondeterminism can change the exact hierarchy across machines.The expected invariant is broad semantic organization rather than an identical tree.
  • Construction statistics: The induced trees contain 49–67 nodes across DeepCoder-Preview, AlphaMed19k, and BarExamQA, with total construction times of 20:18, 20:53, and 1:44 minutes, respectively.These statistics were measured on one node with 8×H100 GPUs.

F Frontier Imbalance

The section distinguishes dataset imbalance from policy-dependent frontier imbalance, showing how adaptive sampling can concentrate training on certain problem types. It highlights that this interaction can reduce minority-type exposure and affect evaluation outcomes, especially across model sizes.

  • F Frontier Imbalance: Dataset imbalance occurs when heterogeneous training data contains many more examples of one problem type, causing non-type-aware samplers to allocate it more training effort.This concentration reflects the dataset’s static composition rather than the current policy’s learning frontier.
  • F Frontier Imbalance: Frontier imbalance occurs when problems currently providing substantial learning signal are unevenly distributed across problem types under the current policy.A productivity-oriented sampler may therefore prioritize a type because it is overrepresented on the frontier, even if it is rare in the dataset.
  • F Frontier Imbalance: An 80%–20% dataset split can produce opposite sampling outcomes: a minority type may dominate the frontier, while reinforcement of both imbalances can starve other regions.Minority types may receive little exposure because they are rare, rarely productive, or skipped by sampling mechanics, creating undertraining feedback loops.
  • F Frontier Imbalance: Dynamic Sampling can skip effective prompts after composite batches fill, allowing dataset and frontier imbalance to shape which problem types receive training exposure.The mechanism generates rollouts, discards zero-variance prompts, and may defer later effective prompts to a future pass.
  • F Frontier Imbalance: For the 8B model, Dynamic Sampling improved strongly on English mathematics evaluations but plateaued on Chinese mathematics evaluations relative to uniform sampling.The dataset was dominated by English mathematics problems, and the observed behavior is consistent with interactions among dataset imbalance, frontier imbalance, and problem skipping, without causal isolation.
  • F Frontier Imbalance: Frontier imbalance is not inherently undesirable: BMC aims to make uneven training effort structured and controllable rather than eliminate it.Disproportionate sampling can be useful when selected regions genuinely provide stronger learning signal.

G Utility-Aware Sampling and the Evaluation-Protocol Tension … I Inter-Batch Diversity (Temporal Awareness)

The paper frames utility-aware curriculum design as a protocol-sensitive use of evaluation information and derives task structure from the policy model’s own representations rather than external taxonomies. It also identifies temporally aware inter-batch diversity as a promising but currently under-validated extension to BMC’s within-batch coverage.

  • G Utility-Aware Sampling and the Evaluation-Protocol Tension: BMC-T uses held-out benchmarks as target distributions that assign utility scores to sampleable regions without allowing target examples into training.This setup tests whether training problem types differ in relevance to a desired capability or evaluation.
  • G Utility-Aware Sampling and the Evaluation-Protocol Tension: Using a benchmark to define BMC-T’s target distribution makes it part of curriculum design, so strict evaluation requires a separate validation or development target set.The benchmark should not simultaneously guide training decisions and serve as the untouched final generalization test.
  • G Utility-Aware Sampling and the Evaluation-Protocol Tension: Validation targets may provide weak utility signals when human-defined relevance or taxonomies poorly align with the policy model’s internal organization of the task space.The paper treats these protocol choices as study-dependent and uses BMC-T primarily for diagnostic and methodological analysis.
  • H External Models: BMC derives its Latent Task Tree from the policy model’s hidden representations because curriculum scheduling should reflect the model’s own task geometry rather than external semantic categories.External models remain useful for labeling, validation, and analysis, but are not used as the source of curriculum structure.
  • H.1 Semantic Embedding Models: Policy-model representations can distinguish curriculum-relevant differences that human or retrieval embeddings miss, while prompts that look different may be treated similarly when they activate similar internal behavior.Latent Task Tree visualizations show coherent, fine-grained clusters despite human-readable labels being added only after construction.
  • H.2 Frontier Model Annotations & Prompt-based Clustering.: Frontier-model annotations offer interpretable taxonomies but may impose an external semantic perspective and can be costly at scale, especially for multimodal datasets.BMC therefore uses such annotations for analysis and visualization rather than constructing the Latent Task Tree from them.
  • H.3 Sparse Autoencoders.: Sparse autoencoders could support interpretable task discovery, but BMC prioritizes curriculum structure over mechanistic interpretability and avoids their added design complexity.The paper instead uses normalized hidden representations for the Latent Task Tree.
  • H.4 Critic-Style Prompt Predictors: BMC replaces a separate productivity predictor with local uncertainty-aware estimates that share evidence across geometrically related prompts and adapt through surprise-based forgetting and uncertainty growth.This design treats the task tree as structure for handling policy-dependent, non-stationary prompt productivity rather than excluding learned predictors as alternatives.

J Limitations … L.3 Round-Robin Initialization

The paper identifies limitations in heuristic tree construction, computational cost, static latent geometry, and uncertainty saturation, while outlining future directions for adaptive rollout allocation, frontier-aware prioritization, structured bandits, and reproducible implementation details. The experiments also use fixed rollout allocation, a single pretraining tree, matched hardware constraints, and round-robin initialization to control comparisons.

  • J.1 Tree Construction Heuristics: Tree construction is heuristic, and evaluating alternative representations, dimensionality reductions, manifold approximations, stopping criteria, or clustering algorithms requires expensive downstream RL ablations.The current pipeline produces coherent, fine-grained trees, but its hyperparameter effects are difficult to isolate because tree generation is cheaper than validating curriculum quality through RL.
  • J.2 Computational Requirements; L.2 Compute: Latent Task Trees may become computationally or memory intensive for sufficiently large models and datasets, while the implementation relies primarily on CPU-based preprocessing.GPU-accelerated PCA, UMAP, and HDBSCAN could reduce construction time, but such an implementation is not included.
  • J.3 Structure Drift: A tree constructed before RL can become misaligned as policy representations drift, although periodic reconstruction did not substantially change the main conclusions over the tested training horizons.Online rebuilding required interrupting and resuming training because latent extraction and RL training could not conveniently use all required models simultaneously.
  • J.4 Belief Bounds and Uncertainty Saturation: 80% of prompts reached maximum uncertainty by the end of the main training runs, suggesting that uncertainty growth or the sigma-rule variance bound may be too aggressive or poorly matched.Possible refinements include sublinear staleness accumulation, alternative saturating transforms, and adjusted sigma-rule bounds; the bounded rule nevertheless improved learning signal and effective ratio in main experiments, with less consistent medical-domain behavior.
  • K.1 Rollout Allocation: The study fixes rollout allocation per prompt to isolate problem-selection effects, leaving adaptive, type-aware allocation across difficulty, diversity, and utility as a future direction.Allocating compute only by difficulty or immediate productivity could reproduce coverage and target-mismatch issues.
  • K.2 Prioritization Mechanisms: Future acquisition functions could prioritize frontier expanders by estimating how training on one problem changes the learnability of nearby problems, rather than selecting only currently productive prompts.This would extend BMC toward actively expanding the frontier of learnability through local manifold effects.
  • K.3 Connections to the Bandit Literature: The framework motivates broader use of structured, causal, combinatorial, pure-exploration, and budgeted bandits to make LLM curricula adaptive, uncertainty-aware, and compute-conscious.These formulations can share information across related arms and reason about how interventions change learnability in other regions.
  • L.1 Hyperparameters; L.2 Compute; L.3 Round-Robin Initialization: Experiments use documented hyperparameters, matched single-H100 hardware constraints, and round-robin initialization that samples every prompt once before adaptive scheduling.BMC and Difficulty Only share the post-round-robin checkpoint and beliefs, while Tree Only shares the checkpoint but uses fixed uniform beliefs.

L.4 Evaluation Datasets … M.7.1 Implementation

The paper evaluates curriculum strategies on diverse mathematics and scientific-reasoning benchmarks, using cost-aware evaluation and diagnostics alongside detailed Bayesian implementation choices. BMC combines reward-variance learning signals, bounded uncertainty modeling, hierarchical batch sampling, non-stationarity adaptation, and structure-aware diagnostics.

  • L.5 Dense-to-Sparse Evaluation Schedule: The dense-to-sparse schedule evaluates every 10 optimizer steps early, interleaves training-set pass@1 and downstream mean@16, then evaluates both every 50 steps after step 250.This reduces evaluation cost while preserving early learning-dynamics measurements, though sparse late curves can miss brief fluctuations or regressions.
  • L.6 Evaluation Curves: Evaluation curves compare Qwen3-8B-base and Qwen3-4B-base across GSPO, GRPO, and variations of BMC-T on DAPO-Math-17k.The corresponding trajectories are presented in Figures 24–27 and linked to bar-graph results in Figures 6–7.
  • M. Additional Algorithmic Details; M.1 Learning Signal; M.1.1 Quantifying the Learning Signal: BMC’s learning signal is a prompt-level statistic distinct from policy reward, with reward variance providing an inexpensive proxy for informative group-relative policy-optimization feedback.Uniform rollout rewards produce zero unnormalized advantages, corresponding to prompts that are uniformly too easy or too difficult; for binary rewards, variance is maximized at p = 0.5.
  • M.1.2 Normalizing the Learning Signal: Learning signals are preferably normalized to [0, 1] across heterogeneous tasks, because unnormalized Gaussian beliefs can create sampling imbalance when reward scales differ.Normalization is not required in principle, but scale differences can cause the scheduler to over-prioritize prompts for reasons unrelated to learning value.
  • M.2 Logit-Normal Beliefs and the Sigma Rule: Implementation uses Logit-Normal beliefs for bounded signals, applying clamping and Gaussian updates in logit space while retaining the original ordering under monotonic transformation.The sigma-rule constant is a = 3, with minimum variance preventing beliefs from collapsing permanently and enabling revisitation as signals evolve.
  • M.3 Sampling Without Replacement; M.4 Saturating Estimation Errors; M.5 Staleness & Uncertainty Growth: BMC’s batch-aware hierarchical descent conditions each subsequent selection on prompts already chosen, avoiding rejection loops and encouraging exploration after high-value prompts are exhausted.The framework also adapts beliefs conservatively: saturating surprise terms reduce confidence in stale evidence, while coverage-adjusted staleness increases uncertainty for unsampled prompts.
  • M.6 Tree Diagnostics; M.6.1 Rarity-Weighted Exposure; M.6.2 Structure Gain; M.7 BMC-T; M.7.1 Implementation: Tree diagnostics measure rarity-weighted exposure and structure gain, while BMC-T implementation uses additive smoothing as a symmetric Dirichlet prior for comparing target relevance among sibling children.Rarity-weighted exposure is diagnostic rather than an optimization objective, and structure gain values greater than one indicate relative alignment with more learning-signal variation than a size-matched random partition.

M.7.2 Alternative Implementations

BMC-T is presented as one proof-of-concept implementation of utility-aware hierarchical curriculum learning, not a unique prescription. Utility may instead be integrated multiplicatively or estimated from alternative signals while remaining explicit alongside productivity and diversity.

  • BMC-T design: BMC-T estimates target relevance from training–target overlap in the Latent Task Tree and adds a normalized utility bonus to productivity during top-down sampling.Productivity measures where the current policy can learn, while utility measures which learnable regions are relevant.
  • Alternative implementations: Utility could modulate productivity multiplicatively as a relevance gate, or be estimated through gradient-based alignment or optimizer-based data valuation.These alternatives may help when target examples are sparse, unavailable, or difficult to compare.
  • Interpretation: BMC-T demonstrates that evaluation relevance can be represented alongside productivity and diversity when sampling target-relevant task-manifold regions rather than difficulty metrics alone.The framework should therefore be interpreted as a proof of concept for utility-aware curriculum design.
Loading 2606.19750v1…