Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models

1 Introduction

• Group-relative RL methods need prompts with variable rollout rewards because identical outcomes yield zero relative advantage and no policy-gradient signal.Dynamic Sampling in DAPO [Yu et al., 2025] regenerates prompts until batches have non-zero reward variance, but incurs substantial wall-clock overhead.Source: §1 · p. 2
• Unlike standard non-stationary bandits, LLM sampling is endogenous: model updates and sampling choices alter future prompt rewards, while related problems can share reasoning patterns.Treating prompts as independent arms can therefore make exploration inefficient.Source: §1 · p. 2
• Difficulty-focused sampling can sacrifice diversity and narrow problem-space coverage, while externally defined groupings may not match the model’s latent organization.The introduction argues curricula should adapt both to evolving difficulty and to the model’s latent problem structure.Source: §1 · p. 2
• Latent Task Trees approximate prompt relationships from model embeddings, while BMC applies Bayesian decision-making over the hierarchy to explore under non-stationary, model-induced dynamics.The task tree provides a multi-scale representation of the task manifold; BMC uses it to guide structured adaptive sampling.Source: §1 · p. 2
• BMC-T biases sampling toward target-relevant regions while preserving structure-aware exploration, motivated by the finding that productivity, diversity, and utility are distinct and non-interchangeable axes.The authors argue downstream evaluation performance alone cannot fully characterize adaptive curricula.Source: §1 · p. 2

Reinforcement Learning for LLMs

• RLVR trains an autoregressive policy to maximize expected verifier reward over the prompt distribution, optionally regularizing or constraining policy updates.For each prompt, the policy samples a response that an automatic verifier scores.Source: §2 · p. 3

Group-Relative Policy Optimization

• Group-relative methods such as GRPO [Shao et al., 2024], DAPO [Yu et al., 2025], and GSPO [Zheng et al., 2025a] use within-prompt response comparisons, avoiding a learned critic.Zero reward variance within a prompt’s rollouts yields no learning signal; Dynamic Sampling in DAPO [Yu et al., 2025] regenerates prompts until reward variation is sufficient, increasing wall-clock time.Source: §2 · p. 3

Structured, Causal, and Non-Stationary Bandits

• Structured bandits transfer information between related arms [Lattimore and Szepesvári, 2020, Slivkins, 2024], while causal bandits treat actions as interventions [Lattimore et al., 2016].In RL training, sampled prompts induce policy updates that can change future rewards on related prompts; this makes non-stationarity partly endogenous to the learner’s sampling decisions.Source: §2 · p. 3

Latent Geometry and the Manifold Hypothesis

• LLM representations are often approximated by lower-dimensional geometric structure [Valeriani et al., 2023, Modell et al., 2025], consistent with the manifold hypothesis [Fefferman et al., 2013].The latent task structure provides a prior for where information and learning effects may transfer across the task space.Source: §2 · p. 3

Latent Task Trees

• Latent Task Trees use policy-representation similarity to organize related prompts into regions, supporting structured sampling and information sharing rather than treating each prompt as an independent arm.The hierarchy reflects the premise that nearby prompts may share solution strategies, failure modes, and learning dynamics.Source: §3 · p. 3
• Because representation-space density, intrinsic dimension, and organization vary locally, the tree recursively approximates regions without imposing a fixed global partition or smooth global manifold.This creates a multi-scale, computationally tractable representation of the task manifold [Li and Sarwate, 2025a, Shafran et al., 2026].Source: §3 · p. 4
• Construction extracts normalized mean embeddings from a selected intermediate policy layer, then applies PCA, UMAP [McInnes et al., 2018], and HDBSCAN recursively to identify child clusters.PCA retains the smallest number of components explaining 95% of variance; HDBSCAN accommodates varying density without a preset cluster count.Source: §3 · pp. 4–5
• Recursion stops when a node passes the Chart Test, HDBSCAN finds at most one cluster, or clusters fall below a minimum size threshold.The Chart Test checks low intrinsic dimensionality with Two-NN [Facco et al., 2017] and local connectivity using a k-nearest-neighbor graph, on PCA-reduced embeddings before UMAP.Source: §3 · p. 5
• The construction needs only model embeddings, avoiding dataset-specific tuning, external embedding models, and annotations, and is presented as applicable across models, domains, languages, and modalities.After embedding extraction, tree construction uses standard unsupervised operations whose cost is described as negligible compared with autoregressive generation.Source: §3 · p. 5

4 Bayesian Manifold Curriculum

• BMC organizes prompts in a Latent Task Tree, a hierarchical approximation of the model’s latent task manifold that shares information across related prompts and abstraction levels.The scheduler allocates training effort across task regions using the tree’s structure.Source: §4 · p. 5
• Reward variability across multiple rollouts defines each prompt’s learning signal, prioritizing binary-reward prompts near a 50% success rate as productive for group-relative learning.The paper follows prior RL-for-LLMs work [Foster et al., 2025, Zhang et al., 2025a, Zeng et al., 2025] and allows optional normalization of the signal.Source: §4 · p. 5

4.1 Top-Down Problem Selection (Hierarchical Thompson Sampling)

• Independent batch agents use hierarchical Thompson sampling to descend from tree regions to prompts, concentrating on productive areas while spreading across similarly productive regions.This balances exploration and exploitation, and a batch-aware variant enforces without-replacement sampling.Source: §4.1 · pp. 5–6

4.2 Non-Stationary Belief Modeling (Bayesian Filtering)

• A Bayesian filter tracks prompt-level signal means, uncertainty, and effective evidence weight, using standardized surprise and unsampled-time staleness to adapt beliefs under policy drift.Beliefs become more uncertain after surprising observations or long periods without feedback, allowing the curriculum to revisit prompts as model capability changes.Source: §4.2 · pp. 6–7

4.3 Bottom-Up Belief Propagation (Empirical Bayes)

• Empirical Bayes propagation aggregates child beliefs upward, accounting for subtree heterogeneity so disagreement increases parent uncertainty and discourages overconfident selection of diverse regions.Recursive updates let observations at one prompt affect expectations and sampling probabilities for related prompts before they are directly observed.Source: §4.3 · pp. 7–8
• BMC supports group-relative RL algorithms with multiple rollouts, inherits Latent Task Tree domain, language, and modality generality, and can extend to continuous-reward settings.The paper names GRPO, DAPO, and GSPO, as well as RLHF and rubric-based RL [Gunjal et al., 2025] as examples.Source: §4.3 · p. 8

5.1 Experimental Setup

• Experiments evaluated BMC on DAPO-Math-17K with Qwen3-4B-Base and Qwen3-8B-Base, primarily using GSPO; methods shared initialization, reward, optimization hyperparameters, and rollout allocation, varying only prompt sampling to isolate its effects.Difficulty Only removes the Latent Task Tree, while Tree Only uses fixed beliefs without experience-based updates; BMC and these ablations use round-robin initialization through step 136.Source: §5.1 · pp. 8–9

5.2 Training Efficiency (Productivity)

• Across model sizes and group-relative algorithms, higher reward variance and effective ratios accompanied faster training-set accuracy gains; curriculum methods improved effective ratio by roughly 40% over uniform sampling.Dynamic Sampling reached an effective ratio of 1.0 through repeated resampling and incurred substantially higher wall-clock time, while BMC and Difficulty Only retained comparable learning-speed benefits with training times near uniform sampling, especially for 8B.Source: §5.2 · p. 9, Figure 4 · p. 10

5.3 Coverage and Information Sharing (Diversity)

• BMC balanced Tree Only’s diversity-focused sampling with Difficulty Only’s productivity focus, while both BMC and Difficulty Only showed structure gain over random partitions.The tree’s alignment with learning signal was present even when it did not guide sampling; BMC uses the tree as a prior to propagate observations across related regions, exploiting existing structure rather than creating it.Source: §5.3 · pp. 10–11, Figure 5 · p. 11

5.4 Evaluation Performance (Utility)

• BMC performed strongly on OOD GPQA-Diamond but had trade-offs on in-distribution math benchmarks, consistent with evaluation relevance depending on which problem types receive productive effort.The authors interpret this as an evaluation deadzone: training problems can provide learning signal while their induced capabilities are weakly measured by the available evaluation suite.Source: §5.4 · pp. 11, 13, Figure 6 · p. 12
• Dynamic Sampling’s perfect effective ratio and highest average learning signal came with poor performance on less-represented benchmarks; Difficulty Only improved their balance but still favored common types.Tree Only emphasized diversity without prioritizing informative problems and did not outperform other methods on any benchmark; Uniform Sampling’s broad coverage occasionally benefited AIME 2025 but lacked systematic prioritization.Source: §5.4 · pp. 12–13

5.5 Utility-Aware Sampling and Approximate Causality

• BMC-T uses target-distribution overlap to bias sampling, but is not a strict held-out evaluation when benchmark examples steer sampling; the authors also caution that its results do not identify prompt-level causal effects.For GPQA-Diamond, the target examples occupied a high-level tree node with no training examples, so BMC-T could not directly steer toward them and would reduce to BMC; periodic tree reconstruction produced only modest downstream changes.Source: §5.5 · pp. 13–15
• BMC-T preserved BMC-like productivity while target choice shifted evaluation profiles: AIME2024 favored English competition math, whereas the full mixture favored broader evaluations, including Chinese math.BMC-T uses target overlap within the Latent Task Tree to bias sampling toward productive, sampleable regions relevant to the target distribution.Source: §5.5 · pp. 13–14, Figure 7 · p. 14

6 Related Work

• Reward-variance and difficulty-aware methods use rollout evidence or capability-aligned signals to prioritize informative prompts.LILO [Foster et al., 2025] measures learnability through success variance across rollouts; DAPO [Yu et al., 2025] increases learning signal at added wall-clock cost, while SRPO [Zhang et al., 2025b] and POLARIS filter saturated prompts. SPEED-RL [Zhang et al., 2025a] screens with partial rollouts, and AdaRFT [Shi et al., 2026] and VCRL [Jiang et al., 2025] use difficulty or reward variance aligned with model capability.Source: §6 · p. 15
• Informativeness predictors estimate reward dynamics, success probability, or difficulty online, while DOTS combines embedding-based prediction with rollout replay to reduce generation cost.GRESO [Zheng et al., 2025b] probabilistically skips prompts predicted to yield negligible reward variance; CurES [Zeng et al., 2025] uses Bayesian success-probability modeling, and PCL selects intermediate-difficulty prompts with an online value model without explicit rollout estimation. DOTS [Sun et al., 2026] interpolates difficulty predictions from a small rollout-labeled reference set.Source: §6 · pp. 15–16
• Bandit curricula adapt prompt distributions, cluster prompts before selection, or treat individual prompts as arms without modeling distribution-level structure.DUMP [Wang et al., 2025b] and SEC [Chen et al., 2025] inspire productivity-based bandit scheduling; other work applies bandits to flat clusters based on difficulty or semantic embeddings [Do et al., 2025, Anonymous, 2026]. MoPPS [Qu et al., 2025], BOTS, and VADE [Hu et al., 2025] instead use Thompson-sampling variants over individual prompts.Source: §6 · p. 16
• The proposed method learns prompt relationships from policy latent representations and uses hierarchical Bayesian selection and belief updates over distributions and individual problems.It treats problem difficulty and policy-perceived problem type as complementary but distinct curriculum factors.Source: §6 · p. 16

7 Conclusion

• BMC and BMC-T balance learning signal, task-type coverage, and downstream relevance through Bayesian updates over a Latent Task Tree built from policy representations.The framework requires only policy hidden representations and rollout rewards, and is presented as applicable across domains, languages, modalities, and reward types.Source: §7 · p. 16

Appendix Overview

• The appendix includes further experiments, discussions, and implementation details, including structure drift, medical-data experiments, limitations, and future directions.It also provides links to code, tree breakdowns, and training logs.Source: Appendix Overview · p. 26

A.1 Contemporary Work

• BMC complements OPUS, GPS, GradAlign, and EBPO by scheduling RL prompts over policy-derived latent task geometry, whereas these methods use optimizer geometry, external embeddings/predictors, policy-gradient alignment, or external clusters.Source: §A.1 · pp. 27–28

A.2.1 Layer Heterogeneity; A.2.2 Manifold Representations & Regularization

• Prior work motivates intermediate-layer representations and a cautious local-geometry view: BMC recursively approximates locally coherent regions rather than assuming a globally smooth manifold.The appendix notes that intermediate layers can capture semantically meaningful structure, while global manifold assumptions may fail; UMAP is used for neighborhood-preserving clustering, not as evidence of global geometry.Source: §A.2.1 · pp. 28–29, §A.2.2 · p. 29

A.2.3 Mechanistic Interpretability & Sparse Autoencoders; A.3 Hierarchical Retrieval and Structured Aggregation

• BMC uses raw intermediate representations to induce task-level prompt geometry for RL sampling, not to recover human-aligned latent features or organize static-corpus retrieval.Any human interpretability of its trees is incidental; the goal is allocating training effort according to the model’s implicit problem-space organization.Source: §A.2.3 · pp. 29–30, §A.3 · p. 30

A.4 Curriculum Learning & Active Learning

• The curriculum-learning discussion argues that productive prompt selection should also account for diversity and structure when constructing batches.The passage contrasts individually informative selection with methods that address redundancy through joint batch selection, including BatchBALD [Kirsch et al., 2019].Source: §A.4 · p. 30

A.5 Structured & Causal Multi-Arm Bandits

• BMC differs from structured bandit methods by deriving its task tree from the policy’s prompt representations rather than a supplied taxonomy or covariance structure.The resulting hierarchy supports structured exploration, belief sharing, and diversity-aware curriculum scheduling.Source: §A.5 · p. 31

B Structure Drift

• BMC keeps its Latent Task Tree topology fixed during training while updating node beliefs, so it tracks reward changes but not evolving prompt representations.The paper investigates drift by comparing trees before and after RL and by periodically reconstructing the tree; fixed topology is motivated by relatively small RLVR parameter updates and prompt-token masking from the policy-gradient objective.Source: B Structure Drift · p. 32

• The comparisons are diagnostic, not causal evidence for a particular RL algorithm, because checkpoints may differ in data, optimization, and model lineage.Tree construction was reproducible across repeated runs under fixed seeds on the same machine; exact reproducibility across hardware or software environments was not assumed.Source: §B.1 · p. 32
• For Qwen3-8B, the post-RL tree retains its depth but becomes wider at its largest level and has a larger maximum branching factor.Source: Table 1 · p. 32, §B.1 · p. 32
• For DeepCoder-Preview, the post-RL coding tree is slightly smaller, with fewer terminal nodes and a lower maximum branching factor, while both trees remain shallow.Source: §B.1 · p. 32, Table 2 · p. 33
• For GURU-92k, the post-RL Guru-7B tree becomes deeper with more internal nodes, while maximum width and branching factor decrease.Source: §B.1 · pp. 32–33, Table 3 · p. 33
• Across three settings, post-RL trees become wider, smaller, or more hierarchical, indicating structure drift without a fixed direction.Source: §B.1 · p. 33
• For BMC, whether periodic tree reconstruction improves curriculum decisions is treated as an empirical design choice, not an obvious requirement.Source: §B.1 · p. 33

B.2 Periodic Tree Construction Experiment

• The experiment compares standard BMC’s fixed Latent Task Tree with a variant that periodically reconstructs it during RL to track possible drift in prompt representations.The reconstruction variant updates the tree every 100 steps after round-robin initialization.Source: §B.2 · p. 33, Figure 8 · p. 34
• Periodic reconstruction slightly improves AIME-style benchmarks, AMC, and the Chinese math aggregate, but slightly reduces MATH500 and GPQA-Diamond performance.The authors describe the differences as modest; Figure 8 compares standard BMC with a tree reconstructed every 100 steps after round-robin initialization.Source: §B.2 · p. 33, Figure 8 · p. 34
• The small changes suggest the static tree remains a reasonable approximation, so the main experiments keep it fixed for simplicity and stability.The authors speculate that reconstruction shifts emphasis among latent regions as representations evolve, potentially benefiting some evaluation distributions while reducing coverage or transfer to others.Source: §B.2 · pp. 33–34

Standard bandit pattern

• Adaptive curricula predict informative prompts to increase the fraction contributing to gradients, addressing identical-reward rollouts without dynamic sampling’s wall-clock overhead.Dynamic sampling resamples until informative groups are found, but can incur substantial overhead.Source: C The Standard Bandit Pattern · p. 35
• The standard bandit pattern treats each prompt as an arm, repeatedly samples according to estimated learning values, and updates sampled prompts’ values from observed rewards.The pattern is descriptive, not limited to formal UCB or Thompson-sampling algorithms; methods may use different statistical representations and sampling rules.Source: C The Standard Bandit Pattern · p. 35

Limitations

• Productivity-only sampling leaves problem-type coverage to chance and can repeatedly favor narrow high-signal regions, while productive prompts may not match a target evaluation distribution.Difficulty predictors can still improve productivity, but productivity alone does not guarantee diversity or benchmark superiority, especially for broad, heterogeneous mixtures and mismatched task regions.Source: C The Standard Bandit Pattern · pp. 35, 37, 38

Results

• MoPPS and Difficulty Only improve learning speed and effective ratio over uniform sampling without dynamic sampling’s wall-clock overhead, but trail BMC on GPQA-Diamond and uniform sampling on AIME2025.Both show their strongest relative performance on in-distribution English mathematics and outperform dynamic sampling on Chinese mathematics evaluations.Source: Figure 9 · p. 36, C The Standard Bandit Pattern · p. 37

BMC extension

• BMC replaces independent prompt-level arms with structured latent task regions, sharing evidence across related prompts and making diversity explicit; BMC-T additionally biases sampling toward regions relevant to target evaluations.This shifts curriculum decisions toward which manifold regions receive training effort and which downstream evaluations that effort may support.Source: C The Standard Bandit Pattern · p. 38

D Medical Data Experiments

• After round-robin initialization, BMC and Difficulty Only see declining effective ratio and learning signal yet improve training-set pass@1 over Uniform Sampling.Source: D Medical Data Experiments · p. 39
• Dynamic Sampling needed roughly three filtering iterations per effective batch on AlphaMed19K versus two in math, suggesting productive prompts are sparser or depleted sooner and increasing wall-clock cost.Source: D Medical Data Experiments · p. 40
• Medical utility was measured by pass@1 on MedQA, MedMCQA, and MedXpertQA, rather than mean@16 because the evaluation sets are larger.Source: D Medical Data Experiments · p. 40
• The AlphaMed19K tree may be finer-grained, making evaluation relevance uneven across regions; this suggestive, noncausal interpretation motivates BMC-T’s explicit utility signal.Source: D Medical Data Experiments · p. 40
• BMC improves training productivity over Uniform Sampling and approaches Dynamic Sampling without its wall-clock overhead, but Dynamic Sampling and Difficulty Only are stronger on several medical evaluations.The results do not show that structure-aware sampling is ineffective in medical question answering; they show that productivity, diversity, and evaluation utility do not necessarily align.Source: Figure 11 · p. 39, D Medical Data Experiments · p. 40
• Qwen3-8B-Base often produced answer-only outputs despite prompt-template trials, so reported results use instruction-tuned Qwen3-8B and leave initialization and domain effects unresolved.Source: D Medical Data Experiments · p. 40

E Tree Visualizations & Statistics

• The visualizations and construction statistics cover eight datasets, from math and coding to medicine, law, finance, heterogeneous reasoning, instruction following, and multimodal geometry.The set includes DAPO-Math-17K, DeepCoder-Preview, AlphaMed19K, BarExamQA, Agentar-DeepFinance-100k, GURU-92k, IF-RLVR, and Geometry3K.Source: E Tree Visualizations & Statistics · p. 41
• BMC operates directly on the unlabeled tree, while displayed labels are coarse interpretive annotations rather than required inputs.The labels were assigned after inspecting 10 representative prompts and word clouds per cluster, with ChatGPT assisting summarization; unclear clusters are marked “???”.Source: E Tree Visualizations & Statistics · p. 41
• Tree construction is reproducible on a given machine with a fixed seed, but implementation- and hardware-level nondeterminism can change the exact hierarchy across machines.The authors expect broad semantic organization to remain similar.Source: E Tree Visualizations & Statistics · p. 41
• Tree construction accounted for 0.802% of DAPO-Math-17K training time and 1.15% of AlphaMed19K training time.The corresponding trees contained 50 and 67 nodes, respectively.Source: Figure 12 · p. 42, Figure 14 · p. 44
• 54-node Geometry3K tree labels were ambiguous for some clusters, suggesting its latent structure reflects visual and multimodal properties beyond surface text [Lu et al., 2021].Labels were assigned from representative prompts and word clouds; the authors note that a finer-grained visual audit could refine them.Source: E Tree Visualizations & Statistics · pp. 41, 49, Figure 19 · p. 49

F Frontier Imbalance

• Dataset imbalance comes from a static majority of one problem type, whereas frontier imbalance is a policy-dependent skew in currently productive problems, or the model’s frontier [2025].As the policy improves, problems can shift between being too easy, too hard, or productively learnable.Source: F Frontier Imbalance · p. 51
• A dataset with 80% type PA and 20% type PB can still have a frontier where 80% of productive problems are PB, so productivity sampling may favor PB.A balanced frontier could instead let a productivity-oriented sampler select useful examples from both types despite the dataset skew.Source: F Frontier Imbalance · p. 51
• Dynamic Sampling discards zero-reward-variance prompts, but effective prompts encountered after its composite batch fills may be deferred, interacting with type or partition imbalances.This skipping can leave minority types with little exposure when they are rare in the dataset, rarely productive, or skipped by the sampling procedure.Source: Figure 22 · p. 52, F Frontier Imbalance · p. 52
• When dataset and frontier imbalances reinforce each other, productivity sampling can further concentrate effort on a majority type and leave other task-space regions undertrained.The 4B model did not show the 8B plateau; a different frontier mixture is one possible explanation, but the interpretation is suggestive rather than causal.Source: F Frontier Imbalance · pp. 51–52
• For the 8B model, Dynamic Sampling improves English mathematics evaluations but plateaus or underperforms Chinese mathematics evaluations versus uniform sampling; the experiment does not isolate the causes.The pattern is consistent with an interaction among dataset imbalance, frontier imbalance, and problem skipping.Source: F Frontier Imbalance · pp. 52–53, Figure 23 · p. 53
• Frontier imbalance is not inherently undesirable; BMC aims to make sampling imbalance structured and controllable, rather than eliminating it or relying on productivity alone.Returning unused effective prompts to the active pool is one implementation alternative that could reduce or eliminate skipping.Source: F Frontier Imbalance · pp. 52–53

G Utility-Aware Sampling and the Evaluation-Protocol Tension

• Held-out benchmarks can define BMC-T’s utility target, but then serve curriculum design rather than untouched final testing; separate development targets preserve strict held-out evaluation.Target examples influence utility scores but are never training examples or sampleable prompts; Latent Task Trees can also help audit whether target sets align with the policy’s latent task manifold.Source: G Utility-Aware Sampling and the… · p. 54
• Changing BMC-T’s target distribution shifted evaluation outcomes even when apparent training productivity remained similar, suggesting some problem types matter more for specific benchmarks.The authors frame BMC-T primarily as a diagnostic and methodological tool, not a resolution of benchmark-awareness or train–test-separation questions.Source: G Utility-Aware Sampling and the… · p. 54

H External Models; H.1 Semantic Embedding Models

• BMC builds Latent Task Trees from the policy’s hidden representations because curriculum-relevant similarity reflects that model’s behavior, and visualizations show coherent, fine-grained clusters.External semantic similarity may not track differences in policy representations, rollout behavior, reward variance, or learning dynamics; human-readable labels are added post hoc.Source: H External Models · p. 55, §H.1 · p. 55

H.2 Frontier Model Annotations & Prompt-based Clustering.; H.3 Sparse Autoencoders.

• Frontier-model labels and prompt-based clustering are analysis tools, not tree structure, because external taxonomies may misrepresent policy geometry and labeling can be costly.Although sparse autoencoders could support task discovery, their training and validation add complexity; BMC instead uses normalized hidden states directly.Source: §H.2 · p. 55, §H.3 · p. 56

H.4 Critic-Style Prompt Predictors

• Rather than train a separate critic to predict prompt utility, BMC uses latent geometry and online Bayesian beliefs, with uncertainty growth and surprise-based forgetting adapting to non-stationarity.The same structure can support explicit control of diversity and utility without requiring a separate critic, teacher, annotator, or embedding model.Source: §H.4 · p. 56

I Inter-Batch Diversity (Temporal Awareness)

• BMC controls intra-batch type coverage but lacks a mechanism against repeated subtree selection across batches; visitation penalties are a proposed, relatively unvalidated extension.A leaf-level visitation penalty could reduce recently sampled regions while preserving intra-batch diversity, but the empirical importance of inter-batch diversity remains less established.Source: I Inter-Batch Diversity (Temporal… · p. 57

J.1 Tree Construction Heuristics; J.2 Computational Requirements

• Tree construction is heuristic, and its choices and hyperparameters are not systematically optimized because comparing trees requires costly downstream RL experiments; large model–dataset combinations may also strain compute and memory.The current CPU-based preprocessing could be accelerated with GPU implementations; very large settings may require memory-efficient methods such as streaming PCA or subsampling-based initialization.Source: §J.1 · p. 58, §J.2 · p. 58

J.3 Structure Drift

• The tree remains fixed during RL, but periodic reconstruction did not substantially change conclusions over the tested horizons; representation drift may matter more in longer or more disruptive training.Reconstruction required interrupting and resuming training in the current implementation, though prompt-level beliefs can be reassigned by prompt identity.Source: §J.3 · p. 58

J.4 Belief Bounds and Uncertainty Saturation

• 80% of prompts reached maximum uncertainty by training’s end, potentially making posterior beliefs less able to distinguish genuinely uncertain prompts from stale ones.The authors suggest sublinear staleness accumulation, a different saturating uncertainty transform, or a better-matched sigma-rule bound as future refinements.Source: §J.4 · p. 59
• The bounded update rule improved learning signal and effective ratio in main experiments, but both BMC and Difficulty Only saw learning signal decrease in medical-domain experiments.Possible explanations include dataset difficulty or reward structure, aggressive uncertainty growth, or a mismatch between the learning-signal statistic and that domain.Source: §J.4 · p. 59

K.1 Rollout Allocation

• Fixed rollout allocation isolates the effect of problem selection; future allocation could account for difficulty, diversity, and utility rather than difficulty or immediate productivity alone.Prior work studies adaptive rollout allocation for harder or more promising problems [Zeng et al., 2025, Li et al., 2025b, Yao et al., 2026, Cheng et al., 2026b].Source: §K.1 · p. 60

K.2 Prioritization Mechanisms

• BMC aggregates productivity across local tree neighborhoods, while future acquisition rules could prioritize productive problems that may unlock nearby, currently unproductive tasks.This frontier-expander idea assumes learnability propagates locally through the latent task manifold.Source: §K.2 · p. 60

K.3 Connections to the Bandit Literature

• Future work could apply structured, causal, combinatorial, pure-exploration, and budgeted bandits to information sharing, interventions, batch selection, frontier discovery, and compute-aware allocation [Lattimore et al., 2016, Chen et al., 2013, Bubeck et al., 2010, Tran-Thanh et al., 2012].These directions extend beyond prompt-level scalar values, simple uncertainty estimates, and difficulty-oriented sampling.Source: §K.3 · p. 60

L.1 Hyperparameters

• Table 4 details hyperparameters central to the experimental setup.Source: Table 4 · p. 61

L.2 Compute

• Experiments used a single H100 GPU node, approximately 32 CPU cores for tree construction, and matched hardware constraints within comparisons.Node interconnect varied across runs between Ethernet and InfiniBand.Source: §L.2 · p. 61

L.3 Round-Robin Initialization

• BMC and its ablations begin with a round-robin pass sampling each prompt once; BMC and Difficulty Only share initial beliefs, while Tree Only uses uniform beliefs.All three begin adaptive sampling from the same post-round-robin policy checkpoint; the pass reduces cold-start comparisons and is not required by the method.Source: §L.3 · p. 62

L.4 Evaluation Datasets

• GPQA-Diamond evaluates graduate-level scientific reasoning beyond the mathematical training distribution, using a selective subset of expert-written questions.GPQA covers biology, physics, and chemistry and is designed to challenge non-experts even with search access [Rein et al., 2023].Source: §L.4 · p. 62
• Mathematical evaluations span AIME 2024/2025, AMC 2023, MATH500, CNMO 2024, and CCEE 2024, covering competition and standardized-exam reasoning.AIME tests olympiad-style problems [Maxwell-Jia, 2024, yentinglin, 2025]; AMC problems are generally shorter and more accessible; MATH500 contains 500 problems; CNMO and CCEE are Chinese benchmarks.Source: §L.4 · p. 62

L.5 Dense-to-Sparse Evaluation Schedule; L.6 Evaluation Curves

• Evaluation is dense early, alternating training-set pass@1 and downstream mean@16 every 10 steps, then measures both every 50 steps after step 250.The accompanying curves cover Qwen3-8B and Qwen3-4B training on DAPO-Math-17k with GSPO or GRPO, plus BMC-T variations; late-stage results are coarse trends.Source: §L.5 · p. 63, Figure 24 · p. 63, Figure 25 · p. 64

M.1 Learning Signal; M.1.1 Quantifying the Learning Signal

• Reward variance serves as an inexpensive prompt-level proxy for informative learning signal, vanishing when all rollouts agree and peaking near 50% success for binary rewards.The proxy reflects within-prompt reward contrast, not the reward optimized by the policy; identical successes or failures provide little relative credit-assignment information.Source: §M.1.1 · pp. 67–68, §M.1 · p. 67

M.1.2 Normalizing the Learning Signal

• Normalizing learning signals to [0, 1] helps compare prompts across heterogeneous scales, though it is optional and unnormalized scale differences can skew sampling.Monotonic transformations to a shared range are also suitable; the Gaussian scheduler can represent unbounded signals, but substantial scale differences may cause imbalance.Source: §M.1.2 · p. 68

M.2 Logit-Normal Beliefs and the Sigma Rule

• BMC uses Gaussian beliefs in logit space to represent bounded learning signals, with a sigma rule bounding uncertainty and a minimum variance preserving revisitation.With eight rollouts, the clamp uses ϵ = 1/64 and rmax = log(63); experiments set the sigma-rule constant to a = 3.Source: §M.2 · pp. 68–69

M.3 Sampling Without Replacement

• Batch-aware hierarchical descent excludes already-selected prompts during sequential sampling, avoiding rejection loops and encouraging selection from other prompts or neighboring regions.Although batch-aware descent is the default, the main BMC experiments used rejection-based sampling because duplicate conflicts were rare.Source: §M.3 · p. 70

M.4 Saturating Estimation Errors

• BMC’s saturating error updates softly forget stale evidence and increase uncertainty after surprises, using bounded, smooth heuristics rather than claiming Bayesian optimality.The exponential prior-weight decay responds to large surprises, while logarithmic uncertainty growth limits amplification; these practical choices are described as simple and effective.Source: §M.4 · pp. 70–71

M.5 Staleness & Uncertainty Growth

• BMC increases uncertainty for recently unsampled prompts, scaling staleness by batch coverage so expected gaps with many arms and small batches accumulate more slowly.A logarithmic transformation makes uncertainty growth diminish rather than letting extreme staleness dominate.Source: §M.5 · p. 71

M.6.1 Rarity-Weighted Exposure

• Rarity-weighted exposure diagnoses whether sampling favors underrepresented clusters beyond dataset-prior expectations; it is not optimized, since maximizing it could ignore learning value.Rank-based weights give rarer clusters greater weight and are less sensitive to outlier cluster sizes than raw inverse-frequency weights.Source: §M.6.1 · p. 72

M.6.2 Structure Gain

• R2 = 0.00875 in the 8B setting: the true tree partition explains modest absolute learning-signal variation but more than size-matched random partitions.Structure gain is a relative alignment diagnostic, not evidence that the tree fully explains the signal.Source: §M.6.2 · p. 73

M.7.1 Implementation

• BMC-T uses target examples only to estimate locally smoothed sibling utilities—not as arms—and adds a support-gated utility bonus to productivity scores during top-down sampling.The tree uses both training and target examples; γ controls utility awareness, and γ = 0 does not necessarily recover standard BMC because target examples can still affect tree topology.Source: §M.7.1 · pp. 73–75

• BMC-T estimates target relevance from training–target overlap in the Latent Task Tree and adds a normalized utility bonus to productivity during top-down sampling.Productivity estimates where the policy can learn; utility estimates which learnable regions match the target distribution.Source: §M.7.2 · p. 75
• Utility could instead modulate productivity multiplicatively or use gradient-based alignment [Yang et al., 2026] or optimizer-based data valuation [Wang et al., 2026].These alternatives may help when target examples are sparse, unavailable, or difficult to combine with training data in a shared tree.Source: §M.7.2 · p. 75
• BMC-T is a proof of concept for representing evaluation relevance alongside productivity and diversity, not a unique prescription for utility-aware curriculum design.It samples target-relevant regions of the task manifold rather than relying on difficulty-based metrics alone.Source: §M.7.2 · p. 75