PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics

Changho Shin, David Alvarez-Melis

arXiv:2609.09099v1cs.LG

TL;DR

Curriculum effects are hard to isolate because difficulty, ordering, exposure, and pacing are coupled. The paper represents curricula as Wasserstein paths over difficulty levels, finding context-dependent outcomes: ordering, endpoint design, and pacing affect where a fixed budget is effective.

  • Problem

    Curriculum methods often couple difficulty definition, ordering, exposure, and pacing, making it difficult to identify which design choices account for observed effects.

    Source: §1 · p. 1

  • Method

    The paper represents curricula as Wasserstein paths of training distributions over difficulty levels, enabling controlled variation of ordering, exposure, endpoint smoothness, pacing, and geometry.

    Source: §1 · pp. 1–2

  • Results

    Across tasks, difficulty axes, and budgets, no single curriculum dominates; easy-to-hard ordering improves hard-level performance over exposure-matched static sampling, and endpoint design and pacing affect where curricula help.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The transport framework supports studying context-dependent curriculum effects and extensions to adaptive pacing through geometry and structured difficulty spaces.

    Source: §1 · p. 2, §9 · p. 9

    Most of the study is intentionally limited to from-scratch training and newly constructed tasks, leaving transfer to pretrained settings, natural data, and tasks interacting with prior knowledge open.

    Source: §9 · p. 10

Abstract

from arXiv · show

Curriculum learning is governed by several coupled design choices---how difficulty is defined, how examples are ordered, how much exposure each level receives, and how quickly training moves across levels---making it hard to isolate what actually helps. We present Wasserstein curriculum paths, a simple transport-based framework that decouples these factors by representing curricula as trajectories of training distributions over discrete difficulty levels. Across a calibrated synthetic suite with 12 tasks and 33 difficulty axes, we use this framework to isolate the effects of ordering, matched exposure, endpoint smoothness, and pacing under fixed training budgets. We find that curriculum effects are strongly context-dependent: no single strategy dominates across tasks, difficulty axes, and budgets, and curricula mainly change where a fixed budget is spent most effectively. Within this framework, easy-to-hard ordering improves hard-level performance relative to exposure-matched static sampling, showing that the benefit is not explained by cumulative exposure alone. We further show that endpoint smoothness and pacing substantially affect where along the difficulty spectrum a curriculum is effective. Finally, we show that the same transport view naturally supports extensions to learned pacing through geometry and to structured difficulty spaces beyond one-dimensional orderings.

1 Introduction

Curriculum effects are difficult to interpret because difficulty, ordering, exposure, and pacing are often coupled. The paper uses Wasserstein paths to study these choices separately and finds that their effects depend on task, difficulty axis, and budget.

1 Introduction

  • Wasserstein paths represent curricula as trajectories of training distributions over ordered difficulty levels, enabling controlled variation of design choices.The framework separates ordering, matched exposure, endpoint smoothness, pacing, and geometry.Source: §1 · pp. 1–2
  • The controlled study uses a calibrated synthetic suite spanning 12 tasks and 33 difficulty axes.The framework is also presented as supporting adaptive pacing through geometry and extensions to structured difficulty spaces.Source: §1 · p. 2
  • Across tasks, difficulty axes, and budgets, no single curriculum dominates; curricula mainly shift where a fixed training budget is used most efficiently.Smooth easy-to-hard progression is particularly efficient on the hardest levels.Source: §1 · p. 2
  • Easy-to-hard progression improves hard-level performance over exposure-matched static sampling, while hard-to-easy ordering often hurts.This indicates that timing matters beyond cumulative exposure alone.Source: §1 · p. 2
  • Endpoint concentration and traversal speed can change whether an easy-to-hard path helps or hurts.Very sharp endpoints are brittle under both overly aggressive and overly conservative pacing.Source: §1 · p. 2
  • The authors position Wasserstein curricula as a framework for studying when curriculum components matter and extending curriculum design.The extensions include adaptive pacing and structured difficulty spaces.Source: §1 · p. 2

2 Related Work

Prior curriculum approaches often combine choices about what examples to show and when, while optimal-transport work has used interpolation to construct intermediate training stages. This paper instead uses paths over a fixed difficulty axis to isolate curriculum components.

2 Related Work

  • Automated curricula use self-paced, teacher–student, bandit, and competence-based schedules, while other work studies difficulty measurement and when ordering helps.Recent ordering studies report that preferred direction can depend on budget, label noise, model capability, task complexity, and difficulty metric.Source: §2 · p. 2
  • Prior optimal-transport curriculum work constructs or sequences intermediate stages between task or environment distributions; this paper analyzes distributions on a fixed difficulty axis.Its focus is isolating ordering, exposure, pacing, and endpoint effects rather than primarily designing new curricula.Source: §2 · pp. 2–3
  • Figure 1 contrasts Wasserstein movement through intermediate levels with linear interpolation that directly mixes the endpoint distributions.The batch-level view shows more mass on intermediate levels during the Wasserstein transition.Source: Figure 1 · p. 3
  • Prior curriculum methods often vary several design choices together, and recent ordering studies do not provide a common framework for separating exposure, endpoints, pacing, and geometry.The paper focuses on which aspects of a curriculum matter given a difficulty structure.Source: §2 · p. 3

3 Preliminaries and Experimental Setup

The setup represents a curriculum as a path of batch-sampling distributions over ordered difficulty levels, with a separate schedule controlling traversal. A calibrated synthetic suite and shared training protocol support controlled comparisons.

3.1 Wasserstein–Geodesic Curriculum

  • At each training step, batches are sampled from a probability distribution over ordered difficulty levels, making a curriculum a sequence of such distributions.The path specifies visited mixtures, while the schedule maps training steps to positions along that path.Source: §3.1 · p. 3
  • Wasserstein interpolation moves probability mass through adjacent levels between endpoint distributions, unlike linear interpolation, which mixes the endpoints directly.Endpoints determine where a path starts and ends; level geometry determines which levels are nearby, and the schedule controls traversal speed.Source: §3.1 · p. 3

3.2 Common Experimental Setup

  • The common setup holds task construction, models, sampling protocol, and budget calibration fixed unless a later experiment varies them.The synthetic suite is used for controlled comparisons; real-data SFT results with pretrained models and noisier difficulty labels show weaker, harder-to-disentangle effects.Source: §3.2 · p. 4
  • The synthetic suite contains 12 tasks, with one difficulty axis varied at a time to form 33 task-by-axis conditions.The within-task axes order examples from easier to harder while keeping task semantics fixed.Source: §3.2 · p. 4
  • Most tasks use small decoder-only Transformers trained from scratch, while language-like tasks fine-tune SmolLM-135M.Source: §3.2 · p. 4
  • Budgets are calibrated separately for each benchmark so static i.i.d. learning curves span early, intermediate, and later learning stages.The default static baseline samples uniformly, while default easy-to-hard runs use reversed endpoint distributions and a linear schedule.Source: §3.2 · p. 4

4 When and How Does Curriculum Learning Help?

Across 33 task-by-difficulty-axis conditions, curriculum effectiveness depends on task, difficulty axis, and budget rather than yielding a universal winner. With a fixed budget, curricula also differ in where accuracy is most data-efficient across difficulty levels.

4.1 When Curricula Help Is Highly Context Dependent

  • No single curriculum dominates across tasks, difficulty axes, and budgets; the ranking shifts with budget.All three methods win substantial subsets of conditions.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2 Curricula Shift the Data-Efficiency Profile Across Difficulty

  • Wasserstein has the highest exposure-adjusted accuracy on the hardest bucket, while static i.i.d. leads on easiest-bucket accuracy and linear on the middle bucket.Wasserstein exceeds linear on hard-bucket exposure-adjusted accuracy in all 99 task-by-axis-and-budget conditions.Source: Figure 2 · p. 5, §4.2 · p. 6
  • With a fixed budget, curricula reshape where training pays off across difficulty rather than producing a uniform gain in final accuracy.Exposure-adjusted accuracy is final bucket accuracy divided by cumulative exposure to that bucket.Source: §4.2 · p. 6

5 Does Ordering Matter?

The section tests whether ordering matters when cumulative exposure is held fixed. Easy-to-hard progression retains hard-bucket gains that exposure-matched static sampling misses, while hard-to-easy ordering often harms performance.

5 Does Ordering Matter?

  • On the hardest level, forward Wasserstein exceeds reverse Wasserstein in 92 of 99 task-by-axis-and-budget conditions, while exposure-matched static sampling misses easy-to-hard Wasserstein’s hard-bucket gains.The exposure-matched static baseline preserves cumulative exposure but removes temporal progression; reverse Wasserstein preserves exposure while reversing progression direction.Source: §5 · pp. 6–7
  • Exposure-matched static sampling misses easy-to-hard Wasserstein’s hard-bucket gains, and hard-to-easy ordering often shifts strength toward easy levels and harms performance.Overall averages can remain similar even when the hard-bucket profiles differ.Source: §5 · p. 7

6 Pacing and Smoothness Matter

The section varies endpoint smoothness and traversal speed along an easy-to-hard Wasserstein path. A broad region of moderate pacing and smoother endpoints performs well, while very sharp endpoints are brittle under both overly aggressive and overly conservative pacing.

6 Pacing and Smoothness Matter

  • The sweep keeps the easy-to-hard Wasserstein path fixed and varies only endpoint smoothness and pacing.Smaller τ sharpens the endpoint distributions, while larger τ smooths them; γ controls traversal speed, with smaller values moving toward harder levels sooner.Source: §6 · p. 7
  • Moderate pacing and smoother endpoints perform well across the medium-budget sweep, while very sharp endpoints are brittle under overly aggressive or conservative pacing.The sweep covers 20 task-by-difficulty-axis conditions with three seeds; the default setting is strong but not uniquely optimal.Source: §6 · pp. 7–8
  • An affinely rescaled geometry stretches the early-to-mid region and compresses the hard end, making the path dwell longer before a sharper final handoff.Source: Figure 5 · p. 8

7 Adaptive Geometry as Learnable Pacing

WARP adapts pacing by changing the geometry between difficulty levels while keeping the easy-to-hard endpoints and direction. The prototype beats fixed Wasserstein at small and medium budgets and matches it at large budget.

7 Adaptive Geometry as Learnable Pacing

  • Changing the geometry changes pacing while preserving the same easy-to-hard endpoints and direction.Source: §7 · p. 8
  • After a short uniform warmup, WARP uses neighboring-level gradient alignment to set relative edge lengths: higher similarity shortens an edge and lower similarity lengthens it.Source: §7 · p. 8
  • WARP beats fixed Wasserstein at small and medium budgets and matches it at large budget.This proof-of-point result suggests adapting geometry can improve pacing along the same easy-to-hard path.Source: §7 · p. 8, Table 2 · p. 8

8 Structured Difficulty Spaces Beyond One-Dimensional Orderings

The section tests whether Wasserstein curricula extend to graph-structured difficulty spaces, using cube and tree benchmarks built from arithmetic operators. Results vary by graph, region, and budget rather than identifying one universally strongest method.

Setup

  • The experiments compare static i.i.d., Wasserstein, and WARP on cube and tree benchmarks using a 2-layer transformer over ten seeds.The benchmarks are structured difficulty spaces built from arithmetic operators.Source: §8 · p. 8

Results

  • On the cube, Wasserstein improves hard-endpoint L8, while static i.i.d. leads on easier regions and at medium and large budgets; on the tree, WARP leads at small budgets and Wasserstein overall at large budgets.The comparison uses ten-seed mean accuracy on the structured benchmarks.Source: Table 3 · p. 9, §8 · p. 9

Setup

  • Each benchmark node represents a level in the same local problem family, but the cube and tree differ in how those levels are connected.The reported effects vary across graph, region, and training budget.Source: §8 · p. 9

9 Conclusion

The conclusion presents Wasserstein interpolation as a controlled lens for isolating curriculum-design choices, while emphasizing that effects vary by context. It also bounds the evidence: most experiments use from-scratch training and constructed tasks, leaving transfer to pretrained and natural settings open.

Conclusion

  • The results do not support a universally best curriculum: ordering, endpoint design, pacing, and geometry matter differently across settings.Easy-to-hard ordering helps beyond cumulative exposure, while pacing and endpoint design change where a fixed budget is most effective.Source: §9 · p. 9

Limitations and Future Work

  • Most experiments use from-scratch training and newly constructed tasks, so transfer to pretrained settings, natural data, and tasks interacting with prior knowledge remains open.The paper also notes that real-data SFT effects weaken substantially with pretrained models, natural benchmarks, and noisier difficulty labels.Source: §9 · p. 10
  • The framework is presented as a basis for studying adaptive pacing and structured difficulty spaces, with learning geometry or pacing from transfer signals identified as future work.Source: §9 · pp. 9–10

Disclosure of LLM Use

The supplied passages report interactive LLM assistance under close author supervision and describe the paper’s related work and constructed task suite.

Disclosure of LLM Use

  • LLM assistants supported parts of task design, coding, exploratory analysis, plotting, and manuscript drafting or revision; authors made final decisions and verified experiments, analyses, figures, tables, and text.Source: Disclosure of LLM Use · p. 10

A Extended Related Work

  • Prior curriculum methods adapt example selection or pacing, while this framework represents curricula as distributional paths and separates the path from its traversal speed.The related-work discussion names self-paced, teacher–student, bandit, and competence-based approaches.Source: A Extended Related Work · p. 13
  • The paper uses Wasserstein geodesics as a common scaffold to compare endpoint choice, smoothness, pacing, and geometry, emphasizing empirical control rather than a new RL-specific scheduler.Source: A Extended Related Work · p. 13

B.1 Task and Dataset Details

  • The task suite varies binary-string parity, bracket completion, and grid completion problems, with difficulty axes including informative bits, string length, bracket types, missing suffix, and blanks.The k-parity design also evaluates unique slices that isolate newly introduced informative bits.Source: §B.1.1 · p. 14, §B.1.2 · p. 14, §B.1.3 · p. 15
  • Other tasks cover detecting invalid bracket strings, deterministic calendar scheduling, and arithmetic expression search, with difficulty varied through bracket types or length, schedule constraints, and number or target ranges.Source: §B.1.4 · p. 15, §B.1.5 · p. 16, §B.1.6 · p. 16
  • The suite also tests ownership updates, indexed span selection, and sequence reversal, varying people or exchange counts, source or output properties, ranges, alphabet size, and sequence length.Source: §B.1.7 · p. 17, §B.1.8 · p. 17, §B.1.9 · p. 18

B.1.10 Run-Length Encoding Dataset

  • Run-length encoding provides another deterministic task, with difficulty axes for sequence length, run count, alphabet size, and maximum run length.Source: §B.1.10 · pp. 18–19

B.1.11 Stack Operations Dataset

This section specifies the stack and queue benchmarks and training setup, then examines exposure, adaptive geometry, structured-space analyses, and variation in curriculum outcomes across budgets, tasks, difficulty axes, and levels. It also characterizes easy-to-hard progression by task family and semantic difficulty-axis category, including hardest-level performance.

B.1.11 Stack Operations Dataset; B.1.12 Queue Operations Dataset

  • Stack Operations tests LIFO behavior across operation-count, alphabet-size, maximum-depth, and pop-rate axes; Queue Operations tests FIFO behavior across operation-count, alphabet-size, and capacity axes.Both tasks report exact-match accuracy on the final stack or queue.Source: §B.1.11 · p. 19, §B.1.12 · p. 20

B.2 Architecture and Training Details

  • Most synthetic tasks use scratch decoder-only transformers with hidden size 256, four layers, and eight heads, while specified tasks use different model sizes or layer counts.Training generally uses AdamW with learning rate 5 × 10−5, weight decay 0.1, gradient clipping at 1.0, and a constant schedule without warmup.Source: §B.2 · p. 20

B.3 Level-Wise Cumulative Exposure

  • Wasserstein interpolation allocates more cumulative exposure to intermediate levels than uniform sampling or linear interpolation, improving sample efficiency at higher levels.The illustrated comparison uses γ = 1 and τ = 1 with the standard W1 distance on ordered levels.Source: §B.3 · p. 20, Figure 7 · p. 21

B.4 Adaptive Geometry in WARP

  • WARP learns nonuniform level spacing from cosine similarity between neighboring level gradients, then follows the same forward Wasserstein geodesic, changing pacing but not endpoints or direction.It begins with uniform sampling during a warmup of 5% of training.Source: §B.4 · p. 21

B.5 Details for Structured Difficulty Spaces

  • The structured difficulty variants use an 8-node operator-modification cube and a 7-node tree, each organized into easy, mid, and hard regions.The cube combines multiplication, parentheses, and negation; the tree branches first by multiplication and parentheses, then into longer-expression and negation leaves.Source: §B.5 · p. 21, Figure 8 · p. 22

C.1 Task-Wise Final-Accuracy Barplots; C.3 Task-Wise Final Level Profiles

  • Curriculum rankings shift across tasks, axes, and budgets, while level profiles reveal where methods trade easy-end accuracy for harder-end gains.The task-wise endpoint plots summarize final overall accuracy; the level profiles locate performance changes along each task’s difficulty levels.Source: §C.1 · p. 22, §C.3 · p. 29

C.2 Budget-Wise Exposure, Accuracy, and Exposure-Adjusted Accuracy

  • Across budgets, static i.i.d. leads on easy-end accuracy, linear on middle-bucket exposure-adjusted accuracy, and Wasserstein on hardest-bucket accuracy.Exposure-adjusted accuracy is final bucket accuracy divided by cumulative exposure to that bucket.Source: §C.2 · p. 27, Figure 21 · p. 28

C.4 Exploratory Analysis of When Easy-to-Hard Progression Helps

  • Easy-to-hard progression is strongest for Sequence Transformation tasks, is mixed for Data-Structure Execution, and often helps on Context Length and Entity Complexity axes.Formal Structure and Constraint and Search tasks also show substantial support, especially on the hardest level; axis categories still do not yield a single winning type.Source: §C.4 · p. 38
  • Answer Length has the strongest hardest-level rate (15/18), while Procedural Complexity is weakest overall but improves on the hardest level.The Answer Length result is driven by axes including Hidden suffix length, Maximum run length, and Selected-output length.Source: §C.4 · p. 39
  • Joint task-family and difficulty-axis patterns suggest easy-to-hard gains are more characterizable by task family than semantic axis, but remain only partially predictive.Figure 34 presents overall and hardest-level win rates across these task-family/category pairs.Source: §C.4 · p. 39, Figure 34 · p. 39
  • Sequence Transformation is strongest with Entity or Procedural Complexity axes, while Constraint and Search is strongest with Context or Answer Length axes.Constraint and Search shows little support on Procedural Complexity; Formal Structure is broadly positive across categories with fewer conditions, while Data-Structure Execution is weak across most cells.Source: §C.4 · p. 39

C.5 Level Transfer Experiments

The transfer experiments distinguish how easy-to-hard curricula help and test whether intermediate levels assist hard targets. The pretrained SFT analyses find weaker and less consistent curriculum advantages than the synthetic setting, with static i.i.d. strongest on average.

C.5 Level Transfer Experiments

  • Easy-to-hard transfer helps through both a target-stage head start and faster learning, with faster learning especially prominent on hardest targets.The fitted aggregate coefficients show both effects growing with source share, while hardest-target pairs lean further toward faster learning.Source: §C.5 · p. 40, Figure 36 · p. 41
  • Hard-to-easy transfer can yield larger gains on easy targets, but its later arrival leaves less margin there and does not offset Wasserstein’s larger hard-bucket gains.The directional comparison uses isolated transfer at full source share against matched no-transfer controls.Source: §C.5 · p. 41, Figure 37 · p. 41

C.6 Bridge Effect Experiments

  • Increasing bridge share raises mean final target accuracy from 23.49% to 25.31%, though the response is not perfectly monotone; bridge effects also positively associate with Wasserstein-over-linear hard-bucket gaps.The reported association is r = 0.37; the authors describe the pattern as consistent with intermediate levels splitting a hard jump into easier ones.Source: §C.6 · p. 42, Figure 38 · p. 42

D Real-World SFT Analyses

  • The real-world SFT analyses evaluate three base models across ARC, MMLU-STEM, and StrategyQA, then use a synthetic-pretraining analogue to probe weaker curriculum effects.The benchmark study follows the benchmark construction and difficulty labels from Hase et al. (2024).Source: D Real-World SFT Analyses · p. 42, §D.1 · pp. 42–43

D.1 Real-World SFT Benchmarks

  • Across all three models, static i.i.d. has the highest mean overall and hardest-level accuracy in every model-budget block; linear and Wasserstein win only isolated contexts.The tables summarize three curricula across 12 benchmark-by-axis contexts per budget block.Source: §D.1 · p. 43, Table 6 · p. 44, Table 7 · p. 44

D.2 Final Level Distributions, Bucket Profiles, and Remaining Efficiency Differences

  • In medium-budget SFT, linear is strongest in the middle bucket, while Wasserstein and WARP are strongest on the hard bucket, with modest, model-dependent gaps.Exposure shifts more than final accuracy across the five-position view, and exposure-adjusted accuracy is more informative than compressed raw bucket accuracies.Source: §D.2 · p. 45, Figure 40 · p. 46

D.3 Easy-to-Hard Ordering Effect Is Weak in Pretrained SFT

  • Across nine model-by-budget blocks, forward Wasserstein never has the highest hard-bucket mean, so easy-to-hard ordering shows no robust advantage under pretrained SFT.The comparison holds geometry fixed using reverse Wasserstein and an exposure-matched static baseline.Source: §D.3 · p. 47, Figure 41 · p. 47

D.4 Exploratory Analysis of When Easy-to-Hard Progression Helps

  • Remaining easy-to-hard wins cluster weakly around ARC and human-annotated difficulty axes, without a consistent pattern across overall and hardest-level accuracy.The reported analysis counts ties among the forward curricula.Source: §D.4 · p. 48, Figure 42 · p. 48

D.5 Controlled Synthetic Pretraining Analogue

The controlled synthetic-pretraining analogue varies pretraining while holding downstream tasks and evaluation fixed. As pretraining grows, curriculum differences and easy-to-hard ordering advantages shrink, but ordering effects persist and static i.i.d. does not uniformly lead on hardest-level accuracy.

D.5 Controlled Synthetic Pretraining Analogue

  • Static mean hardest-level accuracy rises from 84.7% under small pretraining to 93.7% under large pretraining, yet linear and Wasserstein still match or exceed it on many contexts.The intermediate medium-pretraining value is 91.5%.Source: §D.5 · p. 48, Table 9 · p. 49
  • As pretraining grows, easy–hard gaps shrink under both static i.i.d. and Wasserstein, while exposure-adjusted accuracy remains strongest for linear on middle levels and Wasserstein on hard levels.Under Wasserstein, the mean easy–hard gap falls from 5.5 points under small pretraining to 3.3 under large pretraining; easy-level saturation contributes to the narrowing.Source: §D.5 · p. 49, Figure 43 · p. 49
  • The Wasserstein–exposure-matched-static gap falls from 3.9 points under small pretraining to 0.2 under large pretraining, showing that ordering advantages weaken but persist.The Wasserstein–reverse gap also falls, from 13.5 points under small pretraining to 2.3 under large pretraining.Source: §D.5 · p. 50, Figure 44 · p. 50
  • The controlled analogue links increasing pretraining with weaker curriculum effects and suggests reduced room for improvement as one possible mechanism, while noting natural post-training data may also matter.The downstream split excludes examples seen during pretraining.Source: §D.5 · pp. 48, 50

Related papers

Loading 2609.09099v1…