PaperlayerView exact revision on arXiv

Source-linked AI summary

Telescopic Language Models

Zhilin Guo, Boqiao Zhang, Hakan Aktas, Kyle Fogarty, Nursena Koprucu Aslan, Wenzhao Li, Canberk Baykal, Albert Miao, Siyu Hong, Yixiao Liu, Adam Wu, Ashish Kumar Singh, Sakar Khattar, Chenliang Zhou, Weihao Xia, Cristina Nader Vasconcelos, Cengiz Oztireli

arXiv:2609.35769v1cs.CLcs.AI

TL;DR

Serving many compute budgets commonly requires separate training or compression runs, while fixed-exit nested models leave untrained depths unusable. TLM trains a nested Transformer with stochastic prefix supervision and a full-capacity anchor; on a 200M proxy suite, it yields usable models at all twenty prefixes and reduces AULB by 43–44% versus fixed-exit suites.

  • Problem

    Serving different compute budgets often requires a separate model training or compression run for each point, while fixed-exit nested models leave other depths untrained.

    Source: §1 · p. 1

  • Method

    TLM trains one randomly sampled depth prefix against the full target alongside the full model, using a distribution over prefix depths and a full-capacity anchor.

    Source: §3.1 · p. 4, §3.2 · p. 4

  • Results

    A 43–44% reduction in AULB (3.28 vs. 5.73–5.90) accompanies usable TLM models at all twenty prefixes; at full capacity, quality is on par with MLMS at ∼12% lower GPU cost.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The prefix sampling distribution controls the trade-off between coverage across depths and quality at selected exits, making operating points a training-time choice.

    Source: §1 · p. 2, §3.2 · p. 4

    Evidence is at 200M proxy scale on FineWeb-Edu, with one seed per arm, and whether the peak–continuum separation holds at 8B–70B remains open.

    Source: §5 · p. 9

Abstract

from arXiv · show

One deployed language model must often serve many compute budgets, yet serving each budget still means a separate training or compression run per point. We train a Telescopic Language Model (TLM) to be that continuum: a nested-capacity Transformer supervised by stochastic prefix supervision with a full anchor. At every step, one randomly truncated prefix of the capacity axis is trained against the full next-token target, alongside one full-capacity pass, so the trained artifact is a valid language model at every depth. Two forward-backward passes per step, no architectural change, nothing extra at inference. Fixed-exit suites such as Matryoshka Language Model Suites (MLMS) occupy one point in this design space, and the point has a cost: supervising only a few fixed exits leaves the nested model at chance level everywhere else (perplexity 10^2-10^5 in our baselines). On a 200M proxy suite (20B FineWeb-Edu tokens, identical data stream for all methods), a single TLM run is a valid language model at every one of its twenty layer prefixes, in perplexity and on perplexity-sensitive downstream tasks, reducing the area under the quality-budget curve by 43-44% relative to the fixed-exit suites while matching them at full capacity, at ~12% lower GPU cost per run. The prefix sampling density is a dial: concentrating it on a few depths recovers fixed-exit quality there at the price of the continuum, so the operating points become a training-time choice rather than an architectural one. These results indicate that the training objective, not the nesting itself, is what makes a model elastic.

1 INTRODUCTION

Serving different compute budgets typically requires separate training or compression runs, while fixed-exit nested models leave other prefixes unusable. TLM applies stochastic prefix supervision with a full anchor to an unchanged cascade, producing usable operating points across its depths with a sampler-controlled trade-off.

1 INTRODUCTION

  • Nested deployment addresses budgets spanning devices and latency needs, where separate models multiply training cost and post-hoc compression products trail models trained natively at their size.The paper frames stochastic prefix supervision with a full anchor as a broader training principle for ordered-capacity models.Source: §1 · pp. 1–2
  • TLM samples one prefix per step against the full target alongside a full-model pass, using the unchanged cascade and two forward-backward passes per step.The trained artifact can be truncated directly at any depth.Source: §1 · pp. 1–2
  • A 43–44% reduction in AULB (3.28 vs. 5.73–5.90) accompanies usable TLM models at all twenty prefixes, versus MLMS’s off-exit collapse.With uniform sampling, TLM perplexity falls from 81 at k=1 to 15 at k=20.Source: §1 · p. 2
  • Ten TLM operating points extend below MLMS’s 50M exit, reaching 0.11× full latency at PPL 26–81 where MLMS prefixes collapse.The Figure 1 schematic contrasts separate standalone models, three trained MLMS exits, and a TLM continuum from one run.Source: §1 · p. 2, Figure 1 · p. 2
  • At full 200M capacity, uniform TLM reaches 14.99 perplexity, within the MLMS suites’ 14.96–15.42 span; concentrating sampling on exits trades continuum coverage for exit quality.A single exit-focused seed reaches 14.65, a 2.2% edge over MLMS within seed noise, and narrows the gap to a vanilla twin to 4.8% vs. 7.1%.Source: §1 · p. 2
  • The uniform TLM run costs 131 GPU-hours, about 12% less than the 149-hour distillation-free MLMS suite and 1.7× less than the 216-hour vanilla three-model suite, while delivering twenty rather than three operating points.The continuum therefore does not require a separate training run for each budget.Source: §1 · p. 2

2 RELATED WORK

Related work spans nested representations, elastic networks, early exits and random-depth training, and post-hoc compression. TLM combines a sampled capacity prefix with full-target supervision and a full-capacity anchor along a nested language-model depth axis.

Nested and Matryoshka models

  • MatFormer reports usable never-trained intermediate widths, while MLMS trains per-exit heads; TLM instead samples depth prefixes with a full-capacity anchor.The paper attributes MLMS off-exit collapse largely to readout miscalibration: its trained trunk is paired with a head that did not see that prefix.Source: §2 · p. 3

Slimmable networks

  • Slimmable networks, SortedNet, Once-for-All (Cai et al., 2020), and Flextron (Cai et al., 2024) supervise multiple configurations, whereas TLM trains one sampled prefix plus the full model against the task target.Once-for-All additionally distills from the model’s full output.Source: §2 · p. 3

Early exits and random depth

  • Early-exit methods target fixed intended operating points, while random-depth methods such as LayerDrop (Fan et al., 2020) drop layers in a shared-head stack without TLM’s full-capacity anchor.TLM samples a prefix of a width-growing cascade against the full task target; its prefixes can also serve as speculative-decoding drafts (Leviathan et al., 2023).Source: §2 · p. 3

Other budget axes and post-hoc compression

  • MQT-LLaVA samples token budgets and MatryoshkaKV (Lin et al., 2025) samples KV-cache ranks, but neither combines sampled budgets, a full anchor, and full-target matching on an ordered capacity axis.Post-hoc pruning or per-size distillation derives smaller models after training, one compression pass per target.Source: §2 · p. 3

The same objective elsewhere

  • Stochastic prefix supervision with a full anchor was previously applied to 3D Gaussian splatting; TLM transfers the principle to nested Transformer blocks and language-model capacity.The passage links this setting’s expensive per-budget evaluation to the two-pass design’s cost advantage over per-exit supervision.Source: §2 · p. 3

3 METHOD

The method keeps the nested-capacity architecture but replaces fixed-exit supervision with stochastic prefix supervision and a full-model anchor. It evaluates every integer depth using a quality–budget curve and summarizes quality across depth and throughput.

3.1 NESTED-CAPACITY ARCHITECTURE

  • The unchanged cascade stacks Transformer sub-models of increasing width and depth, with each prefix equipped to act as a standalone language model.The method uses the MLMS architecture unchanged.Source: §3.1 · p. 4

3.2 STOCHASTIC PREFIX SUPERVISION WITH A FULL ANCHOR

  • Each step samples one prefix depth, trains it against the full next-token target, and adds a full-model loss, requiring two forward-backward passes.The sampled depth is redrawn from π each step, giving every prefix an unbiased gradient signal over training; optional distillation is not required.Source: §3.2 · pp. 4–5
  • The full-model anchor gives the largest configuration a full-target gradient every step, while shared-trunk prefix gradients can still leave a gap to an independently trained twin.The anchor is designed to protect full-size quality; the residual difference is described as the price of sharing the trunk.Source: §3.2 · p. 4

3.3 THE CONTINUOUS QUALITY–BUDGET CURVE

  • AULB averages token NLL across integer depths, while LODA measures normalized area under the quality–throughput envelope using valid operating points.AULB weights depth equally; LODA weights throughput equally and credits only valid operating points.Source: §3.3 · p. 5
  • Training uses two passes per step regardless of the number of supervised depths, and inference at any prefix costs the cumulative FLOPs of its first k blocks.Unlike fixed-exit suites, any depth is available from one training run.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments compare nested-model objectives on a 200M proxy suite trained on 20B tokens with identical data streams. Uniformly sampled TLM prefixes form a continuum, while fixed-exit suites collapse off-exit; sampler concentration trades coverage for exit quality.

4.2 RESULTS & ANALYSIS

  • AULB falls from 5.73–5.90 for MLMS to 3.28 for TLM, a 43–44% reduction; TLM’s perplexity decreases smoothly from 81 at k=1 to 15 at k=20.Figure 2 shows validation perplexity across all prefixes; MLMS collapses between its trained exits.Source: §4.2 · p. 6, Figure 2 · p. 7
  • TLM wins on both quality–throughput frontier measures, while serving ten points below MLMS’s smallest exit where MLMS has no valid model.Figure 3 depicts validation perplexity and mean seven-benchmark accuracy; its LODA measures area under each quality–throughput envelope.Source: Figure 3 · p. 7
  • The uniform TLM arm matches MLMS at full size: 14.99 perplexity versus the suites’ 14.96–15.42 span.Exit-grid sampling reaches 14.65 PPL, but its full-size edge over MLMS is not decisive given single-seed suites.Source: §4.2 · pp. 7–8
  • At trained exits, MLMS retains lower perplexity: 21.89 versus 24.38 at 50M and 17.47 versus 19.11 at 100M.At the intermediate k=13 depth, TLM-uniform reaches 22.6 PPL while MLMS collapses to 285.Source: §4.2 · p. 8
  • A uniform TLM run costs 131 GPU-h versus 149 for distillation-free MLMS, about 12% less; per operating point, costs are 6.5 versus 50–72 GPU-h.The comparison is billed GPU-hours on one A100; TLM’s twenty points include off-frontier prefixes.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 ABLATIONS

  • Moving sampler mass toward exit depths improves exit quality but sacrifices off-exit coverage; AULB is 3.45 for log-uniform, 3.28 for uniform, and 6.21 for exit-grid.The exit-grid arm’s off-exit k=5 prefix reaches 2,693 PPL; full-size quality is 14.99/14.99/14.65 across the three samplers.Source: §4.3 · pp. 8–9
  • Without the full anchor, 200M perplexity rises to 22.32 while continuum AULB remains 3.41; pass-matched controls attribute the effect to the full-depth gradient.The tested ingredient arms use a log-uniform base sampler; main-text TLM results use the anchor and no distillation.Source: §4.3 · p. 9
  • Log-uniform sampling effectively omits k=1, which reaches 6,740 PPL, while uniform sampling trains every prefix and gives the best AULB.Log-uniform is better at k=2 (40.0 versus 54.7 PPL), its intended small-prefix niche.Source: §4.3 · p. 9

5 CONCLUSION

A single TLM run serves all twenty layer prefixes as usable language models, with full-capacity quality on par with MLMS at about 12% lower GPU cost. Evidence is limited to a 200M FineWeb-Edu proxy setting, and whether the peak–continuum separation holds at 8B–70B remains open.

5 CONCLUSION

  • All twenty layer prefixes remain usable, and the deployment frontier improves: LODA rises 1.8× and AULB falls 43–44%.At full capacity, quality is on par with MLMS at about 12% lower GPU cost.Source: §5 · p. 9
  • The evidence is at 200M on FineWeb-Edu, with one seed per arm; whether peak–continuum separation holds at 8B–70B remains open.Second seeds for the two headline TLM arms and paired bootstrap confidence intervals corroborate the main gaps.Source: §5 · p. 9
  • The continuum’s advantage shrinks on a parameter-count budget axis because prefixes share the embedding and head.The reported budgets are latency and throughput.Source: §5 · p. 9
  • The reported frontier varies depth at fixed width; width, 3B scale, and speculative decoding are left as next steps.Serving fixes a depth per request; switching depth within a generation would require truncating or extending the KV cache across layers, a systems question left for future work.Source: §5 · p. 9

AI-USE STATEMENT

The authors used generative AI tools under human direction to assist with writing, literature checks, figures, ideation, and experiment design. They report that experimental results came from logged runs and that authors reviewed the AI-assisted work and take responsibility for the final content.

AI-USE STATEMENT

  • AI tools assisted writing, manuscript editing, literature and reference checks, figure creation, research ideation, and experiment design under human direction.The authors reviewed all AI-assisted work and checked quantitative claims against recorded metrics.Source: AI-USE STATEMENT · p. 10
  • The authors did not use generative AI to generate synthetic datasets.All experimental results came from the paper’s logged training and evaluation runs.Source: AI-USE STATEMENT · p. 10
  • The authors take responsibility for the final content, including AI-assisted text, claims, and artifacts.Source: AI-USE STATEMENT · p. 10

ETHICS STATEMENT

The work trains language models on public web-text data and reports no human subjects or new data collection. It discloses experiment energy costs as GPU-hours and notes broad positive implications from more efficient training.

ETHICS STATEMENT

  • The study uses FineWeb-Edu, a public web-text dataset, and involves no human subjects or new data collection.Source: ETHICS STATEMENT · p. 10
  • The energy cost of all experiments is disclosed as GPU-hours in Appendix D.Source: ETHICS STATEMENT · p. 10
  • The authors note broad positive implications for environmental costs from more efficient multi-budget model deployment and no specific dual-use concerns beyond language modeling generally.Source: ETHICS STATEMENT · p. 10
Loading 2609.35769v1…