PaperlayerView exact revision on arXiv

Source-linked AI summary

The Capability Manifold and ML Scaling Laws

Syed Ali Raza Zaidi, Maryam Hafeez

arXiv:2609.27588v1cs.LGcs.AI

TL;DR

Predictive loss does not directly characterize distinct downstream capabilities, and the paper frames this as a gap in relating capabilities to resources across the model lifecycle. It introduces a capability manifold with resource-sensitive capability mappings and Jacobians, then applies it to pre-training and test-time scaling. In that application, training and inference scaling correspond to different capability dynamics, with test-time search showing a transition near T > Tc = 1.8.

  • Problem

    Predicting downstream capabilities from scale is less direct than predicting loss, while models exhibit distinct functional capabilities across lifecycle stages.

    Source: §1.1 · p. 1, §3 · p. 7

  • Method

    The paper defines a multidimensional capability manifold using bounded resource transformations, capability-specific mappings, cross-resource interactions, and an analytical Jacobian.

    Source: §1.3 · p. 2, §1.2 · p. 2

  • Results

    Pre-training allocation dictates baseline reasoning velocity while Tmult governs emergent math velocity; search-heavy paths accelerate more steeply and transition near T > Tc = 1.8.

    Source: §3.2 · p. 9

  • Takeaways & Limitations

    The framework provides a basis for analysing efficient scaling, distillation, safety, and capability trade-offs in foundation models.

    Source: §3.3 · p. 9

    Capability definitions are operational coordinates for analysis, not prescriptions for universal benchmarks.

    Source: §3 · p. 7

Abstract

from arXiv · show

Existing machine learning (ML) scaling laws relate predictive loss to compute, model parameters, and data. However, as models are increasingly deployed through agentic harnesses, loss alone is insufficient to characterize downstream performance: models with similar loss can exhibit different capabilities in reasoning, retrieval, planning, and adaptation. Yet, no unified framework connects such capabilities to the coupled resources available across the ML lifecycle. We bridge this gap by introducing a capability manifold, a multidimensional framework mapping downstream capabilities to pre-training, post-training, and test-time resources through bounded scaling functions. Analytical Jacobians quantify capability sensitivity to resource changes and interactions. As an initial application, we embed Kaplan- and Chinchilla-type scaling laws and test-time compute within the framework, demonstrating how existing scaling relationships can be unified as trajectories on a common capability manifold.

1 Introduction

Scalar loss does not directly capture distinct model capabilities or how different lifecycle resources affect them. The capability manifold represents capabilities in a shared multidimensional framework for comparing resource configurations and their effects.

1.1 Motivation

  • Loss alone does not capture how pre-training, post-training, and inference-time resources change distinct model capabilities.Different capabilities may scale at different rates, saturate at different points, or depend on different resource combinations.Source: §1.1 · pp. 1–2

1.2 Capability Aware Design

  • The capability manifold represents model behaviour as a multidimensional capability vector linked to resource configurations, rather than a scalar loss.Its capability definitions provide a common coordinate system, not a universal benchmark prescription.Source: §1.3 · p. 2, §1.2 · p. 2

1.3 Contributions & Organisation

  • The framework formulates bounded resource transformations, capability-specific mappings, and cross-resource interactions, with a Jacobian measuring local capability sensitivity.The paper instantiates it with Kaplan and Chinchilla pre-training relationships alongside test-time scaling to compare alternative capability trajectories.Source: §1.3 · p. 2, §1.2 · p. 2

2 AI & ML Scaling Laws

The section reviews empirical scaling relationships across model lifecycle stages. It distinguishes pre-training, post-training, test-time scaling, and emergent improvements that may appear after a resource threshold.

Pre-training and post-training

  • Pre-training scaling relates parameters, data, and compute to model performance as resources increase.Post-training uses resources such as data, feedback, and optimization compute to improve task-related performance through fine-tuning, preference optimization, and reinforcement learning.Source: §2 · p. 3

Test-time scaling

  • Test-time scaling uses inference compute, context, tools, reasoning traces, or agentic interactions to improve baseline outputs at deployment.Unlike model construction, its scaling variables concern model utilization.Source: §2 · p. 3

Emergent behaviour scaling laws

  • Reasoning performance may not improve monotonically over time, and emergent scaling may appear when resources cross a threshold.Source: §2 · p. 3

2.1 Pre-training Scaling Laws

Pre-training scaling laws relate loss to model size, data, and compute, with empirical power-law relationships under controlled resources. Kaplan- and Chinchilla-type results characterize how these resources should scale together, while downstream capability may remain less predictable than loss.

2.1 Pre-training Scaling Laws

  • Pre-training laws quantify how loss changes with model size, data, and compute, but downstream capability is less predictable from pre-training loss than from size [6].Models with similar scale or loss can have different downstream capability profiles.Source: §2.1 · p. 3, §2.1.2 · p. 5

2.1.1 Parameter Scaling Law

  • A 10× increase in model size yields a 16% reduction in loss in the reported parameter-scaling example [4].The fitted exponent is approximately 0.076, and the scaling law applies when other resources are sufficiently large.Source: §2.1.1 · pp. 3–4

2.1.2 Data & Compute Scaling Laws

  • Kaplan et al. [4] report power-law loss scaling with data and compute, while Chinchilla scaling recommends scaling model size and data similarly to reduce cross-entropy loss [3].The reported data and compute relations span eight orders of magnitude in compute and two in data.Source: §2.1.2 · pp. 4–5
  • Kaplan et al. [4] describe joint loss dependence on N^0.74/D, implying model size must grow faster than data to avoid over-fitting.Their compute-optimal allocation is Nopt ∝ C^0.73 and Dopt ∝ C^0.27; Hoffmann et al. [3] later revisited the model-versus-data scaling conclusion.Source: §2.1.2 · pp. 4–5

2.2 Post-training Scaling Laws

Post-training transforms a pre-trained model using additional instruction, task, preference, or reward data, but unlike pre-training it lacks a single established loss law. Fine-tuning and preference-based scaling depend on resource combinations and task conditions.

2.2 Post-training Scaling Laws

  • Post-training transforms pre-trained parameters to optimize a post-training metric across model, data, feedback, compute, and optimization-step resources.Unlike the comparatively mature pre-training laws, post-training objectives may differ and lack a single loss metric.Source: §2.2 · p. 5

2.2.1 Fine-tuning Scaling

  • SFT loss scaling is multiplicative: resources and the fine-tuning method jointly affect loss, and scaling the pre-training model benefits fine-tuning more than pre-training data.For LoRA, adapted parameter count matters because pre-training model size is no longer an appropriate proxy for loss scaling; results are task- and data-dependent.Source: §2.2.1 · p. 6

2.2.2 Preference and Reinforcement Learning with Human Feedback (RLHF) Scaling

  • RLHF scaling has no universally established law; reported behavior includes power laws, saturation, and regime dependence.Preference-data quality and informativeness may matter as much as quantity.Source: §2.2.2 · p. 6

2.3 Test-time Scaling

Test-time compute adds resources during inference, with observed scaling across several inference strategies but no task-independent functional form.

2.3 Test-time Scaling

  • Empirical test-time scaling includes demonstrations, reasoning compute, repeated sampling, and search, with functional form dependent on task and inference algorithm.These relationships are summarized in Table 1.Source: §2.3 · p. 6

2.4 Emergence and Phase Transition Scaling

Emergence and phase-transition scaling describe capabilities appearing once a model exceeds a critical threshold. The discussion then turns from learning mechanics toward mapping them to downstream capability measures.

2.4 Emergence and Phase Transition Scaling

  • A capability may be absent in smaller language models and appear once model size exceeds a critical threshold [8].The passage also points to an alternative way of inspecting the phase-transition threshold [10].Source: §2.4 · p. 6
  • The section frames the next step as mapping known learning mechanics to downstream capability measures.Source: §2.4 · p. 7

3 Capability Manifold

The section replaces loss-only scaling with a framework relating multidimensional capabilities to resources across the model lifecycle. It formalizes this capability manifold and applies it to compare pre-training and test-time scaling trajectories.

3 Capability Manifold

  • The capability manifold treats operationally defined capabilities as a common coordinate system linked to pre-, post-, and test-time resources, not as universal benchmark prescriptions.The capability vector is given as C = [CK, CR, CP, CT, CM, CA, CC]T.Source: §3 · p. 7

3.1 Mathematical Formulation

  • Its bounded scaling formulation weights resource contributions, permits cross-resource interactions and alternative mappings, and uses a capability Jacobian to characterize scaling.A different function can replace a power-law scaling function when that relation does not hold.Source: §3.1 · pp. 8–9

3.2 Application of Framework

  • The application computes analytical Jacobians for N, D, and T across Kaplan and Chinchilla paradigms with search intensities Tmult ∈ {1.20, 0.80, 0.20}.Test-time search is capped at 10^5 in the stated setup.Source: §3.2 · p. 9
  • Search intensity governs emergent math velocity while pre-training sets baseline reasoning velocity; search-heavy paths accelerate more steeply, and test-time search triggers a phase transition near T > Tc = 1.8.Pure pre-training scaling instead drifts into a capability mismatch zone.Source: §3.2 · p. 9

3.3 Conclusion

  • The conclusion presents the manifold as a basis for analyzing efficient scaling, distillation, safety, and capability trade-offs in foundation models.It frames these analyses as future uses of the framework.Source: §3.3 · p. 9

NeurIPS Paper Checklist

The checklist reports simulation uncertainty and compute details, alongside statements about ethics, broader impacts, assets, and research practices.

7. Experiment statistical significance

  • Simulation plots include shaded error bands representing standard deviation across independent random seeds.The checklist answer is yes.Source: NeurIPS Paper Checklist · p. 11

8. Experiments compute resources

NeurIPS Paper Checklist

  • The paper states that its research adheres entirely to the NeurIPS Code of Ethics.The checklist answer is yes.Source: NeurIPS Paper Checklist · p. 12
  • The paper says broader impacts are discussed in Section 6 and that it releases no high-risk models or sensitive datasets.The broader-impacts checklist answer is yes; safeguards are marked not applicable.Source: NeurIPS Paper Checklist · p. 12

12. Licenses for existing assets

  • Software packages and baseline code frameworks are credited with standard open-source licenses, while no new dataset or standalone software library is introduced.The license checklist answer is yes and the new-assets item is marked not applicable.Source: NeurIPS Paper Checklist · p. 12

NeurIPS Paper Checklist

  • The paper involves neither crowdsourcing nor human-subject research, and LLMs are not a core part of its methodology or pipeline.The corresponding checklist items are marked not applicable.Source: NeurIPS Paper Checklist · p. 12
Loading 2609.27588v1…