PaperlayerView exact revision on arXiv

Source-linked AI summary

MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining

Haoyu Dong, Pengkun Zhang, Mingzhe Lu, Yanzhen Shen, Guolin Ke

arXiv:2509.06806v6cs.CLcs.AI

TL;DR

LLMs struggle to learn standard ML tasks from many in-context demonstrations, despite their broad knowledge and reasoning abilities. MachineLearningLM uses continued pretraining on synthetic SCM-driven tasks with a random-forest warm start and token-efficient prompting. It surpasses strong LLM baselines by an average of about 15% on high-shot out-of-distribution tabular classification and approaches random-forest-level accuracy across hundreds of shots.

  • Problem

    LLMs often fail to learn standard ML tasks from many-shot demonstrations, limiting their ability to use in-context examples for prediction.

    Source: §1 · p. 2

  • Method

    MachineLearningLM continues pretraining on millions of synthetic SCM-based tasks, using a random-forest teacher and then in-context demonstrations to predict unseen instances without parameter updates.

    Source: §1 · p. 3

  • Results

    By ∼15% on average under high-shot settings, MachineLearningLM surpasses leading open- and closed-source LLMs and reaches random-forest-level accuracy within 2% relative on average from 8 to 512 shots.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The model retains general chat capabilities, attaining 75.4% micro accuracy on MMLU in 50-shot evaluation.

    Source: §1 · p. 4

    Pretraining focuses on SCM-synthesized tabular classification, caps classes at K ≤10, and does not yet cover regression, ranking, time-series forecasting, or structured prediction.

    Source: §5 · p. 16

Abstract

from arXiv · show

Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.

1 INTRODUCTION

LLMs often fail to learn standard ML tasks from many-shot demonstrations, while specialized tabular learners lack the broad knowledge and multimodal perception of general-purpose LLMs. MachineLearningLM combines continued pretraining on synthetic tasks with efficient prompting, improving many-shot tabular prediction while retaining broader chat capabilities.

1 INTRODUCTION

  • LLMs often plateau after few demonstrations and remain sensitive to label biases and example order, while tabular-specific models lack broad LLM knowledge and multimodal perception.MachineLearningLM addresses the question of teaching an LLM to do ML in context while preserving general abilities.Source: §1 · p. 2
  • MachineLearningLM uses LoRA-based continued pretraining on millions of SCM-generated tabular tasks, then predicts unseen instances from demonstrations without parameter updates.The synthetic training tasks are designed not to overlap with evaluation datasets.Source: §1 · p. 2
  • A random-forest teacher stabilizes training by first matching its example predictions before the model transitions to self-reliant in-context prediction.The teacher's interpretable decision paths align with chain-of-thought reasoning, although this work distills predicted labels only.Source: §1 · p. 3
  • Tabular serialization, integer normalization to [0, 999], and sequence-level batch prediction enable 3–6× more examples per context and 50× amortization via batch inference.Integer encoding reduces tokenization fragmentation and avoids decimal comparisons as strings.Source: §1 · p. 3
  • At inference, confidence-weighted self-consistency aggregates predictions across shuffled demonstration and feature orders to address long-context position effects.The method combines outputs through a probability-weighted vote over samples.Source: §1 · p. 3
  • By ∼15% on average under high-shot settings, MachineLearningLM surpasses leading open- and closed-source LLMs and reaches random-forest-level accuracy within 2% relative on average from 8 to 512 shots.It also generalizes from a 32k-token pretraining budget to a 131k-token inference budget.Source: §1 · p. 3
  • The model handles natural-language features alongside numerical values without text bucketing or embeddings and retains chat-style general abilities, scoring 75.4% micro accuracy on MMLU in 50-shot evaluation.The introduction reports 73.2% micro accuracy in 0-shot MMLU evaluation.Source: §1 · pp. 3–4

2 METHOD

MachineLearningLM builds synthetic SCM tasks and trains an autoregressive predictor, using a Random Forest warm-up before transitioning to independent in-context prediction.

2.1 PRETRAINING CORPUS SYNTHESIS

  • Thirty percent of structural functions are replaced with multi-output gradient-boosted regressors, while categorical features and class labels are sampled or transformed to vary task structure.Class IDs are permuted, and class imbalance arises naturally from sampled bounds.Source: §2.1 · p. 4
  • Approximately 3 × 10^6 SCMs generate i.i.d. classification tasks with 5–50 features, 2–10 classes, 50 queries, and up to 1,024 demonstrations.Tasks vary in graph structure, mechanisms, feature types, and class formation.Source: §2.1 · p. 4

2.2 OBJECTIVE OF CONTINUED PRE-TRAINING

  • Continued pretraining uses the standard left-to-right log-likelihood objective on a JSON prediction target, encouraging feature-to-label mapping, ID alignment, and valid output formatting.The input combines an instruction/schema header, labeled demonstrations, and unlabeled queries.Source: §2.2 · p. 5

2.3 RANDOM-FOREST MIMIC WARM START

  • A Random Forest teacher supplies per-example predictions during warm-up; task-level guards and a temporary consensus filter screen supervision before the model develops standalone in-context policies.The consensus filter keeps only evaluation examples where the teacher prediction matches ground truth, without changing labels; it is disabled after warm-up.Source: §2.3 · p. 6

2.4 TOKEN-EFFICIENT PROMPTING FOR IN-CONTEXT ML

Compact tabular rows, integer-normalized numbers, and sequence-level batching reduce prompt costs for many-shot in-context prediction. The section reports savings in token cost and explains the trade-offs of batching.

2.4 TOKEN-EFFICIENT PROMPTING FOR IN-CONTEXT ML

  • The prompting design addresses fixed context length and long-context computation through compact example structure, numeric-token compression, and sequence-level batching.These choices target the cost of applying many-shot ICL.Source: §2.4 · p. 7

2.4.1 TABULAR ENCODING

  • Comma-delimited rows organize demonstrations compactly, with commas separating features and a vertical bar separating the label; the header is optional.This replaces scattered natural-language descriptions while retaining column and field meaning.Source: §2.4.1 · p. 7

2.4.2 COMPACT INTEGRAL-BASED NUMBER ENCODING

  • Mapping normalized numeric values to integers in [0, 999] preserves ordinal structure while reducing token fragmentation and avoiding textwise decimal comparisons.The encoding uses z-score normalization before mapping; GPT’s cl100k_base represents each mapped number as a single token.Source: §2.4.2 · pp. 7–8

2.4.3 SEQUENCE-LEVEL BATCH-PREDICTION

  • Sequence-level batching predicts N query samples in one forward pass, amortizing instruction and context overhead, but larger N reduces demonstration capacity and can increase instruction-following errors.The default is N = 50, and query order is randomized to reduce position bias and improve exchangeability.Source: §2.4.3 · p. 8

2.4.4 COMPRESSION AND AMORTIZATION RATIO

  • The amortized token cost per predicted label divides the shared header and row costs across M demonstrations and N queries, assuming the header cost is negligible.Increasing N amortizes the many-shot block; the model represents each row’s cost by R and omits H from the estimate.Source: §2.4.4 · pp. 8–9
  • With delimiters, number normalization yields 2.75× savings for GPT (cl100k_base) and LLaMA-3, and 1.88× for Qwen and LLaMA-2; combined compression estimates are 5.5× and 3.2×, respectively.Structural formatting and number normalization compound to reduce per-row token cost.Source: §2.4.4 · p. 9
  • 136× improvement in amortized token cost per inference sample is observed using Qwen’s tokenizer when comparing the proposed encoding with TabuLa’s.The evaluation compares amortized token cost per inference sample for the two encodings.Source: §2.4.4 · p. 10

2.5 ORDER-ROBUST, CONFIDENCE-AWARE SELF-CONSISTENCY AT INFERENCE TIME

The inference procedure creates shuffled prompt variants to diversify predictions, then aggregates their label probabilities to select the most consistent prediction.

2.5 ORDER-ROBUST, CONFIDENCE-AWARE SELF-CONSISTENCY AT INFERENCE TIME

  • Shuffling demonstrations or features across prompt variants supplies diversity, replacing stochastic sampling of reasoning paths from identical prompts.The procedure is presented as a modification of self-consistency in both its source of diversity and aggregation method.Source: §2.5 · p. 10
  • For each label, the method sums its next-token probabilities across the shuffled prompt variants.Each variant’s probability is extracted for the token representing each possible label.Source: §2.5 · p. 10
  • The prediction is the label with the highest aggregate probability; experiments use V = 5 variants, compared with 32 for TabICL and TabPFN.The variant count is described as far fewer than the 32 used by those methods.Source: §2.5 · p. 10

3 EXPERIMENTS

The experiments evaluate MACHINELEARNINGLM on TALENT classification tasks against general-purpose LLMs and tabular learners, then examine its scaling, general-chat retention, and performance across task types. Results show strong gains over vanilla LLMs and random-forest-level accuracy, alongside limits on some high-shot, high-cardinality, and forecasting tasks.

3.2.1 MACHINELEARNINGLM VS. VANILLA LLMS

  • Across tasks, MACHINELEARNINGLM improves Qwen-2.5-7B-Instruct accuracy by ∼15% and, at 128–1,024 shots, surpasses GPT-5-mini by ∼12% and o3-mini by ∼16% on average.Accuracy gains occur across domains, with virtually no task showing marked degradation; the model also scales monotonically with more demonstrations.Source: §3.2.1 · p. 11, Table 2 · p. 12, §3.2.2 · p. 12

3.2.2 MANY-SHOT SCALING

  • MACHINELEARNINGLM accuracy increases monotonically from 23 to 210 demonstrations within a 131k-token inference budget (4× its 32k-token pretraining budget); o3-mini improves ∼1.8% and GPT-5-mini ∼4.7%.Source: §3.2.2 · p. 12

3.2.5 MACHINELEARNINGLM VS. TABULAR ICL MODELS (TABICL / TABPFN)

  • MACHINELEARNINGLM can trail TabICL modestly at very high shot counts, largely because of LLM compute requirements, while remaining robust to demonstration order.Unlike specialized tabular ICL models, it retains compatibility with external knowledge, heterogeneous inputs, and reasoning-style alignment.Source: §3.2.5 · p. 12, Table 2 · p. 12

3.2.4 COMPETITIVE MANY-SHOT PERFORMANCE VS. STATE-OF-THE-ART TABULAR-SPECIFIC METHODS

  • Without task-specific training, MACHINELEARNINGLM reaches random-forest-level accuracy across 8–512 shots, typically within 2% relative, and beats random forest on roughly 30% of tasks at 512 shots.It also surpasses kNN by >4% relative on average.Source: §3.2.4 · p. 12, Table 3 · p. 13

3.2.3 MACHINELEARNINGLM VS. TABULA-8B

  • MACHINELEARNINGLM supports an order of magnitude more demonstrations per prompt than TabuLa-8B, which is typically limited to an 8k context and often 20–32 shots.The model scales stably to 1,024 shots for most tasks; at 32 shots it also outperforms TabuLa-8B on average.Source: §3.2.3 · p. 12, Table 3 · p. 13

3.2.6 MACHINELEARNINGLM IN GENERAL CHAT WORKFLOWS

  • On MMLU, MACHINELEARNINGLM achieves 73.2% accuracy at 0-shot and 75.4% at 50-shot, comparable to strong general-purpose LLMs.Numeracy-heavy subjects show consistent gains; TabuLa-8B underperforms on MMLU by ∼10%.Source: Table 4 · p. 14, §3.2.6 · p. 14

3.3 EMPIRICAL STUDY

  • MACHINELEARNINGLM remains competitive on numeric-dominant tables and consistently beats vanilla LLMs on mixed text-and-numeric datasets, but falls short on highly symbolic text fields.On imbalanced datasets it tracks random-forest-level performance without majority-class collapse.Source: §3.3 · pp. 14–15
  • Performance falls behind strong tabular baselines on tasks with more than 10 classes and on forecasting tasks, which require temporal inductive bias and higher numeric precision.The authors attribute the high-cardinality weakness to pretraining on tasks with K ≤ 10 classes, which can make labels such as “12” multi-token.Source: §3.3 · pp. 14–15

4 RELATED WORK

Related work spans many-shot ICL, language-model-based tabular learners, specialized tabular ICL models, and agents that call external ML tools. MACHINELEARNINGLM is presented as an LLM-based learner that performs tabular prediction through ICL, combining synthetic-task pretraining with the general capabilities of its backbone.

Many-shot ICL and scaling laws

  • Prior work reports that adding demonstrations can yield diminishing or negative returns, while MACHINELEARNINGLM uses synthetic tabular pretraining to produce monotonic example-count scaling.Related approaches include reinforced or unsupervised ICL (Agarwal et al., 2024), DrICL (Zhang et al., 2025), and semisupervised many-shot ICL (Gu et al.).Source: §4 · p. 15

LM-based ML learner

  • Unlike TabLLM, which requires task-specific fine-tuning and hyperparameter tuning, MACHINELEARNINGLM adapts to new tabular tasks through ICL without inference-time gradient updates.Compared with TabuLa-8B, it targets up to 1,024 demonstrations and uses synthetic SCM-driven tasks with controlled diversity.Source: §4 · p. 15

In-context tabular ML learners

  • TabPFN and TabICL are specialized tabular ICL models, whereas MACHINELEARNINGLM uses an LLM backbone to combine numerical processing with textual interpretation and world knowledge.The passage describes TabICL as a column-then-row architecture that scales PFN-style ICL to larger tables.Source: §4 · p. 15

Tool-using agents for ML vs. in-context learners

  • Tool-using ML agents depend on the external learners and pipelines they invoke, while MACHINELEARNINGLM internalizes prediction and requires neither per-task fine-tuning nor external ML calls.The passage also notes that an MLE agent can call MACHINELEARNINGLM as a tool.Source: §4 · p. 16

5 LIMITATIONS

The paper’s current approach is bounded by its task coverage, context and compute requirements, numerical encoding, warm-start procedure, and tested model configuration.

5 LIMITATIONS

  • Pretraining covers SCM-synthesized tabular classification with at most 10 classes; regression, ranking, forecasting, structured prediction, and non-IID dependencies remain future work.The corpus focuses on IID rows.Source: §5 · p. 16
  • Scaling to many thousands of shots remains challenging because of compute and memory costs, despite inference tests up to approximately 131k tokens.Continued pretraining used a 32k-token context.Source: §5 · p. 16
  • The [0, 999] integer mapping preserves order but coarsens magnitudes and can obscure semantically meaningful constants such as age 18.The authors suggest exploring more expressive encodings as context windows grow.Source: §5 · p. 16
  • RF-mimic warm-up and example-level consensus filtering may bias early training toward tree-like boundaries and easy examples, with some bias potentially persisting after warm-up.The model distills labels rather than rationales.Source: §5 · p. 16
  • Results are limited to a 7B backbone with LoRA rank 8; larger backbones, alternative adapters, and other optimizer or regularization choices were not explored.The authors note these choices may further improve many-shot numeracy.Source: §5 · p. 16

6 FUTURE WORK

The proposed future directions extend MachineLearningLM toward agent workflows, multimodal inputs, longer contexts, interpretability, uncertainty-aware prediction, retrieval, and real-data alignment.

6 FUTURE WORK

  • Generative back-ends could turn synthetic tabular features into weakly labeled text, speech, image, geospatial, or time-series surrogates for multimodal prediction practice.The proposed surrogates are conditioned on latent factors to preserve coarse class semantics.Source: §6 · p. 16
  • Tensor or pipeline parallelism and memory-efficient attention or KV caching are proposed to support substantially more in-prompt examples.The proposal builds on cache-management work for long-context reasoning.Source: §6 · p. 16
  • Narrating each task’s SCM and combining feature attributions or distilled rules could produce more faithful, human-readable prediction rationales.The proposed auxiliary objective describes variables, relations, and mechanisms; examples include SHAP, LIME, inTrees, and RuleFit.Source: §6 · pp. 16–17
  • Future training could use teacher predictive distributions, calibrated uncertainty with abstention, and task-level metric optimization to address uncertainty handling.The proposed evaluation includes risk–coverage; task metrics include AU-ROC, RMSE, and F1.Source: §6 · p. 17
  • Retrieval augmentation could inject relevant examples during pretraining and inference to increase the many-shot example limit.The passage describes prior retrieval-augmented methods as yielding power-law error reductions as data grows.Source: §6 · p. 17
  • Lightweight continued fine-tuning on curated real tabular prediction tasks is proposed to better align the model with practical distributions.Source: §6 · p. 17
  • Pretraining agents on many-shot in-context experiences could help them reuse past textual traces, numerical features, or temporal signals for decisions in dynamic environments.The proposal targets agent memory.Source: §6 · p. 17
  • Combining MachineLearningLM with MLE-agent planners could make it both a solver and an internal tool in end-to-end workflows.The proposed interface targets heterogeneous real-world MLE tasks.Source: §6 · p. 17

7 CONCLUSION

The conclusion presents MachineLearningLM as a continued-pretraining recipe for in-context tabular prediction that retains general knowledge and reasoning. It reports stronger many-shot performance across out-of-distribution classification tasks, with accuracy increasing through 1,024 shots.

7 CONCLUSION

  • MachineLearningLM improves over its backbone and approaches random-forest-level accuracy from 8 to 512 shots, with accuracy increasing further up to 1,024 shots.The conclusion also reports robustness to exemplar-order permutations across diverse out-of-distribution classification tasks.Source: §7 · p. 17
  • The recipe combines millions of SCM-driven synthetic tasks, a Random-Forest warm start, token-efficient encoding, and sequence-level batch prediction without architectural or tokenizer changes.The authors say it preserves general knowledge and reasoning.Source: §7 · p. 17
  • The authors characterize targeted continued pretraining on synthetic prediction tasks as a practical path to scaling many-shot ICL within general-purpose LLMs.Source: §7 · p. 17

A LIST OF SYMBOLS

The symbol list defines notation for the SCM task, batched predictions, model training, and evaluation criteria.

A LIST OF SYMBOLS

  • SCM notation includes the graph G, node index v, variable value x_v, structural function f_v, noise term ε_v, and demonstration set S.The list also identifies M as the number of in-context demonstrations, with a maximum of 1,024.Source: A LIST OF SYMBOLS · p. 23
  • Prediction notation includes N test rows per batched sequence, target predictions y, predicted labels ℓ̂_i, and tokenizer tok(·).The list gives N a default of 50.Source: A LIST OF SYMBOLS · p. 23
  • Training and evaluation notation includes model parameters θ, NLL loss L(θ), token costs H, R, and C, plus accuracy and agreement thresholds.The listed evaluation symbols include random-baseline accuracy p0, Cohen’s κ, balanced-accuracy margin δbacc, macro-F1 margin δF1, and dominant-class fraction τdom.Source: A LIST OF SYMBOLS · p. 23

B MANY-SHOT SCALING PER-TASK EVALUATION

Table 6 reports per-dataset test accuracy on additional domain-representative TALENT datasets across 8 to 1,024 shots.

B MANY-SHOT SCALING PER-TASK EVALUATION

  • Table 6 reports per-dataset test accuracy across 8 to 1,024 shots on more sampled domain-representative TALENT datasets.Source: Table 6 · p. 24

C EXTENDED PER-TASK EVALUATION

Table 7 reports per-dataset test accuracy on extended TALENT datasets from 8 to 512 shots; broader evaluation was deferred because of computational cost.

C EXTENDED PER-TASK EVALUATION

  • Table 7 reports per-dataset test accuracy on extended TALENT datasets from 8 to 512 shots.Source: Table 7 · p. 28
  • Evaluation across all 8–1,024 shots and full datasets was left for a later revision because it is very computation intensive.Source: Table 7 · p. 28
Loading 2509.06806v6…