PaperlayerView exact revision on arXiv

Source-linked AI summary

From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, Mihai Surdeanu

arXiv:2404.07544v3cs.CLcs.AI

TL;DR

The paper asks how well pre-trained LLMs can perform regression from in-context examples without additional training. It evaluates linear and nonlinear tasks and finds that LLMs can rival supervised methods, while their predictions improve toward the best fixed strategy as examples accumulate.

  • Problem

    The paper investigates whether pre-trained LLMs can learn regression functions from in-context examples without being specifically trained for regression.

    Source: §1 · p. 1

  • Method

    The study tests LLMs on regression tasks using in-context examples, including author-created nonlinear datasets, and analyzes how performance changes with the number of examples.

    Source: §1 · p. 1, §4.2 · p. 7, §5 · p. 9

  • Results

    LLMs perform linear and nonlinear regression with performance rivaling supervised methods, and capable models approach the best fixed strategy as in-context examples grow.

    Source: §4.3 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    The results demonstrate that LLMs can perform regression from in-context input-output examples despite not being explicitly trained to do so.

    Source: §7 · p. 10

    Training-data opacity means the authors’ mitigation steps do not eliminate the possibility of data contamination.

    Source: §9 · p. 11

Abstract

from arXiv · show

We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.

1 Introduction

The paper tests whether pre-trained LLMs can learn regression functions from in-context examples without additional training. It reports strong performance and uses synthetic datasets spanning simple and difficult functions to study this ability.

1 Introduction

  • Pre-trained LLMs can learn regression functions from in-context examples without additional supervision or training.The study compares their predictions with supervised and unsupervised models, given examples and a new input.Source: §1 · pp. 1–2
  • Synthetic regression data provide deterministic functions, controllable difficulty, and novel examples less likely to have appeared in model training.The studied functions range from simple linear regression to nonlinear benchmarks such as Friedman #2.Source: §1 · p. 2
  • On a linear task with one informative variable out of two, Claude 3 approaches Linear Regression and outperforms Random Forest and Gradient Boosting without parameter updates.The figure compares Mean Absolute Error for three LLMs and four supervised methods.Source: Figure 1 · p. 1, §1 · p. 2

2 Experimental Setup

The setup evaluates LLMs, supervised models, and heuristic baselines on numeric and symbolic-input regression tasks. Synthetic datasets vary in dimension, function complexity, and input type; models receive common training data and are tested on the same partition.

2.1 Datasets

  • The experiments cover linear, nonlinear, and regression tasks with non-numerical inputs.The nonlinear benchmarks include Friedman #1, #2, and #3, as well as functions created by the authors and randomly initialized neural networks.Source: §2.1 · p. 2, §2.1.2 · pp. 2–3, §2.1.3 · p. 3

2.1.1 Linear Regression Datasets

  • Linear datasets use y = wx + b, with input dimension d from 1 to 10 and varying numbers of informative variables.Inputs are sampled from a standard normal distribution and weights from a uniform distribution between 0 and 100.Source: §2.1.1 · p. 2

2.1.2 Non-Linear Regression Datasets

  • Friedman benchmarks and five newly created nonlinear datasets test functions beyond linear regression.The authors also use outputs from randomly initialized neural networks, including a transformer encoder block.Source: §2.1.2 · pp. 2–3

2.1.3 Regression With Non-Numerical Inputs

  • For symbolic-input regression, five lowercase ASCII characters form a vocabulary mapped to numeric values through a sampled weight vector.The task uses a dot product between the weight vector and the values associated with the input symbols.Source: §2.1.3 · p. 3

2.2 Models

  • Models receive the same training data and test partition; LLMs use training examples as in-context exemplars, while supervised methods train on them.Inputs are rounded to two decimal places, and experiments use different random seeds.Source: §2.2 · p. 3
  • The model set includes 12 LLMs, traditional supervised regressors, and heuristic unsupervised baselines; LLMs use prompting without prompt tuning.Supervised-model hyperparameters use package defaults, and the Figure 2 comparison reports Mean Absolute Error across two linear tasks.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3 Large Language Models Can Do Linear Regression

The linear-regression experiments find that LLMs outperform unsupervised baselines, with some matching or exceeding supervised methods. Rankings across four datasets show strong performance from both closed-source and open-weight models.

3 Large Language Models Can Do Linear Regression

  • Claude 3 ranks third among 31 models on Regression NI 1/3, with mean absolute error 0.14 versus Linear Regression’s 0.12.The task has one informative variable out of three; Claude 3 also largely outperforms Random Forest and Gradient Boosting.Source: §3 · p. 5
  • Mixtral outperforms Random Forest and Gradient Boosting on Regression NI 2/2, extending the result beyond closed-source flagship models.The task has two informative variables.Source: §3 · p. 5
  • Across four linear datasets, Mixtral 8x7B and Yi 34B Chat outperform KNN or SVM, while Claude 3 Opus ranks first on NI 1/1.The heatmap presents datasets vertically and models horizontally, with rank encoded by color.Source: §3 · pp. 5–6
  • LLMs outperform unsupervised baselines on linear regression, while Claude 3 and GPT-4 can exceed strong supervised methods without gradient updates.The rank comparison spans four linear datasets and reports that Claude 3 Opus and GPT-4 consistently outperform methods including Gradient Boosting, Random Forest, and KNN.Source: Figure 3 · p. 5, §3 · p. 6

4 Large Language Models Can Do Non-Linear Regression

Across Friedman and newly designed datasets, in-context LLMs show strong nonlinear-regression performance; the new datasets are intended to reduce the chance of pretraining familiarity.

4.1 Friedman Benchmarks

  • The three Friedman benchmarks include Friedman #2, whose four inputs are drawn from uniform distributions and whose target includes Gaussian noise.Source: §4.1 · p. 7
  • On Friedman #2, Claude 3 outperforms all methods except Linear Regression with Polynomial Features.Source: §4.1 · p. 7

4.2 New Regression Datasets

  • Original #1 mimics a line with oscillations, while Original #2 modifies Friedman-inspired domains and operands to reduce possible pretraining familiarity.The novel datasets are intended to reduce the probability of evaluating models on datasets seen during training, not to make inherently difficult challenges.Source: §4.2 · pp. 7–8

4.3 Discussion

  • Claude 3 Opus outperforms Gradient Boosting and KNN on all five datasets, although LLMs struggle on randomly initialized neural-network datasets.The paper reports that LLMs generally remain better than unsupervised methods on those neural-network datasets.Source: §4.3 · p. 8
  • On Original #1, eight of the ten highest-ranking methods are LLMs, and DBRX outperforms every traditional supervised method without gradient updates.Figure 6 presents relative method ranks across the nonlinear datasets.Source: Figure 6 · p. 8, §4.3 · p. 8

5 How Fast Do Large Language Models Adapt?

The paper examines how regression predictions change as in-context examples accumulate, using regret and curve fitting to assess performance over time. It reports improving performance and sub-linear regret for capable LLMs, while noting that the analysis covers finite, relatively small horizons.

5 How Fast Do Large Language Models Adapt?

  • Claude 3 and GPT-4 can obtain sub-linear regret, and the illustrated two-model, two-dataset examples also show sub-linear growth.The paper describes this as predictions approaching the quality of decisions made by the best fixed strategy informed by hindsight.Source: §5 · p. 9, Figure 7 · p. 9
  • The analysis fits linear, square-root, and logarithmic curves to cumulative regret over examples, choosing the fit with the highest R2; slower growth is better.The curve choices are motivated by their common appearance in theoretical regret bounds.Source: §5 · p. 9, Table 1 · p. 10
  • The regret evidence is empirical and covers finite, relatively small time steps, rather than behavior as T approaches infinity.The authors repeated the experiment three times and averaged cumulative regret; GPT-4 performance also continued to improve across dataset sizes from 20 to 500.Source: §5 · p. 9

6 Related Work

This work extends empirical in-context learning research by testing pretrained language models on regression without training them specifically for in-context learning.

6 Related Work

  • Prior transformer studies trained models for in-context algorithms, whereas this work evaluates pretrained LLMs without task-specific training.Source: §6 · p. 10

7 Conclusion

The paper concludes that pre-trained LLMs can perform linear and non-linear regression from in-context examples without gradient updates, with performance rivaling supervised methods. It also reports that Claude 3 Opus and GPT-4 approach the best fixed strategy as examples grow.

7 Conclusion

  • Pre-trained LLMs perform linear and non-linear regression from in-context input-output examples without gradient updates.Their performance rivals supervised methods such as Linear Regression or Gradient Boosting.Source: §7 · p. 10
  • Claude 3 Opus and GPT-4 approach the quality of decisions made by the best algorithm in hindsight as in-context examples grow.The paper frames this as approaching the best possible fixed strategy.Source: §7 · p. 10

8 Ethics Statement

The authors report that they did not perform additional training and do not envision negative impacts from their results.

8 Ethics Statement

  • The study did not perform any additional training.Source: §8 · p. 11
  • The authors do not envision negative impacts from the results.Source: §8 · p. 11

9 Limitations

This section outlines limitations of the study, including its focus on regression, reproducibility, possible training-data contamination, and the lack of a theoretical explanation for its empirical findings. It also notes mixed results for Claude 3 Opus on regression tasks with more variables, including weaker performance as the number of informative variables increases.

9 Limitations

  • The study focuses primarily on regression, including symbolic-input scenarios with numeric outputs.Source: §9 · p. 11
  • Private LLM performance may change over time and affect reproducibility; open-weight models were included but generally lagged private models, and intermediate results were released.Source: §9 · p. 11
  • Contamination remains possible despite evaluating multiple models, using multiple random seeds on newly introduced datasets, and including Falcon 40B with publicly available training data.Source: §9 · p. 11
  • The paper provides empirical evidence that LLMs can perform regression but offers no theoretical explanation for this capability.Source: §9 · p. 11

6. Regression 3/3, a linear regression with 3 variables, all which are informative

  • Claude 3 Opus ranks among the top three models with one informative variable, but its performance drops as more variables are informative.Source: §6 · p. 26
  • All LLMs outperform unsupervised methods; four LLMs beat every KNN variant across datasets, and Claude 3 Opus and GPT-4 always beat Gradient Boosting.DBRX and Code Llama 70B outperform Gradient Boosting on 4 of 6 datasets.Source: §6 · pp. 26–27
  • On NI 5/10, Claude 3 Opus beats Gradient Boosting and Random Forest but trails Logistic Regression; on NI 3/10, it trails both, with performance similar overall to NI 2/3.Source: §6 · p. 27, Figure 16 · p. 29

F Large Language Models Can Do Non-Linear Regression (Expanded)

Across the expanded analyses, LLMs perform strongly in regression benchmarks, while performance falls on neural-network-generated datasets. The sections also report model ranks, adaptation analyses, real-world evaluations, Claude-family comparisons, and experiment costs.

F Large Language Models Can Do Non-Linear Regression (Expanded)

G Average Model Ranks

  • Claude 3 Opus ranks second overall, outperforming Gradient Boosting and multi-layer perceptrons; DBRX and Mixtral 8x22B also surpass several conventional methods.The overall-rank table covers linear, original, Friedman, neural-network, non-linear, and all-dataset groupings.Source: G Average Model Ranks · p. 34, Table 7 · p. 34

H How Fast Do Large Language Models Adapt? (Expanded)

I Performance on Real-World Datasets

  • The real-world evaluation includes GPT-4 and Claude 3 Opus, and the authors do not propose replacing traditional regression methods.The reported datasets include Liver Disorders, Real Estate Valuation, Diabetes, Servo, and Movies.Source: I Performance on Real-World Datasets · p. 38

3. Diabetes (sklearn (Efron et al., 2004))15

  • The real-world tables report mean absolute error and ranks; on Liver Disorders, SVM ranks first among the methods evaluated.The tables include Claude 3 Opus and GPT-4 alongside selected supervised and unsupervised methods.Source: §3 · p. 38, Table 9 · p. 38, Table 10 · p. 39

J Claude Performance

  • Claude 3 models perform much better than older Claude models; some Claude 2.0 and 2.1 configurations were omitted because they generated invalid outputs.Other reported averages use at least 20 runs.Source: J Claude Performance · p. 40, Figure 30 · p. 40

K Costs

  • The total experiment cost was estimated at under $1200, including approximately $300 on OpenRouter and $600 on OpenAI.The authors attribute the higher OpenAI cost to preliminary experiments using a more expensive GPT-4 model.Source: K Costs · p. 40

L LLMs Justifying Their Prediction

The section finds that model-generated explanations can diverge from predictions, then examines symbolic inputs, rounding, KNN comparisons, and potential contamination.

L LLMs Justifying Their Prediction

M Beyond Numerical Regression

N Effects of Rounding

O Is It Just Better KNN?

P Could It Be Just Contamination?

P.1 LLMs With Known Training Data

The authors test Falcon regression performance and whether naming a dataset changes model accuracy; the dataset-name comparison shows no significant effect after adjustment.

P.1 LLMs With Known Training Data

  • Falcon 40B beat unsupervised baselines and surpassed Gradient Boosting and Random Forest on Regression NI 1/1, though its overall regression ability was weaker than newer, more powerful models.Because of Falcon's context limit, evaluation covered only four datasets.Source: §P.1 · p. 48

P.2 Performance When Knowing The Dataset Name

  • After multiple-comparison adjustment, no dataset-name-awareness p-value remained below 0.05, providing no substantial evidence that dataset names influenced performance.Source: §P.2 · p. 50, Table 14 · p. 51

Q Does The Performance Plateau?

GPT-4's performance continued improving through 500 in-context examples, while non-transformer models showed more limited results than transformer-based LLMs.

Q Does The Performance Plateau?

R Beyond Transformer-Based LLMs

  • RWKV generally beat unsupervised baselines but trailed transformer LLMs, while StripedHyena was generally below some unsupervised baselines.StripedHyena produced 98% invalid responses on Friedman #2, so Friedman #2 and Original #2 were omitted from its evaluation.Source: R Beyond Transformer-Based LLMs · p. 54, Figure 48 · p. 55

Related papers

Loading 2404.07544v3…