PaperlayerView exact revision on arXiv

Source-linked AI summary

LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks

Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, Kangwook Lee

arXiv:2206.06565v4cs.LGcs.CL

TL;DR

LIFT asks whether pretrained language models can solve non-language tasks without architectural or loss-function changes. It converts labeled data into sentences for standard fine-tuning and performs relatively well across low-dimensional classification and regression tasks.

  • Problem

    It is unclear whether language-model fine-tuning can solve non-language tasks without redesigning input/output layers and loss functions.

    Source: §1 · p. 2

  • Method

    LIFT converts labeled samples into sentences and fine-tunes pretrained language models using their unchanged architecture and standard loss function.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    LIFT performs relatively well across a wide range of low-dimensional classification and regression tasks, often nearly matching or slightly outperforming the best baselines.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    The findings suggest that architectural and loss-function changes may not be necessary for applying language models to low-dimensional non-language tasks.

    Source: §1 · pp. 2–3

    LIFT performs poorly with high-dimensional regression features or many classification classes, and is constrained by language-model context length and memory inefficiency.

    Source: §7 · p. 10

Abstract

from arXiv · show

Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks. LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs." We find that LIFT performs comparably well across a wide range of low-dimensional classification and regression tasks, matching the performances of the best baselines in many cases, especially for the classification tasks. We also report experimental results on the fundamental properties of LIFT, including inductive bias, robustness, and sample complexity. We also analyze the effect of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine-tuning. Our code is available at https://github.com/UW-Madison-Lee-Lab/LanguageInterfacedFineTuning.

1 Introduction

LIFT tests whether pretrained language models can learn non-language classification and regression tasks through natural-language interfaces alone, without architectural or loss-function changes. The study finds strong performance across low-dimensional tasks while examining LIFT’s properties, limitations, and improvement techniques.

Motivation and contribution

  • By retaining the original model architecture and loss function, LIFT avoids the task-specific input/output redesigns commonly used for non-language modalities.This enables a natural-language, “no-code machine learning” approach centered on straightforward data-to-sentence conversion.Source: §1 · pp. 2–3

Method

  • LIFT converts labeled samples into sentences, fine-tunes a pretrained language model on them, and parses prompted outputs for classification or regression predictions.Prompts may include feature names and task descriptions or use generic feature names such as x1, x2, ..., xp.Source: §1 · p. 2, Figure 1 · p. 2

Empirical findings

  • LIFT nearly matches or slightly outperforms the best baselines across a wide range of low-dimensional classification and regression tasks.The study also reports high-quality function approximation with GPT-J using 1000 samples, without changing architecture or loss function.Source: §1 · pp. 2–3, Figure 2 · p. 3

Study scope

  • The study evaluates LIFT’s inductive bias, sample efficiency, extrapolation, noise robustness, and dependence on language-model pretraining.These experiments are intended to characterize what LIFT can and cannot do rather than achieve state-of-the-art performance.Source: §1 · p. 3

Extensions

  • Task-specific prompting, prediction calibration, data generation, two-stage fine-tuning, and data augmentation provide additional ways to analyze or improve LIFT.The reported two-stage fine-tuning and data-augmentation techniques improve LIFT’s performance.Source: Figure 1 · p. 2, §1 · p. 3

2 Methodology and Experimental Setup

LIFT converts supervised samples into fixed-template sentences, fine-tunes pretrained language models with standard token-prediction loss, and parses generated text into classification or regression predictions. Experiments use GPT-J and GPT-3 across diverse synthetic, tabular, vision, and regression datasets, comparing against standard learning algorithms.

LIFT training

  • LIFT converts each sample into a templated sentence and fine-tunes the language model using its default token-prediction cross-entropy loss.The generic prompt uses “###” for question/answer separation and “@@@” to mark generation termination; richer prompts can include attribute names and task descriptions.Source: §2 · p. 3

LIFT inference

  • At inference, LIFT parses generated text into class labels or numeric values, treating unmatched classes and failed number parsing as invalid outputs.Invalid outputs trigger temperatures up to 0.75 and at most five attempts; if all fail, regression returns the training-set average, with invalid outputs occurring very rarely [33] [34].Source: §2 · pp. 3–4

Evaluation and baselines

  • Classification uses accuracy and regression uses RMSE and RAE, with LIFT compared against standard classifiers and regressors selected through training-set grid search and cross-validation [45, 46].Baselines include logistic regression, decision trees, KNN, SVM, MLP, random forest, XGBoost, polynomial regression, kernel ridge, gradient boosting trees, Gaussian process, and majority-class classification.Source: §2 · p. 4

Pretrained LMs

  • Experiments apply LIFT to GPT-J with LoRA and GPT-3 through OpenAI’s black-box fine-tuning API [24].GPT-J experiments use AWS p3 instances and local RTX3090 GPUs, while GPT-3 is accessed through the OpenAI API.Source: §2 · p. 4

Datasets

  • The datasets span synthetic, OpenML tabular, MNIST, Fashion-MNIST, permuted vision, and four real regression datasets with controlled noise and varied functions [36] [38] [39] [40].Synthetic regression covers linear, quadratic, exponential, cosine, ℓ1-norm, and piece-wise linear functions, with features sampled uniformly from hypercubes unless otherwise stated.Source: §2 · p. 4

3 Basic Findings of LIFT

LIFT performs competitively on many low-dimensional classification and regression tasks, while its performance depends on task difficulty, sample availability, robustness conditions, and pretrained-model properties. Across additional analyses, LIFT shows distinctive decision-boundary biases, sample-efficiency patterns, and tradeoffs against in-context learning.

3.1 How Well Does LIFT Perform on Standard ML Tasks?

  • For low-dimensional regression, LIFT is comparable to baselines but does not beat the strongest methods, while interpolation and extrapolation generally remain poor.LIFT’s interpolation is good only for linear regression, and it tends to output training-set values during extrapolation.Source: §3.1 · pp. 4–5
  • LIFT/GPT ranks among the top three methods on most classification tasks and achieves 81.17% accuracy on circles, but performance degrades with task difficulty and many classes.On the 100-class Margin dataset, LIFT/GPT-J trails RBF-SVM by nearly 30 percentage points, although LIFT can handle hundreds of features subject to token limits.Source: §3.1 · p. 4, Table 4 · p. 5

3.2 How Many Samples Does LIFT Need?

  • LIFT adapts quickly to new tasks; 1000 samples suffice for small regression RMSE, and it has lower sample complexity than popular baselines for some functions.Classification requires more samples as the number of classes increases because the input and output spaces become more complex.Source: §3.2 · pp. 5–6

3.3 Language-Interfaced Learning: LIFT versus In-Context Learning (ICL)

  • LIFT trained on full datasets consistently outperforms in-context learning, while LIFT and ICL are more comparable when restricted to the same small number of samples.The sample count available to ICL is constrained by the language model’s context length.Source: Table 5 · p. 6, §3.3 · p. 6

3.4 Can We Understand the Inductive Biases of Language Models via LIFT?

  • LIFT/GPT models capture rough shapes of decision boundaries, with axis-parallel forms and fractal patterns resembling tree classifiers and some convolutional networks.The models adapt across datasets constructed with increasingly complex binary decision boundaries.Source: §3.4 · p. 6, Figure 6 · p. 6

3.5 How Robust is LIFT?

  • LIFT/GPT models are among the most robust methods to training outliers, with performance almost unaffected by 2% outliers while baselines suffer large drops.LIFT/GPT-3 is comparably robust to median-3NN and median-5NN across outlier rates, whereas LIFT/GPT-J becomes more vulnerable as contamination increases.Source: §3.5 · pp. 6–7
  • LIFT/GPT-3 tolerates random feature noise but not transferred adversarial attacks on MNIST, indicating vulnerability to adversarial perturbations.The evaluation uses PGD attacks transferred from LeNet-5 across perturbation radii ε ∈ {0.01, 0.1}.Source: §3.5 · p. 7, Table 7 · p. 7

3.6 Does LIFT Need Large-Scale Models Pretrained on Natural Language Data?

  • Pretraining is essential: pretrained natural-language GPTs substantially outperform randomly initialized, programming-language-pretrained, and gibberish-finetuned alternatives.These comparisons indicate that LIFT benefits from pretraining beyond model scale and performs best with natural-language pretraining.Source: §3.6 · p. 7, Table 8 · p. 7

4 Evaluation of LIFT-Specific Learning Properties

LIFT benefits from context-aware prompting with correct feature names, exhibits calibrated predictive variability, and can generate reasonably high-quality MNIST images. These properties support LIFT’s use beyond standard classification and regression settings.

4 Evaluation of LIFT-Specific Learning Properties

  • The evaluation examines LIFT’s behavior through context-aware prompting, calibration, and data generation beyond conventional prediction tasks.These experiments assess LIFT-specific learning properties in a more fine-grained manner.Source: §4 · p. 7, §4.1 · p. 7, §4.3 · p. 8

4.1 Does LIFT Benefit from Incorporating Feature Names?

  • Correct feature names improve LIFT, yielding the best accuracies on TAE, Vehicle, and German and performance comparable to the best model on CMC.All LIFT variants outperform the majority class classifier, while shuffled feature names significantly reduce performance.Source: Table 9 · p. 8, §4.1 · p. 8

4.2 Is LIFT Well Calibrated?

  • LIFT/GPT-J’s prediction standard deviation nearly matches the noisy observations across synthetic regression tasks, indicating calibration; LIFT/GPT-3 shows the same pattern.The experiments vary noise levels across six synthetic regression datasets and use temperature T = 1 for random predictions.Source: §4.2 · p. 8

4.3 Can We Use LIFT for Data Generation?

  • LIFT/GPT-J generates MNIST images with the correct digit shape and reasonably high quality, especially when completing images from their top halves.The tasks generate an image from a digit number or complete an image given its digit number and top-half pixels.Source: Figure 10 · p. 8, §4.3 · p. 8

5 Improving LIFT with Existing Techniques

LIFT is improved through two-stage fine-tuning with synthetic pretext tasks and data augmentation. Two-stage training helps when data are scarce, while noisy training data improves robustness to perturbed inputs.

5.1 Two-Stage Fine-Tuning for LIFT with Synthetic Pretext Tasks

  • Two-stage fine-tuning with synthetic Gaussian pretext tasks improves LIFT over standard fine-tuning when training data are scarce for classification and regression.The method first trains GPT-J for 2–3 epochs on pretext tasks matching the target feature and label or response spaces, then fine-tunes on the target dataset.Source: §5.1 · p. 9, Figure 11 · p. 9

5.2 Data Augmentation

  • Data augmentation significantly improves LIFT/GPT-J accuracy on MNIST test data with Gaussian or signed constant noise.Training uses noisy MNIST images, with perturbations bounded by ϵ ∈[0, 1] per pixel; Table 12 compares clean and perturbed settings.Source: §5.2 · p. 9, Table 12 · p. 9
  • Without augmentation, LIFT/GPT-J is vulnerable to random noise, unlike LeNet-5 and MLP baselines.The augmentation experiment evaluates clean data, Gaussian noise, and signed constant noise, while alternative schemes such as mixup remain future work.Source: §5.2 · p. 9

6 Related Works

Prior work adapts pretrained language models to non-language tasks through architectural or objective changes, parameter-efficient fine-tuning, and multimodal extensions. LIFT instead builds on GPT models trained only on text to pursue architecture- and objective-preserving generalist adaptation while relating to LM adaptability and AutoML.

Fine-tuning for adapting LMs to non-language tasks

  • Parameter-efficient LM fine-tuning includes adapters [25] [26] [27], freezing-based methods [71] [18] [72], distillation [73], and LoRA [24], which LIFT/GPT-J uses.These methods reduce trainable parameters through residual blocks, frozen pretrained parameters, distillation, or low-rank weight updates.Source: §6 · p. 10
  • Unlike common non-language adaptations and FPT [18], LIFT changes neither the architecture nor objective, avoiding the modifications associated with catastrophic forgetting [66] [74].FPT freezes most pretrained parameters and adds modality-specific input and output layers, whereas LIFT requires no such architectural or objective-function changes.Source: §6 · p. 10

Analyzing the adaptability of LMs

  • Adaptability research quantifies the capacity of large LMs [81] [84], including Big-Bench, which evaluates more than 200 diverse tasks.The cited works study how broadly large language models can adapt across tasks and topics.Source: §6 · p. 10

General-purpose models

  • LIFT extends the generalist-model goal [85] [86] [87] [88] [89] [90] across modalities while using one pretrained LM, unlike AutoML [92], which automates pipelines from existing algorithms.LIFT can apply to LM-like generalist models such as GATO [90], and the paper bases it on GPT models trained only on textual data.Source: §6 · p. 10

7 Discussion and Conclusion

LIFT enables pretrained language models to address low-dimensional non-language classification and regression tasks without changing model architecture or loss, while supporting robustness, calibration, data generation, and improvement through prompting, two-stage fine-tuning, and augmentation. Its limitations include poor performance for high-dimensional regression and many-class classification, context-length and memory constraints, and unresolved fairness and broader generalization questions.

Conclusion

  • LIFT/GPT performs relatively well on low-dimensional classification and regression tasks without changing the model architecture or loss function.LIFT converts labeled samples into sentences and fine-tunes pretrained LMs with the standard fine-tuning method and loss function, supporting “no-code machine learning with language models.”Source: §7 · p. 10
  • LIFT/GPT is robust in several practical settings, calibrates predictions, generates realistic data samples, and improves with feature names, two-stage fine-tuning, and data augmentation.Source: §7 · p. 10

Limitations and open questions

  • LIFT/GPT performs poorly when regression features are high-dimensional or classification involves many classes, while context length and memory inefficiency remain constraints.Memory-efficient LMs such as LinTransformer [93] are suggested as one possible way to address the memory issue.Source: §7 · p. 10
  • Future work should examine whether LIFT/GPT resembles ensemble or decision-tree methods, generalizes across modalities and domains, and benefits from more sophisticated numeric encodings for regression.Source: §7 · p. 10

Social impacts

  • Future research should measure fairness risks because feature names in prompts may increase sensitivity to social biases and lead to unfair or harmful predictions.The authors leave measuring embedded bias in LIFT as a future direction.Source: §7 · p. 10

Checklist … A.2 Updatability via Information Retrieval

The appendix supplies implementation, extended-results, and discussion details, while further motivation highlights LIFT’s potential for explainability and compute-efficient updating through information retrieval. The checklist confirms coverage of limitations, societal impacts, reproducibility materials, training details, repeated runs with error bars, and compute resources.

Checklist

  • The paper states that its claims and scope are summarized in the abstract and introduction, with limitations and potential negative societal impacts discussed in Section 7.Source: Checklist · p. 21
  • The checklist reports that code, data, instructions, and training details are provided through the anonymous GitHub repository and the paper.It also states that all training details, including data splits and hyperparameters, are documented.Source: Checklist · p. 21
  • Experiments were run multiple times, tables report means and standard deviations, and compute resources are described in Section 3.Source: Checklist · p. 21

Appendix

  • LIFT’s appendix expands the paper with motivation, related work, datasets, implementations, extended experiments, additional evaluations, visualizations, and further discussion.It organizes these materials across appendices A–F, including detailed results for the main-paper findings and additional functions, datasets, and properties.Source: Appendix · p. 23

A Further Motivations of LIFT

  • The paper presents explainability and information-retrieval-based updatability as additional potential advantages beyond LIFT’s emphasized no-code-ML property.Source: A Further Motivations of LIFT · p. 25

A.1 Explainability

  • LIFT may enable explainability because pretrained language models can explain predictions, although the demonstrated zero-shot explanations are reasonable but far from perfect.The German-credit example uses two consecutive inference calls, and future improvements in few-/zero-shot reasoning could improve explainability.Source: §A.1 · p. 25

A.2 Updatability via Information Retrieval

  • By pairing LIFT with language models that retrieve from databases or the Internet, the framework can support compute-efficient updates under distribution shift.The paper did not use retrieval-equipped models, but states that applying LIFT to them is straightforward because updating the associated database can update the system.Source: §A.2 · p. 25

B Detailed Related Works · C Experimental Setup

The paper situates LIFT within pretrained-LM adaptation, multimodal and tabular learning, and generalist-model research. Unlike prior approaches that modify architectures, objectives, or pretraining, LIFT adapts text-only GPT models to non-language tasks through the language interface alone.

B Detailed Related Works

  • Pretraining followed by fine-tuning has become standard for adapting LMs across NLP tasks, but strong architecture-preserving results have remained mainly limited to language applications.Transformers underpin modern LMs including BERT and GPT variants, whose contextual representations encode broad linguistic knowledge.Source: B Detailed Related Works · p. 25
  • Existing adaptation research extends LMs to images, audio, tables, knowledge bases, numeric tasks, protein folding, and symbolic mathematics, often by adding modality-specific pretraining or task interfaces.The paper emphasizes that LIFT instead relies on GPT models trained only on textual data.Source: B Detailed Related Works · p. 27
  • Prior work studies LM adaptability through benchmarks, synthetic tasks, representation analyses, inductive-bias investigations, and numeracy evaluations, but primarily for downstream language tasks.These studies motivate examining whether LMs can handle non-language tasks without modifying their loss or architecture.Source: B Detailed Related Works · p. 27
  • Fine-tuning methods range from added linear layers and freezing to multi-stage, intermediate, self-supervised, adapter, distillation, and low-rank techniques such as LoRA.LoRA is used to fine-tune GPT-J in the paper, while LIFT avoids input/output-layer and loss modifications that may cause catastrophic forgetting.Source: B Detailed Related Works · p. 27
  • LIFT differs from FPT and other multimodal extensions by adapting text-only GPT models to non-language tasks without changing the architecture or objective.FPT adds input and output layers and changes the objective, whereas LIFT targets basic function-approximation and tabular-classification tasks through the language interface.Source: B Detailed Related Works · p. 27
  • LIFT extends in-context learning’s language-interface principle beyond NLP by reformulating non-language samples as language-modeling inputs and thereby broadening LM application boundaries.In-context methods modify inputs with task examples and rely on prompt generation, search, and tuning, whereas LIFT fine-tunes the model for non-language tasks.Source: B Detailed Related Works · pp. 27–28
  • Tabular learning remains difficult for deep networks because tabular features are heterogeneous, sparse, type-mixed, and weakly correlated, motivating architectures such as differentiable trees, regularized networks, and attention modules.LIFT is presented as the first thorough study of large-LM adaptation for tabular learning without architecture changes, with promising results relative to tree-based methods including Random Forest and XGBoost.Source: B Detailed Related Works · p. 28
  • LIFT shares the generalist-model goal of applying one scalable model across modalities and domains while using a single pretrained LM for all tasks, paralleling AutoML’s emphasis on usability.The approach may also extend to LM-like generalist architectures such as GATO.Source: B Detailed Related Works · p. 28

C.1 Datasets … D.1 Results for Basic Findings of LIFT (Section 3)

The paper evaluates LIFT across synthetic, tabular, vision, and regression datasets, using GPT-based implementations and validation-selected baselines; the appendix extends the main experiments with detailed findings.

C.1 Datasets

  • Regression evaluation uses synthetic linear, quadratic, exponential, cosine, ℓ1-norm, and piecewise-linear functions plus four real datasets.The real datasets are Medical Insurance, CCPP, Servo, and Student, with sample sizes ranging from 167 to 9,568 and feature counts from 4 to 33.Source: §C.1 · pp. 29–30
  • Classification evaluation spans synthetic, real tabular, and vision datasets, including balanced Rolls variants, neural-net-generated tasks, OpenML data, MNIST, and Fashion MNIST.Neural-net datasets use decision-boundary snapshots from 10, 80, and 490 training epochs; MNIST images are center-scaled and cropped to 18 × 18 pixels.Source: Figure 14 · p. 28, §C.1 · p. 29, Figure 15 · p. 29

C.2 LIFT and Baseline Implementation

  • The implementation section documents pretrained models, baseline implementations, computing resources, and fine-tuning and hyperparameter-selection procedures.The experiments use GPT-J and GPT-3 alongside scikit-learn and XGBoost baselines.Source: §C.2 · p. 30, §C.2.1 · pp. 30–31

C.2.1 Pretrained Language Models and Baselines

  • Main experiments use quantized 6-billion-parameter GPT-J for reproducibility and GPT-3 Ada as a reference, while non-numerical regression outputs are treated as invalid.GPT-J uses 8-bit weights; larger GPT-3 variants are compared separately in Section E.2.Source: §C.2.1 · p. 30

C.2.2 Computing Resources

  • GPT-J experiments run on two 24GB RTX 3090 GPUs and AWS EC2 instances, whereas experiments with other models use CPU instances.The computing setup is reported separately from the model and baseline descriptions.Source: §C.2.2 · p. 31

C.2.3 Hyperparameter Selection

  • Hyperparameters are selected using validation results for all methods, with separate candidate grids for LIFT, tree models, nearest neighbors, neural networks, and other baselines.GPT-J epochs, GPT-3 learning-rate multipliers, and model-specific parameters are tuned across predefined discrete sets.Source: §C.2.3 · p. 31

Regression methods

  • Regression model selection uses validation RAE rather than validation loss, with tuned LIFT, polynomial, nearest-neighbor, kernel, neural, boosting, forest, and Gaussian-process methods.Polynomial regression fixes degree 3 because higher degrees can cause out-of-memory errors on high-dimensional datasets.Source: Regression methods · pp. 31–32

D Detailed and Extended Results of Experiments in the Main Paper

  • The appendix extends the main paper’s experiments with primary LIFT findings, learning-property analyses, and techniques intended to improve LIFT.These extended results include the primary findings in D.1, specific learning properties in D.2, and improving techniques in D.3.Source: D Detailed and Extended Results of… · p. 32

D.1.1 How Well Does LIFT Perform on Standard ML Tasks? · D.1.2 How Many Samples Does LIFT Need?

Across standard ML tasks, LIFT performs competitively for classification and real-world regression, but is weaker on high-dimensional, interpolation, and extrapolation regression. The evaluation also examines sample complexity through classification and regression comparisons.

D.1.1 How Well Does LIFT Perform on Standard ML Tasks?

  • LIFT achieves competitive classification performance across synthetic, OpenML, and vision datasets, matching strong methods and remaining effective on nonlinear data.Performance degrades as the number of classes grows, particularly when c=100.Source: Table 17 · p. 32, §D.1.1 · p. 32
  • Regression visualizations compare LIFT/GPTs with baselines after training on either 200 or 1000 samples across several functions.The visualizations use uniformly sampled training coordinates from [−10, 10] and evaluate predictions on [−12, 12].Source: §D.1.1 · p. 32, Figure 21 · p. 33, Figure 22 · p. 34
  • LIFT achieves comparable accuracy to TabNet and TabTransformer, which are deep architectures designed specifically for tabular data.This comparison provides additional evidence that LIFT performs well without task-specific architectural changes.Source: §D.1.1 · p. 32, Table 18 · p. 35
  • LIFT approximates varied low-dimensional functions but fails to match strong baselines and has unsatisfying performance in high-dimensional synthetic regression.For p=1, 2, LIFT can approximate different function types; for p=50, 100, performance is unsatisfying, and LIFT/GPT-J results are unavailable because of resource limitations.Source: §D.1.1 · p. 32, Table 19 · p. 35
  • LIFT/GPT-3 achieves the top 2 regression performance among all evaluated real datasets.Regression performance is measured by RAE(↓).Source: Table 20 · p. 35
  • LIFT has poor interpolation performance except for linear regression and tends to output values seen during training when extrapolating.For extrapolation, outputs outside the training interval often remain within the range of outputs observed on the training data.Source: §D.1.1 · p. 36, Figure 23 · p. 36, Figure 24 · p. 37

D.1.2 How Many Samples Does LIFT Need?

  • The study compares sample complexity across evaluated methods in both classification and regression settings.These comparisons are presented in Figures 25a and 25b, although the supplied passage does not report their numerical results.Source: §D.1.2 · p. 36
  • The extrapolation comparison evaluates LIFT and baselines on six synthetic functions using 200 training samples drawn from [−10, 10].Testing is performed outside the training interval to assess extrapolation behavior.Source: Figure 24 · p. 37

D.1.3 Can We Understand the Inductive Biases of Language Models via LIFT?

LIFT/GPT models adapt to diverse synthetic classification boundaries and remain robust to label corruption, while exhibiting axis-parallel, fractal-like patterns attributed to numerical artifacts from the language interface. Boundary comparisons suggest LIFT/GPT-3 most resembles RBF-SVM, random forest, and MLP behavior, without a single clear overall baseline match.

Visualizing decision boundary

  • LIFT/GPT-J and LIFT/GPT-3 capture the rough shapes of binary, three-class, and five-class decision boundaries, adapting across boundary types.The models were evaluated on neural-net-generated synthetic datasets spanning three classification complexities.Source: §D.1.3 · p. 38, Figure 26 · p. 39
  • LIFT/GPT-3 remains robust under 5% and 20% label corruption, whereas decision trees and XGBoost are most affected and LIFT/GPT-J becomes noisier.LIFT/GPT-3 continues capturing the correct boundary shapes despite corrupted training labels.Source: §D.1.3 · pp. 37–38
  • LIFT decision boundaries are often axis-parallel and fractal-like, likely because decimal-string differences create artifacts in the language interface’s interpretation of numbers.For example, 0.98 and 0.99 differ by one character, while 0.99 and 1.00 differ by three characters.Source: §D.1.3 · pp. 38, 40

Quantifying the similarity of decision boundaries

  • LIFT/GPT-3 shares its closest boundary behavior with RBF-SVM, random forest (E=100), and MLP (W=100), although no single discernible baseline pattern emerges overall.Similarity is measured as the percentage of exact classification matches on sampled points.Source: §D.1.3 · p. 40, Table 27 · p. 40

D.1.4 How Robust Is LIFT? … E.2 Does LIFT Benefit from Larger LMs?

Across robustness, prompting, generation, intermediate fine-tuning, and larger-model studies, LIFT shows strong but condition-dependent behavior: it tolerates several corruptions, benefits from contextual or staged training in selected settings, and gains only modestly from larger pretrained LMs. Additional comparisons indicate that replacing layers can slightly improve performance, while LIFT can generate structured outputs and calibrated predictions under suitable conditions.

D.1.4 How Robust Is LIFT?

  • With 2% outliers, all LIFT models are highly robust; at up to 20%, LIFT/GPT-3 matches median KNN while LIFT/GPT-J fails, and GPT-3 performs well under label corruption; on imbalanced Customers, GPT-J achieves recall 82.61 ± 7.10 and F1 84.43 ± 1.43.LIFT/GPT-3 tolerates Gaussian and signed constant noise at ϵ = 0.01, whereas LIFT/GPT-J is not robust to simple noise.Source: §D.1.4 · pp. 40–41, Figure 28 · p. 41

D.2.1 Does LIFT Benefit from Incorporating Feature Names?

  • Correctly incorporating feature names improves LIFT/GPT-J classification over omitted or shuffled names, but feature names do not necessarily improve regression performance.The classification evaluation uses CMC, TAE, and Vehicle; regression evaluation uses Insurance and Student, where no significant improvements are observed.Source: §D.2.1 · p. 44, Table 34 · p. 44, Table 35 · p. 45

D.2.2 Is LIFT Calibrated?

  • LIFT’s prediction standard deviations align with observed variability across noise levels, indicating calibrated predictions.This behavior is visualized for LIFT/GPT-3 predictions on grid datasets with varying training-noise levels.Source: Figure 37 · p. 45, Figure 36 · p. 45

D.2.3 Can we use LIFT for Generation?

  • LIFT generates reasonable MNIST images when digit and top-half pixels are provided, while unconditional digit generation works mainly at high temperatures such as 0.9 and 1.Small train–test perplexity differences further imply good generative generalization.Source: §D.2.3 · p. 46, Table 39 · p. 47, Figure 38 · p. 48

D.3.1 Two-Stage Intermediate Fine-Tuning for LIFT

  • Two-stage intermediate fine-tuning with synthetic Gaussian pretexts improves ordinary LIFT especially with few training samples, including across synthetic classification tasks and a regression task.The pretexts need only reflect the target dataset’s feature and class counts, not its correlation structure or feature relevance.Source: §D.3.1 · p. 47, Figure 40 · p. 49

E Additional Experiments and Findings (NOT Discussed in the Main Paper); E.1 What Is the Effect of Replacing the Input or Output Layers?

  • Appendix E evaluates architectural replacements, larger models, synthetic classification, ridge regression, and training curves; replacing the output layer slightly improves LIFT, whereas replacing only the input layer performs worst.FPT and FPT (Output Only) are slightly better than LIFT on almost all tested cases, plausibly because output-layer training resembles linear classification.Source: E Additional Experiments and Findings… · p. 49, §E.1 · p. 49, Table 41 · p. 50

D.1.5 Does LIFT Need Large-Scale Models Pretrained on Natural Language Data?

  • The large-language-model comparison uses GPT-J, GPT-3 Ada, Babbage, Curie, and Davinci across classification and regression, finding that larger models outperform smaller ones in many but not all LIFT settings.For LIFT/Rand-GPT-J, only 10%–15% of outputs are valid, and accuracies are computed among valid outputs.Source: §D.1.5 · p. 42, §E.2 · pp. 50–51

E.2 Does LIFT Benefit from Larger LMs?

  • Larger LMs consistently improve in-context classification, but LIFT gains from scaling are usually small and inconsistent across classification settings and regression tasks.With feature names, GPT-J outperforms Davinci on two of three tasks, while without feature names Davinci beats GPT-J on four datasets and loses on one; LIFT usually beats same-model in-context learning.Source: Table 42 · p. 50, Table 43 · p. 50, §E.2 · p. 51

E.3 Quantitative Classification Evaluations on Neural-Net-Based Synthetic Datasets … F Additional Discussion

The evaluations show that LIFT remains competitive on increasingly difficult and corrupted synthetic classification tasks, but fails on ridge regression and requires only a few epochs to perform well. The discussion further addresses regression difficulty, broader impact, classification limitations, and open questions.

E.3 Quantitative Classification Evaluations on Neural-Net-Based Synthetic Datasets

  • LIFT/GPT-3 outperforms logistic regression and decision trees under severe corruption, while LIFT/GPT-J performs worse in corrupted settings; on 3- and 5-class data, both reach about 90% versus MLP/XGBoost at about 92%/91%.Accuracy decreases as decision-boundary complexity or label corruption increases, across clean, 5%, and 20% corruption settings.Source: §E.3 · p. 51, Figure 44 · p. 51

E.4 Can LIFT Perform Ridge Regression via Data Augmentation?

  • LIFT fails to perform ridge regression despite data augmentation, consistent with its demonstrated robustness to outliers.The ridge-regression evaluation reports RAE for LIFT on linear and ridge regression and finds poor performance on the ridge problem.Source: §E.4 · p. 52, Table 45 · p. 52

E.5 LIFT’s Training Curve

  • LIFT’s training loss generally decreases, but regression RAE need not track loss; models are selected by validation accuracy or RAE and often achieve good performance within a few epochs.Figure 46 covers classification accuracy and loss, while Figure 47 covers regression RAE and loss curves.Source: §E.5 · p. 52, Figure 46 · p. 52, Figure 47 · p. 53

F Additional Discussion

  • The additional discussion elaborates on regression-task difficulty and LIFT’s broader impact while addressing classification limitations and open questions.It continues the discussion from Section 7 rather than introducing a separate experiment in the supplied passage.Source: F Additional Discussion · p. 52

F.1 Limitations and Open Questions · F.2 Broader Impact

LIFT faces important limitations in regression, large-class classification, and feature dimensionality, while broader deployment raises fairness and evaluation concerns. The paper identifies encoding-based improvements and open questions about generalist models alongside LIFT’s potential to enable no-code machine learning.

F.1 Limitations and Open Questions

  • LIFT underperforms on high-dimensional regression, and feature names do not consistently improve regression performance.These classification-task phenomena are not consistently observed for regression tasks.Source: §F.1 · p. 53
  • The LM classification loss can misrepresent regression errors, while numerical tokenization may further limit LIFT’s regression performance.RAE reduction does not necessarily imply LM-loss reduction, so RAE is used for model selection; numerical encoding methods may help [133] [135] [170] [173] [174] [175] [176].Source: §F.1 · p. 53
  • Level encoding is proposed to discretize continuous outputs so mismatched bits reflect absolute-value differences, potentially improving regression generalization.For example, values in [0, 3] can map to bins represented as 00, 01, and 11; the paper leaves investigation of this approach for future work.Source: §F.1 · p. 53
  • On 100-class datasets, LIFT/GPT-3 has lower accuracy than RBF-SVM and XGBoost, while LM context length bounds the feature dimension it can handle.More memory-efficient transformer implementations, such as, may mitigate the dimensionality limitation.Source: §F.1 · p. 53
  • Open questions include whether generalist LMs can develop universal models that adapt to any data and whether LIFT applies to models such as GATO [90].Source: §F.1 · p. 53

F.2 Broader Impact

  • By requiring only dataset reformatting and prompt design, LIFT could enable no-code machine learning across applications including credit loaning, disease diagnosis, and criminal sentencing.This simplicity connects LIFT to automated machine learning research [92].Source: §F.2 · p. 54
  • Using LIFT without careful justification may cause discrimination because pretrained-model biases can affect downstream decisions, especially when demographic contexts or feature names are provided.Fairness-aware reweighting [179] [180], data augmentation, and parameter-efficient fine-tuning are suggested mitigation strategies; additional model evaluation is required rather than treating LIFT as a universal solution.Source: §F.2 · p. 54

Related papers

Loading 2206.06565v4…