PaperlayerView exact revision on arXiv

Source-linked AI summary

LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks

Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, Kangwook Lee

arXiv:2206.06565v4cs.LGcs.CL

TL;DR

Non-language tasks commonly require task-specific input, output, and loss designs, raising whether language-model fine-tuning can work without them. LIFT answers this by converting labeled data into sentences and fine-tuning pretrained language models through their standard interface. Across many low-dimensional classification and regression tasks, it performs comparably to strong baselines while exposing useful properties and practical limitations.

  • Problem

    Non-language tasks commonly require redesigned input/output layers and loss functions, motivating the question of whether language-model fine-tuning can avoid those changes.

  • Method

    LIFT converts labeled samples into sentences and fine-tunes pretrained language models without altering their architecture or loss function.

  • Results

    LIFT performs well across a wide range of low-dimensional classification and regression tasks, nearly matching or slightly outperforming the best baselines in most cases.

  • Takeaways & Limitations

    LIFT provides a natural-language-interface framework for non-language prediction, with reported robustness, calibrated predictions, data generation, and improvements from prompting, two-stage fine-tuning, and data augmentation.

  • Takeaways & Limitations

    LIFT performs poorly when regression features are high-dimensional or classification has many classes, and it is constrained by language-model context length and memory inefficiency.

Abstract

from arXiv · show

Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks. LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs." We find that LIFT performs comparably well across a wide range of low-dimensional classification and regression tasks, matching the performances of the best baselines in many cases, especially for the classification tasks. We also report experimental results on the fundamental properties of LIFT, including inductive bias, robustness, and sample complexity. We also analyze the effect of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine-tuning. Our code is available at https://github.com/UW-Madison-Lee-Lab/LanguageInterfacedFineTuning.

1 Introduction

LIFT asks whether pretrained language models can solve non-language tasks without architectural or loss-function changes. It converts labeled data into sentences, fine-tunes the pretrained model conventionally, and performs well across many low-dimensional classification and regression tasks.

  • LIFT addresses whether language-model fine-tuning can solve non-language tasks without changing the architecture or loss function.
  • LIFT converts labeled samples into sentences and fine-tunes a pretrained language model on the resulting sentence dataset.Inference uses the same template with the prediction part left empty, then parses generated output tokens into predictions.
  • LIFT performs well across a wide range of low-dimensional classification and regression tasks, nearly matching or slightly outperforming the best baselines in most cases.
  • The study evaluates LIFT’s inductive bias, sample efficiency, extrapolation, noise robustness, pretraining effects, calibration, data generation, and prompting.
  • Two-stage fine-tuning with synthetic pretext tasks and data augmentation both improve LIFT’s performance.
  • The study’s stated goal is to characterize what LIFT can and cannot do rather than achieve state-of-the-art performance.

2 Methodology and Experimental Setup

The methodology represents supervised datasets as text and applies standard language-model fine-tuning, then parses generated outputs for classification or regression. Experiments cover pretrained models, varied datasets, and standard machine-learning baselines.

  • LIFT training: LIFT training converts each sample into a sentence with a fixed template and fine-tunes language models using the default token-prediction cross-entropy loss.
  • LIFT training: The generic prompt uses feature-value statements followed by a question asking what the output y should be, with separators marking question, answer, and generation end.
  • LIFT inference: During inference, LIFT uses the same template without the answer and end-of-answer parts, then parses generated text into class labels or numeric predictions.
  • LIFT inference: Invalid outputs are handled by increasing decoding temperature, retrying up to five times, and returning the training-set average if all attempts fail.Invalid output occurs very rarely, less than or around 1 in most tested cases.
  • Pretrained LMs: Experiments apply LIFT to GPT-J and GPT-3, using LoRA for GPT-J and black-box API fine-tuning for GPT-3.
  • Datasets: The evaluation spans synthetic, tabular, and vision classification datasets plus synthetic and real regression datasets.
  • Baselines: Classification baselines include logistic regression, decision trees, k-nearest neighbors, Gaussian-kernel SVM, MLP, random forest, XGBoost, and majority-class prediction.

3 Basic Findings of LIFT

LIFT performs competitively on many low-dimensional classification and regression tasks while revealing task-dependent limitations and distinctive learning behavior. Experiments also examine sample efficiency, inductive bias, robustness, pretraining, and comparisons with in-context learning.

  • Regression: LIFT is comparable to baselines on low-dimensional regression but does not surpass the strongest methods because token-level error differs from numerical error.The paper identifies Gaussian-process regression as a stronger baseline and proposes level encoding as a possible improvement.
  • Sample complexity: 1000 samples are sufficient for LIFT to achieve small RMSE on regression tasks, while increasing classification classes requires more samples for adaptation.For some cosine and piecewise functions, LIFT has lower sample complexity than popular baselines.
  • Inductive bias: LIFT/GPT models adapt to decision boundaries of increasing complexity, with rough, axis-parallel shapes resembling tree-based classifiers.The datasets are generated from snapshots of a trained neural network taken at different epochs.
  • Robustness: LIFT is nearly unaffected by 2% regression outliers, but robustness varies with model and attack type.LIFT/GPT-3 is comparably robust to median-based nearest-neighbor baselines across outlier levels, whereas GPT-J becomes more vulnerable as outliers increase; transferred adversarial attacks also defeat LIFT/GPT-3 on MNIST.
  • Pretraining and learning procedures: Pretraining improves LIFT substantially, full-data fine-tuning generally outperforms in-context learning, and natural-language-interfaced variants outperform programming-language-pretrained alternatives.When matched on sample count, LIFT and in-context learning are more comparable; randomly initialized models perform much worse than pretrained models.

4 Evaluation of LIFT-Specific Learning Properties

LIFT's behavior depends on how prompts encode task context, and the study examines its classification, regression, and generative properties. Correct feature names improve performance in most tested datasets, while LIFT also supports calibrated prediction and MNIST image generation.

  • 4.1 Does LIFT Benefit from Incorporating Feature Names?: LIFT uses feature names and task descriptions as context that may exploit pretrained knowledge and improve sample complexity.
  • 4.1 Does LIFT Benefit from Incorporating Feature Names?: Correctly incorporating feature names improves LIFT performance on most evaluated classification datasets, whereas shuffled names substantially reduce it.The reported exception is CMC; all LIFT models outperform the majority-class classifier.
  • 4.2 Is LIFT Calibrated?: The study evaluates LIFT's calibration under varying noise levels using six synthetic regression datasets with 1,000 noisy training samples each.
  • 4.3 Can LIFT Generate Realistic Data?: LIFT is also tested for generative learning on MNIST, including conditional image generation and image completion.
  • 4.3 Can LIFT Generate Realistic Data?: LIFT/GPT-J generates MNIST images of comparably high quality at temperature 1.

5 Improving LIFT with Existing Techniques

The paper improves LIFT with two-stage fine-tuning and data augmentation. Synthetic pretext training helps when real training data is scarce, while noisy-data augmentation improves robustness to perturbed MNIST inputs.

  • 5.1 Two-Stage Fine-Tuning: The two-stage procedure is motivated by separating task-description learning from target-task learning to reduce LIFT's sample complexity.
  • 5.1 Two-Stage Fine-Tuning: Two-stage fine-tuning applies LIFT to synthetic pretext data before real datasets and outperforms ordinary fine-tuning when training data is small.
  • 5.2 Data Augmentation: Data augmentation trains LIFT/GPT-J with noisy MNIST examples to test robustness under clean, Gaussian-noise, and signed-constant-noise conditions.
  • 5.2 Data Augmentation: Without augmentation, LIFT/GPT-J is vulnerable to random noise, unlike the LeNet-5 and MLP baselines.
  • 5.2 Data Augmentation: Training with noisy data significantly improves LIFT/GPT-J accuracy on perturbed test data.The authors identify data augmentation as effective, while leaving mixup and related schemes for future work.

6 Related Works

Related work situates LIFT among fine-tuning methods for adapting language models beyond language tasks, analyses of LM adaptability, and general-purpose models. Existing approaches commonly modify interfaces, losses, or parameterization, whereas LIFT preserves the LM architecture and loss.

  • Fine-tuning for adapting LMs to non-language tasks: Fine-tuning non-language tasks commonly modifies input/output layers or loss functions, including linear layers, adapters, freezing, and distillation methods.
  • Fine-tuning for adapting LMs to non-language tasks: LIFT is motivated in part by Frozen Pretrained Transformer, which adds modality-specific input and output layers while freezing most pretrained parameters.
  • Analyzing the adaptability of LMs: Recent work studies the adaptability and capacity of large language models through efforts such as Big-Bench's benchmark of more than 200 tasks.
  • General-purpose models: LIFT shares the general-purpose goal of extending language models across modalities and domains and can apply to LM-like generalist models such as GATO.

7 Discussion and Conclusion

The paper presents LIFT as a language-interfaced way to apply pretrained LMs to non-language classification and regression without changing architecture or loss. It reports broad low-dimensional performance and several practical extensions, while identifying dimensionality, class count, context length, memory, and fairness boundaries.

  • 7 Discussion and Conclusion: LIFT converts labeled samples into sentences and fine-tunes pretrained LMs with the unchanged architecture and standard loss function.
  • 7 Discussion and Conclusion: Across low-dimensional non-language classification and regression tasks, LIFT/GPT performs relatively well and can calibrate predictions and generate realistic samples.
  • 7 Discussion and Conclusion: LIFT can be improved with in-context feature names, two-stage fine-tuning, and data augmentation.
  • 7 Discussion and Conclusion: LIFT/GPT performs poorly when regression features are high-dimensional or classification involves many classes, and it is restricted by LM context length and memory inefficiency.
  • 7 Discussion and Conclusion: The authors identify potential fairness risks because feature names in prompts may increase sensitivity to embedded social biases and produce unfair or harmful predictions.

Checklist

The paper documents reproducibility, ethics, scope, and appendix materials, while highlighting LIFT’s potential for explainability and retrieval-based updating.

  • Checklist: The authors report that limitations and potential negative societal impacts are discussed in Section 7.
  • Checklist: The study states that its code, data, instructions, training details, repeated runs, error bars, and compute resources are provided.The checklist reports that mean and standard deviation are presented for repeated experiments.
  • Checklist: The appendix contains motivation, related work, datasets, implementations, extended experimental results, visualizations, score tables, and additional evaluations.
  • A.1 Explainability: LIFT may support explainability because large language models can generate explanations of predictions, although the reported zero-shot explanations are imperfect.
  • A.2 Updatability: LIFT may support compute-efficient updating when paired with retrieval-enabled language models, because updating the associated database can update retrieved information.

B Detailed Related Works

Related work frames LIFT as an attempt to extend language-model fine-tuning beyond NLP without the architectural changes commonly used for other modalities.

  • Pretraining and adapting language models: Fine-tuned language models have achieved strong results mainly on NLP tasks, motivating investigation of their adaptation to non-language tasks.
  • Methods for adapting LMs: Prior approaches adapt language models to other modalities by adding input and output layers, freezing parameters, or changing the training setup.
  • LIFT/GPT-J framework: LIFT differs from these approaches by using the language interface without changing the architecture or loss function.
  • In-context few-shot learning: In-context few-shot methods reformulate downstream samples as language-model inputs, but their application to non-language modalities remains unclear.
  • General-purpose models: The related-work discussion places LIFT alongside efforts toward general-purpose models, tabular-data learning, and empirical analyses of language-model adaptability.

C.1 Datasets

The study evaluates LIFT on synthetic, tabular, vision, and regression datasets, including controlled datasets for inductive-bias and function-approximation analyses.

  • Classification datasets: Neural-net-based classification datasets use decision-boundary snapshots from 2-layer networks trained for 10, 40, 80, 210, 320, and 490 epochs.The manuscript tests snapshots at 10, 80, and 490 epochs, with increasing boundary complexity across later snapshots.
  • Regression datasets: Synthetic regression tasks cover linear, quadratic, exponential, cosine, ℓ1-norm, and piecewise-linear functions with smooth, non-smooth, and non-continuous behavior.
  • Classification datasets: The classification suite includes synthetic, real tabular, and vision data, spanning varied feature counts, feature types, class counts, and sample sizes.Vision datasets include MNIST, Fashion MNIST, and permuted variants.
  • Regression datasets: Real regression datasets include Medical Insurance, CCPP, Servo, and Student, with sample sizes from 167 to 9,568 and feature counts from 4 to 33.
  • Models and implementation: The experiments use GPT-J and GPT-3, compare against standard machine-learning baselines, and select hyperparameters using validation results.Model selection for regression uses validation RAE rather than validation loss.

D.1.1 How Well Does LIFT Perform on Standard ML Tasks?

LIFT performs competitively on many classification tasks and some low-dimensional or real regression tasks, but its regression behavior weakens for high-dimensional, interpolation, and extrapolation settings.

  • Classification: LIFT/GPTs perform comparably well across classification tasks, including nonlinear synthetic datasets, OpenML data, and image datasets.
  • Regression: LIFT/GPT-J could not be evaluated on 50D and 120D synthetic regression datasets because of memory-resource limitations.
  • Classification: LIFT achieves comparable accuracy to TabNet and TabTransformer on the evaluated tabular classification tasks.
  • Regression: LIFT approximates several function types in low-dimensional regression, but does not match strong baselines and performs unsatisfactorily in high-dimensional cases.
  • Regression: LIFT/GPT-3 achieves the top 2 regression performance among all real datasets.
  • Interpolation and extrapolation: All methods struggle with interpolation and extrapolation; LIFT has poor interpolation except for linear regression and tends to output seen training values during extrapolation.

D.1.3 Can We Understand the Inductive Biases of Language Models via LIFT?

LIFT models adapt to varied and corrupted decision boundaries, but their boundaries exhibit axis-parallel and fractal patterns linked to numerical token interpretation. Sample-complexity evaluations further show that LIFT generally needs more data initially, while regression performance becomes competitive at larger sample sizes.

  • LIFT needs slightly more training samples than the best baselines before reaching similar performance.
  • LIFT/GPT-3 and LIFT/GPT-J capture the rough shapes of binary, three-class, and five-class decision boundaries.
  • With up to 20% label corruption, LIFT/GPT-3 remains robust while LIFT/GPT-J produces noisier predictions.
  • LIFT decision boundaries are axis-parallel and fractal-like, resembling patterns associated with tree-based classifiers and some convolutional neural networks.
  • Decision-boundary similarity is measured as the percentage of exact classification matches between LIFT/GPT-3 and each baseline.

D.1.4 How Robust Is LIFT?

LIFT shows robustness to several forms of corrupted data, although robustness varies by model and setting. GPT-3 benefits more consistently than GPT-J from larger models, while two-stage intermediate fine-tuning helps especially when training data are scarce.

  • LIFT models are highly robust to 2% regression outliers, while LIFT/GPT-3 remains comparable to median KNN at outlier rates up to 20%.
  • On the Customers dataset with class-imbalance ratio nearly 8, LIFT/GPT-J achieves recall 82.61 ± 7.10 and F1 84.43 ± 1.43.
  • LIFT/GPT-3 tolerates small random MNIST perturbations at ϵ = 0.01.
  • LIFT/GPT-J is not robust to simple noise, and LIFT/GPT-3 outperforms it in corrupted-data settings.
  • Two-stage intermediate LIFT improves original fine-tuning, especially with few training samples, without requiring pretexts to match target-task correlations.
  • Performance gains from larger LMs are inconsistent and relatively small when LIFT is used.

E.3 Quantitative Classification Evaluations on Neural-Net-Based Synthetic Datasets

Quantitative evaluations examine LIFT on increasingly complex neural-network-generated classification boundaries and report learning behavior across classification and regression tasks. LIFT/GPT models approach baseline performance on several complex classification settings but show model- and loss-specific limitations.

  • As boundary complexity or label corruption increases, classification accuracy decreases across methods.
  • For three-class and five-class datasets, LIFT/GPT-J and LIFT/GPT-3 achieve approximately 90%, versus approximately 92% and 91% for the best baselines.
  • At up to 20% label corruption, LIFT/GPT-3 outperforms logistic regression and decision tree, whereas LIFT/GPT-J performs worse than other baselines.
  • LIFT fails to perform Ridge regression, consistent with its reported robustness to outliers.
  • For regression, a decrease in RAE does not necessarily imply a decrease in LM loss, so RAE is used for model selection.
  • LIFT/GPT-J requires only a few epochs to achieve good performance on synthetic regression datasets.

F.1 Limitations and Open Questions

LIFT has clear scope boundaries: regression remains difficult, classification weakens with many classes or high-dimensional inputs, and careful evaluation is needed before broad application. The paper also identifies promising improvements and open questions for extending LIFT’s capabilities.

  • Limitations: Regression performance is poor on high-dimensional functions, and feature names do not consistently improve regression results.These observations distinguish regression behavior from patterns seen in some classification tasks.
  • Limitations: LIFT’s classification loss can misalign optimization with regression accuracy, while tokenization makes numerical understanding difficult.The paper therefore uses RAE for model selection and suggests improved number encodings as a possible aid.
  • Potential improvements: Level encoding discretizes continuous outputs into bins whose canonical representations reflect absolute-value differences, potentially improving regression.This technique is designed to better exploit the classification loss used by language models.
  • Limitations: Classification performance declines when the number of classes is large, and feature dimensionality is bounded by language-model context length.The context-length constraint may be mitigated by more memory-efficient transformer implementations.
  • Open questions: The paper leaves open whether generalist language models can adapt broadly across data types and whether LIFT transfers to models such as GATO.These questions extend beyond the study’s empirical evaluation of LIFT.
  • Practical boundary: LIFT simplifies machine learning to dataset reformatting and prompt design, but deployment requires justification, bias assessment, and additional evaluation.The paper warns that pretrained language-model biases may produce discrimination in downstream decisions.
Loading 2206.06565v4…