PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-LoRA: Instant Transformer Adaption

Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange

arXiv:2506.06105v2cs.LGcs.AI

TL;DR

Task-specific LoRA adaptation requires task-specific data and hyperparameter settings, limiting knowledge transfer and adding engineering overhead. T2L generates LoRA adapters from natural-language task descriptions using a hypernetwork, and is reported to outperform two baselines on benchmark tasks while generating adapters for unseen tasks. It also compresses hundreds of adapters while maintaining task-specific adapter performance.

  • Problem

    Task-specific LoRA fine-tuning requires a dataset and hyperparameter setting for each task, limiting knowledge transfer and adding engineering overhead.

    Source: §1 · p. 1

  • Method

    T2L is a hypernetwork that compresses task-specific LoRAs and generates new adapters zero-shot from natural-language task descriptions.

    Source: §1 · pp. 1–2

  • Results

    T2L outperforms a multi-task LoRA baseline and Arrow Routing on benchmark tasks and generates LoRAs for unseen tasks from language-based descriptions.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    T2L compresses hundreds of LoRA adapters while maintaining task-specific adapter performance and can generalize to unseen tasks with suitable descriptions.

    Source: §1 · p. 2

    T2L is limited to LoRA outputs, and it does not reach task-specific LoRA benchmark performance in a zero-shot manner.

    Source: §7 · p. 9

Abstract

from arXiv · show

While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation of datasets and repeated fine-tuning of the underlying model. Fine-tuning techniques enable practitioners to adapt foundation models for many new applications but require expensive and lengthy training while being notably sensitive to hyperparameter choices. To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting large language models (LLMs) on the fly solely based on a natural language description of the target task. T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass. After training T2L on a suite of 9 pre-trained LoRA adapters (GSM8K, Arc, etc.), we show that the ad-hoc reconstructed LoRA instances match the performance of task-specific adapters across the corresponding test sets. Furthermore, T2L can compress hundreds of LoRA instances and zero-shot generalize to entirely unseen tasks. This approach provides a significant step towards democratizing the specialization of foundation models and enables language-based adaptation with minimal compute requirements. Our code is available at https://github.com/SakanaAI/text-to-lora

1. Introduction

Task-specific LoRA fine-tuning requires datasets and hyperparameter settings, limiting knowledge transfer and adding engineering overhead. T2L uses task descriptions to generate LoRAs and is reported to compress adapters and generalize to unseen tasks.

1. Introduction

  • T2L is proposed to test whether LoRAs can be compressed and new task-specific adapters generated zero-shot from natural-language instructions.Training uses a diverse task distribution from Super Natural Instructions.Source: §1 · p. 1
  • T2L outperforms a multi-task LoRA baseline and Arrow Routing on benchmark tasks, and generates LoRAs for unseen tasks from language-based descriptions.The paper identifies Arrow Routing as a state-of-the-art zero-shot LoRA routing method.Source: §1 · pp. 1–2
  • The proposed hypernetwork generates LoRA adapters from text descriptions in a single forward pass and can be trained by adapter distillation or multi-task fine-tuning.The paper introduces architectures for both training approaches.Source: §1 · p. 2
  • T2L compresses hundreds of LoRA adapters while maintaining task-specific adapter performance, and can generalize to unseen tasks with suitable descriptions.The compression is lossy.Source: §1 · p. 2
  • The paper examines task-description embeddings, training routines, and text-based descriptions, and reports semantically meaningful clusters among generated LoRAs.It also reports evidence explaining why reconstruction-trained T2L cannot generalize.Source: §1 · p. 2

2. Preliminaries

The preliminaries define task datasets and describe single-task and multi-task fine-tuning, LoRA adapters, and hypernetworks. They also introduce architectural variations for generating LoRA weights.

2. Preliminaries

  • Each task dataset pairs inputs and outputs with natural-language descriptions that can describe the dataset generally rather than individual samples.The task collection is denoted D = {D1, . . . , DT}, corresponding to tasks T = {t1, . . . , tT}.Source: §2 · p. 2
  • Figure 2 presents T2L architectural variations, with dashed-box components specific to their architectures and trainable modules and embeddings indicated separately.Its caption identifies r as LoRA rank and d as input and output dimension.Source: Figure 2 · p. 3
  • In multi-task fine-tuning, one adapter is trained to minimize expected loss over the union of task datasets.For a single task, the objective optimizes an adapter for that task’s fine-tuning data.Source: §2 · pp. 2–3
  • LoRA freezes a base model’s pretrained weights and learns low-rank matrices that adapt its selected linear transformations.For a selected transformation, the paper writes the adapted output as h = W0x + BT Ax.Source: §2 · p. 3
  • A hypernetwork generates another network’s parameters from descriptors, providing an indirect encoding whose parameter count is much smaller.Its parameters are trained end-to-end on a downstream task.Source: §2 · p. 3

3. Text-to-LoRA: Learning to Compress and Generate LoRAs

Text-to-LoRA uses task-description-conditioned hypernetworks to generate LoRA weights, with architectures that vary their output parameterization. It can be trained either to reconstruct a LoRA library or directly on fine-tuning data.

3. Text-to-LoRA: Learning to Compress and Generate LoRAs

  • T2L conditions a hypernetwork on task-description vectors and learnable module/layer embeddings to generate LoRA weights, batching inputs for a single forward pass.The text vector represents the task description; the embeddings identify the target module and layer.Source: §3 · p. 3

3.1. Text-to-LoRA Architectures

  • L emits A and B jointly, M shares an output layer for A or B, and S emits one low-rank matrix rank at a time.The variants differ in output spaces, inductive biases, and parameter counts; the S head is smaller than the L and M heads.Source: §3.1 · pp. 3–4

3.2. Training Text-to-LoRA via LoRA Reconstruction

  • Reconstruction training compresses a library of pre-trained LoRAs; one-hot or learnable task embeddings support compression but not zero-shot generation for unseen tasks.Conditioning on natural-language task descriptions enables generation for unseen tasks.Source: §3.2 · p. 4

3.3. Training Text-to-LoRA via Supervised Fine-Tuning

  • SFT trains T2L directly on fine-tuning datasets, avoiding intermediate LoRAs; it can implicitly cluster related tasks when their adapters occupy different minima.The paper reports that reconstruction-trained T2L fails to generalize to unseen tasks in this setting.Source: §3.3 · p. 4

4. Experiments

The experiments evaluate T2L’s ability to compress task-specific LoRAs and generate LoRAs for unseen benchmark tasks. They test multiple architectures and training schemes across a broad set of language-model benchmarks.

4. Experiments

  • The evaluation uses SNI training tasks and ten benchmarks spanning reasoning, math, science, coding, and world knowledge, with task-specific, averaged, and multi-task LoRA baselines.Arrow Routing results are compared indirectly because its adapters, training tasks, and possibly evaluation prompts differ.Source: §4 · p. 4

4.1. LoRA Compression

  • T2L recovers oracle-adapter performance when distilling nine benchmark-specific LoRAs, and exceeds those adapters on several benchmarks.The authors hypothesize lossy compression regularizes oracle weights that overfit on PIQA and Winogrande.Source: §4.1 · p. 5, Table 1 · p. 5
  • Across 16 to 479 training tasks, performance drops as reconstruction error rises, then remains around 65% of oracle performance even above 8 × 10^-4 per-element L1 error.At reconstruction loss below 10^-4, T2L fully recovers oracle performance.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2. Zero-Shot LoRA Generation

  • SFT-trained T2L improves over the multi-task LoRA across unseen benchmarks, though it does not fully close the gap with task-specific LoRAs.It outperforms oracle adapters on a subset of tasks.Source: §4.2 · p. 6, Table 2 · p. 6

5. Ablations and Analyses

The analyses examine how training scale, task embeddings, description alignment, training scheme, and generated-adapter structure affect T2L. SFT performs better than reconstruction training in the reported zero-shot comparison, while aligned descriptions and task-specific activation clusters support its adaptation behavior.

5.1. Increasing Training Compute Proportional to the Number of Training Tasks

  • With proportionally scaled training budgets, T2L generally benefits from more training tasks, though S does not benefit from extended training with 479 tasks.The experiments vary training tasks and scale the budget with dataset size.Source: §5.1 · p. 6, Table 3 · p. 7

5.2. Task Embedding Models

  • gte-large-en-v1.5 and Mistral-7B-Instruct task embeddings yield comparable zero-shot generalization for T2L trained on 128 tasks.The gte model uses the last-layer CLS activation; Mistral uses the last-token activation.Source: §5.2 · p. 6, Table 4 · p. 7

5.3. Varying Task Descriptions

  • Aligned training and unseen evaluation descriptions produce the best-performing reconstructed LoRAs, matching oracle performance; unaligned descriptions perform worse.The comparison uses reconstruction-trained T2L, which had seen benchmark training descriptions.Source: §5.3 · p. 7, Table 5 · p. 7

5.4. Training Schemes

  • 66.3 vs 61.83 benchmark performance averaged over model architectures: SFT-trained T2L outperforms reconstruction-trained T2L.The authors attribute reconstruction’s weaker generalization to the challenge of compressing task-specific adapters that may not lie near one another in weight space.Source: Table 6 · p. 7, §5.4 · p. 7

5.3. Varying Task Descriptions

  • Aligned descriptions can steer a GSM8K response to correctness, and different aligned descriptions can produce different reasoning paths.A low-quality description does not make the base model produce the correct response.Source: §5.3 · p. 7

5.5. Visualization of T2L Activations

  • For unseen task descriptions, SFT-trained T2L’s task-encoder and final-MLP activations cluster by benchmark task; semantically similar tasks also cluster together.The visualization probes each benchmark with three unseen descriptions.Source: §5.5 · p. 8

6. Related Work

Prior hypernetwork work uses learned task identifiers or narrower adaptation settings, while T2L generates LoRA adapters from natural-language task descriptions. The related work contrasts this approach with prefix-based and per-sequence methods and with concurrent description-based adaptation.

Hypernetworks for Adaptation

Hypernetworks for Zero-Shot LLM Adaptation

  • Gisting compresses task descriptions into prefix tokens, while Hyperdecoders generate adapters per input sequence rather than from user-provided task descriptions.The paper argues description-based adaptation gives users more control to steer outputs; Hyperdecoder adapters also cannot be efficiently fused into the base model.Source: §6 · p. 9
  • T2L differs from related hypernetwork work by targeting frontier instruction-tuned models, allowing task-embedder changes, and using generated descriptions.The authors report that generated descriptions increase T2L performance considerably.Source: §6 · p. 9
  • Concurrent work by Lv et al. (2024) assumes few-shot examples in the context vector, whereas T2L requires only a task description.Source: §6 · p. 9

7. Discussion and Limitations

The discussion identifies dependence on high-quality task descriptions as a real-world concern and leaves transfer from smaller to larger same-family models open. The limitations section further bounds the work to LoRA outputs and notes that zero-shot T2L does not yet match task-specific LoRA performance.

Discussion

  • Low-quality real-world task descriptions may degrade generated-adapter performance; transfer from smaller to larger models in the same architecture class remains open.Source: §7 · p. 9

Limitations

  • T2L is limited to LoRA output spaces, and its zero-shot adaptation still falls short of task-specific LoRA benchmark performance.The authors identify direct activation modulation and better inductive biases as possible directions for more efficient modulation and compression.Source: §7 · p. 9

Impact Statement

T2L generates task-specific LoRAs from natural-language descriptions, offering a route to rapid adaptation without per-task fine-tuning datasets or lengthy optimization. The authors report consistent gains over baselines across tested Llama and Gemma models, while noting sensitivity to description quality and uncertain generalization beyond related task types.

Impact Statement

  • T2L enables rapid description-based adaptation without per-task fine-tuning datasets, but poor descriptions may impair reliability and generalization beyond SNI-derived task types needs further study.The impact statement also cautions that T2L may not reach the performance ceiling of extensively fine-tuned adapters on highly complex or specialized domains.Source: Impact Statement · p. 10

A. Generalization to Llama and Gemma Models

B. Training Description Sources

Using SNI-provided task definitions reduced T2L’s zero-shot benchmark performance. The authors suggest that inconsistent templates or levels of detail in crowd-sourced descriptions may make learning and generalization harder.

B. Training Description Sources

  • SNI-provided task definitions reduced T2L’s zero-shot benchmark performance; the authors hypothesize that variation in crowd-sourced descriptions makes learning and generalization harder.Table 9 compares SFT-trained T2L using two training-description sources, but the supplied text does not give the table’s values.Source: B. Training Description Sources · p. 13, Table 9 · p. 13

C. Scaling the Number of Training Tasks with Fixed Compute

With roughly the same number of gradient steps, increasing the number of training tasks improved SFT-trained T2L’s average zero-shot benchmark performance. The authors interpret this as a hint of scalability and positive transfer.

C. Scaling the Number of Training Tasks with Fixed Compute

  • Increasing training tasks improved average zero-shot benchmark performance for SFT-trained T2L under roughly the same gradient-step budget.Table 10 reports results for 64, 128, 256, and 479 tasks; the supplied caption does not provide cell values.Source: C. Scaling the Number of Training… · p. 13, Table 10 · p. 14

D. LoRAs of Similar Tasks

T2L’s adaptation cost is reported as more than four times lower in FLOPs than 3-shot ICL, saving compute within the first question instance. The appendices also examine adapter similarity, model configurations, training, and the FLOPs calculation.

D. LoRAs of Similar Tasks

  • Adapters with task descriptions more similar to a benchmark performed better on it, although similar-description adapters were not similar in flattened A/B parameter space.The analysis measures flattened A/B cosine similarity and compares it with task-description embedding similarity and benchmark-relative performance.Source: D. LoRAs of Similar Tasks · p. 14, Figure 6 · p. 15

E. Hyperparameter Settings

  • Reconstruction training uses 100,000 epochs because each prediction target is an entire LoRA adapter, yielding 479 training samples for 479 SNI tasks.Table 11 and Listing 1 are identified as the sources of training configurations, but their settings are not included here.Source: E. Hyperparameter Settings · p. 14, Table 11 · p. 14

F. Additional Details of T2L Architectures

  • T2L’s architecture variants share a backbone but differ in output heads and embeddings, with L outputting A and B matrices, M one matrix, and S one rank at a time.The variants impose different output spaces and inductive biases; S is described as the most parameter-efficient.Source: F. Additional Details of T2L… · p. 15

G. T2L Intialization

  • Without explicit hypernetwork initialization, training is unstable and often fails; Bias-HyperInit sets the L variant’s output-head weights to zero and its bias to match underlying-layer initialization.The supplied passages also describe initializing the A-head bias and B-head bias to match traditional LoRA initialization.Source: G. T2L Intialization · pp. 15–16

H. Training Details

  • All models fit on one 80GB H100 GPU; reconstruction training takes about 10 minutes for 9 benchmark LoRAs and 10 hours for 479 SNI adapters.For reconstruction, T2L predicts z-scores of low-rank A and B weight entries, then rescales them using elementwise standard deviations and means at test time.Source: H. Training Details · pp. 16–17

I.1. T2L per instance FLOPs

  • 0.856005 TFLOPs/instance is the reported total for a base LLM question instance with T2L, including 0.827 TFLOPs for the base model.The 0.827 TFLOPs figure is reported for the base LLM without ICL; the total adds the stated T2L adaptation costs.Source: §I.1 · p. 17

I.2. Base LLM with 3-shot ICL

  • Based on this calculation, T2L’s adaptation cost is significantly cheaper than 3-shot ICL, with more than 4x FLOPs reduction and compute saved within the first question instance.Source: §I.2 · p. 17

J. Training and Evaluation Datasets

Using flattened ΔW matrices, the appendix reports positive correlation between task-embedding similarity and adapter similarity, but full-matrix reconstruction did not outperform low-rank reconstruction for zero-shot LoRA generation. It also documents the task data, evaluation procedures, and description-generation setup.

J. Training and Evaluation Datasets

J.1. Benchmark Details

  • GSM8K uses chain-of-thought response pre-filling, coding tasks use evalplus, and question-answering benchmarks use task-specific instruction templates without response pre-filling.HumanEval and MBPP use a Python response pre-fill; the provided passages do not specify the exact evaluation details for each benchmark.Source: §J.1.1 · p. 20, §J.1.2 · p. 20, §J.2 · p. 21

K. Utilizing Full Adaptation Matrix vs Low-Rank Matrices

  • Positive correlation appears between task-embedding similarity and adapter similarity in flattened ΔW space, but full-matrix reconstruction did not outperform low-rank reconstruction for zero-shot generation.The authors suggest that many A/B factor permutations can represent a given ΔW matrix.Source: Figure 11 · p. 21, K. Utilizing Full Adaptation Matrix vs… · p. 21

L. Generating Task Descriptions with a Foundation Language Model

  • GPT-4o mini generates 200 descriptions per task from ten prompts, each based on five sampled question-answer pairs, with example pairs and prompts designed to encourage diversity and avoid verbose extras.The generated descriptions are used for training and benchmark tasks.Source: L. Generating Task Descriptions with a… · p. 21, Figure 12 · p. 22

M. Example of Task Descriptions

  • The appendix provides examples of descriptions for three SNI training tasks and benchmark tasks including BoolQ, GSM8K, HumanEval, MBPP, Winogrande, PIQA, HellaSwag, ARC, and OpenBookQA.It also shows random descriptions.Source: Figure 13 · p. 23, Figure 14 · p. 24, Figure 15 · p. 25

N. Scaling Number of Descriptions per Task

  • Varying the number of descriptions per training task produced mixed benchmark-performance results, so the experiments consistently used 128 descriptions per task.Figure 19 presents the variation experiment; its supplied caption does not provide individual values.Source: N. Scaling Number of Descriptions per… · p. 28, Figure 19 · p. 28
Loading 2506.06105v2…