Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations

1 Introduction

• Full fine-tuning is increasingly costly and often infeasible, so this survey focuses on adapting pretrained or instruction-tuned LLMs without it.The motivating costs include training, memory, and deployment overhead.Source: §1 · p. 1
• The survey distinguishes weight-based, prompt-based, and embedding-based adaptation by where task information is encoded.Weights or added parameters, prompts, and task embeddings injected into activations are the respective locations.Source: §1 · pp. 1–2
• Prior PEFT-only and ICL-only surveys do not jointly organize the paradigms, especially emerging ICL-derived embedding methods.The paper says their independent development leaves cross-paradigm relationships and trade-offs underexplored.Source: §1 · p. 2
• The survey proposes a unified taxonomy and analyzes the paradigms’ relationships, strengths, limitations, and high-level trade-offs.The trade-offs include model access, training requirements, inference overhead, parameter efficiency, and deployment constraints.Source: §1 · p. 2
• The survey also identifies open challenges and future research directions for LLM task adaptation.Source: §1 · p. 2

• Task information is encoded in model weights, input prompts, or explicit embeddings injected into model activations; the taxonomy compares how each paradigm builds and injects that information.Source: §2 · p. 2, Table 1 · p. 3
• The comparison captures representative trends rather than strict properties of every method, and the survey traces how methods evolved in light of these trade-offs.Source: §2 · pp. 3–4
• Weight-based methods can perform strongly and stably, often comparably to full fine-tuning, but require parameter access and add training and serving overhead.Source: §2 · p. 3
• Prompt-based methods need only a model’s standard input-output interface, but long prompts and iterative optimization can increase inference costs and outcomes can depend on prompt construction.Source: §2 · p. 4
• Embedding-based methods typically use fewer parameters than weight-based approaches and leave weights unchanged, but require internal activations and learned variants can be unstable or costly to train.For Llama-3.1-8B, representative embedding methods use roughly 0.1K–130K parameters, compared with more than 3M for LoRA adapters.Source: §2 · p. 4

3 Weight-Based Adaptation

• Weight-based methods encode task information in model weights or additional trainable modules, with single-task and multi-task approaches addressing different adaptation challenges.Source: §3 · p. 4

3.1 Single-Task Adaptation

• LoRA variants target expressivity, stability, convergence, rank allocation, training memory, and parameter or storage efficiency through changes to updates and their training.Source: §3.1 · pp. 4–5
• Other PEFT methods use adapters, selective weight updates, activation scaling, combined techniques, or auxiliary networks to reduce trainable parameters or training costs.Source: §3.1 · p. 5

3.2 Multi-Task Adaptation

• Routing methods select or combine task-specific adapters to reduce the overhead of task switching and mixed-task batches, at task or instance level.Source: §3.2 · p. 5
• Model merging combines task vectors from fine-tuned models for multi-task inference without further fine-tuning; it can help when training data are inaccessible, though it typically underperforms multi-task training.Source: §3.2 · p. 5

4 Prompt-Based Adaptation

• Prompt-based adaptation provides instructions or input-output demonstrations in the inference prompt, with methods optimizing instructions or demonstrations to improve stability and performance.Source: §4 · p. 6

4.1 Instruction Optimization

• Instruction optimization commonly uses an optimizer LLM to refine instructions from feedback, including error analysis and iterative updates.Source: §4.1 · p. 6
• Other instruction methods use evolutionary search or optimize auxiliary components, such as soft prompts or the optimizer LLM itself.Source: §4.1 · p. 6
• Agent-skill methods and instruction optimization both modify prompt text without parameter updates, but skills encode reusable procedural knowledge for multi-step action and tool use.Source: §4.1 · p. 6

4.2 Demonstration Optimization

• Demonstration selection methods retrieve nearby examples or promote diversity, while performance is sensitive to which examples are chosen.Source: §4.2 · p. 6
• Demonstration optimization also treats ordering as a selection problem or uses sequential decision-making, including reinforcement-learning policies.Source: §4.2 · pp. 6–7
• When labels are unavailable, methods annotate unlabeled examples; other approaches generate synthetic demonstrations or add reasoning paths.Source: §4.2 · p. 7
• Many-shot ICL can improve performance and robustness to demonstration selection and ordering, but increases memory and computation and may suffer attention dispersion in long prompts.Source: §4.2 · p. 7

5 Embedding-Based Adaptation

• Embedding-based methods inject task embeddings into model activations, typically by addition or concatenation, and divide into ICL-derived and learned embeddings.Source: §5 · p. 7

5.1 ICL-Derived Task Embeddings

• Non-contrastive methods aggregate task-relevant activations from few-shot prompts and select injection locations using validation sweeps or optimization strategies.Some methods inject at attention-head rather than layer level; reported performance can be comparable to few-shot ICL, but evaluations largely target simpler tasks.Source: §5.1 · p. 7
• Contrastive methods derive steering vectors from differences between activations induced by prompt pairs; task-adaptation variants contrast correct demonstrations with incorrect or missing ones.Source: §5.1 · pp. 7–8

5.2 Learned Task Embeddings

• Prompt tuning optimizes learnable continuous embeddings prepended to the input, while later work adapts prompts per instance or transfers knowledge from source-task prompts.Source: §5.2 · p. 8
• Other learned methods optimize embeddings injected into internal activations, including layer-wise vectors and weighted combinations of attention-head outputs.Source: §5.2 · p. 8
• Learned embedding methods can require more training iterations than standard PEFT, and their conceptual similarity to soft prompting is often overlooked.Source: §5.2 · p. 8

Hybrid and Compositional Task Adaptation

• Instruction Prompt Tuning combines a shared soft prompt with task instructions and demonstrations, while how best to combine adaptation paradigms remains underexplored.The combined signals are described as jointly guiding behavior and improving alignment with domain-specific instruction semantics, safety, grounding, and completeness of long-form generations.Source: §6 · p. 8

Data Efficiency and Synthetic Supervision

• How synthetic demonstrations’ fidelity, bias, and diversity affect different adaptation paradigms has not been thoroughly studied.Most methods require a nontrivial number of labeled demonstrations, and LLM-generated data are an increasingly adopted scalable alternative to human-curated supervision.Source: §6 · pp. 8–9

Benchmarks for Evaluating Task Adaptation

• Evaluation should extend beyond established benchmarks to long-form generation, complex multi-step reasoning, and coherent behavior in long-horizon or interactive environments.Such settings may provide deeper insights into robustness, error accumulation, and long-range reasoning capabilities.Source: §6 · p. 9

Beyond Task Adaptation: Encoding Behavioral and Contextual Information

• A research direction is to extend adaptation techniques to encode behavioral and contextual signals, including safety, truthfulness, persona, and long-context knowledge.The passage notes that prompting can be limited for some attributes and long prompts can add inference overhead; recent methods instead encode information in model parameters or embeddings.Source: §6 · p. 9

7 Conclusion

• The survey organizes existing LLM task adaptation methods according to where task information is encoded.It presents this organization as a unified overview of task adaptation.Source: §7 · p. 9
• The framework connects weight-based, prompt-based, and embedding-based adaptations and discusses their relationships and trade-offs.Source: §7 · p. 9
• The survey discusses open problems shared across the adaptation paradigms and aims to serve as a reference for understanding their landscape.Source: §7 · p. 9

Limitations

• The taxonomy and comparisons may not fully capture the most recent developments because the area is progressing rapidly.Source: Limitations · p. 9
• The survey emphasizes high-level methodological distinctions and qualitative trade-offs rather than exhaustive empirical re-evaluation of individual methods.Source: Limitations · p. 9
• The survey focuses exclusively on text-based LLMs, leaving applications to multimodal LLMs and visual generative models for future work.Source: Limitations · p. 9

A Full Taxonomy of Efficient Task Adaptation in LLMs

• Figure 3 gives the full taxonomy and individual methods for each subcategory, unlike Figure 1’s main category branches.Source: A Full Taxonomy of Efficient Task… · p. 24

B.1 Comparison with Previous PEFT-only and ICL-only Taxonomies

• The survey compares its taxonomy with representative PEFT-only and ICL-only surveys selected by citation impact, subfield relevance, and publication or arXiv release year.Source: §B.1 · p. 24

PEFT-only surveys

• The listed representative PEFT-only surveys include works by Han et al. (2024), Wang et al. (2025b), and Mao et al. (2025).Source: PEFT-only surveys · p. 24

ICL-only surveys

• Unlike PEFT-only taxonomies centered on technical designs, this framework separates single-task from multi-task adaptation and includes model merging for settings without training data.Multi-task adaptation includes task switching or mixed-task batched inference; model merging uses accessible task-adapted models.Source: ICL-only surveys · pp. 24, 26
• The prompt-based taxonomy focuses on encoding and adaptation challenges, covering demonstration selection, annotation, generation, many-shot ICL, and recently developed instruction optimization.Earlier ICL surveys sparsely represent instruction optimization and do not systematically categorize several supporting directions.Source: ICL-only surveys · p. 26
• The unified taxonomy spans weight-based, prompt-based, and embedding-based adaptation, organizing methods around task-information encoding and adaptation challenges rather than a single subfield.It contextualizes conceptual connections while preserving insights developed within individual research areas.Source: ICL-only surveys · pp. 24, 26

B.2 Detailed Comparison of Weight-Based and Embedding-Based Adaptations

• The taxonomy classifies task information in model parameters as weight-based and information in activation-space representations, including soft prompts, as embedding-based.This places soft prompting under embedding-based adaptation even though it is often discussed alongside weight-based PEFT.Source: §B.2 · pp. 26–27
• Learned embedding methods and soft prompts share activation-space representations optimized through gradient-based training, a connection previous studies rarely discuss or compare.The taxonomy uses this shared encoding perspective to connect otherwise separately discussed adaptation approaches.Source: §B.2 · p. 27

B.3 Quantitative Meta-Analysis

• LoRA and (IA)3 achieve the highest scores among compared methods; SITE outperforms 10-shot ICL with 1.02K trainable parameters, while FV and LIVE score low.The results indicate that embedding-based methods do not uniformly improve performance, which depends on how task embeddings are constructed and injected.Source: §B.3 · p. 27
• On 10,000 AG_News test queries with Llama-3.1-8B on one NVIDIA A6000 GPU, 10-shot ICL takes longer than LoRA and SITE, while FV takes substantially longer.The runtime difference is attributed mainly to repeatedly processing long demonstration prompts for ICL and FV's additional intervention-head search; trained methods' runtimes include training and evaluation.Source: §B.3 · p. 27
• The comparison covers representative methods, one backbone, selected benchmarks, and specific settings, so it may not reflect trends across models, tasks, implementations, or hardware.The authors characterize it as a limited empirical snapshot rather than a general ranking of adaptation paradigms.Source: §B.3 · p. 27

B.4 Paper Collection Process

• The collection targets representative and influential work across the three adaptation paradigms, not an exhaustive list of related papers.The authors identify foundational papers, then expand the candidate set through citations, venue and baseline prominence, and recent-paper screening.Source: §B.4 · p. 27
• The authors iteratively review citations, baselines, and related-work discussions until the set covers the main methodological directions in each paradigm.They also monitor newly released work relevant to efficient task adaptation.Source: §B.4 · pp. 27–28
• Because the field evolves rapidly, recent or concurrent works may be missing, so the taxonomy is a structured snapshot of the literature at writing time.Source: §B.4 · p. 28

C Use of LLMs in This Work

• Chat-based LLMs were used for sentence-level grammar and clarity edits, which the authors reviewed and verified.The authors state that all conceptual contributions are theirs.Source: C Use of LLMs in This Work · p. 28