PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey

Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Dhagash Mehta, Stefano Pasquali, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Jian Pei, Carl Yang, Liang Zhao

arXiv:2305.18703v7cs.CLcs.AI

TL;DR

Directly applying LLMs to specialized domains faces challenges from domain expertise, knowledge elicitation, and model complexity. This survey systematically categorizes domain-specialization techniques and application domains, outlining the field’s overall picture and trends while identifying catastrophic forgetting as a limitation.

  • Problem

    Direct application of LLMs to domain-specific tasks is hindered by limited domain-specific expertise, knowledge elicitation, and model complexity.

  • Method

    The survey categorizes domain-specialization techniques by their access level to LLMs and reviews application domains that can benefit from specialization.

  • Results

    The survey outlines an overall picture and trends of LLM domain specialization.

  • Takeaways & Limitations

    The taxonomy covers domain-specific tasks across natural, social, and formal science fields and identifies application domains that may benefit from specialized LLMs.

  • Takeaways & Limitations

    Applying specialization to complex LLM architectures may lead to severe catastrophic forgetting.

Abstract

from arXiv · show

Large language models (LLMs) have significantly advanced the field of natural language processing (NLP), providing a highly useful, task-agnostic foundation for a wide range of applications. However, directly applying LLMs to solve sophisticated problems in specific domains meets many hurdles, caused by the heterogeneity of domain data, the sophistication of domain knowledge, the uniqueness of domain objectives, and the diversity of the constraints (e.g., various social norms, cultural conformity, religious beliefs, and ethical standards in the domain applications). Domain specification techniques are key to make large language models disruptive in many applications. Specifically, to solve these hurdles, there has been a notable increase in research and practices conducted in recent years on the domain specialization of LLMs. This emerging field of study, with its substantial potential for impact, necessitates a comprehensive and systematic review to better summarize and guide ongoing work in this area. In this article, we present a comprehensive survey on domain specification techniques for large language models, an emerging direction critical for large language model applications. First, we propose a systematic taxonomy that categorizes the LLM domain-specialization techniques based on the accessibility to LLMs and summarizes the framework for all the subcategories as well as their relations and differences to each other. Second, we present an extensive taxonomy of critical application domains that can benefit dramatically from specialized LLMs, discussing their practical significance and open challenges. Last, we offer our insights into the current research status and future trends in this area.

1 INTRODUCTION

LLMs offer broad task-solving capabilities, but specialized domains require customization because their knowledge, objectives, language practices, and constraints vary. This survey organizes domain-specialization techniques and application domains while identifying research challenges and future directions.

  • Motivation: Domain specialization customizes general-purpose LLMs with contextual data, domain knowledge, objectives, and constraints.These requirements reflect differences across fields, institutions, teams, and application settings.
  • Motivation: General-purpose LLMs are not one-fits-all solvers because professional tasks follow distinct utility functions and social, cultural, religious, legal, and ethical constraints.Domain knowledge may also be proprietary and unsuitable for disclosure to general-purpose models.
  • Challenges: Specialized domains demand current, accurate, and deep knowledge that pretrained LLMs cannot easily maintain as discoveries, regulations, and practices change.Keeping models fresh requires continuous data collection, processing, and computationally intensive retraining.
  • Challenges: Domain-specific concepts, terminology, and entity relationships can cause plausible but inconsistent answers or hallucinations without proper guidance.LLMs predict likely word sequences rather than definitive answers based on structured responses.
  • Contributions: The survey systematically classifies specialization methods by LLM accessibility into black-box, grey-box, and white-box categories, comparing their relationships, advantages, and disadvantages.The taxonomy is intended to help domain experts identify suitable techniques for target problem settings.
  • Contributions: It also categorizes representative application domains, explains their practical significance and open challenges, and synthesizes current research status and future trends.The survey supports cross-referencing techniques across application domains and discusses bottlenecks, open problems, and possible future directions.

2 TAXONOMY OF DOMAIN SPECIALIZATION

The survey organizes LLM domain-specialization techniques by model accessibility and presents their relationships, trade-offs, and a shared four-stage framework.

  • The taxonomy groups domain-specialization approaches into external augmentation, prompt crafting, and model fine-tuning according to LLM accessibility.These correspond to black-box, grey-box, and white-box assumptions, respectively.
  • Alternative taxonomies classify specialization methods by training strategy, intervention level, or evaluation and feedback mechanism.These include fine-tuning, training from scratch, mixed training, pre-training, fine-tuning-time, inference-time, fixed, dynamic, and user-feedback-based evaluation.
  • External augmentation incorporates domain knowledge through external resources or tools without modifying the LLM’s internal structure.It is the most accessible approach for users with limited computational resources or domain-specific data.
  • Prompt crafting designs task- and domain-specific prompts, while model fine-tuning updates the LLM’s parameters to incorporate domain knowledge directly.Prompt crafting can use gradient or loss information, whereas fine-tuning requires the greatest access and resources.
  • The approaches differ in specialization level, computational cost, implementation ease, and generalization.External augmentation and task-specific instructions are often less computationally expensive than updating the LLM’s knowledge, but may not achieve the same performance level.
  • The survey provides a common framework for black-box, grey-box, and white-box methods with four stages: Definition, Augmentation, Optimization, and Evaluation.The Definition stage specifies the domain, its objectives, and its constraints.

3 EXTERNAL AUGMENTATION FOR DOMAIN SPECIALIZATION

External augmentation specializes LLMs by supplying domain knowledge or connecting them to domain tools, improving access to relevant information and extending task capabilities. The survey highlights integration, scalability, adaptability, and information-conflict challenges.

  • External Augmentation: Retrieval augmentation has two forms: Domain Knowledge Augmentation supplies external context, while Domain Tool Augmentation integrates external systems, often through APIs.The former improves depth and accuracy in a field; the latter enables tasks beyond the model’s inherent abilities.
  • 3.1 Domain Knowledge Augmentation: Domain knowledge augmentation enriches LLM performance using documents, knowledge graphs, neural networks, or other domain-specific information.The survey distinguishes explicit knowledge, which is structured and directly usable, from implicit knowledge embedded latently in data or systems.
  • 3.1.1 Utilizing Explicit Knowledge with LLM.: External retrieval can anchor predictions in task-relevant context, allowing specific model outputs to be refined or corrected without frequent retraining.This is especially relevant when retrieved information contradicts the model’s memorized knowledge.
  • 3.1.1 Utilizing Explicit Knowledge with LLM.: An approximately 11-billion-parameter LLM coupled with an external knowledge base achieved similar performance to a 540B-parameter PaLM.LLMs are also used for retrieval reasoning and relevance verification, supporting more transparent retrieval processes.
  • 3.1.2 Utilizing Implicit Knowledge with LLM.: Implicit-knowledge augmentation uses latent representations, attention scores, softmax weighting, and weighted memory vectors to retrieve task-relevant information during inference.This approach can help models access relevant, current information, but transforming domain data into latent vectors requires extra processing.
  • Open Challenges: External-knowledge integration must handle incomplete or conflicting retrieved information, while scalable systems must manage expanding domain data and changing information.Existing methods commonly concatenate retrieved knowledge with the LLM input or intermediate layers.
  • 3.2 Domain Tool Augmentation: Domain tools combine specialized algorithms, knowledge, and functionality with an LLM interface that guides user interaction and resource use.Examples include genomics APIs, formal theorem provers, and sandbox environments for social-behavior simulation.

4 PROMPT CRAFTING FOR DOMAIN SPECIALIZATION

Prompt crafting specializes LLM behavior through task-specific inputs that guide content generation and set output expectations. The survey distinguishes discrete natural-language instructions from continuous learnable vectors.

  • Prompt Crafting: Prompts are task-specific input texts that guide LLM content generation, user-intention adherence, and response expectations.The survey presents prompt approaches as a way to elicit more accurate and less toxic responses.
  • Discrete Prompt: Discrete prompts use manually designed natural-language instructions to elicit domain-specific knowledge from an LLM’s parameter space.They express the task through language rather than learned prompt vectors.
  • Continuous Prompt: Continuous prompts use learnable vectors instead of manually designed text instructions and are discussed as an alternative specialization approach.The survey treats both approaches as mechanisms for domain adaptation and customization.

4.1 Discrete Prompt

Discrete prompting adapts frozen LLMs to new domains and tasks through zero-shot or few-shot natural-language instructions. The survey covers reasoning extensions, output stability, instruction optimization, and open challenges.

  • 4.1 Discrete Prompt: Discrete prompting elicits desired outputs from a frozen LLM using a natural-language prompt and test query, without updating the model’s parameters.The prompt serves as an instruction for applying previously learned knowledge to new situations.
  • 4.1.1 Zero-shot Discrete Prompts.: Zero-shot prompts contain a task description without illustrative examples, whereas few-shot prompts add a few illustrative labeled examples.Zero-shot prompting represents a cold-start setting with no supportive labeled sample.
  • 4.1.1 Zero-shot Discrete Prompts.: PADA generates a target-domain name and domain-related features for a test query before using them to predict task labels.This supports domain adaptation on tasks including sentiment analysis, image classification, semantic segmentation, and rumor detection.
  • 4.1.1 Zero-shot Discrete Prompts.: Zero-shot-CoT uses two-stage prompting: it first elicits reasoning sentences and then uses them to generate the final answer.The method adds “Let’s think step by step” before each answer and reports stronger performance than standard zero-shot prompting on arithmetic, symbolic, and logical reasoning tasks.
  • 4.1.2 Few-shot Discrete Prompts.: Few-shot prompts provide task-intention and output-format illustrations, yielding more stable output formats and more decent downstream-task performance.Chain-of-Thought adds intermediate reasoning steps, while Auto-CoT generates reasoning chains directly to avoid manually designing them for each test example.
  • 4.1.2 Few-shot Discrete Prompts.: Ensemble-based instruction aggregates outputs from multiple instructions, while recursive instruction decomposes an unseen complex task into easier subtasks.Programming-language-format instructions are also explored for structured reasoning tasks.
  • 4.1.3 Open Challenges.: Discrete prompts help LLMs leverage inherent knowledge for new domains, but raw outputs may require post-processing to extract rigorous predictions.The approach is also described as enhancing flexibility and adaptability across diverse situations and tasks.
  • 4.1.3 Open Challenges.: Open challenges include evaluating whether expert-curated or templated instructions are effective and automating their generation, selection, or combination.Domain experts and data scientists can analyze model performance and adjust instructions, although automatic evaluation is preferred.

4.2 Continuous Prompt

Continuous prompts specialize LLMs with trainable vectors while freezing the underlying model, providing a parameter-efficient route to task and domain adaptation. The survey covers prompt construction, initialization, transfer, lifelong learning, and optimization challenges.

  • 4.2 Continuous Prompt: Continuous prompt tuning replaces hard-coded instructions with trainable pseudo-token embeddings optimized using downstream loss while the LLM remains frozen.A template organizes the input and target, maps the sequence into embeddings, and feeds the resulting representation to the LLM.
  • 4.2 Continuous Prompt: Prompt-related parameters can comprise around 0.01% of the LLM’s total parameters, making continuous tuning parameter-efficient and controllable.The method aims to adapt models to customized tasks or domains while preserving general language understanding.
  • 4.2 Continuous Prompt: For classification, a verbalizer maps class labels to vocabulary words after a masked position is introduced into the template.For example, a positive label may map to the word “great.”
  • 4.2.3 Open Challenges.: Open challenges include naïve-prompt convergence, domain-specific improvement, and extending continuous prompt methods beyond preliminary successes.The survey describes naïve prompts as hard to converge and potentially suboptimal across scenarios.
  • 4.2.1 Task-dependent Prompt Tuning.: Prompt content can incorporate task-specific initialization and prior knowledge, including trainable virtual type words aligned with target relations in relation extraction.Optimization is sensitive to initialization because standard updates may remain in a limited neighborhood and fall into a local minimum.
  • 4.2.1 Task-dependent Prompt Tuning.: Prompt initialization influences optimization: random initialization is least efficient for small models, whereas initialization differences become less significant at 11B model size.The survey also reports that source-domain prompts can improve unseen target-domain performance and that well-initialized prompts can accelerate convergence.
  • 4.2.1 Task-dependent Prompt Tuning.: Transferred continuous prompts support cross-task and cross-model adaptation, while lifelong-learning methods reuse previous-task prompts or generate prior-task samples to address catastrophic forgetting.Progressive prompts concatenate prompts optimized for previous tasks with a tunable prompt for the current task.
  • 4.2.1 Task-dependent Prompt Tuning.: Continuous prompts can be prepended, appended, or inserted into input sentences, and Prefix-tuning also places tunable prompts in attention-block activations.Templates can reformulate tasks such as sentence classification as masked-word prediction.

5 MODEL FINE-TUNING FOR DOMAIN SPECIALIZATION

The survey presents model fine-tuning as a route to domain specialization, distinguishing adapter-based methods from task-oriented updates. These approaches tailor LLMs to specialized tasks or domains while addressing parameter, resource, and performance considerations.

  • Fine-tuning on smaller domain-specific datasets can improve performance in particular areas where general pretraining lacks adequate specialized knowledge.
  • Task-oriented Fine-tuning: Task-oriented fine-tuning modifies selected inner parameters to align an LLM with specific tasks, because updating all parameters can be impractical.Researchers therefore seek efficient ways to identify or update only subsets of parameters.
  • Adapter-based Fine-tuning: Adapter-based fine-tuning adds a small number of parameters through modules while keeping the original LLM parameters largely frozen.Adapters are inserted between layers and support domain or task adaptation without major changes to the model’s inner parameters.
  • Adapter-based Fine-tuning: Adapter strategies include modular domain and task adapters, sequential training, adapter fusion, weight averaging, and adapter selection by clustering or semantic similarity.
  • Adapter integration has yielded superior performance across varied settings, while efficient fine-tuning remains important for researchers and smaller organizations.
  • Task-oriented Fine-tuning: Instruction fine-tuning across tasks and datasets uses detailed task descriptions, and effective instructions can substantially improve performance on unseen tasks.The surveyed instruction-tuned model reportedly surpassed zero-shot and few-shot 175B GPT-3 on most unseen tasks.

6 APPLICATIONS OF LLM DOMAIN SPECIALIZATION

The survey reviews domain-specialized LLM applications across social, natural, and formal sciences. It describes uses in content generation, prediction, conversational assistance, biomedical research, Earth science, finance, law, HCI, and software engineering, alongside domain-specific data and ethical requirements.

  • The survey covers specialized LLM applications across social sciences, natural sciences, and formal sciences, including education, finance, law, biomedicine, Earth science, HCI, software engineering, and cybersecurity.
  • Cross-domain applications: Common applications include domain-specific generation and summarization, data-driven prediction and recommendation, conversational agents, expert systems, and code generation or analysis.
  • Biomedicine: In biomedicine, specialized LLMs support biological-function analysis, disease-mechanism and drug-discovery research, protein prediction, medical-record processing, diagnosis, treatment recommendation, and medical-image analysis.
  • Earth Science: Earth-science applications include question answering, dataset recommendation, satellite-data code generation, environmental responses, idea development, and climate-scenario generation.
  • Finance and Law: Finance and law require specialization for complex terminology, regulations, legal language, rulings, precision, formal tone, compliance, and changing domain information.The paper emphasizes ethical guardrails because financial and legal decisions are high stakes.
  • HCI and Software Engineering: HCI and software-engineering models can use conversational data, interaction logs, codebases, issue trackers, and documentation to support interface design, code generation, debugging, review, and documentation.

7 OPEN CHALLENGES AND FUTURE WORKS

The survey identifies recurring challenges in domain specialization, including domain complexity, knowledge balance, explainability, evolution, scalability, and resource demands. It proposes hybrid, automated, structured-knowledge, human-in-the-loop, and active-learning directions for future work.

  • Open Challenges: Domain specialization must handle specialized vocabularies, nuanced terminology, complex knowledge structures, and intricate relationships among domain entities and concepts.
  • Open Challenges: Models must balance domain knowledge with general knowledge, because excessive specialization can impair performance outside the target domain while excessive generality can dilute domain responses.
  • Open Challenges: Explainability remains difficult because increasingly complex models can be opaque, creating a trade-off between model complexity and transparency in high-stakes domains.
  • Open Challenges: Specialized models must adapt continuously to new terminology, concepts, trends, laws, regulations, and discoveries as domains evolve.
  • Open Challenges: Scaling specialization across many or complex domains requires computational resources, domain-specific data, and expertise, while efficient fine-tuning remains essential.
  • Future Works: Future research includes hybrid methods that combine black-box, grey-box, and white-box strategies to balance resource requirements and model performance, especially with scarce domain data.
  • Future Works: Other proposed directions include AutoML or meta-learning for strategy selection, structured world knowledge, human feedback, and active learning from uncertainty.

8 CONCLUSION

The conclusion frames direct LLM use in specialized domains as constrained by domain expertise, knowledge elicitation, and model complexity. The survey responds with taxonomies of specialization techniques and application domains, analyzes their relationships and trade-offs, and identifies practical significance and open challenges.

  • Limited domain-specific expertise, knowledge elicitation, and model complexity hinder direct LLM application to domain-specific tasks across natural, social, and formal sciences.
  • The survey systematically categorizes domain-specialization techniques according to their access level to LLMs.
  • It provides a comprehensive overview of application domains that can benefit from specialized LLMs.
  • The analysis discusses the advantages, disadvantages, and relationships among techniques and domains.
  • The survey aims to help domain experts identify suitable techniques for target problem settings and help data scientists understand practical significance and open challenges.
  • It highlights current research status and future trends while serving as a resource for researchers and practitioners pursuing advances across diverse domains.
Loading 2305.18703v7…