PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Large Language Models to Domains via Reading Comprehension

Daixuan Cheng, Shaohan Huang, Furu Wei

arXiv:2309.09530v4cs.CL

TL;DR

The paper asks whether continued pre-training on domain-specific corpora benefits large generative models, given that raw-corpus training adds domain knowledge but hurts prompting. It transforms raw texts into reading-comprehension texts and reports improved domain-task performance across three domains, with gains also reported on general benchmarks.

  • Problem

    The paper examines whether domain-adaptive pre-training benefits large generative models, since raw-corpus training adds domain knowledge but drastically hurts prompting.

    Source: §1 · p. 2

  • Method

    The method enriches each domain-specific raw text with content-related comprehension tasks and mixes the resulting texts with general instructions.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Across biomedicine, finance, and law, AdaptLLM outperforms the general language model on domain tasks, and reading-comprehension texts improve performance on general benchmarks.

    Source: §5 · p. 7, §9 · p. 10

  • Takeaways & Limitations

    The reported gains on general benchmarks suggest potential for improving general language models across more domains.

    Source: §9 · p. 10

    Limited supervised datasets prevented construction of a finance knowledge-probing test.

    Source: §2 · p. 3

Abstract

from arXiv · show

We explore how continued pre-training on domain-specific corpora influences large language models, revealing that training on the raw corpora endows the model with domain knowledge, but drastically hurts its prompting ability for question answering. Taken inspiration from human learning via reading comprehension--practice after reading improves the ability to answer questions based on the learned knowledge--we propose a simple method for transforming raw corpora into reading comprehension texts. Each raw text is enriched with a series of tasks related to its content. Our method, highly scalable and applicable to any pre-training corpora, consistently enhances performance across various tasks in three different domains: biomedicine, finance, and law. Notably, our 7B language model achieves competitive performance with domain-specific models of much larger scales, such as BloombergGPT-50B. Furthermore, we demonstrate that domain-specific reading comprehension texts can improve the model's performance even on general benchmarks, showing the potential to develop a general model across even more domains. Our model, code, and data are available at https://github.com/microsoft/LMOps.

1 INTRODUCTION

The paper investigates continued pre-training as a way to add domain knowledge to large generative models, after preliminary experiments found that raw-corpus training can hurt prompting. It proposes enriching raw texts with content-related comprehension tasks, then reports improved performance across biomedicine, finance, and law.

1 INTRODUCTION

  • Domain-adaptive pre-training has been effective for natural language understanding models, motivating investigation of whether it also benefits large generative models.The paper frames this as an exploration of continued training on domain-specific corpora.Source: §1 · p. 2
  • The proposed method transforms each raw text into reading-comprehension material by adding tasks relevant to its content and diverse general instructions.The tasks are intended to help retain natural-language question-answering ability based on the raw-text context.Source: §1 · p. 2
  • Experiments report that the method improves performance across domain-specific tasks in biomedicine, finance, and law.The resulting model is called AdaptLLM, and the authors envision extending the approach across more domains.Source: §1 · p. 3
  • Raw-corpus continued pre-training gives LLMs domain knowledge but drastically hurts prompting, motivating a method to preserve prompting ability while learning from domain corpora.The authors also report that the approach improves performance across tasks in biomedicine, finance, and law.Source: §1 · p. 3

2 PRELIMINARY EXPLORATION ON DOMAIN-ADAPTIVE PRE-TRAINING

The preliminary experiments examine domain-adaptive pre-training on biomedicine, finance, and law using prompting, fine-tuning, and knowledge probing. Raw-corpus training improves domain knowledge and fine-tuning results but generally weakens prompting; finance lacks a knowledge-probing test because supervised datasets are limited.

2 PRELIMINARY EXPLORATION ON DOMAIN-ADAPTIVE PRE-TRAINING

  • The experiments continue training general LLaMA on domain-specific raw corpora, then assess prompting, fine-tuning, and knowledge probing within each domain.The domains are biomedicine, finance, and law.Source: §2 · p. 3
  • Fine-tuning improves across all three domains, while prompting drops across most domains after raw-corpus domain-adaptive pre-training.The authors interpret the divergent pattern as domain knowledge gains accompanied by reduced prompting ability.Source: §2 · p. 3, Table 1 · p. 3
  • Knowledge-probing results improve after domain-adaptive pre-training in biomedicine and law, indicating domain knowledge acquisition in those domains.The probing results are reported for biomedicine and law.Source: §2 · p. 3
  • Finance has no knowledge-probing test because supervised datasets were too limited to construct one.Source: §2 · p. 3
  • Limited diversity in domain-specific pre-training corpora may reduce the diversity of input-output patterns and thereby weaken prompting ability.The authors present this as a possible explanation for the prompting decline.Source: §2 · p. 4

3 ADAPTING LARGE LANGUAGE MODELS VIA READING COMPREHENSION

The method turns raw domain texts into reading-comprehension material by following each text with content-related tasks. It also mixes in general instructions to broaden input-output patterns and support prompting ability.

3 ADAPTING LARGE LANGUAGE MODELS VIA READING COMPREHENSION

  • Each raw text is treated as a reading phase followed by question-answering comprehension tasks, with general instructions added to diversify input-output patterns.The tasks aim to help the model answer questions from the text, while the added instructions further support prompting ability.Source: §3 · p. 4

3.1 CREATING READING COMPREHENSION TEXTS

  • Self-supervised task mining uses regex patterns to find intrinsic tasks, then paraphrased and reversed templates turn mined examples into diverse input-output pairs.Table 2 describes mining patterns and corresponding templates; reversing a task exchanges its input and output.Source: §3.1 · p. 4, Table 2 · p. 5
  • Word-to-Text tasks use domain-specific keywords as prompts for sentence generation, while the task can also be reversed to extract keywords from a sentence.Keywords are selected from domain vocabulary absent from the general model vocabulary; sentences with more than three such keywords are chosen for tasks.Source: §3.1 · pp. 4–5
  • Tasks include cause-and-effect questions and premise-hypothesis inference, mined from causal cues and relation verbalizers, with generation templates for entailment.For commonsense reasoning, one template asks for the reason behind a sentence; entailment generation uses the second sentence as its target.Source: §3.1 · p. 5
  • Regex mining may link sentences that are not strictly equivalent, so the method reformats paraphrase classification as sentence generation to reduce dependence on label accuracy.The paper gives sentences linked by “Similarly” as an example of a potentially unreliable match.Source: §3.1 · p. 6
  • Text completion inserts prompts between sentences to elicit the next section and requires no task-specific mining patterns, making it applicable to any raw text.Other mined tasks include summarizing a text or generating an article from its title, and identifying sentence topics.Source: §3.1 · pp. 4, 6

3.2 MIXING WITH GENERAL INSTRUCTIONS

  • Because mined tasks may not cover real-world task diversity, the method mixes reading-comprehension texts with general instructions to cover a wider range of input-output patterns.Source: §3.2 · p. 6

4 EXPERIMENT SETTINGS

The experiments continue LLaMA-7B training on domain corpora and vary the mix of reading-comprehension texts and general instructions. They evaluate domain-specific tasks across biomedicine, finance, and law.

4 EXPERIMENT SETTINGS

  • Pretraining uses PubMed abstracts for biomedicine, FreeLaw opinions for law, and financial news from May 2022 to May 2023 covering over 7,000 stocks.General instructions come from LIMA, WizardLM, and Orca.Source: §4 · p. 6
  • The study continues training LLaMA-7B and reports optimal reading-comprehension-text-to-general-instruction ratios of 1:1 for biomedicine, 1:2 for finance, and 1:1 for law.Source: §4 · p. 6
  • Access to earlier financial news is limited, constraining the available finance pretraining corpus.Source: §4 · p. 6
  • Each raw text yields about two input-output examples on average, with at most two examples per task subcategory and randomly sampled paraphrased or reversed templates.Source: §4 · p. 7
  • Evaluation covers five biomedical tasks, five finance tasks, and three law tasks.The biomedical set comprises PubMedQA, ChemProt, MQP, RCT, and USMLE; finance includes ConvFinQA, FPB, FiQA SA, Headline, and NER; law includes SCOTUS, CaseHOLD, and UNFAIR-ToS.Source: §4 · p. 7

5 MAIN RESULTS

Across the three domains, AdaptLLM counteracts the prompting-performance decline associated with raw-corpus domain-adaptive pretraining and outperforms the general language model. In finance, AdaptLLM-7B performs competitively with BloombergGPT-50B.

5 MAIN RESULTS

  • AdaptLLM counteracts the prompting-performance decline from raw-corpus DAPT and outperforms the general language model across various tasks in biomedicine, finance, and law.Source: §5 · p. 7
  • MedAlpaca's advantage over LLaMA is inconsistent across biomedical tasks; the authors suggest its instructions may not fully infuse domain knowledge or cover varied input-output scenarios.Source: §5 · p. 7
  • AdaptLLM-7B achieves competitive performance with BloombergGPT-50B in finance, while LLaMA-7B scores lower than BloombergGPT-50B.BloombergGPT was trained from scratch on a mixture of financial and general corpora.Source: §5 · p. 7
  • In law, LexGPT has negative prompting results relative to GPT-J, whereas the paper reports positive prompting results for its method.LexGPT uses vanilla domain-adaptive pretraining on raw legal corpora.Source: §5 · p. 7

6 ABLATIONS ON TRAINING DATA

The ablations compare training-data choices for prompting and show benefits from both reading-comprehension texts and general instructions. Reading-comprehension task types also differ in their effectiveness on domain-specific tasks.

6 ABLATIONS ON TRAINING DATA

  • Reading-comprehension texts improve prompting across adapted domains, and adding general instructions improves results further.Compared with general instructions alone, reading-comprehension texts provide domain knowledge that improves domain-specific task performance.Source: §6 · p. 7, Table 5 · p. 8
  • The prompting comparison includes general language models, vanilla DAPT models, AdaptLLM, and domain-specific models MedAlpaca, BloombergGPT, and LexGPT.The caption identifies AdaptLLM-13B in biomedicine as trained from LLaMA-13B and AdaptLLM-6B in law as trained from GPT-J-6B.Source: Table 4 · p. 8
  • Word-to-Text and Natural Language Inference are the most effective comprehension task types on domain-specific tasks.The task-type ablations are reported in Appendix E.Source: §6 · p. 8

7 ANALYSIS OF DOMAIN KNOWLEDGE AND PROMPTING ABILITY

Reading-comprehension texts are intended to impart domain knowledge while improving prompting ability. Across the evaluations, they improve domain-specific fine-tuning and prompting performance, including on most general task types.

Analysis framing

  • The analysis treats domain knowledge and prompting ability as two distinct objectives of reading-comprehension training.Figure 3 separates domain-task fine-tuning results from general-task prompting results.Source: §7 · p. 8, Figure 3 · p. 9

Domain Knowledge

  • Reading-comprehension training consistently improves domain-specific fine-tuning results, and outperforms raw-text training in every adapted domain.The authors suggest diverse comprehension tasks may create a multi-task instruction-tuning setting that benefits single-task fine-tuning.Source: §7 · p. 9

Prompting Ability

  • The prompting evaluation uses zero-shot general benchmarks, including reading comprehension and closed-book question answering, with and without contexts.It includes at least three general tasks per comprehension-task type.Source: §7 · p. 9
  • Converting raw texts to reading-comprehension texts consistently improves prompting across task types, often outperforming the general model even without general instructions.Figure 3 averages task scores within task types and then across the three adapted models.Source: §7 · p. 9, Figure 3 · p. 9

8 RELATED WORK

Related work on domain-specific language models is organized around training from scratch, instruction fine-tuning, and retrieval-augmented prompting. Each approach offers a different route to domain adaptation, with distinct constraints or benefits.

Related-work overview

  • The paper groups these approaches as training from scratch, instruction fine-tuning, and retrieval-augmented prompting.These are the three categories used to organize recent domain-specific LLM work.Source: §8 · p. 9

Training from Scratch

  • Training from scratch is intuitive but carries substantial computational and data requirements, motivating lower-cost adaptation methods.BloombergGPT is presented as an early financial-domain example trained on financial and general corpora.Source: §8 · p. 10

Instruction Fine-tuning

  • Instruction fine-tuning is cost-effective, but limited fine-tuning data may not provide sufficient domain knowledge, while generating large-scale QA data can be costly.The passage describes domain-specific question-answering instruction fine-tuning and LLM-generated QA pairs.Source: §8 · p. 10

Retrieval-augmented Prompting

  • Retrieval-augmented prompting adds external domain information without changing model parameters and can help answer domain-specific questions and address hallucination.Models may need to accept or reject retrieved information when it is incomplete or conflicting.Source: §8 · p. 10

9 CONCLUSION

The paper concludes that continued training on domain-specific reading-comprehension texts can acquire domain knowledge while improving prompting across three domains. Its probing tests use next-token or next-sentence prediction formats aligned with generative language-model pretraining.

9 CONCLUSION

  • Reading-comprehension texts support domain knowledge acquisition and prompting improvements across biomedicine, finance, and law, and also improve general benchmarks.The authors describe this as suggesting potential to improve general language models across more domains.Source: §9 · p. 10

A DOMAIN KNOWLEDGE PROBING

  • The probing tests adapt the pretraining-style prediction format to assess inherent model knowledge without changing architecture or parameters.Inspired by LAMA, the tests use predict-the-next-token or predict-the-next-sentence formats; the reported results show domain knowledge after training on raw or reading-comprehension texts.Source: A DOMAIN KNOWLEDGE PROBING · p. 17, Table 6 · p. 17
  • For biomedicine probing, MedMCQA questions are filtered to remove instruction-following and fill-in-the-blank formats, then evaluated across 21 medical subjects.The reported score is the average across subjects, using per-option token likelihoods without demonstrations.Source: A DOMAIN KNOWLEDGE PROBING · p. 17
  • For law probing, LEDGAR contract provisions are converted from 100-way classification into four-choice prediction, with three distractor classes selected randomly.Evaluation uses the template “{CONTRACT} The topic is” and averages results across 100 law topics.Source: A DOMAIN KNOWLEDGE PROBING · p. 17

B DOMAIN-ADAPTIVE PRE-TRAINING

The section describes continued-training data mixtures and how raw texts are converted into comprehension texts, including domain-specific title sourcing and regex-based example mining.

B DOMAIN-ADAPTIVE PRE-TRAINING

  • Training mixes domain-specific raw or reading-comprehension texts with general instructions at ratios of 1:2, 1:1, or 2:1, and concatenates documents with </s>.Documents may be raw texts, reading-comprehension texts, or general instructions.Source: B DOMAIN-ADAPTIVE PRE-TRAINING · p. 17

C CREATING READING COMPREHENSION TEXTS

  • Summarization titles come from PubMed abstracts, collected finance news, or available law-section titles, with law texts split into sections for text construction.PubMed uses the first sentence as the title; a law section, rather than a whole raw text, becomes the basis for one reading-comprehension text.Source: C CREATING READING COMPREHENSION TEXTS · p. 18
  • Raw texts are truncated to 1,800 tokens, and verbalizers and keywords are expanded into regex patterns to extract matching sentences.The truncation leaves room for comprehension tasks within a maximum sequence length of 2,048 tokens.Source: C CREATING READING COMPREHENSION TEXTS · p. 18

D DOMAIN-SPECIFIC TASKS EVALUATION

The evaluation uses varied prompts and task-specific scoring for prompting, alongside a fixed question-answering template for fine-tuning.

D DOMAIN-SPECIFIC TASKS EVALUATION

  • Prompting randomly samples one of multiple templates per example; multiple-choice predictions use per-token option likelihoods, while text completion uses greedy search.Random sampling is used to mitigate variation from template sensitivity.Source: D DOMAIN-SPECIFIC TASKS EVALUATION · p. 18
  • Tables document regex keywords, task-dataset specifications including demonstration counts, and prompt-template examples with paraphrased variants.Source: Table 9 · p. 19, Table 10 · p. 19, Table 11 · p. 20
  • For UNFAIR-ToS, a prediction is correct when its highest-likelihood option belongs to the gold label set; several challenging biomedical tasks use balanced few-shot demonstrations.The biomedical tasks named are MQP, RCT, and ChemProt.Source: D DOMAIN-SPECIFIC TASKS EVALUATION · p. 19
  • Fine-tuning converts task examples into question-answer pairs with a fixed template and trains for one epoch without warm-up.Evaluation omits demonstrations and uses the same template as training.Source: D DOMAIN-SPECIFIC TASKS EVALUATION · p. 19

E FURTHER ABLATIONS ON COMPREHENSION TASKS

The ablations show that comprehension-task types contribute differently to domain-specific and general-benchmark performance, while further analyses examine task examples, model portability, verbalizers, and raw-corpus patterns.

E FURTHER ABLATIONS ON COMPREHENSION TASKS

  • Word-to-Text, Summarization, and Text Completion are the most frequent mined types; Natural Language Inference removal also caused a substantial drop, while Commonsense Reasoning and Paraphrase Detection had less pronounced effects.The authors suggest the NLI effect may reflect its unique role as the only classification task, and that more diverse examples could help domain-specific performance.Source: E FURTHER ABLATIONS ON COMPREHENSION… · pp. 21–22
  • Removing any comprehension-task type lowered domain-specific task performance, with noticeable declines after removing Word-to-Text, Summarization, or Text Completion.Source: E FURTHER ABLATIONS ON COMPREHENSION… · p. 21
  • Removing Summarization, Word-to-Text, Natural Language Inference, or Commonsense Reasoning lowered corresponding general-task scores, but removing Paraphrase Detection or Text Completion did not.The authors relate these exceptions to a generation-versus-classification mismatch for paraphrase detection and the lack of clear text-completion counterparts in general benchmarks.Source: E FURTHER ABLATIONS ON COMPREHENSION… · p. 22

F ANALYSIS OF DOMAIN KNOWLEDGE AND PROMPTING ABILITY

  • Tables 12 and 13 frame the analysis around fine-tuning on domain-specific tasks and prompting on general benchmarks, contrasting general, raw-text, and reading-comprehension training.Source: Table 12 · p. 22, Table 13 · p. 23

H APPLICABILITY TO OTHER MODELS

  • The method consistently improved domain-specific performance for Pythia-70M, LLaMA-13B, and LLaMA-2-Chat-7B in biomedicine.The LLaMA-13B setup used 512-token sequences and 40k continued-training steps.Source: H APPLICABILITY TO OTHER MODELS · p. 28
  • For LLaMA-2-Chat, comprehension data is converted into multi-turn conversations, while training keeps each input to one conversation because concatenation caused gradient overflow.Source: H APPLICABILITY TO OTHER MODELS · p. 28

I ANALYSIS ON VERBALIZER EFFECT

Loading 2309.09530v4…