PaperlayerView exact revision on arXiv

Source-linked AI summary

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, Ming Zhou

arXiv:2002.01808v5cs.CLcs.LG

TL;DR

Pre-trained models may miss rich knowledge, while updating shared parameters for new knowledge can cause earlier knowledge to fade. K-ADAPTER freezes the backbone and trains separate adapters for each knowledge type; across three knowledge-driven tasks, the adapters improve performance individually and together.

  • Problem

    Existing knowledge-injection methods update pre-trained model parameters, so injecting multiple knowledge types may cause previously injected knowledge to fade and representations to become entangled.

  • Method

    K-ADAPTER keeps RoBERTa fixed and trains independent knowledge-specific adapters for factual knowledge from Wikipedia–Wikidata alignments and linguistic knowledge from dependency parsing.

  • Results

    Across relation classification, entity typing, and question answering, each adapter improves performance, their combination brings further improvements, and K-ADAPTER outperforms RoBERTa consistently.

  • Takeaways & Limitations

    K-ADAPTER supports continual infusion of multiple knowledge types while preserving the original pre-trained model representation and producing versatile knowledge-infused representations.

Abstract

from arXiv · show

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K-Adapter, a framework that retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowledge in this work, including (1) factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis indicates that K-Adapter captures versatile knowledge than RoBERTa.

1 Introduction

Pre-trained language models can struggle to capture rich factual and linguistic knowledge, while existing knowledge-injection methods update shared parameters and may forget earlier knowledge. K-ADAPTER addresses this with knowledge-specific adapters, and experiments show consistent gains across knowledge-driven tasks.

  • Motivation: Unsupervised pre-trained models struggle to capture rich factual knowledge and the meaning of negation.These limitations motivate knowledge injection into models such as BERT and RoBERTa.
  • Motivation: Existing knowledge-injection methods update the entire model, so adding new knowledge may fade previously injected knowledge and entangle representations.Entangled representations also make it difficult to investigate the effect of each knowledge type.
  • K-ADAPTER: K-ADAPTER keeps the original pre-trained representation unchanged while supporting different representations for different infused knowledge types.The framework uses compact neural adapters plugged outside the pre-trained model.
  • K-ADAPTER: K-ADAPTER uses RoBERTa with factual and linguistic adapters trained on aligned Wikipedia–Wikidata knowledge and dependency-parsing data.The adapters are trained independently, reducing trainable parameters relative to RoBERTa.
  • Results: Across six benchmark datasets and three knowledge-driven tasks, K-ADAPTER consistently outperforms RoBERTa and reaches state-of-the-art performance on five datasets.The evaluated tasks are relation classification, entity typing, and question answering.

2 Related Work

Prior work injects varied knowledge through knowledge-specific objectives and often updates BERT parameters. K-ADAPTER instead combines factual and linguistic objectives with frozen base parameters and independently trained adapters for continual knowledge infusion.

  • Prior approaches: Prior methods inject knowledge from sources such as Wikidata, WordNet, entity linking, or replaced entity mentions using knowledge-aware objectives.Examples include entity alignment, relation classification, supersense prediction, and entity discrimination.
  • K-ADAPTER differences: K-ADAPTER considers both predicate or relation prediction and dependency relation prediction as knowledge-infusion objectives.This combines fact-related and linguistic-related supervision.
  • K-ADAPTER differences: K-ADAPTER clamps the original BERT parameters and trains different adapters independently, supporting continual learning without loss of previously injected knowledge.The adapters are not entangled during learning, enabling independent infusion of additional knowledge types.

3 K-ADAPTER

K-ADAPTER keeps RoBERTa frozen and attaches independently trained, knowledge-specific adapters that provide separate representations. The framework instantiates factual and linguistic adapters using relation classification and dependency relation prediction.

  • Framework: K-ADAPTER injects each knowledge type into a separate compact adapter while keeping the pre-trained model representation fixed.Adapters are plugged outside the pre-trained model and consume hidden states from its intermediate layers.
  • Adapter structure: Each adapter contains adapter layers with transformer layers, two projection layers, and a skip-connection across the projections.The design places adapter layers among transformer layers of the pre-trained model.
  • Training and use: Adapters are pre-trained independently on different tasks, and downstream models concatenate outputs from multiple adapters before task-specific layers.A single adapter can provide the input feature alone, while multiple adapters contribute jointly.
  • Implementation: RoBERTaLARGE is frozen during training, while each adapter has about 42M trainable parameters and uses settings N = 2, HA = 768, AA = 12, Hu = 1024, and Hd = 768.The smaller adapter size makes training memory efficient.
  • Factual adapter: The factual adapter is trained for relation classification on 430 relations and 5.5M sentences from aligned Wikipedia abstracts and Wikidata triples.It classifies relation labels for entity pairs using concatenated RoBERTa and adapter features.
  • Linguistic adapter: The linguistic adapter is trained for dependency relation prediction on 1M examples derived from Book Corpus with an off-the-shelf Stanford dependency parser.Its task is to predict the head index of each token.

P R Mi-F1 Acc Ma-F1 Mi-F1

The supplied passages identify Table 2 as reporting entity-typing results on OpenEntity and FIGER, but do not provide its numeric entries or a complete comparison for this section.

  • Results: Table 2 reports results on the OpenEntity and FIGER entity-typing datasets.The available passage provides the table identity but not its values.

4 Experiments

K-ADAPTER is evaluated across three knowledge-driven tasks using factual and linguistic adapters, with experiments showing consistent gains over RoBERTa and strong improvements from combining adapters.

  • Experimental scope: K-ADAPTER is evaluated on entity typing, question answering, and relation classification across six benchmark datasets.The model variants include both factual and linguistic adapters, or either adapter individually.
  • Entity Typing: 1.38% F1 over RoBERTa is achieved on OpenEntity by K-ADAPTER (F+L).On FIGER, it improves over WKLM by 2.88% macro F1, 2.54% micro F1, and 1.60% accuracy.
  • Ablation analysis: Knowledge, rather than additional parameters, accounts for the gains: K-ADAPTER (F) significantly outperforms the knowledge-free ablation.The knowledge-free model performs slightly worse than RoBERTa on OpenEntity.
  • Question Answering: 4.01% F1 over WKLM is achieved by K-ADAPTER (F+L) on SearchQA, with slight gains over WKLM+Ranking.On Quasar-T, it outperforms WKLM by 3.1% F1 and slightly outperforms WKLM+Ranking.
  • Relation Classification: K-ADAPTER models significantly outperform all baselines on TACRED relation classification.The results also show larger gains than RoBERTa+multitask, supporting individually injected knowledge through adapters.
  • Case study and probing: Factual knowledge helps K-ADAPTER correct relation predictions and produce more accurate LAMA objects than RoBERTa.These case-study and probing results indicate richer factual knowledge in K-ADAPTER.

5 Conclusion

K-ADAPTER infuses knowledge while keeping the original pre-trained parameters unchanged, enabling continual addition of knowledge through separate adapters. Experiments show improvements from individual adapters and further gains when combining them.

  • K-ADAPTER keeps the original parameters of pre-trained models unchanged while supporting continual knowledge infusion.New injected knowledge does not affect parameters learned for previously injected knowledge.
  • Factual and linguistic knowledge are infused into RoBERTa through separate adapters pretrained on relation classification and dependency relation prediction.The adapters target factual and linguistic knowledge with different pre-training tasks.
  • Each adapter improves downstream performance individually, while combining both produces further improvement across three knowledge-driven tasks.The evaluated tasks are relation classification, entity typing, and question answering.
  • Analyses suggest that K-ADAPTER captures richer factual and commonsense knowledge than RoBERTa.

A Probing Experiments

The probing experiments evaluate whether K-ADAPTER captures factual knowledge beyond what can be inferred from entity names. K-ADAPTER outperforms RoBERTa on multiple LAMA settings, although BERT performs better than RoBERTa in the reported comparison.

  • Probe setup: LAMA probes factual knowledge by requiring zero-shot answers to cloze-style relational questions without fine-tuning.LAMA-UHN further filters queries that are easy to answer from entity names alone.
  • Settings: The experiments compare ELMo, ELMo5.5B, Transformer-XL, BERTLARGE, and RoBERTaLARGE on LAMA-GoogleRE, LAMA-T-REx, and LAMA-UHN.
  • Results and Discussion: BERTLARGE performs better than RoBERTaLARGE, possibly because byte-level BPE may divide proper nouns into bytes and harm factual learning.The paper therefore excludes BERT from the following experiments.
  • Results and Discussion: K-ADAPTER outperforms other models except BERT by a large margin in the reported LAMA comparison.
  • Results and Discussion: Compared with RoBERTaLARGE, K-ADAPTER improves P@1 by 2.2% on Google-RE and 1.2% on T-REx, and also performs better on LAMA-UHN.

B Pre-Training Details

The factual and linguistic adapters are trained separately while the pre-trained model remains fixed. Their training uses different objectives, sequence lengths, epochs, batch sizes, and learning rates.

  • Factual adapter: During factual-adapter training, the pre-trained model is fixed and the randomly initialized adapter is trained with cross-entropy loss.Training uses five epochs, batch size 128, maximum sequence length 64, and AdamW with initial learning rate 2e-5.
  • Linguistic adapter: During linguistic-adapter training, the pre-trained model is fixed and the randomly initialized adapter is trained with BCEWithLogits loss.Training uses ten epochs, batch size 256, maximum sequence length 128, and AdamW with initial learning rate 1e-5.

C Applying K-adapter on Downstream Tasks

K-ADAPTER applies knowledge-enhanced representations by combining the pre-trained model’s general features with adapter-specific features before the downstream task layer. The framework overview presents knowledge-specific adapters trained on pre-training tasks.

  • Applying K-adapter on Downstream Tasks: For downstream tasks, K-ADAPTER concatenates pre-trained-model output features with adapter features before feeding them to the task-specific layer.This combines general information from the pre-trained model with specific knowledge from the adapter.
  • Applying K-adapter on Downstream Tasks: Figure 5 presents the downstream-use procedure for combining pre-trained and adapter representations before task prediction.
  • Applying K-adapter on Downstream Tasks: Figure 3 overviews K-ADAPTER as training a knowledge-specific adapter on a pre-training task to inject specific knowledge.

D Dataset statistics

This section reports dataset coverage and experimental settings for relation classification, entity typing, and question answering, including implementation choices and fine-tuning procedures.

  • Dataset coverage: The evaluation covers TACRED for relation classification, Open Entity and FIGER for entity typing, and CosmosQA, SearchQA, and Quasar-T for question answering.Dataset statistics are summarized separately for relation classification/entity typing and question answering.
  • Experimental protocol: Experiments select hyperparameters on validation data and use AdamW, with random seed 42 for reproducibility.The reported search includes batch size, learning rate, and warmup-step choices.
  • Knowledge-injection example: Relation classification uses an example in which “Barack Obama was born in Honolulu” yields the relation “Birth-of-place.”The example frames relation classification as a pre-training task for injecting knowledge into K-ADAPTER.
  • Fine-tuning procedure: K-ADAPTER is fine-tuned like RoBERTa or BERT, with adapter parameters fixed during fine-tuning.RoBERTa parameters are trainable and initialized from a Huggingface checkpoint.
  • Experimental protocol: Question-answering experiments use maximum sequence lengths of 256 for CosmosQA and Quasar-T and 128 for SearchQA.CosmosQA uses 3 epochs, SearchQA 2 epochs, and Quasar-T 5 epochs on a single 16G P100 GPU.
  • Experimental protocol: The TACRED experiments run for 5 epochs on four 16G P100 GPUs with maximum sequence length 184.Separate best settings are reported for the factual, linguistic, and combined adapters.

E.4 Probing Experiments

The probing experiments use LAMA to assess knowledge representations in K-ADAPTER while preserving the adapter and backbone parameters during masked-language-model probing.

  • Probing setup: The probing experiments are implemented using LAMA.LAMA is used to evaluate the knowledge infused into the adapters.
  • Probing setup: Because K-ADAPTER does not use MLM pre-training, the experiments add and train a linear MLM layer for predicting masked entities.This layer is introduced before probing rather than changing the original pre-trained model parameters.
  • Probing procedure: During probing, all K-ADAPTER parameters are fixed and only the MLM layer parameters are updated using an MLM loss.The procedure isolates probing through the newly trained prediction layer.
Loading 2002.01808v5…