PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models

Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, Yulia Tsvetkov

arXiv:2305.09955v3cs.CL

TL;DR

General-purpose LLMs are difficult to update and may lack reliable knowledge for knowledge-intensive tasks. KNOWLEDGE CARD adds specialized knowledge models and selectors, with complementary integration approaches. It outperforms vanilla LLMs and other knowledge-augmentation baselines across three tasks and six datasets.

  • Problem

    General-purpose LLMs can hallucinate, struggle with long-tail facts, and are difficult to update with new knowledge, while existing approaches do not adequately support modular and collaborative knowledge sources.

    Source: §1 · pp. 1–2

  • Method

    KNOWLEDGE CARD uses specialized knowledge cards, selectors for relevance and factuality, and bottom-up or top-down approaches to integrate selected knowledge into general-purpose LLMs.

    Source: §1 · p. 2, §2.2 · p. 3, §2.3 · p. 5

  • Results

    KNOWLEDGE CARD outperforms vanilla LLMs, retrieval-augmented LMs, and generated knowledge prompting approaches across three tasks and six datasets.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The framework supports synthesis and updating of knowledge from diverse sources through modular knowledge cards.

    Source: §7 · p. 9

    Knowledge cards initialized from OPT-1.3B sometimes generate low-quality documents with degeneration or topic deviation, which the selectors only partly alleviate.

    Source: B LIMITATIONS · p. 19

Abstract

from arXiv · show

By design, large language models (LLMs) are static general-purpose models, expensive to retrain or update frequently. As they are increasingly adopted for knowledge-intensive tasks, it becomes evident that these design choices lead to failures to generate factual, relevant, and up-to-date knowledge. To this end, we propose Knowledge Card, a modular framework to plug in new factual and relevant knowledge into general-purpose LLMs. We first introduce knowledge cards -- specialized language models trained on corpora from specific domains and sources. Knowledge cards serve as parametric repositories that are selected at inference time to generate background knowledge for the base LLM. We then propose three content selectors to dynamically select and retain information in documents generated by knowledge cards, specifically controlling for relevance, brevity, and factuality of outputs. Finally, we propose two complementary integration approaches to augment the base LLM with the (relevant, factual) knowledge curated from the specialized LMs. Through extensive experiments, we demonstrate that Knowledge Card achieves state-of-the-art performance on six benchmark datasets. Ultimately, Knowledge Card framework enables dynamic synthesis and updates of knowledge from diverse domains. Its modularity will ensure that relevant knowledge can be continuously updated through the collective efforts of the research community.

1 INTRODUCTION

KNOWLEDGE CARD combines specialized knowledge models with quality selectors and two ways to integrate their outputs into general-purpose LLMs. It targets knowledge gaps that are difficult to address with static models and reports gains across three tasks and six datasets.

1 INTRODUCTION

  • Static LLMs can hallucinate, struggle with long-tail facts, and are difficult to update with new knowledge; existing approaches use retrieval or generated knowledge prompting.The introduction also identifies modularity and collaboration as properties existing systems do not adequately support.Source: §1 · pp. 1–2
  • Knowledge cards are specialized LMs trained on diverse domains and sources, serving as modular repositories that generate background information for general-purpose LLMs.Three selectors refine generated documents for relevance, brevity, and factuality.Source: §1 · p. 2
  • Bottom-up activates all cards and selects their outputs for synthesis, while top-down lets the LLM decide whether external knowledge is needed and which cards to activate.The top-down process repeats until the LLM has enough confidence to respond.Source: §1 · p. 2
  • Across three tasks and six datasets, KNOWLEDGE CARD outperforms vanilla LLMs, retrieval-augmented LMs, and generated knowledge prompting approaches.The reported results include a 6.6% Codex improvement on MMLU and at least 55.6% improvement on MIDTERMQA exact match.Source: §1 · p. 2

2 METHODOLOGY

The methodology pairs specialized knowledge cards with selectors that refine generated information, then integrates selected knowledge into a general-purpose LLM through bottom-up or top-down approaches.

2.1 KNOWLEDGE CARDS

  • Knowledge cards are smaller specialized LMs trained on domain- and source-specific corpora, and can be added, removed, or selectively activated as contributors expand shared knowledge.A card generates a knowledge document from a query, which can be prepended to the general-purpose LLM's context.Source: §2.1 · p. 3

2.2 KNOWLEDGE SELECTORS

  • The relevance selector retains top-k documents by query-document embedding similarity, while pruning condenses documents to fit multiple sources within the LLM context limit.Pruning is operationalized as a summarization system and preserves space for in-context learning.Source: §2.2 · pp. 3–4
  • The factuality selector averages scores for preservation of the original document's important points and support from retrieved evidence, then samples from high-scoring documents.Top-k factuality sampling excludes very low-scoring documents while allowing flexibility among the stronger candidates.Source: §2.2 · p. 4

2.3 KNOWLEDGE INTEGRATION

  • Bottom-up activates available cards, applies relevance, pruning, and factuality selection, then concatenates the retained knowledge documents with the query.Multiple cards can provide background from diverse perspectives, with hyperparameters controlling the synthesis process.Source: §2.3 · pp. 4–5
  • Top-down asks the LLM whether more information is needed, selects relevant cards when requested, and repeats knowledge retrieval until the LLM answers no.Card selection can be automatic or explicit; top-down is intended for settings where external knowledge is not always necessary.Source: §2.3 · p. 5

3 EXPERIMENT SETTINGS

The experiments use three task settings, compare KNOWLEDGE CARD with black-box LLMs and knowledge-augmentation baselines, and report results in tables for MMLU, misinformation detection, and MidtermQA.

Implementation

  • The implementation trains 25 specialized knowledge cards from OPT-1.3B and uses MPNet, Pegasus, WikiSearch, FactKB, and VitaminC for selection components.Codex is the default general-purpose black-box LLM.Source: §3 · p. 5

Results overview

  • The reported performance tables cover MMLU, misinformation detection, and MidtermQA.The captions identify model performance on MMLU, BAcc and MaF for misinformation detection, and performance on MidtermQA.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6

Tasks and Datasets

  • The evaluation covers general-purpose QA with MMLU, multi-domain misinformation detection with LUN, and temporal knowledge updating with a newly curated MIDTERMQA benchmark.MMLU uses 5-shot evaluation, LUN uses 16-shot evaluation, and MIDTERMQA uses 5-shot evaluation across open-book, 2-way, and 4-way settings.Source: §3 · p. 6

Baselines

  • Baselines include vanilla black-box LLMs, generated-knowledge prompting methods, and retrieval-augmented language models.Generated-knowledge prompting methods are applied to Codex for a fair comparison.Source: §3 · p. 6

4 RESULTS

KNOWLEDGE CARD improves results across knowledge-intensive tasks, with specialized cards supporting both targeted knowledge updates and multi-domain synthesis. Its performance also depends on selecting factual knowledge and matching the integration approach to the task.

MMLU

  • 6.6% overall-accuracy improvement over Codex was achieved by top-down with explicit selection on MMLU, outperforming all baselines, including Flan-PaLM.Source: §4 · p. 6

Misinformation Detection

  • At least 31.7% and 19.4% balanced-accuracy improvements over Codex were reported across the two- and four-way misinformation settings.Source: §4 · p. 6

Patching LLM Knowledge

  • In-domain knowledge cards, especially Wikipedia and news, produced greater misinformation-detection gains than other separately added cards.Source: Figure 2 · p. 7

Knowledge Selector Study

  • All three knowledge selectors helped misinformation-detection performance, with the factuality selector contributing most.Source: Figure 3 · p. 7

MidtermQA

  • 57.3% exact match improvement over the baselines in the open-book setting was the largest reported gain on MidtermQA.A 1.3B-parameter knowledge card updated the parametric knowledge of 175B Codex.Source: §4 · p. 7

5 ANALYSIS

The analyses examine when knowledge cards help, how their content is selected and delivered, and whether the framework works across models. Results favor in-domain cards, factuality selection, multi-document knowledge synthesis, and compatibility with different LLMs.

Patching LLM Knowledge

  • Adding any of five cards improved misinformation detection, with the most in-domain cards—Wikipedia and news—producing greater gains.Source: §5 · p. 7

Retrieval vs. Specialized LMs

  • KNOWLEDGE CARD outperformed REPLUG in both misinformation settings when both used Wikipedia as the knowledge source.Source: Table 4 · p. 7, §5 · p. 7

Knowledge Selector Study

  • All three selectors helped, but the factuality selector contributed most to misinformation-detection performance.Source: §5 · p. 7

Knowledge Stream Analysis

  • Larger n2 values reduced performance, n1 had marginal impact, and using n3 = 1 performed worse than larger values.n1 controls documents generated by each card, n2 those retained after selection, and n3 those placed in the LLM context; larger n3 supports multi-domain synthesis.Source: §5 · pp. 7–8, Figure 4 · p. 8

LLM Compatibility

  • Both bottom-up and top-down consistently improved TEXT-DAVINCI-003 and GPT-3.5-TURBO across datasets and evaluation metrics.Source: §5 · p. 8, Figure 5 · p. 8

Yes/No in Top-Down

  • Most top-down queries fell into the correct yes/no categories, but non-negligible errors show that knowledge-limit recognition remains imperfect.Some yes answers were incorrect because specialized LMs did not provide enough information.Source: §5 · p. 8, Figure 6 · p. 8

Qualitative Analysis

  • On 88 races where incumbents were not re-elected, Codex answered 1 correctly, versus 63, 77, and 42 for the three KNOWLEDGE CARD variants.The analysis reports that KNOWLEDGE CARD updates black-box LLM knowledge with a much smaller model.Source: §5 · p. 8, Table 5 · p. 8

6 RELATED WORK

Prior work augments LLMs through retrieval, generated knowledge, and modular model approaches, but the paper identifies limits in source flexibility, updating, and black-box compatibility. KNOWLEDGE CARD proposes plug-and-play knowledge cards as a modular, community-driven alternative.

Retrieval-Augmented Language Models

  • Retrieval-augmented LMs span QA, classification, and language modeling, but often rely on one corpus and face coverage and update issues.The paper positions KNOWLEDGE CARD as enabling flexible information seeking across domains and private sources.Source: §6 · p. 9

Modular LMs

  • KNOWLEDGE CARD integrates plug-and-play cards with general-purpose LLMs to reflect knowledge's modular and community-driven nature.Source: §6 · p. 9

Generated Knowledge Prompting

Modular LMs

  • Existing modular LMs include expert routing, adapters, model averaging, fusion, and continual learning, but mostly assume white-box access.Source: §6 · p. 9

7 CONCLUSION

KNOWLEDGE CARD combines specialized knowledge cards, quality selectors, and two integration approaches to augment general-purpose LLMs. Experiments report gains across three tasks and six datasets, and the paper envisions community-driven contributions to modular knowledge.

7 CONCLUSION

  • KNOWLEDGE CARD combines specialized cards, three selectors, and bottom-up and top-down integration to support multi-domain synthesis and external grounding when needed.Source: §7 · p. 9
  • Across three tasks and six datasets, KNOWLEDGE CARD outperformed vanilla LLMs, retrieval LMs, and generated-knowledge prompting approaches.Source: §7 · p. 9
  • The paper envisions KNOWLEDGE CARD as a community-driven framework for modular, collaborative knowledge.Source: §7 · p. 9

A DISCUSSION

KNOWLEDGE CARD supports diverse knowledge needs through replaceable, heterogeneous knowledge cards and natural-language integration. The framework also allows users to select expertise and organize cards hierarchically; combining its two integration approaches remains future work.

Modularity at every turn

  • All framework components can be replaced with future state-of-the-art models, and new cards can be trained and shared as knowledge or domains emerge.The framework also works with TEXT-DAVINCI-003 and GPT-3.5-TURBO; its selectors can likewise be updated as better models become available.Source: A DISCUSSION · p. 18

User-centric LLM adaptation

  • Users can select and activate knowledge cards to give general-purpose LLMs different skill sets and domain expertise for varied expectations.The paper gives textbook factuality, current NLP research, and cooking as examples of differing needs.Source: A DISCUSSION · p. 18

Compatible with diversified forms of knowledge

  • Natural-language integration makes KNOWLEDGE CARD compatible with retrieval systems, knowledge graphs, and search engines, as well as language-model knowledge sources.Language models support flexible information seeking, domain search, and private knowledge sources; the paper contrasts retrieval corpora as harder to share and use.Source: A DISCUSSION · p. 18

Knowledge cards heterogeneity

  • Knowledge cards can vary in size and architecture, enabling both large, broad-domain models and smaller cards trained on dedicated domains.The paper says this flexibility could allow computationally underprivileged researchers to contribute cards that improve black-box LLMs.Source: A DISCUSSION · p. 18

Knowledge cards hierarchy

  • Cards can follow a hierarchy: broad biomedical questions may use a general card, while fine-grained use cases can use cards for individual subdomains.The paper also suggests analogous divisions for NLP subfields and political news in different countries.Source: A DISCUSSION · p. 18

Combining bottom-up and top-down

  • A proposed combination would have the LLM choose candidate cards at each top-down step, then use bottom-up generation over that card pool.Further exploration of this combination is left for future work.Source: A DISCUSSION · p. 18

B LIMITATIONS

The authors identify limitations in knowledge generation, factuality scoring, and the LLM’s decisions about when to seek external knowledge.

Knowledge cards are not perfect knowledge generators

  • OPT-1.3B knowledge cards sometimes generate low-quality documents with degeneration or topic deviation, which the selectors only partly alleviate.Larger cards or specialized training objectives may help, but require more training and computational resources.Source: B LIMITATIONS · p. 19

Factuality selector bias

  • The factuality selector may favor information-rich domains and existing knowledge, while emerging knowledge may receive low scores.The authors say generalizable, adaptable factuality evaluation across varying and emerging domains remains an important problem.Source: B LIMITATIONS · p. 19

Prompting LLMs to seek help

  • Yes/no prompts for whether to seek help are effective but imperfect, as LLMs are occasionally over-confident about their knowledge.The authors identify better strategies for abstaining, acknowledging knowledge limits, and seeking external information as future work.Source: B LIMITATIONS · p. 19

C ETHICS STATEMENT

The authors warn that malicious actors could exploit community contributions by submitting harmful knowledge cards disguised as benign domains, and identify technical defenses and model-sharing rules as possible responses.

Dual-use risk

  • Malicious cards trained on disinformation, hyperpartisan content, or propaganda could be framed as benign domains and deceive users.Proposed responses include integrating adversarial defenses and reinforcing existing rules on model-sharing platforms.Source: C ETHICS STATEMENT · p. 19

D ANALYSIS (CONT.)

The analysis examines how cards are selected, how sensitive top-down help-seeking is to prompt wording, and how card factuality and accumulation relate to system behavior. It reports prompt agreement, variation in factuality scores, and improvement when cards—especially in-domain ones—are added.

Knowledge Card Selection

  • Knowledge-card selection is concentrated on a few clearly relevant cards for most MMLU tasks, but spreads out for tasks such as jurisprudence and high school U.S. history.The authors suggest selection patterns could indicate when a more in-topic card is needed.Source: D ANALYSIS (CONT.) · pp. 19–20

Yes/No Template Sensitivity

  • 79.9% of cases received unanimous yes-or-no responses across three prompts, while 20.1% received mixed responses.The analysis suggests multiple yes/no questions and an ensemble of answers as a possible robustness improvement.Source: Figure 7 · p. 20, D ANALYSIS (CONT.) · p. 20

Factuality Scores of Knowledge Cards

  • Different knowledge cards have varying factuality-score distributions when prompted with MMLU queries, according to the factuality selector.The authors suggest these distributions could guide evaluation of community-contributed cards.Source: D ANALYSIS (CONT.) · p. 20

Knowledge Card Accumulation

  • Adding knowledge cards improves performance in the two-way misinformation setting, with in-domain News especially helpful.The experiment uses the bottom-up approach with ChatGPT; Table 7 covers the gradual addition of five cards.Source: Table 7 · p. 20, D ANALYSIS (CONT.) · p. 20

E EXPERIMENT DETAILS

This section details the evaluation datasets, specialized knowledge-card corpora, and bottom-up and top-down procedures for selecting and integrating card-generated knowledge.

E EXPERIMENT DETAILS

  • The 25 knowledge cards use corpora spanning domains such as biomedical and political knowledge, legal contracts, mathematics, books, reviews, and knowledge graphs.For knowledge graphs, the authors construct textual corpora for training.Source: E EXPERIMENT DETAILS · pp. 20–21
  • Evaluation uses MMLU’s 15,908 four-choice questions, LUN’s 2,999-example official test set, and MidtermQA built from 510 2022 election races.MMLU spans 57 subtasks in four domains; LUN has binary and four-way settings, and MidtermQA has open-book, two-way, and four-way settings.Source: E EXPERIMENT DETAILS · p. 21
  • Bottom-up generates one document per knowledge card, then applies relevance, pruning, and factuality selectors before adding retained knowledge to the answer prompt.The algorithm processes every card for the question before selecting and filtering the resulting knowledge list.Source: E EXPERIMENT DETAILS · p. 23
  • Top-down iteratively asks whether more information is needed, selects one card by relevance automatically or explicitly, filters its knowledge for factuality, then supplies it to the LLM.The procedure can stop when the LLM indicates no further information is needed, or after the maximum number of trials.Source: E EXPERIMENT DETAILS · p. 23
  • Across MMLU’s 57 subtasks, YAGO and Wikipedia are generally the most selected cards, while the 25 cards show varying factuality-score distributions.The selection statistics are for the top-down approach with automatic selection.Source: Figure 8 · p. 24, Figure 9 · p. 25

Related papers

Loading 2305.09955v3…