PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept Bottleneck Large Language Models

Chung-En Sun, Tuomas Oikarinen, Berk Ustun, Tsui-Wei Weng

arXiv:2412.07992v4cs.CLcs.LG

TL;DR

LLM opacity complicates diagnosing unsafe outputs, while interpretable NLP work remains limited in scale and task coverage. CB-LLMs add a concept bottleneck to pretrained black-box models for classification and generation. They match standard black-box performance across both tasks, while classification has higher reported faithfulness ratings.

  • Problem

    LLM opacity hinders diagnosing unsafe outputs, while interpretable NLP research remains limited and prior methods largely focus on small-scale text classification.

    Source: §1 · p. 1

  • Method

    CB-LLMs transform pretrained black-box LLMs using a human-interpretable Concept Bottleneck Layer alongside a linear prediction layer.

    Source: §1 · p. 2

  • Results

    Across classification and generation, CB-LLMs match standard black-box performance; classification also achieves a 1.5× higher average rating on faithfulness evaluation than existing works.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that interpretability supports tracing harmful outputs, steering responses toward safer ones, and unlearning undesired concepts.

    Source: §1 · p. 1, §3.3 · p. 7

    Automatic Concept Correction depends on the sentence embedding model, so its concept scores may not align with human reasoning and may affect CBL learning.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

We introduce Concept Bottleneck Large Language Models (CB-LLMs), a novel framework for building inherently interpretable Large Language Models (LLMs). In contrast to traditional black-box LLMs that rely on limited post-hoc interpretations, CB-LLMs integrate intrinsic interpretability directly into the LLMs -- allowing accurate explanations with scalability and transparency. We build CB-LLMs for two essential NLP tasks: text classification and text generation. In text classification, CB-LLMs is competitive with, and at times outperforms, traditional black-box models while providing explicit and interpretable reasoning. For the more challenging task of text generation, interpretable neurons in CB-LLMs enable precise concept detection, controlled generation, and safer outputs. The embedded interpretability empowers users to transparently identify harmful content, steer model behavior, and unlearn undesired concepts -- significantly enhancing the safety, reliability, and trustworthiness of LLMs, which are critical capabilities notably absent in existing models. Our code is available at https://github.com/Trustworthy-ML-Lab/CB-LLMs.

1 INTRODUCTION

LLM opacity complicates diagnosing unsafe outputs, while interpretable NLP research remains limited in scale and largely focused on classification. CB-LLMs add intrinsic interpretability for classification and generation, with reported black-box-comparable performance.

1 INTRODUCTION

  • LLM opacity hinders detection of misuse and unsafe outputs and makes debugging and diagnosis difficult, while interpretable NLP research remains limited and prior approaches struggle to scale beyond small classification datasets.The supplied related-work passage also identifies text generation as a capability these approaches do not address.Source: §1 · p. 1
  • CB-LLMs transform pretrained black-box LLMs into inherently interpretable models using a human-interpretable Concept Bottleneck Layer and a linear prediction layer.The framework is designed for both text classification and text generation.Source: §1 · p. 2
  • CB-LLMs (classification) match standard black-box accuracy and achieve a 1.5× higher average rating than existing works on faithfulness evaluation.The authors interpret this as providing high-quality interpretability without sacrificing performance.Source: §1 · p. 2
  • CB-LLMs (generation) match standard black-box performance and support controllable, understandable generation, including a chatbot that detects toxic queries and provides controllable responses.The authors also describe tracing reasoning behind harmful outputs, steering responses, and unlearning undesired concepts.Source: §1 · pp. 1–2

2 RELATED WORK

Concept bottlenecks have been explored extensively for image classification, but interpretable language-model work remains limited and constrained in scale and task coverage. CB-LLMs target large text-classification datasets and generation without GPT-4 concept labeling.

2 RELATED WORK

  • Image-domain CBMs use human concept labels or label-free approaches, but cited methods still face concept-label requirements, backbone restrictions, or faithfulness concerns.The passage identifies VLG-CBM [16] as proposing the Number of Effective Concept metric to control information leakage and support fair comparison.Source: §2 · p. 2
  • Prior interpretable language models include TBMs and C3M [17], but TBM is limited to 250 dataset samples and C3M depends on human-annotated concepts.Neither approach studies autoregressive generation, according to the authors.Source: §2 · pp. 2–3
  • CB-LLMs scale to classification datasets exceeding 500,000 samples without GPT-4 concept labeling, match black-box accuracy, and also handle generation.Table 1 compares interpretable language models on scalability, efficiency, accuracy, and interpretability.Source: §2 · p. 3, Table 1 · p. 3

3 CB-LLMS FOR TASK CLASSIFICATION

CB-LLMs add a concept bottleneck to pretrained language models for interpretable text classification. The classification evaluation reports black-box-level accuracy after concept correction, with human-rated interpretability and concept unlearning as further demonstrations.

3.1 METHOD

  • ChatGPT supplies per-class concepts; sentence embeddings score samples (ACS), ACC removes negative and cross-class scores, then the CBL learns concepts for a sparse linear prediction head.ACC uses the class mapping from separately generated concept sets, and the final layer uses nonnegative CBL activations.Source: §3.1 · pp. 3–5

3.2 EXPERIMENT

  • CB-LLMs with ACC match standard black-box test accuracy across datasets and sometimes surpass it; without ACC, they trail by 1–5%.ACC improves classification accuracy by 3.5% on average.Source: §3.1 · p. 4, §3.2 · p. 5, Table 2 · p. 6
  • ACS takes about 1.6 hours on the largest YelpP and DBpedia datasets, versus 8.8 hours to label 1,000 samples with LLMs; CB-LLM training takes approximately black-box fine-tuning time.The 1.6-hour estimate uses all-mpnet-base-v2.Source: §3.2 · p. 6
  • Human evaluations found higher activation-faithfulness ratings and consistent preference for explanations from CB-LLMs with ACC over TBM&C3M.The study evaluated both whether activated neurons matched their concepts and whether neuron contributions supported the predictions.Source: §3.2 · p. 6, Table 3 · p. 6, Table 4 · p. 6

3.3 CASE STUDY ON CONCEPT UNLEARNING

  • After unlearning “Overpriced,” 2,726 test predictions shifted from negative to positive, and 2,162 of those samples strongly entailed the removed concept.The authors present concept unlearning as a potential way to support user intervention on biased, subjective, or unfair elements.Source: §3.3 · p. 7

4 CB-LLMS FOR TEXT GENERATION

This section develops CB-LLMs for autoregressive generation, evaluates concept detection, steerability, and generation quality, and demonstrates toxicity reduction in a chatbot setting.

4.1 METHOD

  • Adversarial training pairs a concept classifier on unsupervised features with training that makes its predictions uniform, reducing concept information in that layer.This is designed to prevent the final prediction layer from over-relying on unsupervised features and to improve steerability.Source: §4.1 · p. 7, Figure 3 · p. 8
  • Generation combines concept-bottleneck and unsupervised neurons, concatenating their outputs before a linear layer produces token logits over the vocabulary.The concept bottleneck and final layer are jointly trained for concept and token prediction.Source: §4.1 · p. 8
  • Module 1 trains with concept and next-token cross-entropy losses plus elastic-net regularization.The concept loss teaches bottleneck neurons their corresponding concepts; the token loss compares next-token predictions with next-token labels.Source: §4.1 · p. 8

4.2 EXPERIMENTS

  • The evaluation covers concept detection, steerability, and generation quality for CB-LLMs (generation).The paper measures generation quality by perplexity and compares steerability with a model trained without the adversarial module.Source: §4.2 · pp. 9–10
  • CB-LLMs achieve concept-detection accuracy with less than a 1% gap versus fine-tuned Llama3-8B, higher steerability than without Module 2, and similar perplexity to the black-box model.Table 5 reports accuracy, steerability, and perplexity; the prose describes the comparisons.Source: §4.2 · pp. 9–10, Table 5 · p. 10

4.3 CASE STUDY ON TOXICITY REDUCTION

  • On toxicity detection, CB-LLMs achieve 0.9996 test accuracy and a 0.9137 steerability score, using interpretable neurons to identify harmful queries and steer responses.The case study fine-tunes a Llama3-8B chatbot and uses separate neurons for query toxicity and benign or toxic response steering.Source: §4.3 · p. 10, Figure 5 · p. 10

5 CONCLUSION

CB-LLMs scale intrinsic interpretability to large text-classification benchmarks and generation tasks. They remain within 1% of black-box performance while supporting concept unlearning and toxicity reduction.

5 CONCLUSION

  • CB-LLMs are presented as the first interpretable model to scale to both large text-classification benchmarks and generation tasks.The conclusion describes the framework as fully automatic and training-efficient.Source: §5 · p. 10
  • CB-LLMs achieve performance nearly on par with black-box LLMs, within a 1% gap, while providing faithful interpretability and steerability.Source: §5 · p. 10
  • The framework supports applications including concept unlearning and toxicity reduction, with the stated aim of enhancing LLM controllability and safety.Source: §5 · p. 10

A.1 PERFORMANCE OF CB-LLMS USING GPT2 AND GEMMA2-2B AS BACKBONES

The appendix reports competitive classification accuracy with GPT2 and Gemma2-2B backbones, compares training time with other pipelines, and visualizes concept connections to class predictions.

A.1 PERFORMANCE OF CB-LLMS USING GPT2 AND GEMMA2-2B AS BACKBONES

  • With ACC, CB-LLMs classification is competitive with the black-box model using GPT2 and Gemma2-2B backbones.The table captions note that some accuracy values surpass the black-box model, but do not provide the values here.Source: Table 6 · p. 13, Table 7 · p. 13

A.2 TIME COST OF BUILDING CB-LLM

  • CB-LLM training takes slightly more time than standard fine-tuning but is significantly faster than the TBM and C3M pipeline.Source: Table 8 · p. 14

A.3 VISUALIZATION OF HOW INTERPRETABLE NEURONS CONNECTED WITH CLASS PREDICTIONS THROUGH FINAL LAYER WEIGHTS

  • For SST2, Yelp, AGnews, and DBpedia, the visualizations show the five concepts most strongly connected to each class, which are closely related to their associated classes.The connections are shown through final-layer weights.Source: §A.3 · p. 15, Figure 6 · p. 15, Figure 7 · p. 15

A.4 MORE EXAMPLES ON CONCEPT UNLEARNING

Unlearning the “Unappetizing food” concept changed many initially negative predictions to positive, including samples that strongly entailed that concept.

A.4 MORE EXAMPLES ON CONCEPT UNLEARNING

  • Unlearning “Unappetizing food” changed 370 predictions from negative to positive; 313 of those samples, or 85%, strongly entailed the concept.Source: §A.4 · p. 18
  • In the example, removing “Unappetizing food” lets clean ambiance and a quiet, relaxing atmosphere dominate, changing the prediction to positive.Source: Figure 10 · p. 18

A.5 VISUALIZATION OF NEURONS IN CB-LLM

The section visualizes highly activated CB-LLM neurons alongside text samples, showing neuron concepts spanning review sentiment, news topics, and entity attributes.

A.5 VISUALIZATION OF NEURONS IN CB-LLM

  • Neuron 159 is labeled “Engaging performances,” matching reviews that praise Cirque shows’ music, stunts, and stagecraft.The examples include a reviewer calling the music the star of the show and another praising the stage as an engineering feat.Source: §A.5 · p. 21
  • Neuron 104 captures “Unattractive store layout,” with reviews describing confusing navigation, congestion, and disorganized shelves.Source: §A.5 · p. 22
  • Neuron 20 groups sports events and achievements, while Neuron 16 captures human-rights violations and advocacy in news texts.Source: §A.5 · pp. 23–24
  • Neuron 10 represents terrorism and security threats, while Neuron 184 represents favorable breakfast options.The examples include reports of bombings and a review praising breakfast choices.Source: §A.5 · pp. 20, 24
  • Neuron 174 captures transportation-related words, while Neuron 71 responds to artists’ birth dates.Source: §A.5 · pp. 25–26
  • Neuron 469 represents publisher and imprint information, with examples describing newspapers including The Star-Ledger and The Anniston Star.Source: §A.5 · p. 26

A.6 EXPLANATIONS FROM CB-LLM

The section presents randomly selected examples of CB-LLM explanations, pairing text samples with concepts associated with their content across datasets.

A.6 EXPLANATIONS FROM CB-LLM

  • Table 10 presents explanations generated by CB-LLM w/ ACC for three randomly selected samples per dataset.Source: Table 10 · p. 27, §A.6 · p. 27
  • A Cirque du Soleil review is associated with engaging storytelling and a mobile stage that moves on a 3D axis.The sample describes a stage that rotates and moves on a 3D axis and recommends the show to repeat or veteran viewers.Source: §A.6 · p. 28
  • A news sample about China extending mobile coverage to previously disconnected ethnic-minority regions is paired with telecommunications and 5G concepts.Source: §A.6 · p. 28
  • A financial-news sample about Carl Icahn and Mylan is paired with concepts including company earnings, IPOs, and portfolio diversification.Source: §A.6 · p. 28

A.7 MTURK SURVEY DESIGN AND INTERFACE

The appendix describes two MTurk evaluation tasks and the prompting process used to obtain concept sets, and identifies a table of results for alternative generation backbones.

A.7 MTURK SURVEY DESIGN AND INTERFACE

  • Task 1 rates agreement between a neuron concept and its five most highly activated samples; Task 2 compares models’ explanations using their top five contributing neuron concepts.Task 1 uses a 1–5 agreement scale, while Task 2 asks workers to choose which model’s explanations are better or equally good.Source: §A.7 · p. 30
  • The MTurk evaluation compares CB-LLM with ACC against a random baseline, uses repeated ratings by different workers, and filters inconsistent task-2 responses.Each question is evaluated three times; task 2 ratings must agree directionally with workers’ model preference.Source: §A.7 · p. 30

A.8 DETAILS OF PROMPTING CHATGPT

  • The concept-set prompts use four human-designed examples for in-context learning and require n ChatGPT queries to obtain the full set.The prompts provide dataset- and class-specific examples; Table 11 contains the designed prompts.Source: §A.8 · p. 32, Table 11 · p. 32

B.1 PERFORMANCE OF CB-LLMS USING LLAMA2-13B AND MISTRAL-7B AS BACKBONES

  • Table 12 reports accuracy, steerability, and perplexity for generation CB-LLMs using Llama2-13B and Mistral-7B backbones.Source: Table 12 · p. 38

B.2 VISUALIZATION OF THE RELATION BETWEEN INTERPRETABLE NEURONS AND TOKEN PREDICTIONS

The section examines connections between interpretable neurons and token predictions through final-layer weights, finding that neuron activation can raise the probability of tokens related to the neuron’s concept.

B.2 VISUALIZATION OF THE RELATION BETWEEN INTERPRETABLE NEURONS AND TOKEN PREDICTIONS

  • The visualizations show the top 10 meaningful tokens most strongly connected to each neuron through final-layer weights.Source: §B.2 · p. 39, Figure 13 · p. 39, Figure 14 · p. 40
  • The connected tokens are closely related to neuron concepts, and increasing a neuron’s activation raises the probability of generating corresponding tokens.Source: §B.2 · p. 39

B.3 EXAMPLES OF STEERING CB-LLM

Setting the “sport” neuron to 100 prompts CB-LLM to generate sport-related text.

B.3 EXAMPLES OF STEERING CB-LLM

  • Setting the “sport” neuron to an activation value of 100 produces sport-related text, illustrating concept steering.Intervening on interpretable neurons can make CB-LLM generate corresponding text.Source: §B.3 · p. 41, Figure 15 · p. 41

B.4 MORE EXAMPLES ON TOXICITY REDUCTION

Figure 16 presents another example of toxicity detection and reduction through steering CB-LLM generation.

B.4 MORE EXAMPLES ON TOXICITY REDUCTION

  • Figure 16 shows another example of toxicity reduction through steering generation with CB-LLM.The figure caption describes toxicity detection and reduction.Source: Figure 16 · p. 41, §B.4 · p. 41

Related papers

Loading 2412.07992v4…