Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneck

1 Introduction

• Existing concept-based explanations can provide global and local insight, but require expert concept curation and example-level annotations and may use hundreds or thousands of concepts.The introduction identifies large concept spaces as a barrier to useful takeaways about global model behavior.Source: §1 · p. 1
• TBM automatically discovers sparse concepts, measures their values, and combines the scores in a white-box prediction layer for text classification and regression.The framework is presented as providing both global and local interpretability without human effort.Source: §1 · p. 2
• Across 12 diverse datasets, TBM is competitive with few-shot GPT-3.5 and finetuned BERT but lags finetuned GPT-3.5; it is especially competitive in sentiment analysis, with room to improve in news and science.The authors characterize average performance as competitive with strong, but not state-of-the-art, black-box baselines.Source: §1 · p. 2
• Human evaluation finds consistently high-quality generated concepts, though generation can produce redundancy or label leakage; measurement is harder for fake-news concepts than for sentiment concepts.The authors suggest measurement difficulty might help explain performance differences across those domains.Source: §1 · p. 2

2 Related Work

• Token-based rationales, similar training examples, and free-text justifications provide only local interpretability, while the paper’s concept-based approach provides both local and global insights into model behavior.Source: §2 · p. 2
• Vision Concept Bottleneck Models use expert-crafted concepts as classifier inputs, while text concept bottlenecks face risks from label leakage and information overload.The related work notes that leakage can make a bottleneck unfaithful and excessive concepts can impede users' general understanding [Lyu et al., 2022; Ramaswamy et al., 2022].Source: §2 · pp. 2–3
• Prior image-classification work used LLMs to automate concept generation, and this paper extends that approach to text with sparsity as an added benefit.A Causal Proxy Model is another related approach, but it relies on human-annotated counterfactual data, unlike the paper's concept curation approach.Source: §2 · p. 3
• SELF-EXPLAIN uses phrases as concepts without a bottleneck, yielding a much larger concept space than TBM's high-level categorical features, which are capped at 30 per dataset.The paper contrasts SELF-EXPLAIN's hundreds of thousands of phrases with TBM's sparse concept space.Source: §2 · p. 3

3.1 Method Formulation

• Each text receives a score for every generated concept; score sign indicates polarity, zero indicates uncertainty or absence, and magnitude indicates intensity.The prediction layer uses these concept scores as its only input during training and inference.Source: §3.1 · p. 4

3.2 Concept Representations

• Each concept is represented by a name, description, question, possible responses, response guide, and mapping from responses to numerical scores.Table 1 gives a hypothetical “Build Quality” concept as an in-context example in the generation prompt.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Concept Generation

• Concept generation targets misclassified examples, prompts an LLM to propose and refine a discriminative concept, and retains it only if the updated TBM improves by threshold γ.The procedure repeats for N cycles; experiments use N = 30 and a training subset of 100 examples.Source: §3.3 · pp. 4–5, §3.5 · p. 5

3.4 Concept Measurement

• For each text, an LLM answers each concept’s question using its description and response guide; the answer is mapped to a numerical score with a supporting text snippet.For example, positive and negative responses map to +1 and −1, while other responses map to 0.Source: §3.4 · p. 5

3.5 Prediction Layer

• A linear regression or logistic regression layer weights and sums concept scores to predict the output, while weights and example-level scores with snippets provide global and local explanations.The passage also notes that other interpretable classifiers operating on numerical concept scores could be used.Source: §3.5 · pp. 5–6

Datasets

• The evaluation covers 12 datasets across sentiment, natural language inference, topic classification, intent detection, and news analysis; most use 250 training and 250 test examples.Three datasets are regression tasks; the remaining datasets are classification tasks.Source: §4 · p. 6

Baselines

• Five baselines comprise finetuned DeBERTa, BERT, and GPT-3.5-turbo, Naive Bayes, and GPT-4 prompted with up to 10 labeled examples.The neural baselines are finetuned for three epochs.Source: §4 · p. 6

Evaluation Metrics

• Evaluation measures end-to-end performance using MSE for regression and accuracy for classification, and uses human annotation to assess concept generation and measurement.The human-evaluation metrics are described in Section 5.Source: §4 · p. 6

Implementation Details

• GPT-4 (GPT-4-0613) performs concept generation and measurement, while Scikit-learn trains the regression and classification prediction layers.Source: §4 · p. 6

5.1 End-to-End Performance

• Non-sentiment gaps were 0.4% on SNLI, 5.2% on AG News, 4.8% on Partisan News, 5.2% on SciCite, 14% on Patent Classification, and 19.5% on Fake News.The comparison covers accuracy on natural language inference, topic classification, intent detection, and news-analysis tasks.Source: Table 3 · p. 7, §5.1 · p. 7
• TBMs are competitive on sentiment tasks but underperform across non-sentiment tasks except SNLI.The authors suggest that tasks with interdependent concepts may be harder to model without a concept hierarchy.Source: §5.1 · p. 7
• On 4 sentiment datasets, TBMs achieved the highest or second-highest performance; average gaps when beaten were 8.7% MSE on regression and 1.4% on classification.The authors characterize the sentiment-domain interpretability–performance trade-off as minimal.Source: §5.1 · p. 7

5.2 Human Evaluation on Concept Generation Module

• Across datasets, most generated concepts were high quality; redundancy was the leading issue (25%), followed by leakage (15%), difficulty (9%), objectivity (6%), and relevance (1%).Three author-annotators rated concepts on six datasets, resolving disagreements by majority vote. Poem Sentiment had high error rates in nearly all aspects except relevance, while Hate Speech concepts mainly showed redundancy.Source: §5.2 · pp. 7–8, Figure 3 · p. 7

5.3 Human Evaluation on Concept Measurement Module

• On CEBaB, median correlation and accuracy were 0.814 and 0.893, versus 0.317 and 0.549 on Fake News; average scores were 0.759 and 0.824 versus 0.305 and 0.571.Crowd workers provided human judgments on CEBaB and Fake News. Measurement was zero-shot, and Fake News errors included hard-to-measure concepts such as Fact Checking, which the LLM assessed without external resources.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4 Analysis of Learning Curves

• Learning curves expose concept-level contributions: adding Customer Recommendation lowered Yelp MSE, while adding Mention of Overpricing left MSE stable.Concept importance is represented by the absolute final-layer weight; appendix examples also use concepts to explain run differences and small-dataset overfitting via spurious concepts.Source: §5.4 · pp. 8–9, Figure 5 · p. 9
• In the SNLI learning curve, the model used concepts about only one input side, including Hypothesis Truthfulness and Premise Verification, which the authors identify as known NLI biases.The authors suggest TBM’s interpretability could help discover dataset biases or annotation artifacts.Source: §5.4 · p. 9

6 Conclusion and Limitations

• TBM provides global and local interpretability through sparse concept-level explanations, with minimal performance trade-offs.The conclusion presents it as a promising general architecture.Source: §6 · p. 9
• Concept measurement currently requires an LLM call for each bottleneck concept at inference, making call count proportional to bottleneck size.The authors suggest distilling from GPT-4 or another strong teacher as a possible scalability improvement.Source: §6 · p. 9
• Concept importance is the regression-weight magnitude for linear regression, or the average absolute concept weight across classes for logistic regression.Source: §6 · p. 9
• Generated concepts can be redundant or leak classification labels; filtering problematic concepts is proposed as future work.Source: §6 · p. 10
• Practitioners can edit natural-language concepts to steer generation and refine measurement prompts, while human-in-the-loop improvements remain future work.Source: §6 · p. 10

A.1 Does the model generate similar concepts across repeated runs?

• Across five CeBaB runs with 250 training and 250 testing examples, TBMs consistently discovered important concepts such as Menu Variety and Food Quality.The learning curves indicated that these recurring concepts were highly important.Source: §A.1 · p. 14
• Final-model MSEs were 0.43, 0.48, 0.29, 0.36, and 0.48; both best-performing models contained task-label-leaking concepts.Source: §A.1 · p. 14

A.2 Can TBMs work on small training sets?

• With 50 CeBaB training examples, performance increased until around five concepts, then began to drop.The three trained TBMs could overfit after generating too many concepts.Source: §A.2 · p. 14
• The decline may reflect overly specific concepts, including a “Menu Misrepresentation” concept absent from the five full-size replications.Source: §A.2 · p. 14
• Limited training data may leave concept importance underdetermined, making assigned weights unstable and prone to poor generalization beyond the training distribution.Source: §A.2 · p. 14

A.3 Learning Curves on all datasets

• Figure 6 shows learning curves for all datasets, in addition to the four reported in Section 5.4.Source: §A.3 · p. 14

• Yelp Reviews, CEBaB, and Hate Speech Detection are regression tasks; all other datasets are classification tasks.Source: B Dataset Details · p. 14

• Quality scores of 1 indicate no problems, whereas scores greater than 1 indicate issues.Source: C Human Evaluation for Concept… · p. 14
• Concepts are evaluated for redundancy, task relevance, label leakage, objectivity, and relative answering difficulty.Difficulty is judged against the labeling task.Source: C Human Evaluation for Concept… · p. 14
• Figures 6 and 7 show concept learning curves across datasets and across different runs on small CEBaB datasets.Source: Figure 6 · p. 15, Figure 7 · p. 16
• Figure 8 maps concepts across CEBaB runs by correlation-proportional distance; Figure 9 shows learning curves for runs with identical starting conditions.Source: Figure 8 · p. 16, Figure 9 · p. 17
• The concept-measurement annotators were 98 graduate-level AI students with good English proficiency, and participation was voluntary.The annotation tasks were optional extra-credit assignments.Source: C Human Evaluation for Concept… · p. 16

D.1 Prompt Structure

• The prompt has instruction, dataset-information, and TBM-state sections, with the prior concepts loaded in the state section to avoid generating duplicates.The instruction section specifies the task, concept format, and three examples; the dataset section supplies labels and labeled examples.Source: §D.1 · p. 19

D.2 Selecting in-context examples

• In-context examples are selected from each training example’s 10 nearest neighbors, choosing 20 examples with the highest neighborhood loss; sampling for the first concept is random.If examples exceed the token limit, the method removes examples with the most common label; if fewer than four remain, it retries with texts truncated by a factor of 0.8.Source: §D.2 · p. 19

D.3 Managing Token Limits

• LLM token limits bound the number of iterations as the TBM state grows; truncating concept generation can cause redundant concepts, while example texts are dynamically truncated to fit the prompt budget.The appendix identifies exceeding the LLM token limit as a cause of concept-generation failure.Source: §D.3 · pp. 19–20

Concept Measurement

• Concept measurement prompts the model to extract relevant text snippets, reason over them, and return a classification; batch inference reduces LLM costs, and unparseable responses receive score 0.The prompt takes the text and concept JSON as input and returns a JSON object containing snippets, reasoning, and the final classification.Source: Concept Measurement · p. 20

E.1 Concept Generation Prompt

• The generation prompt seeks label-associated concepts that distinguish texts across labels, represented in JSON with a name, description, question, response options, guide, and integer mapping.It directs the model to avoid concepts resembling existing or rejected concepts and keep new concepts relevant to the dataset.Source: §E.1 · pp. 20, 22

E.2 Concept Improvement Prompt

• The improvement prompt asks the model to identify concept problems and revise validity, clarity, phrasing, and JSON formatting, with positive/negative/uncertain responses mapped to 1, -1, and 0.It checks response sets for mutual exclusivity and collective exhaustiveness, among other criteria.Source: §E.2 · pp. 22–23
• The improvement prompt checks concepts for six issues and revises leading questions and weak response guides into neutral questions with richer, objective examples.It also flags response sets outside {1, 0, -1, "na"} and recommends restricting them.Source: §E.2 · p. 24

E.3 Concept Measurement Prompt

• The measurement prompt asks the model to extract relevant snippets for each classification, compare them with the response guide, and return a JSON decision while avoiding irrelevant information and assumptions.Figure 10 shows a survey question for measuring “Quality of Service” on CEBaB texts.Source: §E.3 · p. 25, Figure 10 · p. 27, F Supplementary Material · p. 27

F Supplementary Material

• Figure 10 presents an example survey question used to measure the “Quality of Service” concept.Source: F Supplementary Material · p. 27, Figure 10 · p. 27