PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyBottle: A Recipe to Only Keep the Concepts You Really Need

Wolfgang Stammer, Sukrut Rao, Hevra Petekkaya, David Steinmann, Bernt Schiele

arXiv:2610.08552v1cs.AIcs.LG

TL;DR

Annotation-free concept bottlenecks often use large, fixed vocabularies that can exceed task needs and complicate inspection. ANYBOTTLE uses teacher-guided selection from an unsupervised concept pool and trains a predictor for any concept prefix. Across domains and datasets, it matches or exceeds teacher accuracy with far fewer concepts than annotation-free baselines.

  • Problem

    Annotation-free concept bottlenecks typically use large, static vocabularies that can exceed task needs and make concepts harder to inspect.

    Source: §1 · p. 1

  • Method

    ANYBOTTLE selects task-relevant concepts from an unsupervised pool using teacher guidance, then trains a nested-dropout predictor that supports any concept prefix.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    Across diverse domains and datasets, ANYBOTTLE matches or exceeds teacher accuracy with an order of magnitude fewer concepts than annotation-free baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    A small, discovered vocabulary can be as expressive as a much larger fixed one while remaining tractable to inspect.

    Source: §6 · p. 9

    ANYBOTTLE cannot select concepts absent from the SAE dictionary, and its performance remains tied to the teacher’s guidance.

    Source: §5 · p. 9

Abstract

from arXiv · show

Concept bottleneck models (CBMs) make predictions inspectable and intervenable by routing them through human-interpretable concepts, but originally required concept annotations. Annotation-free variants remove this requirement, but typically use large concept vocabularies, static at both training and inference, producing bottlenecks larger than any task or prediction needs and harder to inspect. We propose AnyBottle, a single recipe for building compact, task-specific CBMs. AnyBottle assumes only a frozen backbone and an unsupervised concept pool, such as a sparse autoencoder. A black-box teacher trained on the same backbone then guides selection: each round adds the concept that best explains the bottleneck's current failures, with candidates restricted to regions of teacher/student disagreement. Trained with nested dropout over this selection order, the final bottleneck predicts accurately from any concept prefix, so inference spends fewer concepts on inputs it is confident about early and more on hard ones. Since no stage is modality-specific, a new domain and task requires swapping only the backbone and concept pool. Across six vision and two text datasets and two teacher paradigms, AnyBottle yields bottlenecks with fewer concepts and higher concept consistency than annotation-free baselines, while staying close to the black-box reference. Overall, AnyBottle shows that going annotation-free need not mean going large: a small, discovered vocabulary can be as expressive as a much larger, fixed one.

1 INTRODUCTION

Annotation-free concept bottlenecks often use large, fixed vocabularies that exceed a task’s needs and are harder to inspect. ANYBOTTLE selects task-relevant concepts and trains a predictor that can use a variable-sized prefix; across vision and text datasets, it matches or exceeds teacher accuracy with far fewer, more coherent concepts.

1 INTRODUCTION

  • Large, overcomplete vocabularies can duplicate or split features, complicating debugging and intervention, while individual predictions may use more concepts than necessary.CLIP-like backbones and web-scale concept extractors can also miss fine-grained or specialized target-domain concepts.Source: §1 · p. 1
  • ANYBOTTLE iteratively adds concepts that cover teacher-used input regions missing from the current concepts, then uses nested dropout to support predictions from any concept prefix.Inference can spend fewer concepts on confident inputs and more on hard ones.Source: §1 · p. 1
  • Across six vision and two text datasets, ANYBOTTLE matches or exceeds teacher accuracy using an order of magnitude fewer and individually more coherent concepts than annotation-free baselines.The resulting bottleneck is sized to the task and sample demand.Source: §1 · p. 2
  • The recipe requires no human concept annotations or language-aligned backbone and transfers across domains and modalities by swapping the frozen backbone and unsupervised concept pool.The compact concept space also supports automated naming and targeted interventions to test concept meaning.Source: §1 · p. 2

2 RELATED WORK

Prior concept bottlenecks have moved from annotated concepts toward unsupervised discovery, but most retain fixed vocabularies. Compact selection and variable-budget inference have been explored through distinct approaches; ANYBOTTLE combines residual-based selection from an unsupervised SAE dictionary with teacher-student attribution disagreement.

2 RELATED WORK

  • Most concept bottlenecks moved from supervised annotations to unsupervised or CLIP-based concepts, but still produce a fixed concept layer; ANYBOTTLE builds its bottleneck dynamically.The cited approaches include CLIP-based supervision (Oikarinen et al., 2023; Yang et al., 2023) and unsupervised discovery (Rao et al., 2024; Bhalla et al., 2024; Wittenmayer et al., 2026).Source: §2 · p. 2
  • Compact-selection methods include residual-based CLIP atoms, post-hoc classifier reconstruction, sparse or gated selection, and iterative concept prompting or scoring.Most select from language-aligned banks; ANYBOTTLE instead uses an unsupervised SAE dictionary and filters candidates by teacher-student attribution disagreement.Source: §2 · pp. 2–3
  • Variable-budget approaches use ordered prefix representations, nested losses, or confidence-driven querying, while ANYBOTTLE applies Nested Dropout to its teacher-discovered vocabulary.The cited examples include Matryoshka CBM (Chen et al., 2026) and Variational Information Pursuit (Chattopadhyay et al., 2023a; 2024).Source: §2 · p. 3
  • SAEs provide an alternative to fixed LLM- or VLM-queried vocabularies, and ANYBOTTLE performs teacher-supervised selection over an already-trained, frozen SAE dictionary.Other SAE approaches either discover concepts before naming or inject supervision into the SAE objective.Source: §2 · p. 3

3 ANYBOTTLE

ANYBOTTLE builds a compact, task-specific concept bottleneck from an unsupervised pool, selects concepts using teacher-student behavior, and trains a predictor to work from any prefix. Concept naming follows selection, so only the selected concepts need to be named and reviewed.

3 ANYBOTTLE

  • ANYBOTTLE starts with an SAE-grounded pool, selects concepts by their statistical relationship to a black-box teacher, trains a prefix-usable predictor, and names selected concepts last.The recipe follows a bottom-up order: grounding before semantics.Source: §3 · p. 3

3.1 OBTAINING THE INITIAL CONCEPT POOL

  • A sparse autoencoder trained on frozen-encoder token embeddings supplies shared, nonnegative activations for M candidate concepts.The SAE can first be trained on a general corpus and then fine-tuned to the target task using teacher-output KL divergence and an embedding reconstruction term.Source: §3.1 · p. 4

3.2 ADAPTIVE CONCEPT SELECTION

  • Selection iteratively adds the concept that best explains the current teacher-output residual, retraining the student after each addition to guide the next round.Least-squares fitting includes an intercept so concepts explain deviations from the teacher-output mean rather than matching it.Source: §3.2 · pp. 4–5
  • Candidates are restricted to teacher-attributed tokens missed by the student on high-residual inputs, then ranked to favor strong activations found across distinct samples.The shortlist ranks candidates by mean activation strength times log(1 + distinct-sample count).Source: §3.2 · pp. 4–5
  • Selection stops when candidates or scores are insufficient, the teacher accuracy gap is within tolerance, validation accuracy or explained variance stalls, or a round limit is reached.The returned set is the best-validation-accuracy set, retaining discovery order.Source: §3.2 · p. 5

3.3 NESTED DROPOUT

  • Nested dropout trains the final predictor on sampled prefixes in discovery order, enabling predictions from any prefix length and optional confidence-based early stopping at inference.The adaptive budget uses fewer concepts for confident inputs and more for harder ones.Source: §3.3 · p. 5

3.4 CONCEPT NAMING

  • Agentic naming tests concept boundaries, but the proposed names still warrant human review against domain knowledge to catch concepts linked to shortcuts.A concept may be well-named while the predictor relies on it for a spurious correlation rather than the task.Source: §3.4 · pp. 5–6

4 EXPERIMENTAL EVALUATIONS

Across six vision and two text datasets, the evaluations test accuracy, compactness, concept consistency, adaptive allocation, and transfer across domains and teacher paradigms. AnyBottle generally retains teacher-level performance with fewer concepts, while concept budgets vary with task and teacher difficulty.

4 EXPERIMENTAL EVALUATIONS

  • The evaluation spans six vision and two text classification datasets, four annotation-free CBM baselines, and linear SAE probes, using teacher-gap, concept-count, C2 or AIS metrics averaged over three seeds.The vision datasets cover coarse- and fine-grained tasks and general and specialized domains; the text datasets are AG News and PubMed-20k.Source: §4 · p. 6

4.1 EXPERIMENTAL EVALUATIONS

  • ANYBOTTLE reaches teacher accuracy on most datasets, within two points on Dogs-10 and ISIC-7, and surpasses it on Flowers-102; versus a same-SAE linear probe, it uses 16–60× fewer concepts on five of six datasets.Source: §4.1 · pp. 6–7
  • On Imagenette, Dogs-10, and Mixed-20, C@90 needs at most 27 concepts and C@98 only a handful more; ISIC-7 needs 135 at C@90 and more than double that for C@98.Machines-15 and Flowers-102 need a larger initial budget, but show the same diminishing cost.Source: Figure 4 · p. 7, §4.1 · p. 7
  • Early selection rounds close large teacher-attended blind spots, while later rounds refine coverage with diminishing accuracy gains rather than adding concepts uniformly.On Imagenette, examples include a toothed-chain concept for chainsaw recognition and a small-spheres concept for golf-ball recognition.Source: Figure 4 · p. 7, §4.1 · p. 8
  • Easy gorilla and boxer samples pass 90% confidence with two and three concepts, while a hard ice-bear sample needs 14 concepts for 95%.The nested-dropout head supports stopping when a concept prefix suffices, rather than always using the full budget.Source: Figure 5 · p. 8, §4.1 · p. 8
  • The selection ablation evaluates least-squares scoring, localized candidate search, and prefiltering.Its stated focus is how selection supports compact, consistent, and accurate concept sets.Source: §4.1 · p. 8, Table 1 · p. 8
  • On two text datasets, ANYBOTTLE closely matches the trained MLP teacher, but with the weaker verbalizer teacher it beats the teacher on PubMed-20k while trailing the linear SAE probe.With the verbalizer teacher, it again matches the teacher on AG News.Source: Table 2 · p. 8, §4.1 · p. 9

5 DISCUSSION & LIMITATIONS

ANYBOTTLE combines several components into a compact, annotation-free concept-bottleneck recipe, while the discussion notes limits tied to teacher fidelity, available components, and iterative training cost.

Discussion

  • ANYBOTTLE combines SAE-based concept discovery, teacher-guided residual selection, nested-dropout prediction, and automated naming into compact, accurate task-specific bottlenecks without concept annotations.The recipe’s selection aims to reduce disagreement with an opaque teacher.Source: §5 · p. 9

Faithfulness, Not Correctness

  • A bottleneck may reproduce a teacher’s shortcut because selection targets agreement with the teacher, not whether the selected concept is correct.The authors say a small concept budget may make shortcuts easier to spot and revise in a human-in-the-loop setting.Source: §5 · p. 9

Limits of the Concept Pool, Backbone and Teacher

  • Performance is bounded by the backbone, pretrained components, available SAE concepts, and teacher guidance; task-aware fine-tuning is harder with scarce target-domain data.Concepts absent from the SAE dictionary cannot be found, even when the teacher relies on other dimensions.Source: §5 · p. 9

Computational Efficiency

  • Training cost scales with selection rounds, although adding multiple concepts at once or jump-starting harder problems can reduce rounds.The authors say this cost is offset downstream by substantially lower concept-naming costs.Source: §5 · p. 9

6 CONCLUSION

ANYBOTTLE builds compact task-specific bottlenecks from a frozen backbone and unsupervised concept pool, using teacher-guided selection and nested dropout. The conclusion reports accuracy comparable to or better than teachers with far fewer concepts, and outlines possible extensions and open verification work.

Conclusion

  • Across diverse domains and datasets, ANYBOTTLE matches or exceeds teacher accuracy with an order of magnitude fewer concepts than annotation-free baselines.The authors conclude that a small, discovered vocabulary can be as expressive as a much larger, fixed one while remaining tractable to inspect.Source: §6 · p. 9

Future work

  • Future extensions include other unsupervised decompositions with per-token activations, logic-based heads, and applying the recipe to tasks with suitable teachers.Compositional naming is proposed as an alternative to MAIA alongside logic-based heads.Source: §6 · pp. 9–10
  • An explicit grounding-verification step and human or automated verification of MAIA’s naming remain open extensions.Grounding verification would check whether a concept’s evidence is genuinely correct, not merely residual-reducing.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to appendices documenting experimental configurations, metrics, baselines, and supporting results, and reports three-seed averages with validation-based model selection.

Reproducibility statement

  • Appendix F documents datasets, backbones, teachers, SAE architectures, fine-tuning objectives, hyperparameters, annotation-free baselines, and MAIA evaluation prompts.Appendix F.3 defines the reported metrics, including C@q, C2, and AutoInterp.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix D provides per-dataset breakdowns, an accuracy-gap stopping-tolerance ablation, and full text results.Source: REPRODUCIBILITY STATEMENT · p. 10
  • All results are averaged over three seeds, with model selection based on held-out validation accuracy.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The authors disclose generative AI assistance across research design, implementation, interpretation, and manuscript preparation, while specifying tasks it did not support and noting that code review was not exhaustive.

AI use statement

  • Generative AI assisted with hypothesis refinement, methodology and experiment design, coding ANYBOTTLE and baselines, interpreting outputs, and several manuscript and figure tasks.Source: AI USE STATEMENT · p. 10
  • The authors report that AI tools were not used to generate synthetic datasets, develop theoretical models, formulate mathematical claims, write proofs, or conduct qualitative analysis.Source: AI USE STATEMENT · p. 10
  • Authors reviewed AI-generated text and figures and visually checked some generated code for plausibility, but did not systematically test or exhaustively review all code.Source: AI USE STATEMENT · p. 10

Supplementary Materials

The supplementary material discusses interpretability risks and related work, points to an algorithmic overview, and introduces additional evaluations.

A IMPACT STATEMENT

  • Automated concept names and teacher-faithful bottlenecks require caution: a bottleneck can reproduce a teacher’s shortcut, and names need human review before consequential use.The paper also warns that plausible-sounding explanations may invite unwarranted trust even when inaccurate.Source: A IMPACT STATEMENT · p. 15

B ADDITIONAL RELATED WORKS

  • Agentic naming is costly across a large SAE dictionary, while AnyBottle’s compact selection makes that approach practical for the surviving concepts.The discussion connects concept selection with naming strategy, contrasting agentic naming with cheaper non-agentic approaches.Source: B ADDITIONAL RELATED WORKS · p. 15

C ALGORITHM

D ADDITIONAL RESULTS

D.1 RAW ACCURACY ACROSS VISION DATASETS

Across vision datasets, AnyBottle’s patch-level concept consistency compares favorably with linear SAE probes using far fewer concepts; raw accuracy bars are not directly comparable across methods.

D.1 RAW ACCURACY ACROSS VISION DATASETS

  • Raw balanced test accuracies are broadly comparable across methods on most datasets, with the largest spread on ISIC-7.Because methods use different backbones, these raw values do not establish which bottleneck is better; they show that no method is degenerate in absolute accuracy.Source: §D.1 · pp. 15–16, Figure 6 · p. 16

D.2 PATCH-LEVEL CONCEPT CONSISTENCY VERSUS THE LINEAR SAE PROBE

  • AnyBottle has higher patch-level C2 than the full 8192-feature linear SAE probe on every dataset, using two to three orders of magnitude fewer concepts.Against the probe’s active features, it matches or exceeds consistency on four of six datasets; Dogs-10 favors the probe, while Machines-15 is within error bars.Source: §D.2 · pp. 16–17, Figure 7 · p. 17

D.3 PER-DATASET SELECTION STRATEGY ABLATION

The supplementary ablations examine selection components, representation choices, task-aware SAE fine-tuning, stopping tolerance, and text performance across teacher paradigms.

D.3 PER-DATASET SELECTION STRATEGY ABLATION

  • Across datasets, the full selection method has the highest patch consistency, while removing components generally reduces it; no configuration wins on teacher accuracy gap everywhere.The full selection performs best on average, though alternatives can do slightly better on individual datasets.Source: §D.3 · p. 17, Table 3 · p. 18

D.4 CONTINUOUS AND DISCRETE BOTTLENECKS

  • Continuous bottlenecks use fewer concepts on every dataset and match or improve teacher gap and patch consistency on most, while discrete bottlenecks retain binary semantics.The representation choice can be made per deployment.Source: §D.4 · p. 17, Table 4 · p. 19

D.5 TASK-AWARE SAE FINE-TUNING

  • Task-aware SAE fine-tuning reduces concept count and improves consistency together on Imagenette, Machines-15, and Flowers-102, but has less clear benefit on Dogs-10 and Mixed-20.On ISIC-7, it closes the accuracy gap and improves consistency at the cost of a larger concept budget; across all six datasets it does not systematically reduce consistency.Source: §D.5 · p. 18

D.6 INFLUENCE OF THE ACCURACY-GAP STOPPING TOLERANCE

  • Tightening δacc generally reduces selected-set size at a dataset-dependent cost to validation accuracy gap, but can mislead when validation signals are unreliable.On ISIC-7, targeting the teacher’s validation performance costs test performance; the authors recommend treating δacc as a budget–accuracy trade-off knob.Source: §D.6 · pp. 18–19, Table 5 · p. 19

D.7 GENERALIZATION TO TEXT IN DETAIL

  • For the MLP-classifier teacher, AnyBottle reaches most of its maximum text accuracy with a smaller fraction of concepts, whereas the verb-classifier teacher needs nearly its entire, much smaller pool.Table 6 reports concept counts, C@90/95, AutoInterp scores, and accuracy gaps across text domains and teacher paradigms.Source: §D.7 · p. 19, Table 6 · p. 20

E QUALITATIVE EXAMPLES

The qualitative examples show selection rounds closing blind spots, adaptive inference using different concept budgets by sample difficulty, and concepts that vary in consistency and naming accuracy.

E QUALITATIVE EXAMPLES

  • Automated naming can be more specific than the visual pattern supports: an Imagenette concept named for a fuel nozzle also fires on chainsaw trigger housings.Compactness makes such cases tractable to inspect but does not guarantee the generated name is correct.Source: E QUALITATIVE EXAMPLES · p. 20
  • Early selection rounds close large, class-defining blind spots, while later rounds refine coverage with diminishing accuracy gains; ISIC-7 needs 314 rounds to plateau.Several datasets use a class-focused jump-start round before residual-driven selection.Source: E QUALITATIVE EXAMPLES · p. 20, Figure 8 · p. 21
  • Across datasets, easy images converge within a handful of concepts, while harder images draw on a much larger share of the discovered budget.Figures 9 and 10 show two correctly classified test images per dataset, one requiring many firing concepts to reach 90% confidence and one requiring few.Source: E QUALITATIVE EXAMPLES · p. 20, Figure 9 · p. 22, Figure 10 · p. 23
  • Most sampled concepts are recognizable across their ten strongest activating images, but examples from ISIC-7 and Machines-15 are visibly looser.ISIC-7 has the lowest C2 scores among the six datasets, consistent with its specialized domain and less-adapted SAE pool.Source: E QUALITATIVE EXAMPLES · p. 20, Figure 11 · p. 24, Figure 12 · p. 25

F.1 DATASETS

The appendix materials cover AnyBottle’s evaluation setup for vision and text tasks, as well as prompt guidance for labeling concepts from exemplars and testing hypotheses.

F.1 DATASETS

  • Six vision datasets span coarse and fine-grained labels, small and large class vocabularies, and generic and specialised imagery; two text datasets cover general-domain topics and biomedical sentence roles.The ImageNet-derived subsets use 700 training images per class, split 90/10 for training and validation, with selected official validation images held out for testing.Source: §F.1 · p. 26

F.2.1–F.2.6

  • Vision baselines use distinct concept sources and designs: text concepts, incrementally mapped candidates, a frozen SAE dictionary, or class-conditioned NMF concepts with gated sparse classification.These correspond respectively to Label-free CBM, Res-CBM, DN-CBM, and UCBM; Table 9 lists selection and training hyperparameters, while linear probes use the full SAE dictionary or only evaluation-active neurons.Source: §F.2.1 · p. 27, §F.2.5 · p. 28, Table 9 · p. 29

F.2.1 VISION BACKBONES AND TEACHERS

  • Vision comparisons use method-specific backbones and separately trained teachers, reporting teacher–student gaps and concept metrics within each method’s embedding space under common splits, three seeds, and held-out validation selection.AnyBottle uses CLIP-DINOiser; baselines use their own configurations, so absolute accuracy is not comparable across methods.Source: §F.2.1 · pp. 26–27

F.2.2 VISION SAE: ARCHITECTURE AND FINE-TUNING

  • AnyBottle’s vision pool is an 8,192-atom Matryoshka BatchTopK SAE, fine-tuned separately per dataset with teacher-output KL divergence, while selection considers all dictionary atoms rather than only the SAE’s active latents.The SAE has 512-dimensional activations, 12 active latents per token, and six nested groups; selection does not depend on that nesting structure.Source: §F.2.2 · pp. 27–28

F.2.3–F.2.4

  • Text experiments use either a Gemma-2-2B MLP teacher or a Gemma-3-1B-it verbalizer teacher, with concepts from distinct SAE checkpoints and a redundancy filter for overly similar decoder directions.The verbalizer predicts from renormalized next-token probabilities over one specified single-token class name; text SAEs are fine-tuned per dataset from shared pretrained checkpoints.Source: §F.2.3 · p. 28, §F.2.4 · p. 28

F.2.7 AGENTIC CONCEPT NAMING; F.4 PROMPTS

  • MAIA names selected visual concepts using activating examples, heatmaps, and optional synthetic image tests, and its prompt directs the agent to identify the activation-driving attribute and report a label, description, and confidence.The prompt distinguishes localized parts from diffuse scene or texture concepts and asks the agent to base its description on gathered evidence.Source: §F.2.7 · p. 29, §F.4 · pp. 31–32

F.3 METRICS

  • Compactness is measured by selected concept count K and the smallest prefix reaching 90%, 95%, or 98% of the student’s full-budget accuracy; full-dictionary probes instead report active SAE features.Accuracy is balanced accuracy, and explained variance is reported as a diagnostic of how much teacher-output variance the selected concepts explain.Source: §F.3 · pp. 29–30
  • Concept coherence is assessed with C2 for vision and AutoInterp score for text; the text judge rates held-out documents from a concept description, and full-dictionary SAE AIS is estimated from stratified active-feature samples.AnyBottle’s image C2 uses localized patches, whereas baseline C2 uses image-level activations, so the granularities are not directly comparable.Source: §F.3 · p. 30

F.4 PROMPTS

  • Concept hypotheses should distinguish texture, shape, parts, and context, and test candidate attributes with edits or crops when tools are available.If causal tests are unavailable, labels should stay within what the exemplars support and disclose in [CONFIDENCE] that they were not causally tested.Source: §F.4 · p. 32
  • The exemplar-evidence prompt treats mined activations and their visual summary as evidence, explicitly excluding API examples from the final label.For the local-VLM run, raw exemplar images are summarized rather than repeated in the reasoning prompt because they exceed the context window.Source: §F.4 · p. 32
  • The final-answer prompt asks for a selective, complete description and concise labels based on the most confident hypotheses, using OR when a neuron is selective for multiple concepts.It also requests multiple hypotheses only for distinct concepts.Source: §F.4 · p. 32
Loading 2610.08552v1…