PaperlayerView exact revision on arXiv

Source-linked AI summary

MAxBench: A Multinomial Concept Recovery Benchmark

Divya Appapogu, Freya Behrens, Yonatan Belinkov, Aaron Mueller

arXiv:2609.13072v1cs.LGcs.AIcs.CL

TL;DR

Multinomial concepts can have many possible geometries, so it is unclear which representations and recovery methods are most suitable. The paper introduces geometry-agnostic MAXBENCH, evaluates representations through sampling, and finds that affine subspaces often perform strongly, while no method consistently outperforms prompting.

  • Problem

    Multinomial concepts can have many possible geometries and unknown dimensionalities, leaving the most suitable representations and recovery methods unclear.

    Source: §1 · pp. 1–2

  • Method

    MAXBENCH uses sampling-based steering and exclusivity and completeness metrics to compare multinomial concept representations without requiring a particular geometry.

    Source: §1 · p. 2

  • Results

    Affine subspaces steer more reliably and have greater recall than rank-one or linear subspaces, manifolds are competitive when applicable, and no method consistently outperforms prompting.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    The findings underscore the value of studying concepts with varied structures and developing methods that better capture their underlying geometry.

    Source: §7 · p. 12

    Experiments rely mostly on LLM judges, which do not always agree with human judgments, and are limited to a single intervention layer.

    Source: §8 · p. 12

Abstract

from arXiv · show

Fine-grained control of language model behaviors (e.g., steering) is among the more actionable outcomes of interpretability research. For binary concepts such as refusal, a single direction in activation space often suffices for steering. However, many concepts are not binary: Animals and Countries contain many subcategories, each with multiple instances. For these concepts, the search space over possible representation geometries is far larger than for binary concepts; it is thus not clear what geometries are most appropriate, nor what methods are most effective at recovering them. In this work, we introduce MAxBench, a geometry-agnostic evaluation framework for multinomial concept representations based on sampling from the recovered concept representation. We use MAxBench to compare 10 localization methods (covering 5 geometry types) across 6 concepts and 4 models. Using this framework, we find that (i) affine subspaces steer more reliably and have greater recall than rank-one or linear subspaces; (ii) much of this advantage is due to better non-zero offsets rather than the choice of bases; (iii) manifold steering is competitive with the best methods when applicable; and (iv) no method consistently outperforms prompting, in alignment with prior findings on binary concepts. These findings underscore the importance of expanding the scope of interpretability research and meta-evaluation to concepts with more varied structure.

1 INTRODUCTION

Binary concepts can often be steered with one direction, but multinomial concepts may have varied geometries and unknown dimensionalities. The paper introduces MAXBENCH, a geometry-agnostic evaluation framework, and uses sampling-based steering to compare concept-localization methods.

1 INTRODUCTION

  • Multinomial concepts create a larger geometry search space than binary concepts, and their representation dimensionality may vary across concepts and be unknown in advance.Possible structures include directions, Gaussian clouds, subspaces, and nonlinear manifolds.Source: §1 · pp. 1–2
  • The paper studies methods for recovering multidimensional concept geometries and proposes sampling-based interventions to steer multinomial concepts in language models.Source: §1 · p. 2
  • MAXBENCH evaluates multinomial concept representations through exclusivity and completeness metrics and is agnostic to the chosen geometry.The benchmark uses sampling-based steering to measure these qualities for localization methods.Source: §1 · p. 2
  • Across the benchmark, affine subspaces steer more reliably and have greater recall than rank-one or linear subspaces; offsets explain much of the advantage, manifolds compete when applicable, and no method consistently outperforms prompting.The comparison covers 10 discovery methods across 6 concepts, 5 geometries, and 4 models.Source: §1 · p. 2

2 PRELIMINARIES

The paper treats concept representations as activation-space geometries that should capture a concept's boundaries and internal variation. It considers linear, affine, and nonlinear structures, then frames recovery as locating low-dimensional structures from activations despite incomplete data.

2 PRELIMINARIES

  • A concept representation is a geometric object in activation space intended to capture a semantic concept's boundaries and variance while being complete and exclusive.Source: §2 · p. 3

2.1 GEOMETRIES

  • The considered geometries include linear and affine subspaces and nonlinear manifolds; affine subspaces add an offset to a linear span, while nonlinear manifolds may curve.The studied manifold methods currently apply only to ordered concepts such as DAYS and YEARS, leaving unordered concepts like ANIMALS for future work.Source: §2.1 · p. 3

2.2 DATA

  • Concept recovery locates a low-dimensional structure in residual-stream activations at a given layer, with activations observed before concept-related tokens across diverse contexts.Source: §2.2 · p. 3
  • Because concept datasets may omit instances, recovery methods should generalize to unseen instances; the benchmark dataset includes six concepts and a random baseline.The six concepts are ANIMALS, COUNTRIES, VEHICLES, PLANTS, DAYS, and YEARS.Source: §2.2 · p. 3

3 MAXBENCH: A MULTINOMIAL CONCEPT RECOVERY BENCHMARK

MAXBENCH evaluates multinomial concept representations by sampling from recovered geometries, intervening on language-model activations, and scoring the resulting outputs. It covers localization methods ranging from directions and affine subspaces to nonlinear manifolds, with scores designed to measure concept fidelity and breadth.

3 MAXBENCH: A MULTINOMIAL CONCEPT RECOVERY BENCHMARK

  • MAXBENCH samples points from recovered concept geometries, intervenes on model activations, and scores outputs for concept-representation quality.The pipeline has three stages: localization, sampling-based intervention, and scoring for completeness and exclusivity.Source: §3 · p. 4

3.1 CONCEPT LOCALIZATION METHODS

  • Localization methods differ in their assumed geometry, parameter-setting approach, and supervision, including whether they use positive examples or positive–negative pairs.Affine methods recover a k-dimensional subspace with basis L and offset b.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1.1 LINEAR

  • Affine-subspace bases are estimated using PCA, factor analysis, probes, or pairwise class-mean differences, with Schatten regularization encouraging low-rank probe weights.The estimated directions form the subspace basis L.Source: §3.1.1 · p. 5
  • SAEs provide selected decoder-feature directions, while MFA selects a Gaussian component associated with the target concept as its representation.SAE feature selection uses Matthews Correlation Coefficient; the selected MFA component supplies its centroid and factor-loading matrix.Source: §3.1.1 · p. 5

3.1.2 NON-LINEAR

  • The nonlinear method projects concept-value centroids into a low-dimensional space and fits a cubic spline through them to form a manifold.The method follows Wurgaft et al. (2026).Source: §3.1.2 · p. 5

3.2 SAMPLING-BASED EVALUATION

  • Affine-subspace samples use Gaussian coefficients and the offset, while manifold samples are drawn along spline segments between concept-value centroids.The affine sampling scale is selected to keep samples within the typical activation-space range; a closed spline includes its final-to-first segment.Source: §3.2 · p. 6

3.3 SCORES

  • The concept score measures target-concept presence, while normalized instance- and category-level entropies measure diversity and breadth.The diversity score sums the two normalized entropies and ranges from 0 to 2, analogous to recall.Source: §3.3 · pp. 6–7
  • MAXBENCH combines concept, fluency, instruction-following, and diversity scores, while additive one-direction steering substantially reduced output diversity.Its harmonic mean rewards reliable target-concept steering alongside broad coverage and fluent, instruction-following output.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments use four instruction-tuned models and six concept datasets, with evaluation across data splits and sampled interventions. An LLM judge scores intervention outputs, and the rank figure reports MAXBENCH scores by rank for Gemma-3-1B.

4.1 MODELS

  • Experiments use four instruction-tuned models, extracting activations from selected later-to-middle layers; comparing other layers is left to future work.The models are Llama-3.1-8B, Gemma-3-1B, Gemma-3-27B, and Qwen-3.5-4B.Source: §4.1 · p. 7

4.2 DATASET

  • The six datasets cover hierarchical unordered, categorical ordered, and continuous ordered concepts; the intended entity is among top-five predictions with at least 70% accuracy across concepts.Evaluation uses category-held-out and entity-held-out splits for hierarchical concepts, and held-out values for ordered concepts.Source: §4.2 · pp. 7–8

4.3 LLM JUDGE

  • Interventions use sampled geometry points on five test prompts, with 30 samples per prompt; an LLM judge scores concept, fluency, instruction following, and diversity.The experiment tunes hyperparameters on five separate validation prompts; the judge also extracts concept instances and assigns categories where applicable.Source: §4.2 · p. 8, §4.3 · pp. 8–9
  • For Gemma-3-1B, Figure 2 plots mean MAXBENCH score against rank by concept and method, with dataset-split confidence intervals and standard-deviation bands.Methods without a rank hyperparameter appear as dashed lines; Schatten Probe uses rounded effective rank.Source: Figure 2 · p. 9

5 RESULTS

Across the evaluated linear methods, affine subspaces outperform centered linear variants, with the offset often contributing more than the recovered basis. Results also cover prompting, spline fitting, generalization to unseen instances, and correlations among evaluation metrics.

5.1 LINEAR METHODS

  • Multidimensional methods consistently achieve higher MAXBENCH scores than one-dimensional methods across concepts.The comparison is summarized in Table 2.Source: §5.1 · p. 9
  • Affine-method MAXBENCH scores are generally stable across evaluated ranks, with slight increases for some concepts, provided representations use an appropriate offset.This suggests much of the benefit can be obtained with relatively low-dimensional subspaces.Source: §5.1 · pp. 9–10
  • Affine variants achieve higher MAXBENCH scores than linear counterparts across models, and the isotropic baseline’s performance suggests centering can matter more than basis directions on Gemma-3-1B.The isotropic baseline performs substantially worse on other models, where recovered basis directions also matter.Source: §5.1 · p. 10, Figure 3 · p. 10
  • Prompting is less competitive for smaller models such as Gemma-3-1B but outperforms all methods on larger models.The prompting baseline elicits the target concept without intervening on hidden activations.Source: §5.1 · p. 10
  • Intermediate-layer embedding representations perform better across models than representations taken directly from concept-instance token embeddings.The authors interpret this as evidence that steering operates over more than concept-value token identities.Source: §5.1 · p. 10

5.2 NON-LINEAR METHODS

  • On DAYS, spline fitting with the complete set of concept values outperforms fitting with held-out values and achieves the highest overall performance among evaluated methods.The method requires an ordered concept sequence; Table 3 reports DAYS MAXBENCH scores across four models.Source: Table 3 · p. 9, §5.2 · pp. 10–11

5.3 GENERALIZATION BEYOND CONCEPT DATA

  • Many methods often generate unseen concept instances, suggesting recovered representations generalize beyond training examples; notably, unsupervised Factor Analysis and MFA produce the largest number of unseen instances.Source: §5.3 · p. 11

5.4 METRIC META-EVALUATION

  • Concept score is somewhat correlated with instance and category diversity, negatively correlated with instruction score, while fluency has relatively weak correlations with other metrics.The authors note diversity can only be maximized when generated outputs contain concept-related values, and stronger concept injection can make staying on the original instruction harder.Source: §5.4 · p. 11, Figure 4 · p. 11

6 RELATED WORK

Related work moves from one-dimensional linear concept assumptions toward multidimensional and nonlinear geometries. Prior evaluation approaches assess fit or intervention outcomes, while this work targets multinomial concepts and whether their full space is covered.

Concept geometry

  • Concept geometry research has expanded from linear directions to multidimensional structures including convex hulls, polytopes, vector magnitudes, dense features, and manifolds.The paper focuses on affine subspaces as a tractable intermediate between one-dimensional and more complex geometries.Source: §6 · p. 11

Evaluating concept representations

  • Prior evaluation work measures geometric fit, downstream interventions, or toy settings with known ground truth, but does not assess whether the full concept space is covered.The paper’s diversity score addresses that coverage gap, while its focus is multinomial concepts with varied internal structure.Source: §6 · p. 12

Multidimensional/Multinomial concept steering

  • Most steering methods intervene along a single direction, while prior alternatives include circular-latent coordinate replacement and MFA-based movement within local subspaces.This work develops an evaluation framework for multinomial steering, a topic only preliminarily explored in prior work.Source: §6 · p. 12

7 CONCLUSION

The paper introduces a sampling-based framework and MAXBENCH for evaluating multinomial concept localization. It finds advantages for affine subspaces, competitive manifold steering when applicable, and no consistent win over prompting on sufficiently capable models.

Conclusion

  • The paper proposes sampling-based steering evaluation and MAXBENCH to benchmark multinomial and multidimensional concept localization methods.The framework compares methods across concepts and geometric structures.Source: §7 · p. 12
  • Affine subspaces more reliably recover multinomial concepts than rank-one or linear subspaces, with much of the advantage attributed to non-zero offsets rather than basis choice.Manifold steering is competitive with the best methods when applicable.Source: §7 · p. 12
  • No method consistently outperforms prompting when models are sufficiently capable.The authors conclude that interpretability research may benefit from studying broader concept types and methods that better capture their geometry.Source: §7 · p. 12

8 LIMITATIONS AND FUTURE WORK

The authors identify limitations in evaluation judgments, intervention scope, sampling, and the geometries covered by current benchmarked methods.

Limitations

  • Most experiments rely on LLM judges, whose judgments do not always agree with human judgments.This limits confidence in evaluations based on those judgments.Source: §8 · p. 12
  • Experiments are limited to a single intervention layer.The passage does not establish how results vary across layers.Source: §8 · p. 12

Limitations and future work

  • The sampling procedure requires hyperparameter tuning for reliable steering; future work could seek sampling methods that avoid it.Source: §8 · p. 12
  • Although MAXBENCH is geometry-agnostic, the benchmarked methods are largely affine, motivating extensions to more nonlinear representations.Source: §8 · p. 12

A DATASETS

The datasets cover six concepts and a random control, with categories defined for the concepts. Across the concept datasets, all evaluated models exceed 70% Top-5 accuracy on target-entity prediction.

A DATASETS

  • Categories for animals derive from WordNet, plant categories from growth habits, and categories for the remaining two concepts are grouped manually using GPT-5.The paper also reports generating six concept datasets and one control dataset.Source: A DATASETS · p. 18
  • All models achieve above 70% Top-5 accuracy across the concept datasets, suggesting the preceding hidden activation generally contains information about the target entity.Top-5 counts a target as correct when its beginning appears among the five highest-probability next-token predictions.Source: A DATASETS · p. 18
  • The evaluation includes six concept datasets and a random control dataset, assessed across all four models.Top-1, Top-5, and continuation accuracy are averaged over prompts and entities within each concept.Source: A DATASETS · p. 18

B CONCEPT LOCALIZATION METHODS

The localization methods recover concept directions or higher-dimensional geometries from activations, with approaches differing in labels, fitting procedures, and how they construct subspaces. The resulting representations are used for sampling intervention points.

B CONCEPT LOCALIZATION METHODS

  • DiffMean-r1 learns a rank-one direction by subtracting mean negative-example activations from mean positive-example activations; ReFT-r1 jointly learns concept detection and steering in one direction.ReFT-r1 uses a slightly different paired-response dataset from the other methods.Source: B CONCEPT LOCALIZATION METHODS · pp. 18–19
  • The linear probe learns a classifier from concept labels and uses leading left singular vectors of its weight matrix as the subspace basis, with the global activation mean as offset.The classifier bias is not used as the offset because it lies in the output space rather than the residual-stream space.Source: B CONCEPT LOCALIZATION METHODS · pp. 19–20
  • PCA and Factor Analysis produce affine subspaces centered at the concept mean, with PCA retaining leading singular vectors and Factor Analysis fitting latent factors with Gaussian noise.Factor Analysis parameters are estimated using expectation-maximization, and its intervention points use the benchmark's Gaussian sampling procedure.Source: B CONCEPT LOCALIZATION METHODS · p. 20
  • DiffMean-Pairs builds a basis from pairwise differences between class centroids, while setting the affine offset to the global activation mean.It retains leading left singular vectors of the stacked difference matrix.Source: B CONCEPT LOCALIZATION METHODS · pp. 20–21
  • Schatten Probe regularizes a factorized classifier weight matrix toward low rank, then uses its effective-rank singular vectors and the global activation mean to define an affine subspace.The effective rank determines the number of retained basis vectors.Source: B CONCEPT LOCALIZATION METHODS · p. 21
  • MFA selects a Gaussian component using its positive-versus-negative concept score, whereas SAE ranks features by Matthews Correlation Coefficient and uses their decoder directions.The selected MFA component is treated as an affine subspace; SAE feature sampling strategies are evaluated separately.Source: B CONCEPT LOCALIZATION METHODS · p. 21

C SAMPLING EVALUATION RESULTS ACROSS MODELS

The cross-model appendix reports scores across ranks and compares composite and component metrics. The reported Diversity Score has a special definition for concepts without category structure.

C SAMPLING EVALUATION RESULTS ACROSS MODELS

  • For DAYS and YEARS, which lack category structure, the reported Diversity Score is defined as twice instance diversity, so instance and category diversity do not always sum to it.Source: C SAMPLING EVALUATION RESULTS ACROSS… · p. 21
  • Figures 5 and 6 plot MAXBENCH score by rank for Gemma-3-27B and Llama-3.1-8B, respectively, with confidence intervals computed across 10 dataset splits.The figures show means at each concept, method, and rank; methods without a rank parameter appear as dashed lines.Source: Figure 5 · p. 22, Figure 6 · p. 22
  • Tables 8 and 9 report test-set Concept, Fluency, Instruction, Diversity, AXBENCH, and MAXBENCH scores for Gemma-3-1B and Gemma-3-27B, selecting each method's rank by mean MAXBENCH.Values are means and standard deviations over data splits.Source: Table 8 · p. 23, Table 9 · p. 23

D UNSEEN INSTANCES ANALYSIS

The supplementary analyses examine generalization to unseen instances, offsets, MFA and SAE selection, diversity scoring, and sampling-scale tuning. They report unseen-instance generation by supervised methods as well as a consistent advantage for sampling around the concept mean offset.

D UNSEEN INSTANCES ANALYSIS

  • Unseen concept instances appear not only for MFA and SAE but also for supervised methods fit on the concept dataset, suggesting sampling can extend beyond entities used during localization.The analysis covers ANIMALS, VEHICLES, PLANTS, and COUNTRIES, checking extracted entities against the dataset.Source: D UNSEEN INSTANCES ANALYSIS · p. 23

F MFA GAUSSIAN SELECTION

  • MFA components are scored by how much more responsibility they assign to positive than negative activations, and the highest-scoring component is retained.MFA models are trained on PILE activations; the selection procedure uses temperature-scaled posterior responsibilities.Source: F MFA GAUSSIAN SELECTION · p. 24

E AFFINE VS LINEAR COMPARISON

G SAE FEATURE SELECTION

  • SAE features are ranked by their best MCC across nine thresholds, with decoder columns used as activation-space directions and ranks swept from 1 to 16.Sampling can use selected features independently or jointly as a subspace; combination sampling performs slightly better overall across models and ranks.Source: G SAE FEATURE SELECTION · pp. 27–28, Figure 11 · p. 27

H DIVERSITY SCORES AND NORMALIZATION

  • Diversity is measured at both category and instance levels using normalized Shannon entropy; instance-level normalization depends on whether the full instance support is known.When the complete support is unknown, the method estimates it using N; the authors note a larger normalizer may be needed if outputs average more than one instance.Source: H DIVERSITY SCORES AND NORMALIZATION · pp. 28–29

I HYPERPARAMETER TUNING

  • Sampling standard deviation is calibrated separately for each method and rank by selecting the candidate that places the largest fraction of samples within observed activation-norm bounds.When candidates tie, the largest standard deviation is selected; steering then interpolates between the original activation and a sampled point, with alpha chosen by grid search.Source: I HYPERPARAMETER TUNING · p. 29

J RANDOM CONTROL RESULTS

The random-noun control tests whether localization methods recover low-dimensional structure from semantically coherent concepts or impose it through their geometric constraints. Gemma-3-1B and Llama-3.1-8B show the expected pattern of lower MAXBENCH scores and improving performance as rank increases; some results for Gemma-3-27B and Qwen-3.5-4B fall below the control.

J RANDOM CONTROL RESULTS

  • The control dataset uses random nouns that do not form a coherent semantic concept, unlike the semantically coherent values in the concept datasets.The authors have no reason to expect the random nouns to lie in a shared low-dimensional subspace.Source: J RANDOM CONTROL RESULTS · pp. 29–30
  • Gemma-3-1B and Llama-3.1-8B show low MAXBENCH scores that improve as rank increases, while some Gemma-3-27B and Qwen-3.5-4B results fall below random control.Below-control scores may mean the tested geometries do not capture those concepts well, or that localization methods fail to recover their underlying structure in that model.Source: J RANDOM CONTROL RESULTS · p. 30, Figure 12 · p. 30

K METRICS CORRELATION

This section evaluates generated outputs using LLM-judge assessments and entity-extraction conventions for instruction-following, fluency, diversity, and out-of-distribution instances. It also examines prompting and localization performance and correlations between concept and diversity scores.

K METRICS CORRELATION

  • Concept scores correlate with both diversity scores; instruction score is somewhat negatively correlated with concept score, while fluency score is mostly neutral.Both Pearson and Spearman correlations are reported.Source: K METRICS CORRELATION · p. 30, Figure 16 · p. 32

L PROMPTING BASELINE

  • Prompting outperforms localization methods across most models, with performance generally increasing by model size; it does worse for Gemma-3-1B but performs very well for Gemma-3-27B.Source: L PROMPTING BASELINE · p. 30

M LLM JUDGE PROMPTS

  • An instruction-tuned Qwen-3.6-27B model judges concept presence, fluency, and instruction following, and processes extracted entities for taxonomy and seen–unseen classification.Concept evaluation uses concept-specific prompts; fluency and instruction-following prompts are shared across concepts.Source: M LLM JUDGE PROMPTS · p. 34
  • The rubric assigns 2 when output meaningfully addresses the task and could plausibly fulfill it, 1 when it overlaps but visibly prevents an important requirement, and 0 when it makes no meaningful attempt.Truncation is never penalized; a visible, unrecoverable failure on a hard formatting requirement receives 1.Source: M LLM JUDGE PROMPTS · pp. 38–39
  • The fluency judge evaluates readability alone, assigning 1 for a limited local defect and 0 for severe garbling or disruptive entity lists.It ignores relevance, correctness, and truncation itself; multiple gibberish tokens or three or more disconnected category items qualify for 0.Source: M LLM JUDGE PROMPTS · pp. 39–40
  • Animal and vehicle extraction prompts require canonical names while preserving breed- or model-level distinctions and assigning a taxonomy from the supplied list.They also include generic terms and metaphorical uses in extraction.Source: M LLM JUDGE PROMPTS · pp. 40–41
  • Day-of-week extraction includes indirect references and canonicalizes them, whereas year extraction captures explicit four-digit years while excluding other numbers.The prompts call for empty lists when no relevant references are found.Source: M LLM JUDGE PROMPTS · p. 41
  • Plant and country prompts canonicalize mentions, retain species-level plant distinctions, and assign taxonomy labels; country extraction also includes demonyms and vague references.The prompts request empty lists when no relevant mentions are present.Source: M LLM JUDGE PROMPTS · p. 42
  • The judge flags extracted instances as OOD when they do not match the complete provided list of known instances, using case-insensitive matching.The prompt frames this as OOD detection for the concept and supplies the known and extracted instances as inputs.Source: M LLM JUDGE PROMPTS · pp. 42–43
  • Matching allows basic morphological variations but excludes synonyms and related concepts, ignores generic terms, and preserves each extracted instance’s original name.Source: M LLM JUDGE PROMPTS · p. 43
Loading 2609.13072v1…