Source-linked AI summary
BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models
Lindia Tjuatja, Graham Neubig
TL;DR
Corpus-level perplexity and predetermined benchmarks may miss fine-grained differences between language models, and relevant data slices can be unknown in advance. BehaviorBox combines contextual embeddings with model probabilities to discover interpretable features where models differ. Across comparisons varying model size, family, and post-training, it uncovers distinctions that conventional metrics can mask, while its experiments remain limited to a subset of language-model tasks and datasets.
Problem
Corpus-level perplexity can mask fine-grained subgroup and domain differences, while relevant data partitions may be unknown in advance.
Source: §1 · p. 1Method
BehaviorBox combines contextual embeddings and evaluated-model probabilities, then uses sparse autoencoder features to identify and describe coherent text groups.
Source: §1 · p. 1, §3 · p. 3, §5.1 · p. 4Results
Takeaways & Limitations
BehaviorBox can serve as a hypothesis-generation tool for further behavioral analysis and support more informed model development and deployment decisions.
Source: §9 · p. 9The experiments cover only a subset of language-model tasks and datasets, and the method depends on embedding and probability-estimate quality.
Source: §10 · p. 10
Abstract
from arXiv · showhide
Language model evaluation is a daunting task: prompts are brittle, corpus-level perplexities are vague, and the choice of benchmarks are endless. Finding examples that show meaningful, generalizable differences between two LMs is crucial to understanding where one model succeeds and another fails. Can this process be done automatically? In this work, we propose methodology for automated comparison of language models that uses performance-aware contextual embeddings to find fine-grained features of text where one LM outperforms another. Our method, which we name BehaviorBox, extracts coherent features that demonstrate differences with respect to the ease of generation between two LMs. Specifically, BehaviorBox finds features that describe groups of words in fine-grained contexts, such as "conditional 'were' in the phrase 'if you were'" and "exclamation marks after emotional statements", where one model outperforms another within a particular datatset. We apply BehaviorBox to compare models that vary in size, model family, and post-training, and enumerate insights into specific contexts that illustrate meaningful differences in performance which cannot be found by measures such as corpus-level perplexity alone.
1 Introduction
Corpus-level perplexity and predetermined benchmarks can miss fine-grained performance differences, while relevant data partitions may be unknown in advance. BehaviorBox discovers interpretable text features where one language model outperforms another, including differences between models with near-identical perplexity.
1 Introduction
- Corpus-level perplexity on large, diverse data can mask finer-grained differences among subgroups and domains.Predetermined slices also require knowing relevant partitions ahead of time and having sufficient metadata to construct them.Source: §1 · p. 1
- BehaviorBox discovers fine-grained, human-interpretable features of text where one language model performs better than another.Its bottom-up approach can find features spanning documents and domains without requiring those domains to be partitioned in advance.Source: §1 · p. 1
- BehaviorBox compares contextual embeddings with evaluated-model probabilities, then uses sparse autoencoder features to describe groups of data.The embeddings capture text semantics and usage while probabilities represent model performance.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Across 7B and 13B Llama 2 and OLMo 2 models, BehaviorBox finds size-, family-, and post-training-related differences, including stylized text and dialogue contexts.It also identifies formatting and syntactic differences between models with near-identical perplexity.Source: §1 · p. 2
- The authors present BehaviorBox as augmenting existing evaluation methods with a more holistic and detailed view of language-model performance.The available passage is truncated after stating this contribution.Source: §1 · p. 2
2 Background
Slice finding seeks salient groups where systems underperform, while behavioral evaluation explains how outputs change with input features. BehaviorBox adapts learned representations to language-model comparison and can help discover hypotheses for further testing.
2.1 Slice Finding
- Slice finding automatically identifies salient data groups where a model underperforms, addressing the difficulty of knowing relevant groups beforehand.Earlier approaches often relied on metadata categories that may be absent or unspecified.Source: §2.1 · p. 2
- Prior learned-representation slice-finding methods cluster semantically similar underperforming samples and have focused mainly on image classification and constrained NLP tasks.Examples include George, Spotlight, and Domino.Source: §2.1 · p. 3
- BehaviorBox uses contextual representations for language modeling and compares models rather than judging whether one generated output is incorrect.The paper frames correctness as less clear in text generation than in the tasks explored by prior work.Source: §2.1 · p. 3
2.2 Behavioral Evaluation in NLP
- Behavioral explanations relate input features to system outputs and should faithfully capture behavior in human-interpretable form.Examples include examining how negation affects sentiment predictions.Source: §2.2 · p. 3
- Errudite and CheckList test practitioner-specified hypotheses, whereas BehaviorBox offers complementary hypothesis discovery for later exploration in other evaluation frameworks.The hypotheses for those existing frameworks still need to be specified ahead of time.Source: §2.2 · p. 3
3 Method Overview: BEHAVIORBOX
BehaviorBox is a three-part pipeline for comparing language models: it creates performance-aware representations, extracts coherent features, and labels data groups that indicate performance differences. Experiments characterize behavior at roughly the word level to balance granularity and salience.
3 Method Overview: BEHAVIORBOX
- BehaviorBox generates aligned contextual embeddings and model probabilities, extracts coherent performance-related features, and labels the resulting data groups.These are the pipeline’s three stages: data generation, feature extraction, and labeling.Source: §3 · p. 3
- Although the method can use units from tokens to documents, the experiments characterize performance at roughly the word level.The chosen abstraction balances granularity and salience, while avoiding token alignment challenges and the parsing difficulties of larger units such as code or mathematical expressions.Source: §3 · p. 3
4 Data Generation
BehaviorBox builds per-word representations by aligning contextual embeddings with language-model probabilities. It averages token embeddings within words and aggregates constituent-token probabilities by multiplication.
4 Data Generation
- BehaviorBox uses contextual word embeddings as semantic information and evaluated-model probabilities as performance measures.The contextual embeddings come from Longformer’s last hidden layer.Source: §4 · p. 3
- To align representations across tokenizers, the method determines word boundaries, combines or splits token probabilities as needed, and handles text beyond model context windows.It uses Longformer’s whitespace-oriented pre-tokenizer to determine word boundaries, averages constituent-token embeddings, and multiplies their probabilities.Source: §4 · pp. 3–4
- Each resulting datapoint is a 770-dimensional vector: 768 Longformer embedding dimensions followed by two language-model probabilities.Source: §4 · p. 4
5 Extracting Features
BEHAVIORBOX extracts fine-grained, performance-linked text features by decomposing contextual representations into sparse components and grouping words with strong feature activations. The pipeline combines contextual embeddings with language-model probabilities, aiming for coherent, interpretable slices that capture differences between models.
5 Extracting Features
- BEHAVIORBOX uses sparse autoencoders to extract performance-relevant features, then groups words whose representations strongly activate each feature.The features are linear decompositions rather than hard partitions, so a word in context can involve multiple features.Source: §5 · p. 4
5.1 Sparse Autoencoder Training
- The desired features should be coherent, fine-grained, and capture performance differences between models; these goals inform the training and regularization choices.The supplied passage identifies these criteria but does not specify the sparsity-enforcement operation.Source: §5.1 · p. 4
- The autoencoder encodes concatenated contextual word embeddings and LM probabilities into a sparse representation, then reconstructs the input from it.The encoder and decoder are trained to minimize reconstruction error.Source: §5.1 · p. 4
5.3 Balancing Context and Performance Awareness
- Probability features are up-weighted so they account for 70% of the input magnitude, helping balance performance information against embedding features.The authors note that probabilities could otherwise be overwhelmed by the many embedding features in the L2 loss.Source: §5.3 · p. 5
6 Processing and Labeling Features
The authors filter SAE activations into representative word groups, retain groups with sufficiently large probability differences, and use an LLM to label and validate feature coherence. They manually correct inaccurate labels when possible and discard features that cannot be corrected or do not form coherent groups.
6.1 Processing and Filtering SAE Features
- For each feature, the authors select up to 50 highest-activation words, apply a dynamic activation cutoff, and exclude features with fewer than 10 non-zero activations.Features with fewer than 50 activations use all available samples; the cutoff retains activations in the top 75% or above 25% of the maximum.Source: §6.1 · p. 5
- A feature is retained only if its filtered samples exceed a median probability difference of 0.1 or a median log-probability difference of 1.Using both thresholds captures large absolute differences and large relative differences, including for rare phenomena.Source: §6.1 · p. 5
- The activation cutoff is acknowledged as somewhat arbitrary, although some cutoff was needed to choose representative word groups.Source: §6.1 · p. 5
6.2 Labeling Procedure
- An LLM first labels coherent groups and then validates each label; authors correct inaccurate labels when possible and discard uncorrectable or incoherent features.Features may be discarded when fewer than 10 examples match the label or the group is not coherent.Source: §6.2 · p. 5
7 Differentiating Model Performance with BEHAVIORBOX
The experiments qualitatively inspect BEHAVIORBOX comparisons across model size, post-training, and family, using diverse Dolma text and perplexity as a baseline. The discovered features reveal context-specific strengths that do not always track aggregate perplexity.
7 Differentiating Model Performance with BEHAVIORBOX
- The study compares model size, post-training, and family through qualitative inspection, sampling about 80 million words from six Dolma sources.The models include Llama 2 and OLMo 2 at 7B and 13B, with base and post-trained variants.Source: §7 · pp. 5–6
7.1 Model Size
- Both 13B models show an approximately 1-point perplexity drop versus their 7B counterparts, alongside more features favoring 13B in both families.Figure 2 presents representative features for the size comparisons.Source: Figure 2 · p. 6, §7.1 · p. 7
- For size comparisons, 7B Llama performs better on tab sequences in narrative text, while 13B models show advantages in typography, bibliography, and long-tail styles.The larger-model advantages include vernacular and non-standard spellings.Source: §7.1 · p. 7
7.2 Post-training
- Across Llama and OLMo comparisons, post-trained models favored conversational phrases and common multi-word grammatical constructions; Llama2-7B-Chat outperformed its base on 17 out of 26 features despite significantly worse perplexity.Source: §7.2 · p. 7
7.3 Model Families
- Across same-size model families, the largest number of distinct features appears despite a perplexity difference below 0.1.Some features persist across 7B and 13B comparisons; OLMo favors certain separators and reference-section HTML, while Llama features include quotation marks and tab sequences.Source: §7.3 · p. 7
8 BEHAVIORBOX Features Distinguish Texts Generated by Different LMs
The authors test whether BEHAVIORBOX features predict differences in strings generated by Llama-13B and OLMo-13B. Six of eight string-frequency hypotheses were significant, and a larger sample supported the “to-morrow” hypothesis for Llama-13B.
9.1 Free-generation analysis
- The analysis asks whether features found on corpus data also appear in open generations, comparing string frequencies between models.The authors note that how probabilities of predetermined strings translate to open-generation behavior remains an open question.Source: §8 · pp. 7–8
- The authors tested four strings associated with better-performing features for each model, including “morrow” in “to-morrow” for Llama-13B.They compared frequencies in 500 sampled generations per model, each 400–600 words long, generated by conditioning on BOS at temperature 1.Source: §8 · p. 8
- Six of eight tested string-frequency hypotheses were significant when comparing free generations from Llama-13B and OLMo-13B.Table 2 marks significant results at p < 0.05; L > O indicates a higher generation frequency for Llama-13B.Source: §8 · p. 8, Table 2 · p. 8
- In 5,000 unfiltered free generations, “morrow” occurred 20 times for Llama-13B and zero times for OLMo-13B, supporting the hypothesis despite no conclusive result in the initial sample.“Morrow” was absent from the sampled strings used in the initial test.Source: §8 · p. 9
9 Conclusion
BEHAVIORBOX combines contextual embeddings and model probabilities with a sparse autoencoder to identify interpretable data slices where models differ. The experiments show that it can expose nuanced differences often masked by perplexity, while the number of salient features distinguishing models does not track perplexity differences.
9 Conclusion
- The pipeline combines contextual embeddings and model probabilities into performance-aware representations, then uses a sparse autoencoder to extract human-interpretable performance-difference features.The resulting features identify coherent data slices where one model outperforms another.Source: §9 · p. 9
- BEHAVIORBOX uncovers differences in formatting, domain-specific language, and syntactic patterns that conventional metrics such as perplexity can mask.The paper reports that perplexity deltas do not correlate with the number of salient features distinguishing models.Source: §9 · p. 9
- BEHAVIORBOX can serve as a hypothesis-generation tool for further behavioral analysis and support more informed model development and deployment decisions.The authors describe it as a step toward more transparent and actionable insights into language-model behavior.Source: §9 · p. 9
10 Limitations
The method depends on compatible contextual embeddings and probability estimates, and its feature extraction and labeling have additional sources of uncertainty. Its generalizability beyond the evaluated tasks and datasets remains to be fully explored.
10 Limitations
- Misalignment between contextual embeddings and performance signals can obscure meaningful differences, while aggregating token probabilities into word-level metrics may add noise when tokenization differs.These limitations concern representation alignment and the aggregation of model probabilities.Source: §10 · p. 10
- The sparse autoencoder may miss behavioral nuances and is sensitive to sparsity and probability-feature weighting; automated labels can also be inconsistent or suboptimal.The labeling process relies on a strong LLM annotator.Source: §10 · p. 10
- Experiments cover only a subset of language-modeling tasks and datasets, leaving generalizability to other tasks, domains, and non-textual modalities to be explored.The authors identify broader evaluation as future work.Source: §10 · p. 10
Ethical Considerations
BEHAVIORBOX offers practitioners tools to understand language-model behavior and differences between models. The authors see potential for helping identify issues before deployment, while cautioning against relying on its output alone.
Ethical Considerations
- The framework is intended to help practitioners understand language-model behavior and compare models.Source: Ethical Considerations · p. 10
- The authors say the tool has potential to ease ethical deployment by identifying possible issues before deployment so they can be addressed.Source: Ethical Considerations · p. 10
- The authors foresee few ethical risks from the framework but caution users against drawing conclusions from its output alone.Source: Ethical Considerations · p. 10
A Appendix
The appendix details SAE training choices, feature thresholding and annotation, then presents fine-grained feature labels across model comparisons. It also notes that feature sets can vary across training runs and that some discovered features recur.
A.1 SAE Hyperparameters and Training
- Higher probability-feature weighting improves sign consistency but can reduce feature coherence and increase dead latents; the authors use a 0.7 fraction and note variation across random seeds.They did not systematically vary all SAE hyperparameters; different initializations sometimes found partially disjoint feature sets, which the authors suggest could be combined across runs.Source: §A.1 · pp. 12–13
A.2 Thresholding Features
- Median probability or log-probability differences are used for thresholding because t-tests were unreliable for non-Gaussian differences; the chosen thresholds filter out most features below 70% consistency.Figure 5 plots median probability and log probability against consistency, with thresholds of ±0.1 and ±1, respectively.Source: §A.2 · p. 13, Figure 5 · p. 14
A.3 LLM Annotation
- LLM annotation first labels coherent groups, then scores and revises labels using top-activating words and contexts; qualitative analyses retain labels scored at least 1 and manually verify them.Samples not described by a feature label are additionally filtered out.Source: §A.3 · pp. 13, 16
A.4 Feature Labels
- Feature labels include document-structure and formatting patterns, while repeated features tend to arise with large probability differences and many nearly identical contexts.Examples include semicolons in reference entries, HTML line-break symbols in geographic references, and “wich” in Petroleum V. Nasby titles and signatures.Source: §A.4 · pp. 17, 21, 22