Inseq: An Interpretability Toolkit for Sequence Generation Models

Motivation

• Sequence generation attribution produces an importance matrix across inputs and generation outcomes, while dynamically incorporating prior generated tokens.Large vocabularies and non-trivial decoding strategies further complicate generation analyses.Source: §1 · pp. 1–2

Contribution

• Inseq centralizes feature attribution methods for sequence-to-sequence and decoder-only models in the Transformers ecosystem.The library draws partly on Captum and supports fair comparisons across techniques.Source: §1 · p. 2

Capabilities

• Inseq supports accessible analyses through a concise interface, customizable attribution targets, constrained decoding, optimized batching, and visualization utilities.The introduction states that users can integrate analyses into sequence-generation experiments with just 3 lines of code.Source: §1 · pp. 1–2

Contribution

• The toolkit aims to make sequence models first-class subjects of interpretability research and support explainable generative applications.Its stated goal is to centralize practical capabilities and encourage future methodological development.Source: §1 · p. 2

Feature Attribution for Sequence Generation

• Sequential feature attribution research has examined machine translation, efficient computation, contrastive explanations, and attribution-based detection of hallucinations.These studies cover alignment, biases, training dynamics, generation factors, and practical diagnostic uses.Source: Feature Attribution for Sequence… · p. 2

Tools for NLP Interpretability

• Only a few interpretability libraries support sequential attribution, leaving substantial tooling gaps for generation-focused analyses.LIT and Ecco are notable examples, while several other popular tools do not support sequence models.Source: Feature Attribution for Sequence… · pp. 2–3
• LIT requires custom wrappers for Transformers models and may be inconvenient for small examples because of its advanced interface.These factors limit usability for researchers needing extracted scores or working with custom models.Source: Feature Attribution for Sequence… · pp. 2–3
• Inseq is designed for methodical quantitative analyses with repeated evaluations, whereas Ecco emphasizes qualitative visualization of model internals.The libraries therefore provide orthogonal benefits despite marginal functional overlap.Source: Feature Attribution for Sequence… · p. 3

3 Design

• Inseq combines sequence models from Transformers with attribution methods mainly sourced from Captum.Although currently limited to text tasks, its modular design is intended to support additional frameworks and modalities without substantial redesign.Source: §3 · p. 3

3.1 Guiding Principles

• The library is intended to support interpretability across a broad range of sequence-generation models rather than narrowly targeting particular architectures or tasks.New cutting-edge methods should be included to enable fair comparisons with established methods.Source: §3.1 · p. 3
• Inseq is designed to scale from interactive toy-model analyses to dataset-wide statistics over large language-model predictions.This principle motivates an optimized interface across varied use cases, models, and setups.Source: §3.1 · p. 3
• The design targets non-experts through high-level abstractions and sensible defaults while supporting experienced users’ custom solutions.Beginner-friendly and extensible interfaces are both explicit design goals.Source: §3.1 · p. 3
• Table 1 organizes supported attribution methods into gradient-based, internals-based, and perturbation-based groups, alongside built-in step functions.The table also marks methods that can attribute arbitrary intermediate layers.Source: Table 1 · p. 3

4.1 Feature Attribution and Post-processing

• Inseq groups attribution methods by gradient-based, internals-based, and perturbation-based approaches and supports intermediate-layer attribution.It also includes attention attribution, Discretized Integrated Gradients, and configurable attribution of model layers.Source: §4.1 · p. 3
• For encoder-decoder models, users can include generated prefixes in attributed inputs or attribute source tokens separately for analyses such as word alignment.Including the prefix accounts for recently generated tokens when explaining subsequent behavior.Source: §4.1 · p. 4
• Aggregation converts neuron- or subword-level scores into interpretable token-, word-, or sentence-level importance values and can chain multiple operations.PairAggregator additionally supports contrastive analyses by combining attribution maps.Source: §4.1 · p. 4

4.2 Customizing generation and attribution

• Inseq can attribute generated outputs step by step or constrain decoding to analyze a specified target string, enabling contrastive minimal-pair comparisons.Constrained attribution can provide model justifications for desired outputs, subject to limitations when targets differ greatly from natural model outputs.Source: §4.2 · p. 4
• Custom step functions extract scores such as probabilities and entropy alongside attributions, and any built-in or custom step function can serve as an attribution target.This supports uncertainty analysis, contrastive attribution, and MC-Dropout-based weighting.Source: §4.2 · p. 4

4.3 Usability Features

• Inseq supports batching, span-focused attribution, command-line and notebook visualization, JSON serialization, and attribution of quantized models.A minimal evaluation found minimal discrepancies between 8-bit and full-precision attribution outputs for the Section 5.2 study.Source: §4.3 · pp. 4–5

5.1 Gender Bias in Machine Translation

• In Turkish-to-English translation, the model exhibits strong gender bias: “she” is produced for only 5 of 49 occupations, while gender-neutral variants are never produced.The study ranks occupations using pronoun probabilities and attribution scores, then correlates them with U.S. labor statistics.Source: §5.1 · p. 5
• Occupation attributions from Gradients and Input × Gradient significantly correlate with labor statistics, whereas pronoun probability correlates weakly with occupational gender stereotypes.Integrated Gradients shows a weaker correlation than the other attribution methods.Source: §5.1 · p. 5
• In an English-to-Dutch example, attributions associate a masculine mistranslation with preceding context and leraar, while replacing it with leerkracht increases occupation importance for the correct feminine pronoun.The analysis suggests sequential attention and motivates context-aware approaches to pronoun translation.Source: §5.1 · p. 6, Table 3 · p. 6

5.2 Locating Factual Knowledge inside GPT-2 with Contrastive Attribution Tracing

• The GPT-2 analysis reproduces an early important site associated with the last subject token alongside the expected importance of terminal-layer states.These patterns concern recalling factual statements such as the location of the Eiffel Tower.Source: §5.2 · p. 6, Figure 3 · p. 6
• CAT uses contrastive factual pairs and Layer Gradient × Activation to identify layer activations important for predicting correct over wrong targets.The method computes scores for 1000 Counterfact statements and compares averaged results with prior causal estimates.Source: §5.2 · p. 6
• GPT-2 CAT closely matches prior layer-importance findings while locating factual knowledge with one forward and backward pass.CAT attributes minimal pairs of correct and wrong factual targets to intermediate layer activations, approximating feature importance without causal guarantees.Source: §5.2 · pp. 6–7

• Inseq is an easy-to-use toolkit for interpreting sequence generation models.Source: §6 · p. 7
• Inseq targets generation tasks including machine translation, code synthesis, and dialogue generation, rather than classification alone.Source: §6 · p. 7
• The authors argue that Inseq can accelerate new analysis techniques and support progress in explainable language generation.Source: §6 · p. 7

Broader Impact and Ethics Statement

• Attribution methods in Inseq lack clear guarantees about identifying specific model behaviors and should not be used in high-risk or user-facing contexts.The authors recommend case-by-case validation because explanations may reflect users’ own biases.Source: Broader Impact and Ethics Statement · p. 7
• Inseq does not explicitly evaluate attribution faithfulness or plausibility, and its supported methods remain incomplete as interpretability research rapidly evolves.The authors invite users and method developers to contribute additional methods.Source: Broader Impact and Ethics Statement · p. 7
• The gender-bias case study simplifies gender as binary for easier evaluation and encourages future studies to consider non-binary gender and other marginalized groups.Source: Broader Impact and Ethics Statement · p. 7
• The authors state that measuring bias is complex, especially in multilingual settings, and do not claim a definitive analysis of the machine-translation models.The case study primarily demonstrates how attribution methods can explore social biases and showcases Inseq functionality.Source: Broader Impact and Ethics Statement · p. 7

A Authors’ Contributions

• The project’s listed contributors handled writing, library development, attribution implementations, case-study experiments, validation, and overall process soundness.Source: A Authors’ Contributions · p. 13

B Additional Design Details

• Figure 4 depicts Inseq’s model–method hierarchy, whose architecture-specific and framework-specific classes support extending the toolkit to new modeling architectures and frameworks.Source: B Additional Design Details · p. 13

• Switching the gendered pronoun causes a large probability drop for highly gender-stereotypical professions in the synthetic translation pair.The example follows the template from Section 5.1 and is presented in Figure 5, with results similar to Table 2.Source: C Example of Pair Aggregation for… · p. 13

• Figure 6 presents code from the Section 5.2 case study alongside attribution scores for contrastive examples in the evaluated dataset.Source: D Example of Quantized Contrastive… · p. 13

• Occupation rankings from attribution metrics and probabilities are correlated with U.S. labor statistics using the bls_pct_female column.Source: E Gender Bias in Machine Translation · p. 13
• 49 Turkish occupation terms and their English translations form the gender-bias case-study list.Source: Table 4 · p. 13
• Future Inseq releases plan support for gradient-based, internals-based, and perturbation-based attribution methods.Source: Table 5 · p. 13

Planned developments

• Future releases plan broader attribution-method support, improved interactive visualization, interoperability with ferret, and sequential instance attribution methods.Source: F Planned Developments and Next Steps · p. 13

Framework design

• The framework combines abstract attribution categories with architecture-specific model classes in its model and attribution design.Source: Figure 4 · p. 14

Attribution workflow

• The example loads a Turkish-to-English model with Integrated Gradients, computes forced-decoding probabilities, and aggregates subword and hidden-dimension scores into token attributions.Source: F Planned Developments and Next Steps · p. 14

Visualization

• Paired attribution outputs can be differenced and rendered as HTML for comparison.Source: F Planned Developments and Next Steps · p. 14, Figure 5 · p. 14

Contrastive analysis

• Contrastive attribution examples visualize Layer Gradient × Activation scores across GPT2-XL intermediate layers for false-to-true factual targets.Source: Figure 6 · p. 15