PaperlayerView exact revision on arXiv

Source-linked AI summary

FailSAE: Towards Interpretable Failure Prediction for Vision-Language Models via Sparse Autoencoders

Jie Ma, Zongxi Liu, Yi Zhu

arXiv:2609.04276v1cs.CV

TL;DR

Existing failure-prediction methods provide limited insight into representation changes during errors. This paper uses SAE latent activations in a three-stage failure-aware prediction framework, which outperforms evaluated baselines and supports concept-level analysis of failures.

  • Problem

    Existing failure-prediction methods can estimate errors but provide limited insight into how model representations change during failures.

    Source: Introduction · p. 1

  • Method

    The paper formulates failure prediction as classification over SAE latent activations and introduces a three-stage failure-aware training pipeline.

    Source: Introduction · p. 1

  • Results

    The framework outperforms existing baselines, and analyses indicate that failure-aware training encourages more class-specific SAE concepts.

    Source: Introduction · p. 1

  • Takeaways & Limitations

    Comparing concepts activated in correct and incorrect predictions reveals a shift from class-specific concepts toward ambiguous or style-related concepts.

    Source: Introduction · p. 1

    Existing confidence-score and auxiliary-classifier methods provide limited insight into representation changes during failures.

    Source: Introduction · p. 1

Abstract

from arXiv · show

Vision-language models (VLMs), such as CLIP, have achieved strong performance across multimodal tasks by aligning visual and textual representations in a shared embedding space. As VLMs are increasingly used for high-stakes domains, failure prediction becomes critical for risk-aware deployment and human intervention. Existing failure prediction methods typically rely on confidence scores or auxiliary classifiers. Although these methods are effective on predicting VLM failures, they provide limited interpretability. In this work, we investigate the use of Sparse Autoencoders (SAEs) for interpretable failure prediction in VLMs. We formulate failure prediction as a classification task over sparse SAE latent activations and introduce a three-stage failure-aware training pipeline that encourages the learned latent directions to remain interpretable while becoming more informative for failure prediction. Our experiments show that the resulting framework outperforms the evaluated baselines in failure prediction. Further analysis suggests that failure-aware training encourages SAE latent directions to capture more class-specific concepts. We also use the SAE to provide a concept-level analysis of how model representations change during failures, revealing a shift from class-specific concepts toward more ambiguous or style-related concepts. Finally, we explore how the learned SAE latent directions can support runtime failure recovery.

Introduction

Existing failure-prediction methods offer limited insight into representation changes during errors, while SAE-based interpretations remain underused for prediction. The paper trains SAEs for failure prediction and reports stronger baseline performance, more class-specific concepts, and a shift in activated concepts during failures.

Introduction

  • Failure prediction methods based on confidence scores or auxiliary classifiers provide limited insight into how internal representations change during failures.The paper identifies this interpretability gap as a motivation for its work.Source: Introduction · p. 1
  • The method treats failure prediction as classification over SAE latent activations and uses a three-stage failure-aware training pipeline to make latent directions more predictive.The intended result is accurate prediction alongside concept-level understanding of model representations.Source: Introduction · p. 1
  • The framework outperforms existing failure-prediction baselines, and failure-aware training encourages SAE directions to represent more class-specific concepts.The authors also describe the work as the first study of SAEs for VLM failure prediction.Source: Introduction · pp. 1–2
  • During failures, class-specific concepts are suppressed while ambiguous or style-related concepts become more active.The analysis compares concepts activated during correct and incorrect predictions.Source: Introduction · p. 1
  • The paper also explores runtime recovery strategies using image masking and SAE latent interventions to correct misclassifications.This is presented as an exploration of recovery strategies.Source: Introduction · pp. 1–2

Related Work

Prior work studies failure prediction, out-of-domain detection, and SAE-based interpretation, but the paper distinguishes these aims and uses learned SAE directions for interpretable failure prediction without manually curated concepts.

Related Work

  • Confidence-based methods use probability- or entropy-derived uncertainty, while training-based methods learn separate predictors; both approaches have interpretability limits, and confidence methods can be ineffective on overconfident errors.A concept-score approach offers interpretability but depends on manually defined concepts that may not scale or reflect encoded features, potentially limiting accuracy.Source: Related Work · p. 2
  • The approach uses SAE latent directions learned from model representations rather than requiring a manually curated set of failure-discriminative concepts.This contrasts with a prior interpretable method based on human-level concept scores.Source: Related Work · p. 2
  • Out-of-domain detection asks whether samples come from a different distribution, whereas failure prediction targets erroneous outputs and also covers in-domain and adversarial errors.Out-of-domain samples may increase failure likelihood but do not necessarily produce incorrect predictions.Source: Related Work · p. 2
  • SAEs decompose representations into sparse latent features associated with human-understandable concepts, and prior work has applied them to language and vision-language models.This paper explores leveraging SAEs for failure prediction.Source: Related Work · p. 2

Failure-aware SAE and Failure Prediction SAE Architecture

The section describes an SAE that represents token embeddings with candidate concept directions, a failure-prediction head that classifies four outcomes, and a three-stage training pipeline.

Failure-aware SAE and Failure Prediction SAE Architecture

  • SAEs encode token embeddings into latent activations whose directions can represent candidate concepts, then decode those activations to reconstruct the embeddings.The SAE uses a latent dimension of 49,152.Source: SAE Architecture · p. 2

Failure Prediction

  • The prediction head classifies outputs as Correct, Error-ID, Error-OOD, or Error-ADV using masked SAE activations.The four labels distinguish correct outputs from in-distribution, out-of-distribution, and adversarial failures.Source: Failure Prediction · pp. 2–3
  • For prediction, the framework averages SAE activations across image tokens, retains the K largest activations, and feeds them to a four-class head.The top-K mask sets unselected activation values to zero.Source: Failure Prediction · p. 3

Failure-aware training

  • Training first reconstructs token embeddings with sparse SAE activations, then trains the head with the SAE frozen, and finally fine-tunes both jointly.The reconstruction loss combines embedding reconstruction error with L1 regularization for sparsity; the final stage sums SAE and cross-entropy losses.Source: Failure-aware training · p. 3

Performance of Failure Prediction

FailSAE is evaluated for predicting correct classifications and three types of VLM failures, and it outperforms the evaluated baselines on CLIP-B/16. Ablations examine its training stages and top-K masking, while additional experiments test performance across other VLMs.

Performance of Failure Prediction

  • Evaluation distinguishes Correct, Error-ID, Error-OOD, and Error-ADV, with binary evaluation grouping all three error types against Correct.The FailPred dataset includes misclassified clean, corrupted, and attacked images, alongside correctly classified samples.Source: Experimental settings · p. 3

Overall Performance

  • On CLIP-B/16, FailSAE consistently achieves the best performance across all three datasets and evaluation metrics, with accuracies of 88.0%, 76.3%, and 73.0% and binary-setting AUROCs of 93.9%, 85.6%, and 81.4%, respectively.Source: Overall Performance · p. 4

Ablation Study

  • The raw Embeddings baseline performs worst, and removing either training stage hurts, especially Stage 3, whose joint fine-tuning benefits from Stage 2's head warm-up.Stage 2 provides initialization for joint fine-tuning, while Stage 3 improves the usability of sparse features for failure prediction.Source: Ablation Study · p. 4
  • At K=300, accuracy reaches 88.1%, 76.8%, and 73.0% across the three datasets; Figure 2 plots accuracy against K.The latent dimension is 49,152, and the tested K values range from 50 to 49,152.Source: Figure 2 · p. 4, Impact of Top-K Latent Masking · p. 4
  • Accuracy is unstable above K=300: at K=500 it is 87.9%, 74.9%, and 72.0%, while using all 49,152 latents yields 87.7%, 74.5%, and 72.1%.The authors infer that most discriminative power lies in a small latent subset and that additional latents may be redundant or distracting.Source: Impact of Top-K Latent Masking · p. 4

Performance on Various VLMs

  • On CLIP-B/32, SigLIP-B/16, and CLIP-L/14, FailSAE consistently achieves high accuracy and AUROC.The authors describe the method as having no VLM-architecture requirement and as easily transferable to other VLMs.Source: Performance on Various VLMs · p. 4

Impact of failure-aware training on SAE

Failure-aware training makes SAE activations more class-specific while largely preserving reconstruction error and decoder orthogonality. Human evaluation and label-entropy analysis both indicate stronger concept specificity, especially among the most active directions.

Failure-aware training promotes class-specific concepts

  • Reconstruction error changes from 0.0015 for Standard SAE to 0.0019 for Failure-aware SAE, while decoder orthogonality changes from 0.39 to 0.38.The paper characterizes these changes as minimal, with the SAE retaining its core functionality while improving failure prediction.Source: Impact of failure-aware training on SAE · p. 5
  • Average label entropy falls from 2.598 for Standard SAE to 1.955 for Failure-aware SAE, indicating concepts are more associated with specific image classes.Failure-aware training also shifts sporadically activated directions toward lower activation frequencies, while mean activation-value distributions remain largely similar.Source: Impact of failure-aware training on SAE · p. 6

Qualitative evaluation of learned concepts

  • Among the 2,458 selected Failure-aware SAE directions, 627 (25.5%) meet the class-consistent concept criterion, compared with 457 (18.6%) for Standard SAE.Directions were ranked by mean activation, and their reference images were assessed for a shared class-consistent concept.Source: Impact of failure-aware training on SAE · p. 6
  • Among the top 1% most strongly activated directions, 44.1% from Failure-aware SAE represent meaningful concepts, versus 23.8% from Standard SAE.The evaluation selected the top 5% of 49,152 directions for human assessment and judged whether reference images shared a class-consistent concept.Source: Impact of failure-aware training on SAE · p. 6

Concept-Level Analysis of Failures

Across three failure types, incorrect predictions activate fewer class-specific concepts and more ambiguous or style-related concepts than correct predictions. Under OOD corruptions, class-specific directions can disappear from the top-activated set as ambiguous or style-related directions enter it.

Fewer class-specific concepts are learned when failures occur

  • Top-activated directions are selected by counting activations above threshold τ and taking the 1,000 directions with the most activations at class or dataset level.The directions are grouped as class-specific when label entropy is low, or ambiguous/style-related when it is high.Source: Concept-Level Analysis of Failures · p. 6
  • Across Error-ID, Error-OOD, and Error-ADV, failed predictions activate fewer class-specific directions and more ambiguous or style-related directions than correct predictions.The analysis uses CLIP ViT-B/16 and compares top-activated directions for correct and incorrect predictions.Source: Concept-Level Analysis of Failures · p. 6

Visualization of emerging and disappearing SAE latent directions during failures

  • Under OOD corruptions, many class-specific directions disappear from the top-activated set, while ambiguous or style-related directions enter it.Examples include a direction whose reference images depict the interior or underside of a triumphal arch disappearing, and a direction with stylistically similar but semantically unclear images emerging.Source: Concept-Level Analysis of Failures · p. 7

Runtime Failure Recovery

The paper explores two runtime recovery strategies: occluding candidate image regions to locate spatial corruption, and suppressing SAE directions associated with an incorrect prediction. The reported patch-attack recovery rate is 81.2%, and a Truck example is corrected through latent intervention.

Locating and removing spatial corruption

  • 81.2% of ImageNet-1K failures caused by patch attacks are recovered by masking the region identified through changes in predicted failure risk.Candidate regions are occluded in turn; a rise in Correct-class confidence above 0.85 marks a likely corrupted region, which is then masked before reclassification.Source: Runtime Failure Recovery · p. 7

SAE Latent intervention

  • Latent intervention suppresses activations associated with the incorrect class, decodes the modified activations into intervened CLIP embeddings, and updates the classification.If the failure predictor still signals an error, additional directions associated with the incorrect class are iteratively ablated.Source: Runtime Failure Recovery · p. 7
  • In the illustrated OOD example, suppressing Automobile-associated directions changes the prediction from Automobile to the ground-truth label Truck.The failure prediction model identifies the initial misclassification as Error-OOD.Source: Runtime Failure Recovery · p. 7

Conclusion

The framework uses SAEs for interpretable VLM failure prediction, and its analyses connect failures with changes in model representations.

  • The proposed framework accurately predicts model failures.Source: Conclusion · p. 7
  • Failure-aware training encourages SAE latent directions to capture more class-specific concepts.Source: Conclusion · p. 7
  • Comparing latent directions activated during correct and incorrect predictions provides an interpretable view of representation changes during failures.Source: Conclusion · p. 7
Loading 2609.04276v1…