PaperlayerView exact revision on arXiv

Source-linked AI summary

CRANE: Post-Hoc Counterfactual Retrieval for Multimodal Classifiers

Franco Rugolon, Ioanna Miliou, Panagiotis Papapetrou

arXiv:2610.31594v1cs.LG

TL;DR

Multimodal counterfactuals need to account for relationships across input modalities, yet prior counterfactual retrieval had not been extended beyond unimodal settings. CRANE retrieves target-class examples using model-agnostic strategies, and Multimodal Consensus Retrieval achieves the best or joint-best outcome on three of four datasets. Its real-instance retrieval grounds explanations in the joint training distribution, while unimodal strategies remain competitive when one modality dominates.

  • Problem

    Nearest-unlike-neighbour retrieval had not been extended to multimodal classifiers, and the relative merits of alternative search spaces for counterfactual quality and diversity remained unexplored.

    Source: §I · pp. 1–2

  • Method

    CRANE is a post-hoc, model-agnostic framework that retrieves target-class training examples through strategies including per-modality neighbourhood intersection.

    Source: §I · p. 2, 4) Multimodal Consensus Retrieval (MC-R) · p. 4

  • Results

    MC-R achieved the best or joint-best outcome on three of four datasets and perfect outcome on Real or Fake Jobs (0.000).

    Source: §VI · p. 7

  • Takeaways & Limitations

    MC-R grounds counterfactuals in the joint training distribution, while unimodal retrieval remains competitive when one modality dominates the decision boundary.

    Source: §I · p. 2, §VI · p. 7

    IF-NN fails to produce valid counterfactuals across the evaluated datasets and fusion strategies because the penultimate layer is optimized for discrimination rather than within-class clustering.

    Source: §VI · p. 7

Abstract

from arXiv · show

Counterfactual explanation methods have been introduced for unimodal settings, with dedicated frameworks for tabular, timeseries, text, and image data, yet no post-hoc method has been designed for classifiers that learn from multimodal inputs. Existing explainability methods for multimodal classifiers apply modality-specific attribution independently to each input, which fails to yield coherent explanations and does not transfer to arbitrary pre-trained multimodal architectures. To address these shortcomings, we present CRANE (Counterfactual Retrieval via Agnostic Nearest-neighbour Explanations), the first post-hoc, model-agnostic framework for nearest-neighbour multimodal counterfactual generation that retrieves real training examples from the target class as candidate explanations, ensuring plausibility by construction. Across four heterogeneous datasets and three fusion strategies, we show that Multimodal Consensus Retrieval (MC-R), which intersects per-modality neighbour sets to identify instances jointly close to the query in every modality, emerges as the strongest multimodal strategy, achieving the best or joint-best outcome on three of four datasets while grounding every explanation in the joint training distribution. Unimodal strategies remain competitive when a single modality dominates the decision boundary.

I. INTRODUCTION

Multimodal counterfactuals must remain coherent across modalities, but existing generators and retrieval methods are framed around single input spaces. CRANE addresses this gap with model-agnostic retrieval strategies and evaluates alternative similarity spaces against counterfactual quality and diversity objectives.

I. INTRODUCTION

  • Independent modality-specific counterfactuals can be locally plausible yet globally incoherent, producing clinically meaningless multimodal explanations.For example, a female survivor paired with a rising PSA trajectory is inconsistent.Source: §I · p. 1
  • Unimodal nearest-unlike-neighbour retrieval returns real target-class examples, but had not been extended to multimodal classifiers.Retrieval avoids synthesizing a new sample and therefore grounds candidates in observed data.Source: §I · p. 1, §II · p. 2
  • Different retrieval search spaces encode different similarity notions, and their relative merits for counterfactual quality and diversity remained unexplored.The considered spaces include one modality, concatenated modality embeddings, and the multimodal model's latent representation.Source: §I · pp. 1–2
  • CRANE is a post-hoc, model-agnostic nearest-neighbour framework supporting retrieval strategies from unimodal search to multimodal fusion.It supports arbitrary combinations of tabular, timeseries, text, and image modalities and extends counterfactual quality objectives to heterogeneous inputs.Source: §I · p. 2

II. RELATED WORK

  • Most explainability methods and reviewed counterfactual approaches assume homogeneous inputs, leaving multimodal counterfactual generation comparatively underexplored.Existing counterfactual methods cover tabular, timeseries, text, and image data, while retrieval work targets flat tabular or embedding-space inputs.Source: §I · p. 1, §II · p. 2

III. COUNTERFACTUAL RETRIEVAL VIA AGNOSTIC NEAREST-NEIGHBOUR EXPLANATIONS

CRANE formulates multimodal inputs as tuples spanning modality-specific spaces and treats the classifier as a black box. Its formulation accommodates multiple data types when suitable per-modality distance and plausibility functions are specified.

A. Problem Formulation

  • A multimodal input is a tuple whose components belong to the respective modality spaces, which together form a product space.The formulation allows one or more modalities and defines each space by its representation format and domain constraints.Source: A. Problem Formulation · p. 2
  • The framework can cover any data type with suitable distance and plausibility functions, and this work explores tabular, timeseries, text, and image inputs.Adding a modality requires specifying its distance and plausibility functions.Source: A. Problem Formulation · p. 2
  • CRANE requires only the classifier's prediction function and assumes no access to its internals, gradients, or training procedure.The classifier maps multimodal inputs to label scores, with the predicted label given by the highest-scoring class.Source: A. Problem Formulation · p. 2
  • Given a factual input, labelled training set, and different target label, the goal is a target-predicted counterfactual assessed using multimodal versions of standard quality objectives.The objectives are defined in Section III-A1.Source: A. Problem Formulation · p. 2

1) Objectives:

CRANE evaluates candidate counterfactuals using outcome, proximity, sparsity, and plausibility objectives. It normalizes and aggregates modality-specific scores so that heterogeneous inputs can be compared.

1) Objectives:

  • All four objectives are minimized, with zero denoting a perfect candidate; modality-specific scores are mapped to [0, 1] before aggregation.Experiments use uniform modality weights, although dataset-specific weights may be substituted when prior knowledge indicates differing modality importance.Source: 1) Objectives · pp. 2–3
  • Outcome measures whether and how decisively the candidate attains the target label, using prediction-flip validity and a continuous score deviation.The outcome objective reaches zero when the classifier has full confidence in the target label.Source: 1) Objectives · p. 3
  • Proximity measures deviation from the factual input using normalized modality-specific distances, while sparsity measures how many factual components must change.For tabular and timeseries inputs, proximity uses normalized mean absolute error; text and image use normalized cosine distance.Source: 1) Objectives · p. 3
  • Sparsity is modality-specific: it counts changed tabular features, threshold-exceeding timeseries timesteps, and token-level text edits, with image changes counted across pixel elements.The image measure is the fraction of pixel elements whose absolute difference exceeds a threshold.Source: 1) Objectives · p. 3
  • Plausibility measures how typical a candidate is relative to the target-class training distribution using modality-specific normalized outlier scores.The scores use Local Outlier Factor models fitted on target-class training data; timeseries are flattened before fitting.Source: 1) Objectives · p. 3
  • For k = 1, retrieval returns a real target-class training instance; for k > 1, only the first candidate is guaranteed to be observed, while later candidates are component-wise medians.The plausibility objective is applied uniformly to all candidates.Source: 1) Objectives · p. 3

B. CRANE Retrieval Strategies

CRANE searches target-class training examples and assembles candidates from the nearest neighbours. Its strategy family varies the space used for the search while retaining a shared black-box interface.

B. CRANE Retrieval Strategies

  • Retrieval strategies search the training set for the k instances nearest to the factual input among examples with the target label, then assemble candidates from those neighbours.The key design choice is the space in which nearest-neighbour search occurs.Source: B. CRANE Retrieval Strategies · p. 3
  • The strategy family ranges from unimodal search to multimodal fusion, reflecting different choices of search space.Each strategy takes the factual input, target label, and k as input.Source: B. CRANE Retrieval Strategies · p. 3
  • All strategies return an ordered candidate list and treat the classifier as a black box.They share the same interface: (x, y⋆, k).Source: B. CRANE Retrieval Strategies · p. 3

1) Unimodal Retrieval:

CRANE retrieves target-class neighbours in a single modality’s feature or embedding space, then constructs candidates according to whether that modality is structured or unstructured.

1) Unimodal Retrieval:

  • Four unimodal strategies rank target-class training instances by distance in one modality’s feature or embedding space.Text and image distances use encoder embeddings, while structured modalities use feature-space distances.Source: 1) Unimodal Retrieval · pp. 3–4
  • For tabular and timeseries data, the j-th candidate is a progressive median over the j nearest neighbours.At j = 1 it is a real training instance; for j > 1 it is a smoothed aggregate that may not match an observed row.Source: 1) Unimodal Retrieval · p. 4
  • For text and images, each candidate is the literal item from the corresponding rank in embedding space, so every candidate remains an observed training instance.As j increases, selected instances come from progressively more distant embedding-space regions.Source: 1) Unimodal Retrieval · p. 4

2) Early-Fusion Nearest Neighbour (EF-NN):

Early-fusion retrieval ranks neighbours in concatenated modality embeddings, whereas intermediate-fusion retrieval searches the classifier’s learned latent representation.

2) Early-Fusion Nearest Neighbour (EF-NN):

  • Early-fusion retrieval concatenates all modality embeddings and ranks neighbours by distance in that joint space.Candidates are assembled from the jointly ranked neighbours using the modality-specific materialisation rules.Source: 2) Early-Fusion Nearest Neighbour… · p. 4

3) Intermediate-Fusion Nearest Neighbour (IF-NN):

  • Intermediate-fusion retrieval ranks neighbours using distances between penultimate-layer activations of the multimodal classifier.Candidate construction uses the modality-type materialisation rules; this strategy requires a deep neural network with accessible intermediate activations and does not apply to non-differentiable models.Source: 3) Intermediate-Fusion Nearest… · p. 4

4) Multimodal Consensus Retrieval (MC-R):

MC-R selects candidates from the intersection of per-modality neighbour sets, while MPS combines separately retrieved modality prototypes into synthetic candidates.

4) Multimodal Consensus Retrieval (MC-R):

  • MC-R intersects per-modality top-k target-class neighbour sets and ranks shared candidates by summed modality distance.k is chosen to keep the consensus set nonempty; its first candidate is a real training instance and candidates are jointly close in every modality.Source: 4) Multimodal Consensus Retrieval (MC-R) · p. 4

5) Modality-wise Prototype Synthesis (MPS):

  • MPS concatenates independently retrieved modality prototypes, allowing candidates even when no training instance is jointly close across modalities.Because the modalities can come from different neighbours, MPS candidates are synthetic composites and lack the structural plausibility guarantee of MC-R and unimodal retrievers.Source: 5) Modality-wise Prototype Synthesis… · pp. 4–5

IV. EXPERIMENTS

The experiments cover four publicly available datasets across different application domains, modality combinations, and class structures.

A. Datasets

  • Four publicly available datasets span diverse domains, modality combinations, and class structures.Table I summarizes the datasets used in the experiments.Source: A. Datasets · p. 5, Table I · p. 5
  • Sepsis mortality prediction uses hourly clinical timeseries and static tabular features, with an approximately 7.4% mortality rate.Evaluation uses 5-fold stratified cross-validation, with F1 as the primary metric and AUC also reported.Source: A. Datasets · p. 5
  • Long COVID emotion classification uses Italian tweet text and tabular engagement and structural features; fear and anger dominate, while joy is the minority class.Fear accounts for 41.6%, anger 32.9%, and joy 8.8%.Source: A. Datasets · p. 5
  • Hateful Memes pairs image and caption inputs for binary hate-speech classification, with 63.7% non-hateful and 36.3% hateful examples.Source: A. Datasets · p. 5
  • Real or Fake Jobs combines structured posting fields with three text fields to classify postings as real or fraudulent.Fraudulent postings account for 4.8% of the data.Source: A. Datasets · p. 5

B. Experimental Setup

The experimental design spans multiple classifier families and fusion strategies, while retrieval is configured with k = 20. Evaluation averages four quality objectives across cross-validation folds and ablation combinations, with defined scope restrictions for certain strategies and datasets.

1) Classifier Families and Fusion Strategies:

  • Classifiers span early, intermediate, and late fusion, with deep MLPs and non-deep ensembles or ROCKET-based models; IF-NN is limited to deep classifiers.All other CRANE strategies are evaluated across both classifier families.Source: 1) Classifier Families and Fusion… · p. 5

2) CRANE Configuration:

  • CRANE retrieval uses k = 20; MC-R searches 200 neighbours per modality before intersection, and encoder/distance ablations range from 6 to 84 combinations by dataset.Results are averaged across ablation combinations.Source: 2) CRANE Configuration · p. 5

3) CRANE Strategies per Dataset:

  • IF-NN is evaluated only on deep classifiers, and unimodal strategies are omitted when their modality is absent.For example, image strategies are not evaluated on Sepsis, Tweets, or Jobs.Source: 3) CRANE Strategies per Dataset · p. 5

C. Evaluation Protocol

  • Four lower-is-better objectives are averaged across five cross-validation folds and pooled test queries, with all test instances predicted as undesired-class cases included.For Long COVID Tweets, evaluation focuses on sadness→joy; other class pairs are excluded because they lack a clearly defined opposite.Source: C. Evaluation Protocol · p. 6

V. RESULTS

Intermediate fusion is strongest or joint-strongest on macro-F1 across datasets, while EF-NN is the most reliable generator by mean outcome. Generator performance varies by dataset and metric, and changing k affects outcome and plausibility differently across strategies.

A. Classifier Performance

  • Intermediate fusion is strongest or joint-strongest on macro-F1 across datasets, including 0.717 versus 0.589 for early fusion on Tweets and 0.932 versus 0.886 on Jobs.Late fusion achieves the highest Sepsis AUROC at 0.871; Hateful Memes macro-F1 ranges from 0.669 to 0.713.Source: A. Classifier Performance · p. 6

B. Generator quality

  • 0.079 mean outcome: EF-NN is the most reliable generator overall, while unimodal generators achieve the best proximity and plausibility.Fig. 2 summarizes metric distributions across the benchmark; all metrics are lower-is-better.Source: Figure 2 · p. 6, B. Generator quality · p. 6
  • On Jobs, MC-R achieves 0.000 outcome and 0.137 sparsity; on Sepsis, MPS leads proximity and plausibility while TimeSeriesNN has the lowest outcome.On Tweets, TabularNN has the best outcome (0.217); Memes sparsity is fixed at 0.500 because distinct normalized training images differ at every pixel.Source: B. Generator quality · p. 6
  • Fusion strategy shifts mean outcome by up to 0.58 on Tweets but below 0.05 on Sepsis and Jobs, while proximity, sparsity, and plausibility remain largely stable.Table III averages across fusion strategies, so the classifier’s fusion strategy is an additional source of outcome variance.Source: B. Generator quality · p. 6

C. Effect of k on Counterfactual Quality

  • As k increases from 1 to 50, outcome improves for nearly all generators, but MC-R plateaus after k = 5 at 0.116 to 0.115 by k = 50.IF-NN has the largest absolute improvement yet remains the worst generator at k = 50.Source: C. Effect of k on Counterfactual Quality · p. 6
  • Across k, ImageNN plausibility remains exactly 0.527 because every image candidate is a real training instance, while sparsity remains nearly flat across k for all generators.Fig. 3 averages metric trends across all datasets and fusion strategies, showing unimodal generators with dashed lines and multimodal generators with solid lines.Source: C. Effect of k on Counterfactual Quality · p. 7, Figure 3 · p. 7

VI. DISCUSSION

MC-R is the strongest multimodal strategy overall, while unimodal retrieval remains competitive when one modality dominates. The discussion also ties counterfactual outcomes to classifier quality and identifies limits of latent-space retrieval.

VI. DISCUSSION

  • On Sepsis, TimeSeriesNN achieves 0.021 outcome, while on Hateful Memes TextNN (0.070) outperforms ImageNN (0.182).The results suggest temporal clinical trajectories dominate Sepsis decisions and text dominates meme-classification decisions; on imbalanced Sepsis and Jobs, the large majority-class retrieval pool partly aids performance.Source: §VI · p. 7
  • IF-NN fails to produce valid counterfactuals across datasets and fusion strategies, even at k = 50.The discussion attributes this to discriminatively trained penultimate layers not reliably grouping target-class neighbours; explicitly clustered latent representations may overcome the limitation.Source: §VI · p. 7
  • Counterfactual quality partly depends on classifier quality: Jobs classifiers exceed F1 0.88, while Hateful Memes classifiers score F1 0.669–0.713.On Jobs, every generator except IF-NN reaches near-zero outcome; on Hateful Memes, outcome is substantially higher across generators.Source: §VI · p. 7
  • MC-R achieves the best or joint-best outcome on three of four datasets, including 0.000 outcome and 0.137 sparsity on Jobs.Its intersection retrieves instances jointly close across modalities, grounding explanations in the joint training distribution; outcome saturates at k = 5.Source: §VI · p. 7

VII. CONCLUSION

  • The conclusion presents CRANE as supporting arbitrary modality combinations and identifies baseline comparisons, expert evaluation, more clinical datasets, and generative strategies as future work.It characterizes MC-R across four datasets and three fusion strategies as combining strong outcomes, joint-distribution grounding, and computational efficiency.Source: §VII · p. 7

Related papers

Loading 2610.31594v1…