Source-linked AI summary
Counterfactual Transition Graphs: Evaluating Cross-Class Transition Quality
Syed Muhammad Hamza Zaidi, Szymon Bobek, Grzegorz J. Nalepa, Myra Spiliopoulou
TL;DR
The paper addresses the limits of instance-level counterfactual evaluation by asking how a classifier connects its classes through plausible edits. It constructs a prototype-based Counterfactual Transition Graph and finds that confident boundaries can be among the hardest to cross in distribution, while generator choice separates on-manifold from off-manifold conclusions.
Problem
Counterfactual evaluation usually studies one example at a time, leaving pairwise class differences and global transition structure insufficiently explicit.
Method
The paper builds a class-level graph whose edge weights aggregate prototype-to-prototype counterfactual transition reliability using class proximity and interchangeable generators.
Results
The six-class graph shows non-trivial transition topology: the most confidently separated boundaries are among the hardest for in-distribution counterfactuals to cross, while gradient and replacement generators disagree on manifold-sensitive transitions.
Takeaways & Limitations
Counterfactual reachability can diagnose class structure and distinguish classifier behavior from explainer artefacts through generator agreement or disagreement.
Takeaways & Limitations
The approach was evaluated on only one dataset, with three prototypes fixed per class; transfer and prototype-count effects remain open.
Abstract
from arXiv · showhide
Counterfactual (CF) explanations for time-series classifiers are usually evaluated one example at a time: what minimal edit flips this single window's prediction? We argue that the more informative question for diagnostic interpretability is structural: how does the classifier connect its own classes to each other? We propose a counterfactual transition graph (CGT) in which each node is a class and each edge weight is the CF reliability of the transition from one prototype to another under a proximity aware retrieval sweep. On a six-class hand-movement task, we induce a CGT that reveals a non-trivial topology, which is not predicted by the binary confusion matrix: it shows that counterfactual reachability does not align with classifier accuracy and even runs counter to it (Spearman $ρ=-0.37$ over the 15 pairs), i.e. the boundaries the classifier separates most confidently are among those an in-distribution edit can least often cross. Our framework is method agnostic, i.e. any CF-explainers can be used. Presently, we use it to juxtapose replacement-based CFs with gradient-based CFs; gradient-based methods reach almost any class by stepping off the data manifold, while replacement-based methods stay on it and fail on precisely the rigid boundaries.
1 Introduction
Counterfactual explanations are usually assessed per instance, but the paper asks what class-to-class edits reveal about counterfactual plausibility and classifier structure. It proposes a prototype-based transition graph and finds that highly confident class boundaries can be hardest to cross with in-distribution edits.
- Counterfactual explanations ask for the smallest plausible change that flips a classifier’s prediction to a chosen class.
- The paper shifts evaluation from individual edits to the global pattern of class-to-class transitions and their plausibility.
- The Counterfactual Transition Graph aggregates prototype-anchored transition reliability for every class pair under retrieval strategies.
- The hand-movement study finds that the strangest counterfactuals occur between classes separated with high confidence.
- The SenseGlove use case combines multimodal joint readings, temporal dependencies, and movement classes with different kinematic characteristics.
- Testing transfer beyond this concrete hand-movement use case remains future work.
2 Related Work
Related work evaluates time-series counterfactuals through perturbation, substitution, and retrieval-based generation, typically emphasizing instance-level validity, proximity, and plausibility. The paper identifies a remaining gap: evaluations rarely analyze pairwise differences in transition difficulty between classes.
- Time-series counterfactual methods include direct perturbation, subsequence substitution, and retrieval of nearby unlike examples.
- Perturbation-based methods modify the queried series, while substitution-based methods insert or patch subsequences from other examples.
- Counterfactual evaluations commonly measure validity, proximity, and plausibility, with time-series extensions including smoothness, continuity, temporal coherence, and signal dynamics.
- Existing approaches largely focus on instance-level properties rather than differences between specific class pairs.
- Actionability concerns whether modifications can be implemented in real-world settings while preserving immutable features and semantic relationships.
3 Materials
The study uses SenseGlove hand-movement recordings and a temporal graph classifier that combines anatomical graph convolutions with bidirectional temporal encoding. Four generation cores provide contrasting perturbation, substitution, contextual retrieval, and residual-attenuation edit mechanisms.
- The SenseGlove dataset contains recordings from 34 subjects performing six labeled hand movements, including Clothes-Peg, Fingertip-Touch, and four cube rotations.
- The four cube rotations form a size-and-direction taxonomy that the transition graph can test against classifier behavior.
- Each hand window contains 20 anatomical joint channels with angular velocity and acceleration features sampled at 100 Hz.
- The classifier represents the hand as an anatomical graph, applies graph convolutions per time step, uses a bidirectional LSTM for temporal encoding, and classifies with an MLP head.
- The study compares direct perturbation, local substitution, context-aware substitution, and residual attenuation, all operating on non-overlapping windows.
4 Our approach
The CFTG turns instance-level counterfactual generation into a class-level graph whose edge weights quantify reliable, plausible transitions between prototype representations. It also separates counterfactual reachability from classifier difficulty to diagnose rigid and porous class boundaries.
- The CFTG represents each class with typical prototypes and weights class-pair edges by aggregated counterfactual transition reliability.Prototypes are selected by proximity to class centroids, with multiple compact representatives per class.
- A transition attempt edits a source prototype toward a target class, then records whether the classifier crosses the boundary and how plausible the result is.Validity is binary, while plausibility measures proximity to the target prototypes and is independent of validity.
- CP-proximity averages transition validity across prototypes, folds, retrieval strategies, and both directions between a class pair.Pooling both directions is justified because the pairwise classifier defines one shared boundary.
- CF-rigidity is the complement of CP-proximity, so high rigidity indicates mostly invalid crossings whereas low rigidity indicates a porous boundary.Classifier difficulty is pairwise classification error, and their interplay is measured with Spearman correlation.
- The graph provides a benchmark for comparing counterfactual generators and shows a clear-cut Clothes-Peg/Fingertip-Touch boundary where replacement methods remain on-manifold but fail, while gradient methods succeed off-manifold.This construction supports generator-agnostic evaluation through per-pair transition edges.
5 Experimental Setup
The experiments evaluate the CFTG on six SenseGlove hand movements using pairwise classifiers, prototype-based transition sweeps, and four counterfactual generators. The setup also explicitly limits claims by leaving sensitivity to design choices for future work.
- The evaluation uses six SenseGlove classes, 34 subjects, 150-sample windows over a 20-node hand graph, and five-fold subject-grouped cross-validation.
- The study trains 15 pairwise classifiers, one for each class pair and fold, with a mean validation accuracy of 0.97.Pairwise specialization removes multi-target ambiguity and gives each graph edge a single-boundary interpretation.
- Counterfactuals are generated with replacement-based GACP and GACP-Context and gradient-based NEPO and Residual using three prototypes per class and validity threshold 0.5.Replacement methods sweep nine retrieval specifications, while gradient generators run once per setup.
- The workflow depends on choices such as prototype count, retrieval settings, and distance metrics, whose systematic sensitivity analysis is outside the present scope.
- Table 1 compares binary accuracy, pooled edge weight, successful-flip plausibility, and validity for each generator across the 15 class pairs.Rows are sorted by increasing CP-proximity, and the graph uses these quantities to juxtapose transition structure with classifier accuracy.
6 Results
The six-class CFTG reveals transition structure that differs from classifier difficulty: Clothes-Peg boundaries are rigid despite easy separation, while generator choice distinguishes on-manifold reachability from off-manifold flips.
- Transition graph versus confusion matrix: 13,500 transition attempts produce a six-class graph whose pooled edge weights summarize mean validity, plausibility, and per-generator validity by class pair.The graph is compared with binary classifier accuracy to assess transition structure.
- Transition graph versus confusion matrix: 0.56 is the highest pooled edge weight, for Fingertip-Touch to Small-Rubik rotation; other porous pairs include Fingertip-Touch to Rubik counter-clockwise and Rubik clockwise to Small-Rubik counter-clockwise at 0.49.The most rigid boundaries involve Clothes-Peg.
- Transition graph versus confusion matrix: The most confidently separated pairs, including T1,T3 and T1,T4, are among the hardest for in-distribution counterfactuals to cross, whereas T1,T5 reverses this pattern.Figure 4 compares independently min-max-rescaled classifier difficulty and CF rigidity; connector length represents ranking disagreement.
- Transition graph versus confusion matrix: Spearman ρ = −0.37 with p = 0.18 indicates a weak negative rank association between classifier difficulty and rigidity across 15 class pairs, without statistical significance.The authors attribute caution partly to the small number of pairs and uncorrected multiple testing.
- A singleton cluster and non-taxonomic clusters: The rigidity dendrogram isolates Clothes-Peg as a singleton and mixes object sizes and rotation directions rather than recovering the human taxonomic grouping.Same-object opposite-direction pairs are only moderately porous, with w = 0.30 for T3,T4 and w = 0.35 for T5,T6.
- On-manifold and off-manifold transitions: Gradient generators achieve mean validity 0.84 and 0.83, while replacement generators achieve 0.19 and 0.30 and collapse near zero on rigid Clothes-Peg boundaries.Gradient methods reach rigid boundaries through highly implausible, off-manifold flips; replacement methods remain on-manifold and fail where no in-distribution path exists.
- On-manifold and off-manifold transitions: Generator disagreement is greatest on boundaries that can be crossed only off the manifold, so agreement defines a method-robust skeleton while high-disagreement edges should remain generator-sensitive.The graph turns an isolated Fingertip-Touch to Clothes-Peg counterfactual into a dark edge toward a T1 attractor.
7 Discussion and Conclusion
The paper introduces the CFTG as a class-level instrument for evaluating counterfactual explainers and finds that high classifier performance does not imply high-quality counterfactuals. Its proof-of-concept remains limited to one dataset, while future work targets broader benchmarks and ensemble explainers.
- Discussion and Conclusion: The CFTG aggregates valid transition attempts between dataset classes, with classes as nodes and edge weights as proportions of valid transitions.The construction is intended to evaluate counterfactual generation cores.
- Discussion and Conclusion: On the SenseGlove hand-gesture dataset, high classifier performance does not imply high counterfactual quality.The evaluation is presented as a proof of concept.
- Limitations and future work: The approach was tested with multiple counterfactual generator cores but only on one dataset, limiting evidence about generality across datasets.The authors identify dataset choice as pivotal and plan to collect further datasets with suitable class separability and visualizable validity.
- Limitations and future work: Future work includes native multiclass counterfactuals, a second time-series benchmark, and an ensemble explainer treating each class-to-class transition as a separate decision problem.The ensemble direction follows directly from the pairwise transition graph.