PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Multimodal Classification with Linear Discriminant Tree Ensembles

Mojtaba Moattari

arXiv:2608.20384v1cs.AI

TL;DR

Transformers offer strong multimodal classification but limited transparency, motivating a more explainable approach. The paper develops tree-based ensembles with modified feature importance and reports competitive results alongside improved indicator detection and interpretability.

  • Problem

    Transformers provide strong multimodal classification but limited transparency for classification, interpretation, and feature extraction, motivating competitive and explainable alternatives.

  • Method

    The framework combines tree-based ensemble classifiers with feature encoders, inductive biases, and a modified importance metric that emphasizes nodes where positive instances predominate.

  • Results

    LDF achieves 86.3% accuracy, while tree-based variants exceed IMR on F1-mod and modified feature importance outperforms default feature importance at sample level.

  • Takeaways & Limitations

    The results support tree-based ensembles as a suitable approach for imbalanced multimodal classification with more interpretable feature indicators than default importance.

  • Takeaways & Limitations

    LDA-node projection fits a separate LDA model at every tree node, potentially making training prohibitive for very large datasets without further optimization.

Abstract

from arXiv · show

Multimodal affect and behaviour classifiers that fuse heterogeneous text, audio, and visual streams must simultaneously achieve competitive accuracy and produce human-understandable explanations of the cues driving their decisions -- a dual objective that current high-capacity models, notably Transformers, only partially address. While Transformers attain strong predictive performance, their distributed representations and deep nonlinearity make it difficult to assign meaningful importance weights to individual multimodal features, limiting their use in trust-sensitive applications such as clinical affect monitoring and educational assessment. We address this gap by developing a framework based on tree-based ensembles that balances accuracy and interpretability. The framework encodes each modality into tokens, extracts and clusters concepts to reduce dimensionality, routes the fused modalities through tree-based ensemble classifiers, and interprets trends using a novel modified feature importance metric. The modified importance reduces the influence of the negative class in binary classification tasks, thereby improving indicator or marker detection. The proposed tree-based ensembles -- Linear Discriminant Tree (LDT), Linear Discriminant Forest (LDF), and Linear Discriminant AdaBoost (LDAB) -- achieve F1-mod gains of 4.3\% over the Multimodal Transformer and accuracy gains of 3.0\% over the primary interpretable multimodal baseline, Interpretable Multimodal Routing (IMR). The proposed multimodal feature importance extracts salient inter-modal concepts with substantially higher human-annotator agreement scores than default feature importance (62.2\% vs.\ 43.2\% on IEMOCAP; 46.7\% vs.\ 32.1\% on CMU-MOSI).

1 Introduction

The paper targets multimodal classifiers that combine competitive performance with human-understandable explanations, addressing transparency and computational limitations in Transformer-based approaches. It proposes a unified tree-ensemble framework with context encoding, LDA-node projection, and modified feature importance, evaluated through multiple research questions and datasets.

  • Motivation: Transformer-based multimodal models achieve strong performance but remain computationally demanding and difficult to interpret at feature and instance levels.Their distributed representations, depth, and cross-modal attention complicate local and global importance analysis.
  • Proposed framework: The paper proposes a unified framework that combines multimodal context encoding, tree-based ensembles, and a modified interpretability metric.The framework is designed to compare ensemble types systematically while balancing competitive accuracy and explainability.
  • Proposed framework: LDA projection is inserted at every tree node and extended across DT, Random Forest, and AdaBoost variants, producing LDT, LDF, and LDAB.The authors characterize this node-level discriminant projection as unexplored in ensemble learning.
  • Interpretability: The modified feature-importance method up-weights branches dominated by positive instances to make positive-class cues more interpretable and fault-diagnosable.This addresses standard importance scores that can be dominated by nodes discriminating against negative instances.
  • Context encoding: Three context-integration schemes—K-means, eigenvalue-based binary hierarchical clustering, and multi-sense clustering—are compared for token encoding before fusion.The paper reports that the choice of scheme substantially affects accuracy and interpretability.
  • Evaluation: Experiments on IEMOCAP, CMU-MOSI, and a custom Multimodal Mathematics dataset evaluate accuracy, interpretability, LDA projection, and context-clustering effects.The research questions explicitly compare tree ensembles with Transformers, test modified importance, assess LDA-node projection, and examine clustering choices.

2 Related Work

Prior work combines interpretable tree ensembles, discriminant projections, and concept-based multimodal explanations, but existing approaches leave important gaps in ensemble generalization and cross-modal attribution.

  • Tree ensembles: Random Forest, AdaBoost, and XGBoost aggregate tree predictions through parallel bootstrap sampling, iterative instance reweighting, and sequential residual minimization.These ensemble mechanisms provide the basis for extending linear-discriminative node projections beyond a single tree.
  • Linear-discriminant trees: Fisher’s LDA forms oblique tree splits by maximizing between-class variance relative to within-class variance.The projection uses class means and the inverse pooled covariance matrix.
  • Research gap: The proposed framework extends LDA-based node projection to RF, AB, and XGB while weighting feature importance by positive-class contribution.This targets the limited positive-label emphasis of standard tree importance.
  • Research gap: The framework embeds LDA projectors simultaneously in DT, RF, and AB, enabling unified comparison under one multimodal pipeline.Earlier work considered single trees or discriminant forests without the same boosting extension.
  • Concept-based interpretability: Existing concept-based methods do not jointly provide multimodal concept fusion and class-discriminative tree-node importance.The proposed framework addresses these limitations across text, audio, and visual modalities.

3 Proposed Methods

The proposed method combines LDA-based oblique node projections with feature encoding, context integration, class balancing, and tree ensembles for multimodal classification.

  • Multimodal framework: The framework combines class balancing, feature coding, and tree-based ensemble classifiers after reducing the dimensionality of all modalities.The encoder represents utterances using sentence, BoW, or bag-of-n-grams formats.
  • Class balancing: Subsampling addresses class imbalance through condensed-nearest-neighbour, Near-Miss, edited-nearest-neighbour, random, or binary-hierarchical grouping strategies.These methods remove selected majority-class instances before classification.
  • Context integration: Context integration assigns modality tokens to clusters using K-means, eigenvalue-based binary hierarchical clustering, or multi-sense clustering.Binary hierarchical coding preserves semantic proximity through numerically adjacent cluster-path codes.

3.4 Node Projectors

The framework uses several projector types for oblique-tree nodes, including discriminant, independent-component, and principal-component methods. Separating positive- and negative-class subspaces helps identify projections that reduce node impurity.

  • Oblique-tree node projectors include Histogram-Based Discriminant Dependency Analysis, Fast Independent Component Analysis, Linear Discriminant Analysis, and Principal Component Analysis.
  • Separating the feature space into positive-class and negative-class subspaces helps identify projections that minimize node impurity.

3.5 Feature Extractor

Audio and visual modalities use dedicated affective-computing feature extractors, while the Multimodal Mathematics dataset receives OCR and image preprocessing into separate sparse modalities.

  • COVAREP extracts audio representations, while FACET and classical HOG and PHOG descriptors provide visual features.
  • For Multimodal Mathematics, Tesseract OCR extracts text, and image quantisation, blurring, dilation, and resizing produce sparse pixel-intensity and location modalities.

3.6 Datasets, Baselines, and Hyper-parameters

The framework is evaluated on three multimodal benchmarks against interpretable and non-interpretable baselines, with results averaged across runs and hyper-parameters tuned through search procedures.

  • Datasets: The evaluation uses IEMOCAP for emotion recognition, CMU-MOSI for sentiment classification, and a custom Multimodal Mathematics dataset.
  • Datasets: IEMOCAP and CMU-MOSI contain textual, visual, and audio modalities, with accuracy and F1 reported following prior work.
  • Baselines: IMR is the primary interpretable baseline, while MulT and LMF-MulT are secondary non-interpretable reference models.
  • Evaluation: Results are averaged over 10 independent runs, with paired t-tests and Holm correction used for significance testing against IMR.
  • Hyper-parameters: Table 1 reports hyper-parameter search spaces and final tuned values for each classifier, with the best configuration selected by 5-fold cross-validation on training data.
  • Hyper-parameters: Tree depth and estimator count are selected by grid search, while LDA regularisation and learning rates are tuned through Bayesian optimization.
  • Feature representations: COVAREP supplies 74 acoustic features, and the framework identifies informative action units using cross-modal feature components and highly weighted action units.

3.9 Modified Feature Importance

The modified feature-importance framework targets positive-class indicators by reweighting tree-node gains and supports global and local multimodal explanations.

  • Motivation: Standard feature importance can favor nodes separating the majority negative class, whereas the modified measure emphasizes positive outcomes in assessment settings.
  • Node weighting: The node-level reweighting rewards splits whose right branch contains more positive instances than the left branch.
  • Node weighting: The denominator normalizes by correctly directed positive and negative instances, limiting trivially large gains from high-cardinality nodes.
  • F1-mod metric: F1-mod uses TP, FP, FN, and the negative-to-positive ratio m, and is equivalent to an Fβ measure under the stated imbalance condition.
  • F1-mod metric: When the negative class is at least four times larger than the positive class, F1-mod penalizes false positives less heavily than standard F1.
  • Explanation procedure: Algorithm 2 computes cluster-level concept importance from a trained ensemble and can produce global plots or sample-specific visualizations.

3.10 Algorithm for General and Local Feature Importance

Algorithm 2 computes and visualises global and local multimodal feature importance using the framework’s ensemble-tree models.

  • Algorithm 2 computes and visualises both global and local multimodal feature importance from the framework’s trained ensemble-tree models.

4 Results

The proposed ensembles generally match or exceed baseline classification performance, while LDA-node projection and contextual feature encoding improve results. Modified feature importance also yields more useful multimodal interpretations, although accuracy and interpretability can trade off.

  • Performance Comparison (RQ1 & RQ3): LDF achieves 67.8% F1-mod H on IEMOCAP, exceeding IMR’s 50.0% by 17.8 percentage points.
  • Performance Comparison (RQ1 & RQ3): LDF reaches 84.2% classification performance on Multimodal Mathematics, approaching Multimodal Transformer performance.
  • Ablation Study (RQ4): Removing context integration drops F1 by up to 29 percentage points, while removing the feature encoder reduces performance to 49–53%.
  • Interpretability Scores: Modified FI achieves 24.87/36 versus 15.55/36 for default FI on IEMOCAP and 16.82/24 versus 11.46/24 on CMU-MOSI.
  • Interpretability Scores: LDF consistently leads interpretation because its subspace-inclusive modified feature importance uses LDA node projections across all tree nodes.
  • Interpretability Scores: The most accurate proposed ensemble typically has the lowest interpretability score, indicating an accuracy–interpretability trade-off.
  • Feature Encoding: Bigram encoding represents analogy and contextual clarity more effectively than unigram encoding across trials.

5 Conclusion and Future Work

The framework combines LDA-based node projection, context embedding, feature encoding, and modified feature importance in tree-based multimodal ensembles. It reports competitive classification and improved interpretability, while identifying computational, demographic, linguistic, and binary-classification scope limits.

  • LDF achieves 86.3% accuracy and 83.5% F1 on IEMOCAP happy, while reaching 84.2% F-measure on Multimodal Mathematics.
  • Modified FI achieves a 45% average interpretability score versus 25% for default FI, with higher sample-level agreement on IEMOCAP and CMU-MOSI.On IEMOCAP, scores are 24.87/36 versus 15.55/36; on CMU-MOSI, 16.82/24 versus 11.46/24.
  • LDA-node variants consistently outperform standard tree ensembles on accuracy and F1-mod, with LDF exceeding IMR by 17.8 percentage points on IEMOCAP happy.
  • Removing context embedding lowers F1 by up to 29 percentage points, while K-means with bigram encoding provides the best reported balance of accuracy and interpretability.
  • LDA-node projection increases training time because a separate LDA model is fitted at every tree node, especially for deep trees and large feature spaces.Inference speed is comparable to standard trees, but training may be prohibitive for very large datasets without further optimization.
  • Interpretability benefits were validated only on English-language, North American speaker populations, limiting demonstrated generalization across languages, cultures, and emotional norms.
  • The modified feature importance is formulated for binary targets; extending it to multiclass settings requires one-vs.-rest decomposition or reformulating positive-class weighting.

Appendix B Visual Feature Descriptions

The appendix associates visual FACET features with facial Action Units, expressions, emotions, and sentiment valence, while documenting AU numbering and non-exclusive emotional associations.

  • FACET features map facial Action Units to associated facial expressions, emotions, and sentiment valence, with FACS numbers provided for each AU.An asterisk marks emotions not exclusively associated with the listed AU because the AU also occurs in other emotional contexts.
Loading 2608.20384v1…