"What is Relevant in a Text Document?": An Interpretable Machine Learning Approach

Leila Arras; Franziska Horn; Grégoire Montavon; Klaus-Robert Müller; Wojciech Samek

Source-linked AI summary

"What is Relevant in a Text Document?": An Interpretable Machine Learning Approach

Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, Wojciech Samek

arXiv:1612.07843v1Dec 23, 2016cs.CLcs.IRcs.LGstat.ML

TL;DR

The paper addresses how to make accurate text classification decisions understandable at the word level. It adapts LRP to decompose CNN and BoW/SVM predictions, then uses the resulting relevances for document representations and explanatory-power assessment. Although the models perform similarly in classification, the CNN shows higher explainability, while LRP-based document vectors reveal clearer semantic structure.

Problem
Accurate text classifiers are useful for annotating massive collections, but understanding how and why they categorize documents remains important.
Method
The paper adapts LRP to assign word-wise relevance scores for CNN and BoW/SVM topic classifiers and uses them to construct document vectors and assess explanatory power.
Results
Although CNN and BoW/SVM models have similar classification performance, the CNN is more explainable, and its LRP-weighted document vectors show clearer semantic clustering than baseline weightings.
Takeaways & Limitations
LRP supports fine-grained and dataset-wide inspection of words influencing classifier decisions and extends classification outputs with document-level semantic information.
Takeaways & Limitations
The work is a first step toward applying LRP in NLP, with broader suitability for other architectures and classification problems left as expected future applications.

Abstract

from arXiv · show

Text documents can be described by a number of abstract concepts such as semantic category, writing style, or sentiment. Machine learning (ML) models have been trained to automatically map documents to these abstract concepts, allowing to annotate very large text collections, more than could be processed by a human in a lifetime. Besides predicting the text's category very accurately, it is also highly desirable to understand how and why the categorization process takes place. In this paper, we demonstrate that such understanding can be achieved by tracing the classification decision back to individual words using layer-wise relevance propagation (LRP), a recently developed technique for explaining predictions of complex non-linear classifiers. We train two word-based ML models, a convolutional neural network (CNN) and a bag-of-words SVM classifier, on a topic categorization task and adapt the LRP method to decompose the predictions of these models onto words. Resulting scores indicate how much individual words contribute to the overall classification decision. This enables one to distill relevant information from text documents without an explicit semantic information extraction step. We further use the word-wise relevance scores for generating novel vector-based document representations which capture semantic information. Based on these document vectors, we introduce a measure of model explanatory power and show that, although the SVM and CNN models perform similarly in terms of classification accuracy, the latter exhibits a higher level of explainability which makes it more comprehensible for humans and potentially more useful for other applications.

1 Introduction

The paper adapts layer-wise relevance propagation to identify words that explain text-classification decisions, comparing a CNN with a bag-of-words SVM. It also introduces relevance-based document vectors and a measure of model explanatory power.

Motivation: The paper addresses the need to process massive text collections automatically while making classification decisions understandable at the word level.The introduction situates this need among NLP tasks such as topic categorization, sentiment analysis, translation, extraction, and summarization.
Related foundation: The study combines prior input-space explanation techniques with LRP, which was developed and tested for explaining neural-network predictions in image categorization.This work extends that explanatory approach into text classification.
Contributions: The paper applies LRP to NLP by decomposing classifier predictions into relevance scores assigned to individual words.The method is demonstrated on text documents and compared across a word-embedding CNN and a BoW/SVM classifier.
Contributions: LRP is evaluated qualitatively through document heatmaps and category-level representative-word lists, and quantitatively against sensitivity analysis.The paper reports that LRP identifies relevant words better than sensitivity analysis.
Contributions: The authors introduce relevance-weighted document vectors that capture semantic regularities in their original feature space.These representations condense semantic information from word embeddings into document-level vectors.
Contributions: A proposed explanatory-power measure shows that CNN and BoW/SVM models can have similar classification performance but substantially different explainability.The comparison makes explainability a distinct property from predictive accuracy.

2 Related Work

Prior work explains classifier decisions using model-specific and model-agnostic techniques, including input-feature decompositions and gradient-based salience for text. These approaches differ in generality, computational cost, and validation strategy.

Classifier explanations: Explanation methods have been developed for additive, kernel-based, hierarchical, and convolutional neural-network classifiers.The cited literature includes both classifier-specific approaches and methods for decomposing outputs onto input variables.
Model-agnostic methods: Model-agnostic explanation methods based on random sampling are general but require processing the whole sample to produce one explanation.Their generality therefore comes with additional computational cost.
Text explanations: In text classification, gradient magnitudes have been used to extract salient sentences and validate them through downstream classification performance.The validation compares selected sentences with random and heuristic selection strategies using an external classifier.

3 Interpretable Text Classification

The section presents a word-embedding CNN and LRP procedure that classify documents and trace predictions back to relevant words. It also adapts relevance decomposition to a BoW/SVM baseline and constructs document summary vectors.

3.1 Representing Words and Documents: The method represents each document as a matrix of concatenated word2vec embeddings before CNN processing.The matrix has shape D × L, where D is embedding dimension and L is document length.
3.2 Predicting Category with a Convolutional Neural Network: The CNN applies position-equivariant convolutional filters, ReLU activations, and max pooling before logistic classification.Convolution produces F feature maps, and max pooling creates position-invariant features for the final classifier.
3.3 Explaining Predictions with Layer-wise Relevance Propagation: LRP explains a target classification score by propagating relevance backward from the output to individual input variables.The top-layer relevance is initialized from the target class score, then redistributed through the network to identify contributions to or against the decision.
3.4 Word Relevance and Vector-Based Document Representation: Word-level relevance scores highlight important words and can be combined with word2vec embeddings to form document summary vectors.These vectors lie in the same semantic space as word2vec representations.
3.5 Baseline Methods: The BoW/SVM baseline uses word-frequency statistics and lacks the CNN’s learned hidden representations and distributed semantic similarity.Its document summary vector is constructed from word relevances over a binary bag-of-words representation.

4 Quality of Word Relevances and Model Explanatory Power

The section evaluates word-relevance explanations intrinsically and compares model explanatory power through an extrinsic summary-vector procedure. The proposed index uses KNN performance on summary vectors as a common basis for comparing models.

4.1 Measuring the Quality of Word Relevances through Intrinsic Validation: Intrinsic validation evaluates relevance methods by inspecting document heatmaps or compiling the most relevant words for categories.The quantitative comparison uses word deletion: a word is relevant when removing it strongly decreases the classification score.
4.2 Measuring Model Explanatory Power through Extrinsic Validation: Comparing different models requires a common evaluation basis because classifier-specific performance changes after word removal are not directly comparable.The proposed alternative depends only on word relevances and their semantic extractiveness.
4.2 Measuring Model Explanatory Power through Extrinsic Validation: The explanatory-power procedure computes document summary vectors using CNN or BoW/SVM relevances and the model’s predicted class as the decomposition target.Summary-vector generation is therefore unsupervised with respect to the target class.
4.2 Measuring Model Explanatory Power through Extrinsic Validation: The procedure normalizes summary vectors and applies uniformly weighted Euclidean KNN across random half-splits with different K values.KNN accuracy is averaged over 10 random splits for each K.
4.2 Measuring Model Explanatory Power through Extrinsic Validation: The explanatory power index is the maximum KNN accuracy across K, with higher values indicating greater explanatory power.CNN and BoW/SVM summary vectors occupy different spaces: word2vec semantic space for CNN representations and bag-of-words space for Eq. 8.

5 Results

The experiments show that LRP identifies relevant words, produces more meaningful semantic document representations than baselines, and explains CNN predictions better than BoW/SVM predictions.

LRP identifies relevant words in text documents through heatmaps and representative-word lists.The evaluation includes document-level and dataset-level analyses.
LRP-based document vectors group documents according to topics in a two-dimensional PCA projection without using true labels.
LRP explanations are more semantically meaningful than sensitivity-analysis, uniform, and TFIDF weighting baselines.
The CNN model produces better explanations than the BoW/SVM classifier despite similar classification performance.

5.1 Experimental Setup

The study evaluates topic categorization on the 20Newsgroups dataset using CNNs and a BoW/SVM classifier, with preprocessing that truncates neural-network inputs to 400 tokens. Classification performance is similar across model types.

Experimental Setup: The 20Newsgroups 20news-bydate dataset contains 11,314 training and 7,532 test documents across twenty evenly distributed categories.
Preprocessing: Documents are tokenized and filtered to retain alphabetic, hyphen, dot, and apostrophe characters, while punctuation, numbers, and dates are removed.
Preprocessing: Neural-network input sequences are truncated to a fixed length of 400 tokens, while the SVM additionally uses lowercasing.
Models: The experiments compare three CNNs with filter sizes 1, 2, and 3 against a BoW/SVM classifier.
Results: The linear SVM performs on par with the neural networks, so CNN nonlinearity provides no considerable classification-accuracy advantage on this dataset.

5.2 Identifying Relevant Words

LRP assigns word-level relevance scores that expose which words support or oppose a target class. CNN2 explanations are sparser and more semantically focused than SVM explanations, while embeddings extend relevance to unseen words.

Document-level analysis: CNN2 heatmaps concentrate relevance on semantically meaningful words, whereas SVM heatmaps can emphasize insignificant words and misleading negative contributions.
Document-level analysis: CNN2 assigns relevance per word position, while SVM relevance is tied to bag-of-words features and is therefore context-independent for repeated words.
Document-level analysis: Max-pooling produces sparse CNN explanations by selecting the first most relevant feature occurrence in each feature map.
Unseen words: Word2vec embeddings allow the neural network to assign relevance to unseen words, whereas SVM ignores words absent from its training vocabulary.
Dataset-level analysis: Dataset-wide LRP analysis ranks words by relevance to a target class and retrieves the thirty most relevant words as class representatives.

5.3 Document Summary Vectors

The paper constructs document summary vectors by weighting word representations with classifier-derived relevance scores. LRP weighting yields the clearest topic structure in PCA, outperforming uniform and TFIDF representations and producing more interpretable layouts than SVM-based alternatives.

Summary-vector construction: Document summary vectors are weighted combinations of word2vec or one-hot word vectors designed to transfer semantic regularities into document representations.
PCA evaluation: CNN2 LRP-weighted summary vectors form a clear-cut clustered structure in two-dimensional PCA, unlike uniform or TFIDF weighting.
PCA evaluation: The PCA visualizations color documents by true category, while LRP/SA weightings are computed using each model’s predicted class.
Model comparison: Element-wise LRP produces denser and better-separated bundles than SA weighting, while SVM LRP and SA produce partly cross-shaped layouts.
Interpretability: LRP-based semantic extraction is more meaningful than the baseline methods, and the neural network is more explainable than BoW/SVM.

5.4 Quantitative Evaluation

Deletion experiments show that LRP relevance scores identify words supporting or inhibiting CNN classification decisions, while LRP-weighted document vectors provide stronger semantic extraction than baselines and higher explanatory power for CNN2 than BoW/SVM.

How good does LRP identify relevant words ?: LRP identified pertinent words in all deletion experiments, producing the largest classification-accuracy changes when highly or weakly relevant words were removed.The experiments deleted words in relevance order, reclassified documents, and measured accuracy as words were removed.
How good does LRP identify relevant words ?: SA was largely unable to identify words opposing a classification decision, whereas its lowest relevances often marked words with no classifier impact.This contrasts with LRP’s ability to capture both positive- and negative-relevant words.
How good does LRP identify relevant words ?: CNN2 and CNN3 showed steeper performance declines than CNN1 after deleting the most relevant words, suggesting sensitivity to surrounding-word meaning and possible bigram or trigram use.The paper characterizes the latter interpretation as weak evidence and notes that CNN2 and CNN3 behave similarly.
Quantifying the Explanatory Power: LRP-weighted word2vec and one-hot document representations significantly outperformed corresponding TFIDF representations at significance level 0.05.The comparisons used a corrected resampled t-test.
Quantifying the Explanatory Power: CNN2 produced semantically more meaningful summary vectors and higher explanatory power than the BoW/SVM classifier, while element-wise LRP outperformed baseline methods.Explanatory power was evaluated using maximum mean KNN accuracy across varying neighbor counts and random data splits.
Quantifying the Explanatory Power: The LRP-weighted document vectors exhibited semantic neighborhood regularities in feature space, supporting their use as vector-based document representations.The authors connect this representation to possible aggregation of word representations into phrases, sentences, or paragraphs.

6 Conclusion

The paper demonstrates that LRP can identify words important to classifier decisions and produce document vectors with semantic properties. It presents this work as a first step toward applying LRP across NLP models and tasks.

Conclusion: LRP supports fine-grained document-level analysis and dataset-wide introspection of words important to classifier decisions.The resulting relevance information extends classification outputs with links between decisions and input words.
Conclusion: The resulting document vectors support a model explanatory-power measure and may serve visualization and search tasks through vector-based document similarity.The paper expresses similarity as a dot product between vectors.
Conclusion: The work is a first step toward applying LRP to character-based and recurrent classifiers, sentiment analysis, and hidden-layer relevance analysis.These are presented as expected areas of suitability or potential contribution rather than demonstrated evaluations in this work.

Contributions

The supplied contributions passage contains only author-contribution roles and does not state the paper’s research contributions.

Contributions: The authors divided responsibility across the theoretical framework, experiment design, experimentation, manuscript writing, revision, figure design, and final drafting.The passage states that final drafting was performed equally by all authors.