DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

1 Introduction

• Multimodal document models combine heterogeneous inputs, yet guidance for extending vision- and language-focused XAI methods and metrics to this setting is limited.The modalities include text, layout, and visual/image features.Source: §1 · p. 2
• Baseline selection, feature grouping, and attribution aggregation require modality-specific choices, but existing document-understanding XAI literature offers no principled guidelines.Examples include choosing text-token versus layout-coordinate baselines and aggregating text at token, word, or line level.Source: §1 · p. 2
• Many document tasks produce multiple predictions per input, whereas typical XAI methods and metrics assume single targets, making direct application infeasible or ill-suited.Semantic entity recognition independently classifies each token, producing numerous explanation targets per sample.Source: §1 · p. 2
• The paper frames four questions about multimodal and multi-target adaptation, method quality, metric agreement, and analyzing explanations for model insights.Its contributions also include multi-target adaptations, a metric meta-evaluation, and dataset-wide diagnostics of modality reliance, task-irrelevant features, and spatial attribution patterns.Source: §1 · p. 2
• DocXEval benchmarks 13 XAI methods and 17 metrics across 3 tasks, 8 datasets, and 4 models; Occlusion is most faithful and interpretable overall.Only a few methods perform consistently across all evaluation criteria.Source: §1 · p. 2

2 Related Work

• Multimodal document models combine modalities through differing fusion strategies, including early embedding fusion, cross-attention, and multimodal attention.LayoutLM fuses text and position embeddings and attends between image and text tokens; LiLT omits images, while DocFormer uses a multimodal attention layer.Source: §2 · p. 3
• Feature-attribution methods estimate feature relevance and include backpropagation-based approaches using gradients, modified gradient rules, or activation differences.Perturbation-based approaches instead perturb inputs and measure output impact; they are model-agnostic but generally add computational overhead.Source: §2 · p. 3
• Document-focused XAI includes feature-attribution methods for image classification, layout analysis, and hologram verification, but prior work remains focused on unimodal explanations and single-target scenarios.Examples include DocXplain for document image classification and gradient-based methods for layout analysis.Source: §2 · p. 3
• Multimodal XAI has focused primarily on vision-language tasks, while document-domain work includes visual-input-only DocVXQA for generative models.Prior multimodal studies often emphasize qualitative analysis or use only a small subset of evaluation metrics.Source: §2 · pp. 3–4

3 DocXEval: Our Framework

• DocXEval extracts modality-specific inputs, generates target-specific explanations, scores them with standardized metrics, and aggregates results for analysis across datasets, models, and tasks.The framework considers text, 1D sequence positions, 2D layout, and visual features.Source: §3 · pp. 4–5

3.1 Input and Preprocessing

• Token-level inputs are attributed in their embedding spaces to standardize feature units across models, while visual inputs are attributed over raw pixels.Embedding-level attribution provides common dimensionality across token modalities and a uniform representation for gradient- and perturbation-based methods.Source: §3.1 · p. 5

3.2 Feature Segmentation and Aggregation

• Feature segmentation groups discrete inputs by word and visual inputs by image patch, enabling more interpretable and comparable explanations.Attribution scores are summed within groups and normalized by group size so larger groups do not gain importance merely from having more features.Source: §3.2 · pp. 5–6

3.3 Baseline Selection

• A multimodal extension of Replace selects per-modality baselines by measuring model sensitivity across baseline combinations and perturbation levels, then choosing the least-sensitive strategy for each modality.For LayoutLMv3, image sensitivity was generally lower with Mean, spatial sensitivity with Zero Embedding than [PAD], and text sensitivity with [MASK].Source: §3.3 · pp. 6–7, Figure 2 · p. 7

3.4 Attribution Targets

• Attribution targets are the predicted-class probability for classification, each word’s predicted entity-label probability for SER, and predicted start- and end-position probabilities for QA.SER generates one target per word, while QA uses separate start and end targets.Source: §3.4 · p. 7

3.5 XAI Method Adaptations

• For multi-target backpropagation, DocXEval computes the Jacobian across targets after one forward pass; perturbation methods accumulate scores for all targets during each perturbed pass.LIME and KernelSHAP create one neighborhood dataset and fit separate surrogate models per target; attention attributions are limited to modalities represented as separate model tokens.Source: §3.5 · p. 8

3.6 XAI Metric Adaptations

• Metrics evaluate aggregated feature groups jointly across modalities, with Completeness using raw attributions and Infidelity adding noise independently to each modality.Unordered or random perturbation metrics can accumulate effects across targets in one pass, whereas ordered-perturbation metrics cannot use that multi-target adaptation.Source: §3.6 · p. 9

4 Experimental Setup

• The benchmark covers three classification datasets, four SER datasets, and one QA dataset across eight datasets.Classification uses RVL-CDIP, Tobacco3482, and DocLayNet; SER uses FUNSD, CORD, SROIE, and WildReceipts; QA uses DocVQA.Source: §4 · p. 9
• Four models span text-only BERT and RoBERTa, multimodal LiLT, and fully multimodal LayoutLMv3.The selected models vary in their modality fusion strategies and inputs.Source: §4 · p. 9
• Training used 40 epochs for classification, 50 for DocVQA, and 100 for SER, with early stopping based on validation performance.Source: §4 · p. 9
• The study adapts 13 methods across backpropagation, perturbation, and attention families, excluding Attention Flow for its high computational cost on large sequences.The families include seven backpropagation-based, four perturbation-based, and two attention-based methods.Source: §4 · p. 11
• Sampling or interpolation methods use 200 samples or steps, DeepSHAP uses 100 training-set reference samples, and Occlusion uses modality-specific windows and strides.Occlusion windows are 8 tokens with stride 4 for text, position, and layout, and 3 × 16 × 16 with stride 3 × 8 × 8 for images.Source: §4 · p. 11
• The 17 evaluation metrics cover axiomatic soundness, faithfulness, complexity, and robustness, with specified perturbation levels and shared modality-wise masks and baselines.The evaluation also uses 1% feature-group chunks where applicable and test-set subsets, with experiments run on NVIDIA H200 and RTX A6000 GPUs.Source: §4 · p. 11

5.1 Rank Analysis of XAI Methods

• OCC, RA, and IG are the most consistently strong methods overall, although many method-rank differences are statistically indistinguishable and mon and infid are unreliable in this setting.Basic backpropagation methods rank low across most metrics; mon often returns 0, while infid scores are nearly identical across methods except RA and AR.Source: §5.1 · p. 12, Figure 4 · p. 13

5.2 Metric Agreement Analysis

• Faithfulness metrics show moderate-to-strong agreement in several groups, but infid and mon correlate with none; complexity metrics s-comp, spar, and ent-comp correlate moderately, unlike eff-comp.faith-corr and sens-n rankings are strongly correlated across perturbation levels; infid and mon also show statistical indistinguishability and unreliable behavior in the rank analysis.Source: §5.1 · p. 12, §5.2 · p. 13

5.3 Computational Efficiency of Multi-Target Adaptations

• Multi-target adaptations make unordered SER metrics 176–229× faster than ordered metrics, while ordered metrics remain unadaptable and take about 2,287ms per sample-target.For XAI methods, OCC speedups roughly track targets per sample (about 25× on CORD and up to 189× on FUNSD); LIME reaches 7×–17× and backpropagation methods 1.5×–1.7×.Source: §5.3 · p. 14, Figure 8 · p. 15, Figure 7 · p. 15

5.4 Qualitative Analysis

• In sampled explanations, OCC and RA often emphasize target-neighborhood text or layout, whereas IG can spread importance broadly and produce counteracting attributions.Classification examples show image reliance for DocLayNet and method-specific modality patterns for RVL-CDIP; SER examples show OCC and RA focusing locally while IG can attend far from targets. For QA, OCC highlights answer-neighborhood text and position, while RA emphasizes the answer token and some image regions; IG highlights question text and aligned tokens.Source: §5.4 · pp. 14, 15, 19

5.5.1 Modality Contribution Analysis

• Modality contributions vary by dataset and explainer: RA often emphasizes images, while OCC and IG show more mixed or text-focused patterns across the reported examples.On RVL-CDIP, RA indicates up to 80% image contribution while IG favors text; on DocLayNet, RA relies almost exclusively on images, and OCC varies by class. OCC and IG indicate negligible image importance on CORD and FUNSD, while RA shows image reliance for some CORD entity types and DocVQA start tokens.Source: §5.5.1 · pp. 19–20

5.5.2 Spatial Attribution Locality Analysis

• On SROIE and CORD, OCC and RA show greater horizontal than vertical locality, while IG has higher spread; WildReceipts examples show RA concentrated near target words.OCC also shows high horizontal locality for SROIE ADDRESS, COMPANY, and DATE entities.Source: §5.5.2 · p. 22

5.5.3 Top-k Word Frequency Analysis

• Stop words dominate top-attributed words across tasks, and Tobacco, Cigarette, and Healthy appearing across unrelated classes suggest reliance on dataset-specific artifacts.Some attributed terms align with labels, including Mr., Dear, and Dr. for letters, University, Department, and Vol. for scientific publications, and Contract for government tenders.Source: §5.5.3 · pp. 22–23
• For December and 31, high-frequency top-k words with no direct class relevance appear alongside Company and Financial; manuals attributions include PDF, IVM, and AFP, whose relevance is uncertain.The passage notes that the manuals abbreviations could reflect either class-relevant patterns or spurious dataset artifacts.Source: §5.5.3 · p. 23
• On FUNSD, FAX, DATE, Account, and Facsimile recur across entity types; on CORD, PCS, TOTAL, and SUBTOTAL dominate, while frequent numbers may reflect training-data prevalence; DocVQA attributions are largely stop words.For DocVQA, source and number also appear frequently among top-k attributions for start and end answer tokens.Source: §5.5.3 · p. 24

6.1 Key Findings

• OCC stood out as the most reliable method overall, while RA’s modality coverage is architecture-limited and IG tends to produce complex attribution maps.OCC is costly, though shared perturbations reduce its multi-target overhead; completeness remains a trade-off because IG satisfies it but is difficult to interpret, while KS performed considerably worse on other criteria.Source: §6.1 · p. 24
• Several faithfulness and complexity metrics agreed moderately to strongly, but Monotonicity and Infidelity were unreliable and some methods were statistically indistinguishable.The authors recommend comparing methods across multiple datasets and models; eff-comp had negligible correlation with other complexity metrics.Source: §6.1 · pp. 24–25
• Modality reliance varied across tasks and labels, while stop-word dominance and class-associated terms suggested potential dataset biases requiring deeper inspection.OCC indicated stronger image reliance in classification and text and layout contributions in SER; stop words in CLS may reflect frequency bias.Source: §6.1 · p. 25
• On SROIE, OCC focused on horizontal neighborhoods for ADDRESS, DATE, and COMPANY, whereas RA often focused only on the target token.Unexpected focus on distant, irrelevant tokens could signal a potential spurious correlation warranting investigation.Source: §6.1 · p. 25
• The three top-ranking methods showed limited attribution consensus, so reliable dataset-wide insights call for methods with established faithfulness; the authors identify a need for domain-specific methods.OCC consistently ranked among the top methods for faithfulness and is described as a reasonable choice.Source: §6.1 · pp. 25–26

6.2 Limitations

• The evaluation covered four Transformer models, only two document-specific multimodal models, and just one model processing all four modalities; findings may not generalize to other model families.The authors cite unavailable open-source architectures and pretrained weights as a primary constraint.Source: §6.2 · p. 26

6.3 Broader Impact

• The released code and unified pipeline support consistent evaluation, and the task-agnostic adaptations can be applied beyond multimodal document understanding.The framework is presented as a benchmark and practical guide for future explainability research.Source: §6.3 · p. 26

6.4 Future Work

• Future work includes hierarchical Shapley decomposition, cross-modal interaction attribution, counterfactual explanations, and domain-specific diagnostics and visualizations.The authors note that current attribution methods reveal which features models rely on, but not why.Source: §6.4 · p. 26

7 Conclusion

• DocXEval evaluated 13 XAI methods across 3 tasks, 8 datasets, 4 models, and 17 metrics; OCC emerged as most faithful and interpretable overall, while Monotonicity and Infidelity were unreliable.The analyses also found variable modality reliance, spatial attribution patterns, and potential dataset biases.Source: §7 · pp. 26–27

A.1 Axiomatic Metrics

• Completeness measures whether raw feature attributions sum to the model-output difference between input and baseline; its multi-target adaptation computes that difference across targets in one step.A score of zero indicates perfect completeness, and IG satisfies the property by construction.Source: §A.1 · p. 33

Appendix A Metric Descriptions and Adaptations

• For non-sensitivity, the adaptation uses group-aggregated scores and relative output variance under feature-group perturbations, with chunking and simultaneous computation across targets.With a fixed reference baseline, each perturbation is deterministic and N = 1; scores are normalized by feature count.Source: §A.1 · pp. 33–34

A.2 Complexity Metrics

• Complexity metrics use group-aggregated attribution scores; effective complexity estimates the proportion of top-attributed features needed to influence output, with lower values indicating concentration on fewer features.Effective complexity is computed independently for each sample-target pair because its ordered perturbation procedure does not support multi-target adaptation.Source: §A.2 · p. 34

A.3 Faithfulness Metrics

• AOPC measures average output change as features are removed by attribution rank; higher MoRF scores indicate more faithful explanations.Features are removed using modality-wise baselines in equal-percentage chunks; MoRF, LeRF, and random orderings are evaluated, but target-specific ordering prevents efficient multi-target computation.Source: §A.3 · p. 35
• ABPC measures the gap between descending and ascending perturbation curves; larger values indicate that removing highly attributed features has greater impact.The implementation derives ABPC from AOPC and reports it at full feature removal.Source: §A.3 · pp. 35–36
• Sufficiency evaluates whether top-attributed features retain the prediction by adding them back from a baseline; lower scores indicate better faithfulness.Multi-target adaptation is not possible for ordered perturbation metrics such as Monotonicity and Faithfulness Estimation, so they are computed per sample-target pair.Source: §A.3 · pp. 36–37
• Infidelity measures attribution-impact discrepancy under perturbation; its adaptation applies Gaussian noise directly to each modality and handles all targets per perturbation.Lower values indicate more faithful attributions.Source: §A.3 · p. 36
• Sensitivity-N compares summed group attributions with output changes from removing randomly sampled feature groups.It is evaluated at 20%, 40%, and 60% group perturbation levels, with output changes accumulated across targets simultaneously.Source: §A.3 · p. 36

A.4 Robustness Metrics

• Max and average sensitivity measure explanation changes under input perturbations; lower values indicate more robust explanations.The implementation samples modality-specific uniform noise, and multi-target explanation mode is used to recompute explanations.Source: §A.4 · pp. 37–38

B.1 Training Hyperparameters

• Models use Adam or Adam-W with linear warmup, validation-based early stopping at patience 10, and dropout 0.1.LayoutLMv3 input images are resized to 224 × 224.Source: §B.1 · p. 39

B.2 Explainer Hyperparameters

• XAI method hyperparameters are shared across tasks; sampling or interpolation methods use 200 samples or steps, while DeepSHAP uses 100 training-set baseline samples.Internal computation batch size and multi-target gradient batch size are configured to fit available GPU memory; multi-target implementations are extended and released in torchxai.Source: §B.2 · pp. 39–40

B.3 Metric Hyperparameters

• Perturbation metrics use 1% chunks where supported, and feature grouping and removal share the attribution segmentation masks and modality-wise baselines.Max Sensitivity and Infidelity use Captum implementations extended for multi-target scenarios; remaining metrics and adaptations are available in torchxai.Source: §B.3 · p. 40

C.1 Per-Task XAI Method Rankings

• On SER, Monotonicity and Infidelity show broad statistical indistinguishability; on CLS, Monotonicity and Sparsity do likewise, suggesting these metrics provide no meaningful evaluation results there.QA reports rankings only because the limited observations prevent pairwise significance testing.Source: §C.1 · p. 40, Figure C1 · p. 42
• Per-task mean-rank patterns broadly match the overall rankings, with minor task-specific variations.Lower mean ranks indicate better performance.Source: §C.1 · p. 40, Figure C2 · p. 42

C.2 Per-Task Metric Agreement

• Task-wise faithfulness and complexity correlation patterns were largely consistent with the aggregate results in Fig. 5.Source: §C.2 · p. 51

C.3 AOPC Curves

• AOPC curves cover all XAI methods across dataset-model combinations for CLS, SER, and QA, with higher MoRF and lower LeRF curves indicating better faithfulness.Both curve types are plotted relative to a random baseline.Source: §C.3 · p. 51

• The complete results cover 17 metrics, 13 XAI methods, 8 datasets, and 4 models, including per-dataset mean scores used for rankings and correlation analyses.Source: §C.4 · p. 51
• Table C5 reports modality contribution fractions for each model across all SER datasets.Source: Table C5 · p. 52
• Table C6 reports modality contribution fractions for each model on DocVQA.Source: Table C6 · p. 52

C.5 Additional Qualitative Results

• Additional qualitative results for LayoutLMv3 and LiLT use OCC, IG, and RA explanations.Source: §C.5 · p. 53

C.6 Dataset-wise Modality Contributions

• Tables C4–C6 report modality contributions for LayoutLMv3 and LiLT across CLS, SER, and QA, computed with OCC, IG, and RA.Source: §C.6 · p. 53