PaperlayerView exact revision on arXiv

Source-linked AI summary

When Explanations Cannot Be Read: Measuring and Correcting SHAP and LIME Rendering for Right-to-Left Languages

Rameesha Zia, Muhammad Shahid Iqbal Malik

arXiv:2609.28565v1cs.LGcs.AIcs.CL

TL;DR

RTL labels in SHAP and LIME plots can be unreadable even when the underlying attributions remain valid. The paper introduces a rendering layer that applies complex text layout and per-language font selection without changing those attributions. Its results show that the common workaround fails for Urdu and degrades under Matplotlib 3.11.0, while SHAP-RTL remains correct under both versions.

  • Problem

    SHAP and LIME visualizations assume left-to-right reading order, so RTL text can be misordered or have broken letterforms even though its underlying attributions remain valid.

    Source: §1 · p. 3, §3 · p. 7

  • Method

    SHAP-RTL applies complex text layout and per-language font selection at the label level while preserving attribution values, feature ordering, and model outputs.

    Source: §1 · p. 3

  • Results

    0.998 CER: the reshape-and-reorder workaround fails for Urdu, and SHAP-RTL returns identical scores under both Matplotlib versions in all four languages.

    Source: §1 · p. 3, §8 · p. 25

  • Takeaways & Limitations

    A dedicated label-rendering layer avoids dependence on the plotting library’s text engine and keeps rendering correctness stable across its two tested versions.

    Source: §8 · p. 25

    Layout asymmetry remains unresolved, and OCR measures machine recoverability rather than native-speaker readability.

    Source: §8 · p. 25

Abstract

from arXiv · show

Post hoc explanation methods such as SHAP and LIME are widely used to interpret text classifiers, but their visualizations are mainly designed for left-to-right languages. When applied to right-to-left (RTL) languages such as Urdu, Arabic, Persian, and Hebrew, the attribution values remain mathematically valid, while their visual presentation fails. Tokens appear out of sequence, connected letterforms break apart, and plot layouts do not follow the natural reading direction. This study addresses this gap as a visualization problem rather than a limitation of the explanation methods themselves. We present SHAP-RTL, a rendering layer that corrects reading direction and script shaping in SHAP and LIME visualizations, with per-language font selection, while preserving the original attribution values, feature ordering, and model outputs. The approach is evaluated on Urdu, Arabic, Hebrew, and Persian hate and offensive-language datasets using TF-IDF and logistic regression classifiers. Rendering correctness is measured by an OCR round trip over 200 feature words per language. Default rendering yields character error rates of 0.820 to 0.979, meaning the label no longer carries its token; the common reshape-and-reorder workaround fails for Urdu at 0.998, worse than no correction; and the Matplotlib 3.11.0 text rewrite inverts that workaround, while SHAP-RTL remains correct under both versions. The framework also verbalizes the same attributions as short contextual explanations in the reader's language, constrained to the identified features. Evaluation in this paper concerns rendering correctness; assessment of the generated explanations is left to future work. The study highlights the importance of language-aware visualization in making post hoc explainability more accessible across different writing systems.

1. Introduction

SHAP and LIME visualizations can misrender right-to-left text even when their underlying attributions remain valid. The paper treats this as a rendering problem and proposes a label-level correction, measuring its correctness across four languages.

1. Introduction

  • SHAP and LIME assume left-to-right visual order, so RTL labels can appear out of sequence and cursive letters can break apart despite valid attributions.The paper frames this as a human readability problem that has received little attention in explainability research.Source: §1 · p. 3
  • The proposed SHAP and LIME rendering layer applies complex text layout and language-specific fonts while preserving attribution values, feature ordering, and model outputs.The paper also provides contextual verbalizations of attributions, but does not evaluate them.Source: §1 · pp. 3–4
  • The study covers Urdu, Arabic, Persian, and Hebrew hate and offensive-language datasets, with Urdu as the primary case because Nastaliq shaping is less represented in NLP tooling.The paper identifies the Matplotlib text rewrite as a recent change that does not resolve font selection or layout geometry.Source: §1 · p. 3
  • The paper categorizes RTL plot failures as incorrect direction, shaping failure, and uneven layout, examining them across four languages and eight plot families.It introduces a quantitative measure of rendering correctness and evaluates it with an OCR round trip over 200 feature words per language.Source: §1 · p. 3
  • 0.998 CER: the common reshape-and-reorder workaround fails for Urdu, while Matplotlib 3.11.0 inverts that workaround across all four languages.The paper also reports that the 3.11.0 rewrite repairs default rendering for Arabic and Persian.Source: §1 · pp. 3–4

2. Post hoc XAI methods

Post hoc methods explain predictions after training without changing the model, and SHAP and LIME assign importance to input features. Their RTL readability problem arises in the shared plotting path, not in their attribution computations.

2. Post hoc XAI methods

  • SHAP and LIME are post hoc methods that explain predictions from a fixed trained model, with SHAP assigning feature contributions and LIME fitting a local surrogate.SHAP uses token masking and output changes; LIME perturbs one instance and fits a weighted linear model in its neighborhood.Source: §2 · p. 4, §2.1 · p. 5, §2.2 · p. 6

2.1 SHAP

  • SHAP defines feature attribution through Shapley values, averaging a feature’s marginal effect across subsets of other features.Exact computation quickly becomes intractable for inputs with more than a few tokens, so practical systems use approximations.Source: §2.1 · p. 5

2.2 LIME

  • LIME explains one prediction at a time by perturbing an instance and fitting a distance-weighted local linear model whose coefficients indicate feature influence.For text, it typically deletes words and measures changes in classifier confidence.Source: §2.2 · p. 6

2.3 Locating the directional assumption

  • The LTR assumption belongs to the plotting path: SHAP and LIME pass label strings and attribution values to a general-purpose library that determines text order and shaping.This shared rendering path explains why changing attribution computation is not the correction point.Source: §2.3 · p. 6

3. Problem Analysis

RTL rendering failures affect SHAP and LIME across languages and plot types even though their underlying attributions remain valid. The failures involve direction, shaping, and layout, while the correction and reported behavior depend on the rendering path and library version.

3. Problem Analysis

  • Default SHAP and LIME plots show RTL text out of sequence or broken while attribution values and local surrogate coefficients remain mathematically valid.The issue is presentation rather than computation, and similar failures occur across both methods and multiple RTL languages.Source: §3 · pp. 6–7, §3.2 · pp. 7–8

3.1 Three failure mechanisms

  • The three failure mechanisms are directional inversion, shaping failure in connected scripts, and layout asymmetry that retains LTR plot geometry.Table 1 marks each mechanism’s presence by plot family and language group; the rendering layer addresses direction and shaping, not layout asymmetry.Source: §3.1 · p. 7, Table 1 · p. 8

3.2 Observed misalignment across the four languages

  • Urdu, Arabic, Persian, and Hebrew show related failures across SHAP and LIME, with Urdu’s Nastaliq shaping making its labels especially difficult to read.Hebrew mainly exhibits direction errors, while Urdu can exhibit both direction and shaping errors; Urdu is identified as the hardest of the four to render correctly.Source: §3.2 · p. 8

3.3 Scale of the affected population

  • Arabic, Urdu, and Persian together account for well over half a billion speakers, while Hebrew supplies a control case with non-connecting letters and established NLP resources.The study centers on the first three for their speaker scale and includes Hebrew for its distinct orthography and annotated dataset.Source: §3.3 · pp. 9–10, Table 2 · p. 9

3.4 Version scope of the reported defect

  • Matplotlib 3.10.8 and earlier exhibit the reported default failures; Matplotlib 3.11.0 improves direct text handling but still lacks per-language font selection.Under 3.11.0, the common reshape-and-reorder workaround becomes harmful because its presentation forms conflict with the new shaping path.Source: §3.4 · p. 10

4. Problem statement and research questions

The paper frames RTL explainability as two distinct problems: unreadable visualizations despite valid attributions, and feature scores that do not explain why words matter in context. Its research questions ask how to correct rendering and verbalize attributions in the reader’s language.

Problem statement

  • SHAP and LIME can produce valid RTL attributions whose labels are misaligned, whose letters separate, and whose layouts disregard reading direction.The stated solution target is the visualization layer, not the attribution values or model.Source: §4 · p. 10
  • Even a readable plot shows important words and scores but does not explain why those words mattered in the specific post or how they form a pattern relevant to the classifier.The paper identifies comprehension in the reader’s own context and language as a particular concern.Source: §4 · p. 10

Research questions

  • The research questions ask how to render SHAP and LIME consistently with RTL direction without changing attributions, and how to verbalize them in the reader’s language while staying feature-constrained.Source: §4 · p. 11

5. Proposed solution

SHAP-RTL modifies the rendering stage for RTL labels while preserving the underlying explanation outputs, and adds a language-model stage that verbalizes the same attributions. Its rendering paths accommodate script and font differences, while layout asymmetry remains outside the implemented correction.

Rendering layer

  • SHAP-RTL modifies SHAP and LIME rendering while leaving the model, tokenizer, explainer, attribution values, feature ordering, and clustering unchanged.RTL labels are detected automatically, and the original plot interfaces are retained.Source: §5 · p. 11

Rendering implementation

  • For plot labels, HarfBuzz shapes and orders text using the selected font and FreeType rasterizes it as an image placed at the original label position.The image-based labels cannot be searched or read by screen readers; PIL fallback does not join Arabic-script letters and triggers a warning.Source: §5 · p. 11
  • The label-level correction avoids replacing the plotting backend, which would affect the whole figure and would not itself provide font selection or geometry correction.The alternative mplcairo route depends on optional Raqm support absent from standard binary distributions.Source: §5 · p. 12

Language-specific font selection

  • Fonts are selected by language rather than Unicode script alone, with Nastaliq fonts for Urdu and distinct Arabic, Persian, and Hebrew choices.Urdu, Arabic, and Persian share the Arabic Unicode block but have different typographic requirements.Source: §5 · p. 12

Contextual explanations

  • The language-model stage receives source text, prediction, ordered features, and signed attributions, then produces a short target-language explanation constrained not to add or rerank features.Generated text is presented as a user-friendly interpretation of SHAP results, not independent evidence about the classifier.Source: §5 · p. 12

6. Case study and experimental setup

The study tests RTL rendering for SHAP and LIME on four language datasets, using a controlled TF-IDF/logistic-regression setup and an OCR round trip to evaluate whether rendered feature labels remain readable.

6.1 Datasets

  • The four corpora span two scripts, two language families, and three annotation schemes, with Hebrew serving as a non-joining control for separating direction and layout issues from shaping.The datasets cover Urdu, Arabic, Hebrew, and Persian hate or offensive-language classification tasks.Source: §6.1 · pp. 13–14

6.2 Classifier and experimental configuration

  • TF-IDF word unigrams and bigrams feed balanced-weight logistic regression, with an 80:20 stratified split and seed 42; the linear model keeps feature-to-token correspondence direct.SHAP and LIME explanations are generated for up to 300 correctly and incorrectly classified instances; rendering comparisons preserve the underlying values.Source: §6.2 · p. 14

6.3 Classifier performance

  • All four classifiers exceed their majority-class baselines on test accuracy and Macro F1.Urdu reaches 0.629 accuracy versus a 0.483 majority baseline; Arabic, Persian, and Hebrew reach 0.801, 0.738, and 0.684 accuracy, respectively.Source: §6.3 · pp. 14–15, Table 4 · p. 14

6.4 Evaluation protocol for rendering correctness

  • Rendering correctness is measured by OCR character error rate (CER), the normalized edit distance between each source feature and its OCR-recovered label.The sample contains 200 feature words per language, stratified by length and including bigrams; both Matplotlib versions are tested with fonts and OCR settings held constant.Source: §6.4 · p. 15
  • Four rendering conditions are compared: default output, the reshape-and-reorder workaround, an independent HarfBuzz/FreeType reference pipeline, and SHAP-RTL.A larger-scale same-pipeline render provides an OCR reference for language-specific recognition limits.Source: §6.4 · p. 15

7. Results

Default SHAP and LIME plots make RTL feature labels difficult or impossible to recover, while SHAP-RTL corrects label rendering across the evaluated languages and plotting-library versions. Results also show plot-specific geometry limits and demonstrate contextual explanations generated from the same attribution values.

7.1 Rendering correctness

  • CER ranges from 0.820 to 0.979 under default rendering, while SHAP-RTL reaches the OCR reference in every language and matches the independent reference within 0.010 CER in every cell.The evaluation uses 200 feature words per language under Matplotlib 3.10.8; residual OCR error for Urdu and Hebrew reflects recognition limits rather than rendering.Source: §7.1 · p. 16, Table 5 · p. 16
  • The reshape-and-reorder workaround scores 0.998 CER for Urdu, marginally worse than no correction, because substituted presentation-form characters are missing from Noto Nastaliq Urdu.The paired-bootstrap difference against full shaping is 0.646 with p < 0.001.Source: §7.1 · p. 16

7.1.1 Behavior across library versions

  • Under Matplotlib 3.11.0, the text rewrite improves default rendering, but the pre-reordering workaround degrades across languages; SHAP-RTL returns identical scores under both versions.The second bidirectional pass reorders text that the workaround has already placed in visual order.Source: §7.1.1 · pp. 16–17, Table 6 · p. 17

7.2 Addressing right-to-left content in the SHAP method

  • Corrected SHAP beeswarm plots restore Urdu and Arabic label shaping and order, while the Urdu–Hebrew bar-plot comparison separates Urdu’s shaping and direction issues from Hebrew’s direction issue.The corrected beeswarm labels remain aligned with their dense point rows.Source: Figure 3 · p. 17, §7.2 · p. 18, Figure 4 · p. 18
  • Corrected waterfall labels make feature contributions easier to follow, but the force plot retains a left-to-right geometry despite rendering its labels correctly.The force plot’s horizontal flow runs from the base value toward the final prediction.Source: §7.2 · pp. 18–19, Figure 5 · p. 19, Figure 6 · p. 19
  • Corrected Arabic decision plots avoid visible label collisions, and corrected heatmaps keep feature names readable in their row positions across instances.Together, the examples cover dense decision plots and global heatmap explanations.Source: §7.2 · pp. 19–20, Figure 7 · p. 20, Figure 8 · p. 20

7.3 Addressing right-to-left content in the LIME method

  • LIME shows the same default RTL ordering and shaping problems as SHAP for Urdu and Arabic, and the rendering layer corrects the labels in both cases.The Arabic example uses LIME’s local linear model approach, supporting the paper’s account of a visualization-layer issue.Source: §7.3 · pp. 20–21, Figure 9 · p. 21, Figure 10 · p. 21

7.4 Contextual explanation stage

  • The contextual explanation stage returns short target-language accounts of which terms drive classification and how they function in context, using the same attribution values for target-language and optional English outputs.The Urdu example illustrates contextual information not captured by attribution values alone.Source: §7.4 · pp. 22–23

8. Conclusion

The rendering layer brings RTL attribution labels to the OCR reference level, while a common reshaping workaround fails for Urdu and changes behavior under Matplotlib 3.11.0. The conclusion also identifies unresolved layout, comprehension, and explanation-faithfulness limits, alongside ethical and availability considerations.

8. Conclusion

  • Under Matplotlib 3.10.8, default-output CERs were 0.820 for Arabic, 0.838 for Persian, 0.842 for Hebrew, and 0.979 for Urdu; the rendering layer reached the OCR reference level in all four.The evaluation used an OCR round trip over 200 feature words per language.Source: §8 · p. 24
  • The reshape-and-reorder workaround fails for Urdu at 0.998 and degrades across all four languages under Matplotlib 3.11.0, while the rendering layer scores identically under both versions.The package rasterizes labels outside the plotting library's text engine.Source: §8 · p. 25
  • Plot geometry remains left-to-right, and native-speaker research is needed to establish whether OCR-level recoverability improves comprehension.Word-level tokenization is a precondition because Arabic-script subword fragments have no valid isolated form; contextual-explanation faithfulness remains unevaluated.Source: §8 · pp. 25–26

Ethics Statement

  • Figures reproduce offensive terms from the corpora to demonstrate label legibility, and readers are warned before viewing them.Source: Ethics Statement · p. 26
  • The contextual explanation stage sends source text and attributions to a third-party language-model API, so deployed use with personal data requires local hosting or an agreement covering the transfer.The rendering layer works without this optional stage.Source: Ethics Statement · p. 27
  • Fluent generated explanations may sound authoritative without being faithful, so the paper frames them as presentations of supplied attributions rather than evidence about the classifier.The stage is instructed to describe the supplied attribution set rather than reason independently.Source: Ethics Statement · p. 27

Funding

Data and code availability

  • The package is available; its SHAP/LIME derivative code retains the MIT License with attribution, and distributed fonts use the SIL Open Font License.The benchmark datasets remain subject to their authors' stated terms.Source: Data and code availability · p. 27

Related papers

Loading 2609.28565v1…