Source-linked AI summary
Explainable Software Analytics
Hoa Khanh Dam, Truyen Tran, Aditya Ghose
TL;DR
Software analytics often favors predictive accuracy, yet complex models can be hard for practitioners to understand, a concern associated with trust and adoption. The paper argues for explainability alongside accuracy and proposes research directions for models and evaluation. It concludes that explainability can support understanding and trust, while other trust dimensions also warrant attention.
Problem
Method
Results
The paper argues that explainability should be as important as prediction accuracy and an important evaluation measure for software analytics models.
Source: §1 · p. 1Takeaways & Limitations
Explainability can facilitate practitioners’ understanding of predictions and trust, while reproducibility, repeatability, and awareness of limitations also merit attention.
Source: §6 · p. 4The paper notes that explainability does not cover all trust dimensions; reproducibility and repeatability also deserve attention.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Software analytics has been the subject of considerable recent attention but is yet to receive significant industry traction. One of the key reasons is that software practitioners are reluctant to trust predictions produced by the analytics machinery without understanding the rationale for those predictions. While complex models such as deep learning and ensemble methods improve predictive performance, they have limited explainability. In this paper, we argue that making software analytics models explainable to software practitioners is as \emph{important} as achieving accurate predictions. Explainability should therefore be a key measure for evaluating software analytics models. We envision that explainability will be a key driver for developing software analytics models that are useful in practice. We outline a research roadmap for this space, building on social science, explainable artificial intelligence and software engineering.
1 INTRODUCTION
Software analytics commonly prioritizes prediction accuracy, while complex models can be difficult for practitioners to understand. The paper argues that explainability matters alongside accuracy and outlines research questions for making it central to software analytics.
1 INTRODUCTION
- Accuracy dominates software analytics evaluation, while powerful models such as SVMs, ensembles, and deep neural networks are difficult for practitioners to interpret.The field applies machine learning to tasks including defect prediction, effort estimation, API recommendation, and risk prediction.Source: §1 · p. 1
- Lack of explainability can undermine trust and adoption; developers need a trace and justification for predictions to act appropriately, such as fixing a flagged defect.High test performance may not hold on unknown future data, and unexpected predictions make trust especially important.Source: §1 · p. 1
- The paper argues that explainability should matter as much as accuracy and should be an important measure for evaluating software analytics models.It envisions explainability as a driver of models that deliver practical value.Source: §1 · p. 1
- The proposed research questions concern good explanations, building explainable models and generating explanations, and evaluating explainability.The paper calls for an interdisciplinary approach drawing on social sciences, AI, and software engineering.Source: §1 · pp. 1–2
2 WHAT IS A GOOD EXPLANATION IN SOFTWARE ENGINEERING?
The section defines explainability in terms of human understanding and distinguishes transparency of a model from explanations of individual decisions. It frames useful explanations as potentially causal, contrastive, and plural according to software engineers’ needs.
2 WHAT IS A GOOD EXPLANATION IN SOFTWARE ENGINEERING?
- Explainability measures how well a human observer understands the reasons behind a model decision; it can be global transparency or local, post-hoc explanation of a specific prediction [18].Global explainability concerns the whole decision process; local explainability concerns an individual decision.Source: §2 · p. 2
- Explanations can address a plain fact or contrast a prediction with a different property, object, or time.The four forms ask why an object has a property, rather than another, why two objects differ, or why its classification changed over time.Source: §2 · p. 2
- A causal explanation may be difficult because machine-learning algorithms often provide correlations, while contrastive explanations can focus on differences between cases.The section adopts pragmatic explanatory pluralism: acceptable explanation types vary with engineers’ interests, expertise, and motivation.Source: §2 · p. 2
3 EXPLAINABLE SOFTWARE ANALYTICS MODELS
Explainability can come from transparent models and components, but simplicity alone does not guarantee interpretability. The section calls for addressing the trade-off between explainability and predictive accuracy, including through multi-objective approaches and further work on black-box models.
3 EXPLAINABLE SOFTWARE ANALYTICS MODELS
- Transparency can be provided at the level of the whole model, its components, or its learning algorithm [18].In this approach, the model itself explains how a decision is made.Source: §3 · p. 2
- Decision trees, rules, and linear regression tend to be more transparent than deep neural networks, SVMs, and ensembles, but the figure frames accuracy against explainability.Trees expose decision structure and feature relevance; rules express predictions as IF–THEN statements.Source: §3 · p. 2, Figure 1 · p. 2
- Model components also need intuitive explanations: features should be meaningful to users, while anonymized or highly engineered features can be difficult to interpret.Linear-model parameters may represent strengths of association between features and outcomes.Source: §3 · pp. 2–3
- Simple linear models and large decision trees are not always interpretable, and feature selection or preprocessing can change the direction of a feature–outcome association.The association between normalized cyclomatic complexity and defectiveness may be positive or negative depending on the included features.Source: §3 · p. 3
- The paper calls for treating explainability and accuracy as competing objectives, with Pareto-based approaches as one way to address the trade-off.An evolutionary search for issue-resolution-time prediction optimized accuracy while minimizing model complexity.Source: §3 · p. 3
- Further directions include tuning simple models for better prediction and making black-box models more explainable; evolutionary search has improved SVM performance over deep learning for Stack Overflow question-link prediction [9].The section presents these as complementary research directions.Source: §3 · p. 3
4 EXPLANATIONS FOR DEEP MODELS
Deep networks’ architectural flexibility makes their decisions difficult to understand, motivating methods that inspect their internals, design explanation into architectures, or use external explainers. The section describes these approaches and cautions that post-hoc explanations may not reflect a model’s true operation.
4 EXPLANATIONS FOR DEEP MODELS
4.1 Seeing through the black-box
- Visualizing neuron activations and weights can reveal how a network works internally and help estimate feature importance [4] [26] [15].Examples include token embeddings, CNN feature maps, and RNN activation patterns.Source: §4.1 · p. 3
4.2 Designing explainable architectures
- Explainable architectures can use attention to show which components contribute to an outcome, or pair predictions with natural-language explanations [3] [11].Linguistic explanations may be post-hoc and need not reflect the model’s true inner workings; structured rules, constraints, or knowledge graphs are another option.Source: §4.2 · p. 3
4.3 Using external explainers
- Knowledge distillation trains a simpler student model using a large teacher network’s predictions, transferring predictive power and domain knowledge [12].The student can benefit from knowledge that might be difficult to learn directly from raw data.Source: §4.3 · p. 3
- LIME explains an individual instance by changing its input and observing the complex network’s behavior [24].For example, changing a code loop by removing “++” can indicate that the increment is a relevant feature if the network’s behavior changes.Source: §4.3 · p. 3
5 EVALUATION OF EXPLAINABILITY
Explainability is qualitative and difficult to measure, so evaluation needs evidence-based measures. The section considers model size as a simple proxy, notes its limitations, and proposes practitioner experiments to assess explanations in real tasks.
5 EVALUATION OF EXPLAINABILITY
- Because explainability is qualitative and difficult to measure, the field needs evidence-based methods for evaluating software analytics systems.The section sketches potential directions for that evaluation.Source: §5 · p. 4
- Model size can serve as a simple explainability proxy, measured by parameters, rules, or tree and graph structure dimensions.The underlying assumption is that larger models are harder for engineers to understand.Source: §5 · p. 4
- Size-based indicators capture syntax rather than semantics, and a study of 100 non-experts found larger trees and rule lists more understandable.Similar experiments could be conducted with software practitioners.Source: §5 · p. 4
- Practitioner experiments can assess explanation quality on real software tasks through productivity, bug resolution, comparisons with human explanations, or user tasks [7].Such human studies are expensive and difficult to conduct properly.Source: §5 · p. 4
6 DISCUSSION
Explainability is argued to support practitioners’ understanding and trust in machine predictions. The discussion also identifies reproducibility and repeatability as trust dimensions, and raises awareness of limitations for AI-based analytics agents.
6 DISCUSSION
- Trust in software analytics also involves reproducibility and repeatability, while AI-based agents may need awareness of their own limitations.Reproducibility means reproducing reported results with the same code and data; repeatability means applying the same method to new data for similar results.Source: §6 · p. 4