Source-linked AI summary
Sentiment Polarity Detection for Software Development
Fabio Calefato, Filippo Lanubile, Federico Maiorano, Nicole Novielli
TL;DR
Existing sentiment tools disagree with human annotations in developers’ communication channels and show a negative bias. Senti4SD is a classifier for software developers’ artifacts that improves polarity classification relative to an off-the-shelf baseline, while the authors identify replication and coding-guideline needs.
Problem
Existing sentiment analysis tools disagree with human annotations of developers’ communication channels and misclassify neutral content as negative.
Method
Senti4SD is a sentiment polarity classifier for software developers’ artifacts, supported by manually annotated Stack Overflow data and lexical and semantic features.
Results
19% improvement in precision for the negative class is reported for Senti4SD, while neutral posts misclassified as negative decrease from 27.6% with SentiStrength to 13.8%.
Takeaways & Limitations
The manually annotated dataset provides a resource for emotion-polarity research on Stack Overflow and supports replication of the classifier.
Takeaways & Limitations
The authors acknowledge that replications are needed to assess whether the dataset is representative of developers’ communication style.
Abstract
from arXiv · showhide
The role of sentiment analysis is increasingly emerging to study software developers' emotions by mining crowd-generated content within social software engineering tools. However, off-the-shelf sentiment analysis tools have been trained on non-technical domains and general-purpose social media, thus resulting in misclassifications of technical jargon and problem reports. Here, we present Senti4SD, a classifier specifically trained to support sentiment analysis in developers' communication channels. Senti4SD is trained and validated using a gold standard of Stack Overflow questions, answers, and comments manually annotated for sentiment polarity. It exploits a suite of both lexicon- and keyword-based features, as well as semantic features based on word embedding. With respect to a mainstream off-the-shelf tool, which we use as a baseline, Senti4SD reduces the misclassifications of neutral and positive posts as emotionally negative. To encourage replications, we release a lab package including the classifier, the word embedding space, and the gold standard with annotation guidelines.
1. Introduction
Sentiment analysis tools trained on general-purpose or non-software data can misclassify developers’ technical communication, especially neutral problem descriptions, motivating Senti4SD. Senti4SD combines domain-relevant features and Stack Overflow training data, improving polarity classification against SentiStrength.
- Motivation: Off-the-shelf sentiment tools trained on movie reviews or general-purpose social media may not agree with human annotations of developers’ communication channels.
- Motivation: Existing tools exhibit a negative bias by misclassifying neutral technical texts, including bug reports and problem descriptions, as emotionally negative.
- Senti4SD: Senti4SD is a sentiment classifier using lexicon-based, keyword-based, and semantic features to address domain-dependent sentiment in software engineering.
- Evaluation: 19% improvement in precision for the negative class and 25% improvement in recall for the neutral class were observed versus the SentiStrength baseline.
- Resources: The study built a gold standard of 4,423 Stack Overflow posts and released the classifier, embedding resources, gold standard, and annotation guidelines.
- Evaluation: Evaluation across feature settings provided evidence of better performance even with a minimal set of training documents.
2. Research Methods
The research process combines theoretical framing, manual annotation, natural language processing, and machine learning to construct and evaluate a software-development sentiment classifier. It proceeds through four sequential phases using Stack Overflow content and comparisons with off-the-shelf tools.
- Research process: The study uses qualitative and quantitative methods, including manual coding, natural language processing for feature extraction, and machine learning for classifier training.
- Research process: The research process is organized into four sequential phases covering theoretical framing, annotation, feature extraction, and classifier training and evaluation.
- Resources: The study releases the full lab package, including Senti4SD, the distributional semantic model, and the gold standard.
- Annotation: The annotation sample contains questions, answers, and comments from Stack Overflow, with text preprocessed by removing URLs, code snippets, and HTML tags.
- Annotation: Coders received training and completed a pilot study before final annotation, while Kappa measured interrater reliability and majority voting assigned gold labels.
- Evaluation: The gold standard was used to train and evaluate the classifier against off-the-shelf tools representing the state of the art for social-media sentiment analysis.
3. Background
The paper frames emotion analysis through continuous and discrete models, then motivates technical sentiment classification using lexicons, supervised features, and distributional semantics. Off-the-shelf tools rely on prior word polarity, whereas the proposed Senti4SD combines lexicon, keyword, and semantic features for software-development text.
- Emotion Modeling: Emotion modeling follows either continuous dimensions, such as polarity and arousal, or a limited set of discrete basic emotions.The continuous view models valence and activation, while the discrete view assumes a finite basic-emotion set.
- Emotion Modeling: The paper maps Shaver et al.’s hierarchical emotion labels to positive, negative, and neutral polarity for annotation.This mapping informs the annotation guidelines.
- Polarity Detection with SentiStrength: SentiStrength assigns prior sentiment scores to words and produces positive, negative, and neutral text classifications.It also handles mixed sentiment, intensifiers, negations, and emoticons.
- Research Motivation and Approach: To address off-the-shelf limitations in empirical software engineering, the paper trains Senti4SD on technical developer texts using lexicon, keyword, and semantic features.SentiStrength and SentiStrength-SE provide comparison points for benchmarking.
- Polarity Detection with SentiStrength: Dictionary-based tools cannot adjust word polarity to contextual meaning because they do not model word similarity.This limitation motivates semantic representations that capture contextual relationships.
- Distributional Semantic Models: Distributional semantic models represent words as vectors learned from contextual patterns, enabling similarity-based comparisons.Traditional models count corpus occurrences, whereas neural word embeddings predict contexts.
4. Dataset: A Gold Standard for Emotion Polarity in Software Development
The authors construct a manually annotated Stack Overflow gold standard covering multiple post types and polarity classes. Training and annotation procedures yield a balanced 4,423-post dataset with substantial interrater agreement after excluding mixed-polarity disagreements.
- Gold Standard Construction: The gold standard contains 4,423 Stack Overflow posts for training and evaluating emotion-polarity classification.It represents 92% of the 4,800 items initially annotated.
- Gold Standard Composition: The final dataset is distributed as 35% positive, 27% negative, and 38% neutral posts.Neutral posts are those in which no emotions were observed.
- Sampling and Coding: The dataset includes questions, answers, question comments, and answer comments labeled positive, negative, or neutral.The analysis therefore covers 3x4 combinations of emotion styles and post types.
- Sampling and Coding: Annotation used 4,800 opportunistically sampled items balanced by post type and SentiStrength polarity scores.Posts were sampled from Stack Overflow content and preprocessed to remove code snippets, URLs, and HTML tags.
- Sampling and Coding: Coders assigned positive, negative, neutral, or mixed labels using guidelines grounded in a hierarchical emotion framework.Joy or love mapped to positive, anger, sadness, or fear to negative, surprise depended on context, and opposing emotions were mixed.
- Reliability and Final Dataset: Interrater reliability was substantial, with weighted Cohen’s Kappa ranging from .66 to .80 and averaging .74.Observed agreement ranged from .73 to .85, averaging .79.
- Reliability and Final Dataset: The authors resolved disagreements by majority voting but excluded posts with opposite polarity labels, including mixed cases, from the gold standard.Excluded mixed cases represented 3% of the annotated items.
5. Emotion Polarity Classifier: Feature Description and System Setup
Senti4SD combines lexicon-based, keyword-based, and semantic features tailored to Stack Overflow text. Its semantic features use a distributional semantic model trained on software engineering data, while the classifier uses a high-dimensional linear SVM setup.
- Feature groups: Senti4SD combines generic sentiment lexicons, dataset-derived n-grams, and word representations from a software-engineering semantic model.The three feature groups capture prior polarity, Stack Overflow-specific wording, and distributional semantics.
- Lexicon-based features: Lexicon-based features summarize token polarity, sentiment scores, emoticons, emphasis, and the polarity of the final token.Examples include positive and negative token counts, summed and maximum scores, and end-token indicators.
- Keyword-based features: Keyword-based features represent unigram and bigram counts alongside uppercase, elongated words, emoticons, and laughter slang.The complete keyword-based feature set contains 76,346 features.
- Semantic features: Semantic features compare each Stack Overflow post vector with positive, negative, neutral, and subjective prototype vectors.Posts are represented by summing word vectors, and the resulting similarities are computed in a DSM trained with CBOW word2vec.
- Semantic features: The DSM used 600-dimensional vectors trained on more than 20 million Stack Overflow posts from questions, answers, and comments.The corpus included 3.8 million questions, 5.9 million answers, and 11.6 million comments from a dump updated to September 2015.
- System setup: Senti4SD retains the full high-dimensional feature set and trains a linear SVM, tuning its cost parameter with 10-fold cross-validation.The setup uses information gain to rank features while avoiding feature selection because it can lose information.
6. Evaluation
The evaluation trains Senti4SD on manually labeled Stack Overflow data and tests its ability to classify unseen posts. Compared with SentiStrength, it improves selected class-level metrics, and incremental experiments attribute gains to the combined feature design.
- Evaluation setup: The gold standard is split into stratified training and test sets, with the final model evaluated on unseen Stack Overflow data.The main split uses 70% for training and 30% for testing.
- Baseline comparison: Senti4SD is compared with SentiStrength and SentiStrength-SE after mapping predictions to positive, neutral, and negative labels.SentiStrength is selected as the baseline because it outperforms SentiStrength-SE in the reported evaluation.
- Results: 19% improvement in precision for the negative class and 25% improvement in recall for the neutral class are reported against the SentiStrength baseline.The evaluation reports performance using recall, precision, and F-measure for individual classes and overall.
- Prediction differences: 84 neutral posts recognized only by Senti4SD are classified mainly as negative by SentiStrength, including 69 of 84 cases.For positive posts, 13 are recognized only by Senti4SD, while 10 are missed by it and mostly labeled neutral.
- Feature analysis: The n-gram-only model reaches F = .69, while the full feature set reaches F = .87 in the reported incremental-feature results.Adding semantic features raises performance to F = .81, and adding lexicon features increases negative recall to .89 and neutral precision to .87.
- Robustness analysis: Senti4SD outperforms SentiStrength in all three alternative train-test splits, including one using 30% of the gold standard for training.The comparisons use Chi-squared tests, with reported significance assessed at p-value < 0.05.
7. Discussion
Senti4SD reduces several negative-bias errors observed with SentiStrength on Stack Overflow, while exposing trade-offs in negative recall and long-post classification. Its domain-specific training, released resources, and feature analysis support software-engineering sentiment research.
- 28% of neutral Stack Overflow posts are misclassified as negative by SentiStrength, while Senti4SD reduces this error to 13.8%.
- Senti4SD improves neutral recall from .64 to .80 and negative precision from .67 to .80 relative to SentiStrength.These correspond to 25% improvement in neutral recall and 19% improvement in negative precision.
- The precision gain is accompanied by negative-class recall decreasing from .96 to .89.
- Long posts can still be misclassified when neutral sentences outweigh negative lexicon cues, motivating sentence-level annotation and a sentence-based Senti4SD version.
8. Threats to Validity
The study’s validity is bounded by its Stack Overflow setting, sampling procedure, and subjective manual annotation, although coder agreement supports gold-standard reliability.
- Results may differ outside Stack Overflow, so replication across the broader software developer ecosystem is needed.
- Manual emotion annotation is subjective because affect perception can vary with personality and personal dispositions.
- Clear guidelines, theoretical grounding, and majority agreement among three coders were used to mitigate annotation subjectivity.
- Average weighted Cohen’s Kappa = 0.74 indicates good interrater reliability for the gold standard.
- Sampling through SentiStrength overrepresented neutral posts and emotion-bearing vocabulary, while the negative class remained underrepresented.
- SentiStrength’s sampling role also produced a challenging baseline, with F = .82 on the resulting gold standard.
- Senti4SD currently assigns mixed-sentiment posts to either positive or negative classes, motivating future binary classifiers for mixed sentiment.
9. Related Work
Related work applies sentiment and emotion mining to software-engineering artifacts, but domain-specific classifiers address limitations of general-purpose tools. Senti4SD combines multiple feature types and achieves balanced class performance relative to competing approaches.
- Software-engineering researchers developed sentiment tools because off-the-shelf analyzers show limited performance and negative bias in technical documents.
- Prior work mined developer emotions in issue comments using VAD metrics and classifiers for anger, joy, sadness, and love.
- Other approaches include domain dictionaries and document-structure features for IT tickets, achieving overall F = .85 compared with Senti4SD’s F = .86.
- Senti4SD yields more balanced negative and neutral performance than the IT-ticket classifier and uses a larger, balanced dataset.
- On technical texts, SentiStrength-SE outperforms SentiStrength, but on Stack Overflow posts it trails both SentiStrength and Senti4SD.
- On the Stack Overflow gold standard, SentiStrength-SE has F = .78, SentiStrength F = .82, and Senti4SD F = .87.
- Senti4SD introduces word embedding into software-engineering sentiment analysis, using distributional semantic features and prototype polarity vectors.
10. Conclusions
The paper presents Senti4SD, trained on manually annotated Stack Overflow posts, combining lexical, keyword, and semantic features. It reports improved handling of sentiment polarity and releases resources for replication.
- Senti4SD is a sentiment polarity classifier for software developers’ artifacts, trained and tested on over 4K manually annotated Stack Overflow posts.
- The publicly available gold standard and annotation guidelines support further studies and replication of the annotation experiment.
- Its semantic features use a distributional semantic model built with word embedding and word2vec on over 20 million Stack Overflow documents.
- The released semantic model supports future research on word embedding for software-engineering text categorization and information retrieval.
- Combining lexicon-based, keyword-based, and semantic features addresses negative bias in off-the-shelf sentiment analysis tools.
- 19% improvement in precision for the negative class and 25% improvement in recall for the neutral class are reported against SentiStrength.
- Future work targets additional linguistic features, mixed sentiment, other software-engineering repositories, broader benchmarks, and emotion labels.
Appendix A: Coding Guidelines
The appendix specifies how developers’ Stack Overflow posts are manually coded for emotions and polarity using a structured schema that permits positive, negative, neutral, and mixed labels.
- The annotation study covers developer-authored technical documents and uses randomly selected Stack Overflow questions, answers, and comments.
- The coding schema asks annotators to identify emotions including love, joy, surprise, anger, sadness, and fear.
- Annotators may use second- and third-level emotion categories as references when selecting labels.
- The appendix includes examples of annotated posts to illustrate application of the coding guidelines.
- Posts receive positive, negative, neutral, or mixed polarity according to their annotated emotions; absence of emotion is neutral.
- The schema defines allowed and disallowed combinations of emotion and polarity values.