Code and Named Entity Recognition in StackOverflow

• Software-related named entities remain difficult to identify in natural-language text, despite their relevance to applications such as software knowledge-base extraction and similar-question search.Examples include variable and application names.Source: §1 · p. 1
• SoftNER addresses ambiguity and reliance on code snippets by combining sentence-level context with corpus-level information extracted from code snippets.For example, “list” may refer to a data structure or a variable name.Source: §1 · p. 1
• The paper introduces a manually annotated StackOverflow corpus and in-domain representations, including BERT and ELMo trained on 152 million sentences.The authors report that their in-domain BERT increases F1 by more than 21 points over off-the-shelf BERT.Source: §1 · pp. 1–2
• 79.10% F1 on StackOverflow and 61.08% F1 on GitHub: SoftNER extracts 20 software-related named entity types.The authors describe the performance as sufficiently strong to be practically useful.Source: §1 · p. 2

2 Annotated StackOverflow Corpus

• The StackOverflow corpus was built from 1,237 question-answer threads, manually annotated with 20 entity types; 40% of threads were double-annotated for development and test sets.For each question, annotators labeled four answers when available, including the accepted and most-upvoted answers.Source: §2 · p. 2

2.1 Annotation Schema

• The schema defines 20 fine-grained entity types spanning code and natural-language mentions, with annotation guidance on span length, modifiers, and consulting relevant code when needed.Annotators also updated, corrected, or added annotations from user-provided code tags.Source: §2.1 · p. 2
• 59.73% of code-related entities are unmarked by users, while only 75.54% of code-marked text is actually code-related.The authors cite this mismatch as a reason human-annotated data is important for training and evaluating software-domain NLP tools.Source: §2.1 · p. 3

2.2 Annotation Agreement

• StackOverflow annotation agreement was 0.62 token-level Cohen’s Kappa before adjudication.Four computer-science student annotators used BRAT, and disagreements were discussed with an adjudicator.Source: §2.2 · p. 3

2.3 Additional GitHub Data

• The GitHub dataset contains 6,510 sentences and 10,963 entities of 20 types from readme files and issue reports across 143 repositories.Its token-level Cohen’s Kappa agreement was 0.68.Source: §2.3 · p. 3

2.4 StackOverflow/GitHub Tokenization

• The authors designed SOTOKENIZER because existing web-text tokenizers can split code-related tokens, adding regular-expression rules to avoid those splits.Examples include splitting “math.h” and “a==b” into multiple tokens.Source: §2.4 · p. 3

3 Named Entity Recognition Models

• SoftNER combines contextualized word representations with domain-specific signals to address unseen tokens, polysemy, and context dependence in software-related entity recognition.Its architecture uses attention to combine the embeddings, followed by a linear-CRF tagger.Source: §3 · p. 3

3.1 Input Embeddings

• Each input word receives an in-domain BERT representation plus binary predictions from independent Code Recognizer and Entity Segmenter networks.The Code Recognizer estimates whether a word can be part of a code entity regardless of context; the Entity Segmenter predicts whether it belongs to any named entity in the sentence.Source: §3.1 · p. 4

3.1.1 In-domain Word Embeddings

• BERT, ELMo, and GloVe embeddings were pretrained on 152 million StackOverflow sentences containing 2.3 billion tokens.The paper motivates in-domain embeddings because software texts intermix programming tokens and natural-language words, unlike general book or Wikipedia texts.Source: §3.1.1 · p. 4

3.1.2 Context-independent Code Recognition

• The Code Recognizer classifies tokens using language-model probabilities, spelling patterns, and code-snippet-trained FastText character n-gram embeddings.Its inputs include unigram word and 6-gram character probabilities from Gigaword and StackOverflow code-snippet language models; predictions above 0.5 are converted to SoftNER input vectors.Source: §3.1.2 · p. 4

3.1.3 Entity Segmentation

• 90.41% precision on the development set was achieved by the Entity Segmenter, which identifies entity spans without assigning their fine-grained categories.It fine-tunes in-domain BERT with word-frequency and code-markdown features; the paper notes that markdown tags are noisy but helpful to segmentation performance.Source: §3.1.3 · pp. 4–5

3.2 Embedding-Level Attention

• Embedding-level attention weights contextual BERT, Code Recognizer, and Entity Segmenter representations before their weighted sum is passed to a linear-CRF BIO tagger.The attention weights are computed from bidirectional-GRU representations and a learned context vector.Source: §3.2 · p. 5

3.3 Implementation Details

• SoftNER and its two auxiliary models are trained separately; segmentation concatenates BERT embeddings with 100-dimensional markdown and 10-dimensional word-frequency features.SoftNER receives 850-dimensional vectors from the code recognizer and entity segmenter.Source: §3.3 · p. 5

4.3 Results

• SoftNER outperforms existing approaches on the StackOverflow NER corpus across precision, recall, and F1.Table 2 covers the development and test sets; fine-tuning BERTOverflow improves F1 by more than 10 points over original BERT.Source: Table 2 · p. 6, §4.3 · p. 6

4.1 Data

• The evaluation uses 9,352 training, 2,942 development, and 3,115 test sentences, plus a 5,312-token code-recognition dataset split into 4,312 training and 1,000 test instances.The token labels were independently assigned by two annotators; disagreements were discarded before the split.Source: §4.1 · p. 6

4.2 Baselines

• Baselines include a feature-based linear CRF, ELMo-based BiLSTM-CRF models, and fine-tuned out-of-domain and in-domain BERT.One BiLSTM-CRF variant combines ELMo with code-recognizer and entity-segmenter embeddings through attention.Source: §4.2 · p. 6

4.4 In-domain vs. out-of-domain Word Embeddings

• 13.64 F1 points separate in-domain ELMo from in-domain GloVe, and 15.71 separate it from out-of-domain ELMo; in-domain ELMo and BERT also outperform fine-tuned out-of-domain BERT by 12.92 and 12.76 F1 points.The reported embedding gains are more pronounced than those cited for newswire and biomedical NER; unseen-token accuracy rises from 41% to 70%, and unseen tokens make up 38% of tokens inside gold entity spans.Source: §4.4 · pp. 6–7

4.5 Evaluation of Auxiliary Systems

• Removing code-recognizer vectors, entity-segmenter vectors, or embedding-level attention lowers F1 by 2.19, 3.69, or 2.81 points, respectively.The code recognizer beats both baselines in F1, though the most-frequent-label baseline has higher precision and poorer recall on unseen tokens; FastText embeddings and character- and word-level features are important to the recognizer. The segmenter reaches 88.09 F1, with 90.41% precision and 85.89% recall; word-frequency and code-markdown features add 1.57 and 2.66 F1 points, respectively. Stanford NER scores 10.5 F1 on this task.Source: §4.5 · pp. 7–8

4.6 Error Analysis

• SoftNER reduces segmentation-mismatch errors by 89.36% and entity-type-mismatch errors by 13.54% relative to fine-tuned BERTOverflow.The paper attributes reductions in both categories to auxiliary outputs from the segmenter and code recognizer.Source: §4.6 · p. 8

5 Domain Adaptation to GitHub data

• The GitHub evaluation uses readme files and issue reports from 143 randomly sampled repositories, with ELMo trained on 4 million sentences from 5,000 GitHub repositories.The compared SoftNER model uses StackOverflow ELMo embeddings.Source: §5 · p. 8
• 61.08 F1 on GitHub compares with 79.10 F1 on StackOverflow; SoftNER is within 1.61 F1 points of the top BiLSTM-CRF model using GitHub ELMo.The authors attribute the overall drop to domain mismatch and suggest that GitHub contains fewer code-related entity tokens than StackOverflow.Source: Table 8 · p. 8, §5 · p. 8
• Adding code-recognizer and segmenter vectors to GitHub ELMo embeddings yields no significant gain; the authors leave semi-supervised learning and other domain-adaptation approaches for future work.Code-related entity tokens comprise 63.20% of GitHub data and 77.21% of StackOverflow data.Source: §5 · p. 8

6 Related Work

• Prior NER research spans newswire, social media, biomedical, multilingual, and code-switched text, alongside methods for rare entities such as external context and distant supervision.The section describes CoNLL 2003 as a widely used newswire benchmark and notes contextual representations in state-of-the-art approaches cited there.Source: §6 · p. 9
• Earlier software-domain NER annotated 4,646 StackOverflow sentences with five entity types using a feature-based CRF; this work presents 15,372 sentences with 20 fine-grained types and an attention-based neural NER model.Source: §6 · p. 9

7 Conclusion

• The new corpus contains 15,372 StackOverflow and 6,510 GitHub sentences annotated with 20 fine-grained software entities.Source: §7 · p. 9
• Many corpus ambiguities require long-distance context, making the dataset a benchmark for contextual word representations.Source: §7 · p. 9
• SoftNER outperforms state-of-the-art NER models on the dataset, and code recognition consistently helps improve NER performance.The authors suggest the corpus, StackOverflow-specific BERT embeddings, and tagger may be useful for code retrieval, software knowledge-base extraction, and automated question-answering.Source: §7 · p. 9

A Feature-Based CRF Baseline

• The CRF baseline uses orthographic, contextual, gazetteer, code-markdown, and regular-expression features to extract software entities.Source: A Feature-Based CRF Baseline · p. 13
• Contextual features are less helpful because neighboring words such as “in the” recur across several software-entity types.Source: A Feature-Based CRF Baseline · p. 13
• Table 9 presents dev-set performance for the CRF baseline with varying input features.Source: Table 9 · p. 13

B Attentive BiLSTM CRF with ELMoVerflow

• The model combines contextual ELMo embeddings with code-recognizer and entity-segmenter representations before BiLSTM-CRF tagging.The authors report that multi-level attention was empirically helpful for the software domain.Source: B Attentive BiLSTM CRF with ELMoVerflow · p. 13
• Table 10 reports a dev-set ablation study of Attentive-NER on the StackOverflow NER corpus.Source: Table 10 · p. 13
• Embedding-level attention uses a bidirectional GRU and a learned context vector to weight ELMo, code-recognizer, and entity-segmenter representations for each word.Source: B Attentive BiLSTM CRF with ELMoVerflow · p. 13

Weighted Word Representation

• Word-level attention assigns weights to the aggregated word vectors using a bidirectional GRU and a learned word-level context vector.Source: Weighted Word Representation · p. 13
• The weighted word representation is fed to a BiLSTM-CRF for entity-category prediction; Figure 4 depicts the attentive architecture.Source: Weighted Word Representation · p. 13, Figure 4 · p. 14
• 9.7 higher F1 on the test set, with segmentation errors reduced by 80.33% and entity-type errors by 23.34% versus BiLSTM-CRF.Source: Weighted Word Representation · p. 13

C Entity Segmentation with ELMoVerflow

• The segmenter concatenates ELMo embeddings with 100-dimensional code-markdown and 10-dimensional word-frequency features, then passes its binary output through an embedding layer as an Attentive-NER auxiliary input.It follows the Attentive-NER architecture and training setup, with a different input feature configuration.Source: C Entity Segmentation with ELMoVerflow · p. 14
• 84.3 F1 and 97.4% accuracy: the ELMoVerflow segmentation model achieves these scores on the StackOverflow NER development set.Table 11 reports the segmentation model's development-set performance.Source: C Entity Segmentation with ELMoVerflow · p. 14, Table 11 · p. 14
• Adding word-frequency and code-markdown features improves the segmenter's F1 score by 1.2 and 2.1 points, respectively.The ablation study reports these gains for the hand-crafted features.Source: C Entity Segmentation with ELMoVerflow · p. 14, Table 11 · p. 14