Code and Named Entity Recognition in StackOverflow

Introduction

Recently there has been significant interest in modeling human language together with computer code Quirk et al. (2015); Iyer et al. (2016); Yin and Neubig (2018), as more data becomes available on websites such as StackOverflow and GitHub. This is an ambitious yet promising direction for scaling up language understanding to richer domains. Access to domain-specific NLP tools could help a wide range of downstream applications. For example, extracting software knowledge bases from text Movshovitz-Attias and Cohen (2015), developing better quality measurements of StackOverflow posts Ravi et al. (2014), finding similar questions Shirani et al. (2019) and more. However, there is a lack of NLP resources and techniques for identifying software-related named entities (e.g., variable names or application names) within natural language texts.

In this paper, we present a comprehensive study that investigates the unique challenges of named entity recognition in the social computer programming domain. These named entities are often ambiguous and have implicit reliance on the accompanied code snippets. For example, the word ‘list’ commonly refers to a data structure, but can also be used as a variable name (Figure 1). In order to recognize these entities, we propose a software-related named entity recognizer (SoftNER) that utilizes an attention network to combine the local sentence-level context with corpus-level information extracted from the code snippets. Using our newly annotated corpus of 15,372 sentences in StackOverflow, we rigorously test our proposed SoftNER model, which outperforms both a BiLSTM-CRF and fine-tuned BERT model when identifying 20 types of software-related named entities. Our key contributions are the following:

A new StackOverflow NER corpus manually annotated with 20 entity types, including in-line code within natural language sentences (§2). We demonstrate that NER in the software domain is an ideal benchmark for testing the effectiveness of contextual word representations, such as ELMo Peters et al. (2018) and BERT Devlin et al. (2019), due to its inherent polysemy and salient reliance on context.

An in-domain trained neural SoftNER tagger for StackOveflow (§3) that can recognize 20 fine-grained named entity types related to software developing. We also tested its performance on GitHub data of readme files and issue reports.

A code token recognizer (§3.1) that utilizes StackOveflow code snippets to capture the spelling patterns of code-related tokens, and consistently improves the NER tagger.

In-domain pretrained ELMo and BERT representations (§3.3) on 152 million sentences from StackOverflow that significantly outperforms off-the-shelf ELMo and leads to more than 21 points increase in F1 score over off-the-shelf BERT.

Overall, our named entity tagger (SoftNER) achieves a 79.10% F1 score on StackOverflow and 61.08% F1 score on GitHub data for extracting the 20 software related named entity types. We believe this performance is sufficiently strong to be practically useful. We have released our data and code, including the named entity tagger, our annotated corpus, annotation guideline, a specially designed tokenizer, and pre-trained StackOverflow BERT and ELMo embeddings.

Annotated StackOverflow Corpus

In this section, we describe the construction of our StackOverflow NER corpus. We randomly selected 1,237 question-answer threads from StackOverflow 10-year archive (from September 2008 to March 2018) and manually annotated them with 20 types of entities. For each question, four answers were annotated, including the accepted answer, the most upvoted answer, as well as two randomly selected answers (if they exist). Table 1 shows the statistics of our corpus. 40$\%$ of the question-answer threads were double-annotated, which are used as the development and test sets in our experiments (§4). We also annotated 6,501 sentences from GitHub readme files and issue reports as additional evaluation data.

We defined and annotated 20 types of fine-grained entities, including 8 code-related entities and 12 natural language entities. The code entities include mentions of Class, Variable, In Line Code, Function, Library, Value, Data Type, and HTML XML Tag. Whereas the natural language entities include mentions of Application, UI Element, Language, Data Structure, Algorithm, File Type, File Name, Version, Device, OS, Website, and User Name.

Our annotation guideline was developed through several pilots and further updated with notes to resolve difficult cases as the annotation progressed.Our annotation guideline is available at: https://github.com/jeniyat/StackOverflowNER/. Each entity type was defined to encourage maximum span length (e.g., ‘SGML parser’ instead of ‘SGML’). We annotated noun phrases without including modifiers (e.g., ‘C’ instead of ‘Plain C’), except a few special cases (e.g., ‘rich text’ as a common File Type). On average, an entity contains about 1.5 tokens. While Variable, Function and Class names mostly consist of only a single token, our annotators found that some are written as multiple tokens when mentioned in natural language text (e.g., ‘array list’ for ‘ArrayList’ in Figure 1). The annotators were asked to read relevant code blocks or software repositories to make a decision, if needed. Annotators also searched Google or Wikipedia to categorize unfamiliar cases.

The annotators were asked to update, correct, or add annotations from the user provided $\langle$code$\rangle$ markdown tags. StackOverflow users can utilize $\langle$code$\rangle$ markdowns to highlight the code entities within the natural language sentences. However, in reality, many users do not enclose the code snippets within the $\langle$code$\rangle$ tags; and sometimes use them to highlight non-code elements, such as email addresses, user names, or natural language words. While creating the StackOverflow NER corpurs, we found that 59.73% of code-related entities are not marked by the StackOverflow users. Moreover, only 75.54% of the $\langle$code$\rangle$ enclosed texts are actually code-related, while 10.12% used to are highlighting natural language texts. The rest of cases are referring to non-code entities, such as Software Names and Versions. While markdown tag could be a useful feature for entity segmentation (§3.1.3), we emphasize the importance of having a human annotated corpus for training and evaluating NLP tools in the software domain.

2 Annotation Agreement

Our corpus was annotated by four annotators who are college students majored in computer science. We used a web-based annotation tool, BRAT Stenetorp et al. (2012), and provided annotators with links to the original post on StackOverflow. For every iteration, each annotator was given 50 question-answer threads to annotate, 20 of which were double-annotated. An adjudicator then discussed disagreements with annotators, who also cross-checked the 30 single-annotated questions in each batch. The inter-annotator agreement is 0.62 before adjudication, measured by token-level Cohen’s Kappa Cohen (1960).

3 Additional GitHub Data

To better understand the domain adaptability of our work, we further annotated the readme files and issue reports from 143 randomly sampled repositories in the GitHub dump Gousios and Spinellis (2012) (from October 29, 2007 to December 31, 2017). We removed all the code blocks from the issue reports and readme files collected from these 143 repositories. The resulting GitHub NER dataset consists of 6,510 sentences and 10,963 entities of 20 types labeled by two in-house annotators. The inter-annotator agreement of this dataset is 0.68, measured by token-level Cohen’s Kappa.

4 StackOverflow/GitHub Tokenization

We designed a new tokenizer, SOTokenizer, specifically for the social computer programming domain. StackOverflow and GitHub posts exhibit common features of web texts, including abbreviations, emoticons, URLs, ungrammatical sentences and spelling errors. We found that tokenization is non-trivial as many code-related tokens are mistakenly split by the existing web-text tokenizers, including the CMU Twokenizer Gimpel et al. (2011), Stanford TweetTokenizer Manning et al. (2014), and NLTK Twitter Tokenizer Bird et al. (2009):

Therefore, we implemented a new tokenizer, using Twokenizerhttps://github.com/myleott/ark-twokenize-py as the starting point and added additional regular expression rules to avoid splitting code-related tokens.

Named Entity Recognition Models

The extraction of software-related named entities imposes significant challenges as it requires resolving a significant amount of unseen tokens, inherent polysemy, and salient reliance on context. Unlike news or biomedical data, spelling patterns and long-distance dependencies are more crucial in the software domain to resolve ambiguities and categorize unseen words. Taken in isolation, many tokens are highly ambiguous and can refer to either programming concepts or common English words, such as: ‘go’, ‘react’, ‘spring’, ‘while’, ‘if’, ‘select’. To address these challenges, we design the SoftNER model that leverages sentential context to disambiguate and domain-specific character representations to handle rare words. Figure 2 shows the architecture of our model, which consists of primarily three components:

An input embedding layer (§3.1) that extracts contextualized embeddings from the BERTbase model and two new domain-specific embeddings for each word in the input sentence.

A embedding attention layer (§3.2) that combines the three word embeddings using an attention network.

A linear-CRF layer that predicts the entity type of each word using the attentive word representations from the previous layer.

For each word in the input sentence, we extract in-domain BERT Devlin et al. (2019) representations and two new domain-specific embeddings produced by (i) a Code Recognizer, which represents if a word can be part of a code entity regardless of context; and (ii) an Entity Segmenter, that predicts whether a word is part of any named entity in the given sentence. Each domain-specific embedding is created by passing a binary value, predicted by a network independent from the SoftNER model. We describe the two standalone auxiliary models that generate these domain-based vectors below.

Texts in the software engineering domain contain programming language tokens, such as variable names or code segments, interspersed with natural language words. This makes input representations pre-trained on general book or Wikipedia texts unsuitable for software domain. We pre-trained different in-domain word embeddings, including BERT (BERTOverflow), ELMo (ELMoVerflow), and GloVe (GloVerflow) vectors on the StackOverflow 10-year archivehttps://archive.org/details/stackexchange of 152 million sentences and 2.3 billion tokens (§3.3).

1.2 Context-independent Code Recognition

Humans with prior programming knowledge can easily recognize that ‘list()’ is code, ‘list’ can be either code or a common English word, whereas ‘listing’ is more likely a non-code natural language token. We thus introduce a code recognition module to capture such prior probability of how likely a word can be a code token without considering any contextual information. It is worth noting that this standalone code recognition model is also useful for language-and-code research, such as retrieving code snippets based on natural language queries Iyer et al. (2016); Giorgi and Bader (2018); Yao et al. (2019)

Our code recognition model (Code Recognizer) is a binary classifier. It utilizes language model features and spelling patterns to predict whether a word is a code entity. The input features include unigram word and 6-gram character probabilities from two language models (LMs) that are trained on the Gigaword corpus Napoles et al. (2012) and all the code-snippets in the StackOverflow 10-year archive respectively. We also pre-trained FastText Joulin et al. (2016) word embeddings using these code-snippets, where a word vector is represented as a sum of its character ngrams. We first transform each ngram probability into a $k$-dimensional vector using Gaussian binning Maddela and Xu (2018), which has shown to improve the performance of neural models using numeric features Sil et al. (2017); Liu et al. (2016); Maddela and Xu (2018). We then feed the vectorized features into a linear layer, concatenate the output with FastText character-level embeddings, and pass them through another hidden layer with sigmoid activation. We predict the token as a code-entity if the output probability is greater than $0.5$. This binary prediction is then converted into a vector and used as an input to the SoftNER model.

1.3 Entity Segmentation

The segmentation task refers to identifying entity spans without assigning entity category. Entity segmentation is simpler and less error-prone than entity recognition as it does not require a fine-grained classification of the entity types. In fact, a segmentation model (Entity Segmenter) trained on our annotated StackOverflow corpus can achieve 90.41% precision on the dev set (details in §4.5), predicting whether each token is a part of entity in the given sentence. Our segmentation model fine-tunes the in-domain BERT after concatenating it with two hand-crafted features:

Word Frequency represents the word occurrence count in the training set. As many code tokens are defined by individual users, they occur much less frequently than normal English words. In fact, code and non-code tokens have an average frequency of 1.47 and 7.41 respectively in our corpus. Moreover, ambiguous token that can be either code or non-code entities, such as ‘windows’, have a much higher average frequency of 92.57. To leverage this observation, we include word frequency as a feature, converting the scalar value into a $k$-dimensional vector by Gaussian binning Maddela and Xu (2018).

Code Markdown indicates whether the given token appears inside a $\langle$code$\rangle$ markdown tag in the StackOverflow post. It is worth noting that $\langle$code$\rangle$ tags are noisy as users do not always enclose inline code in a $\langle$code$\rangle$ tag or sometimes use the tag to highlight non-code texts (details in §2.1). Nevertheless, we find it helpful to include the markdown information as a feature as it improves the performance of our segmentation model.

The inclusion of hand-crafted features is influenced by Wu et al. Wu et al. (2018), where word-shapes and POS tags were shown to improve the performance of sequence tagging models.

2 Embedding-Level Attention

For each input word $w_{i}$ in the input sentence, we have three embeddings: BERT ($w_{i1}$), Code Recognizer ($w_{i2}$), and Entity Segmenter ($w_{i3}$). We introduce the embedding-level attention ${\alpha}_{it}$ ($t\in\{1,2,3\}$), which captures each embedding’s contribution towards the meaning of the word, to combine them together. To compute ${\alpha}_{it}$, we pass the input embeddings through a bidirectional GRU and generate their corresponding hidden representations $h_{it}=\overleftrightarrow{GRU}(w_{it})$. These vectors are then passed through a non-linear layer, which outputs $u_{it}=tanh(W_{e}h_{it}+b_{e})$. We introduce an embedding-level context vector $u_{e}$, which is randomly initialized and updated during the training process. This context vector is combined with the hidden embedding representation using a softmax function to extract weight of the embeddings: ${\alpha}_{it}=\frac{exp({u_{it}}^{T}u_{e})}{{\sum}_{t}exp({u_{it}}^{T}u_{e})}$. Finally, we create the word vector by a weighted sum of all the information from different embeddings as $word_{i}={\sum}_{t}{\alpha}_{it}h_{it}$. The aggregated word vector $word_{i}$ is then fed into a linear-CRF layer, which predicts the entity category for each word based the BIO tagging schema.

3 Implementation Details

We use PyTorch framework to implement our proposed SoftNER model and its two auxiliary components, namely code recognition and entity segmentation. The input to the SoftNER model include 850-dimensional vectors extracted from both the code recognizer and the entity segmenter.

We pre-trained BERTbase, ELMo and GloVe vectors on 152 million sentences from the StackOverflow, excluding sentences from the 1,237 posts in our annotated corpus. The pre-training of the 768-dimensional BERTbase model with 64,000 WordPiece vocabulary took 7 days on a Google TPU. The pre-training of 1024-dimensional ELMo vectors took 46 days on 3 NVIDIA Titan X Pascal GPUs. The pre-training of 300-dimensional GloVe embeddings Pennington et al. (2014) with a frequency cut-off of 5 took 8 hours on a server with 32 CPU cores and 386 GB memory.

We train the SoftNER model and the two auxiliary models separately. Our segmentation model follows the simple BERT fine-tuning architecture except for the input, where BERT embeddings are concatenated with 100-dimensional code markdown and 10-dimensional word frequency features. We set the number of bins $k$ to 10 for Gaussian vectorization. Our code recognition model is a feedforward network with two hidden layers and a single output node with sigmoid activation.

Evaluation

In this section, we show that our SoftNER model outperforms all the previous NER approaches on the StackOverflow and GitHub data. We also discuss the factors pivotal to the performance of our model, namely pre-trained in-domain BERT embeddings and two domain-specific auxiliary tasks.

We train and evaluate our SoftNER model on the StackOverflow NER corpus of 9,352 train, 2,942 development and 3,115 test sentences we constructed in §2. We use the same data for our segmentation model but replace all the entity tags with an i-entity tag. For the code recognition model, we created a new lexicon of 6,000 unique tokens randomly sampled from the training set of the StackOverflow NER corpus. Each token was labelled independently without context as code, ambiguous or non-code by two annotators with computer science background. The inter-annotator agreement was 0.89, measured by Cohen’s Kappa. After discarding disagreements, we divided the remaining 5,312 tokens into 4,312 train and 1,000 test instances. Then, we merged ambiguous and non-code categories to facilitate binary classification. We name this dataset of 5312 individual tokens as SOLexicon.

2 Baselines

We compare our model with the following baseline and state-of-the-art approaches:

A Feature-based Linear CRF model which uses the standard orthographic, context and gazetteer features, along with the code markdown tags and handcrafted regular expressions to recognize code entities (details in Appendix A).

A BiLSTM-CRF model with in-domain ELMo embeddings (ELMoVerflow; details in §3.3). This architecture is used as the state-of-the-art baseline named-entity recognition models in various domains Lample et al. (2016); Kulkarni et al. (2018); Dai et al. (2019).

An Attentive BiLSTM-CRF model with in-domain ELMo embeddings as well as domain-specific embeddings from the code recognizer and the entity segmenter. This model combines these three word embeddings using an attention network and then utilizes a BiLSTM-CRF layer to predict the entity type of each input word (details in Appendix B).

A Fine-tuned out-of-domain BERT model where we fine-tune the original BERTbase cased checkpointhttps://github.com/google-research/BERT on our annotated corpus.

A Fine-tuned in-domain BERT model where we fine-tune the in-domain pre-trained BERTbase (BERTOverflow; details in §3.3) cased checkpointhttps://github.com/lanwuwei/BERTOverflow/ on our annotated corpus.

3 Results

Table 2 shows the precision (P), recall (R) and F1 score comparison of different models evaluated on the StackOverflow NER corpus. Our SoftNER model outperforms the existing NER approaches in all the three metrics. Fine-tuning over in-domain trained BERT (BERTOverflow), in particular, improves F1 score by more than 10 points in comparison to using the original BERT.

4 In-domain vs. out-of-domain Word Embeddings

Table 3 shows the performance comparison between in-domain and out-of-domain word embeddings. We consider off-the-shelf BERT Devlin et al. (2019), ELMo Peters et al. (2018) and GloVe Pennington et al. (2014) vectors trained on newswire and web texts as out-of-domain embeddings. When using the BiLSTM-CRF model Lample et al. (2016); Kulkarni et al. (2018); Dai et al. (2019), we observe a large increase of 13.64 F1 score when employing in-domain ELMo (ELMoVerflow) representations over in-domain GloVe (GloVeOverflow), and an increase of 15.71 F1 score over out-of-domain ELMo. We found that fine-tuning out-of-domain BERT Devlin et al. (2019) outperforms the out-of-domain ELMo (Table 3), although it underperforms in-domain ELMo (ELMoVerflow) by 12.92 F1 score and in-domain BERt (BERTOverflow) by 12.76 F1 score (Table 2). Similarly, in-domain ELMo outperforms the out-of-domain fine-tuned BERT by 10.67 F1 score on Github data (Table 8; more details in §5).

It is worth noting that, the performance improvements from contextual word embeddings are more pronounced on our software domain than on newswire and biomedical domains. Original ELMo and BERT outperform GloVe by 2.06 and 2.12 points in F1 respectively on CoNLL 2003 NER task of newswire data Peters et al. (2018); Devlin et al. (2019). For biomedical domain, in-domain ELMo outperforms out-of-domain ELMo by only 1.33 points in F1 on the BC2GM dataset Sheikhshabbafghi et al. (2018).

We hypothesized that the performance gains from the in-domain contextual embeddings are largely aided by the model’s ability to handle ambiguous and unseen tokens. The increase in performance is especially notable (41% $\xrightarrow{}$ 70% accuracy) for unseen tokens, which constitute 38% of the tokens inside gold entity spans in our dataset. This experiment also demonstrates that our annotated NER corpus provides an attractive test-bed for measuring the adaptability of different contextual word representations.

5 Evaluation of Auxiliary Systems

The domain-specific vectors produced by the Code Recognizer and the Entity Segmenter are also crucial for the overall performance of our SoftNER model. Table 4 shows an ablation study. Removing code recognizer vectors and entity segmenter vectors results in a drop of 2.19 and 3.69 in F1 scores respectively. If we replace embedding-level attention with a simple concatenation of embeddings, the performance also drop by 2.81 F1. In addition, we evaluate the effectiveness of our two domain-specific auxiliary systems on their respective tasks.

Table 5 compares the performance of our code recognition model with other baselines on the SLexicon test set (§4.1), which consists of 1,000 random words from the train set of StackOverflow NER corpus classified as either a code or a non-code token. The baselines include: (i) a Most Frequent Label baseline, which assigns the most frequent label according to the human annotation in SOLexicon train set; and (ii) a frequency baseline, which learns a threshold over token frequency in the train set of StackOverflow NER corpus using a decision tree classifier. Our model outperforms both baselines in terms of F1 score. Although the most frequent label baseline achieves better precision than our model, it performs poorly on unseen tokens resulting in a large drop in recall and F1 score. The ablation experiments show that the FastText word embeddings along with the character and word-level features are crucial for the code recognition model.

Table 6 shows the performance of our segmentation model on the dev set of our StackOverflow corpus, where the entity tags are replaced by an I-Entity tag. Our model achieves an F1 score of 88.09 and with 90.41% precision and 85.89% recall. Incorporating word frequency and code markdown feature increases the F1 score by 1.57 and 2.66 points respectively. The low 10.5 F1 score of Stanford NER tagger Manning et al. (2014), which is trained on newswire text, demonstrates the importance of domain-specific tools for the software engineering domain.

6 Error Analysis

Based on our manual inspection, the incorrect predictions made by NER systems on StackOverflow data can be largely classified into the following two categories (see examples in Table 7):

Segmentation Mismatch refers to the cases where model predicts the boundary of entities incorrectly. Our SoftNER model reduces such segmentation errors by 89.36% compared to the fine-tuned BERTOverflow baseline.

Entity-Type Mismatch refers to the errors where a code entity (e.g., names of variables) is predicted as a non-code entity (e.g., names of devices), and vice-versa. Our SoftNER model reduces such entity type errors by 13.54% compared to the fine-tuned BERTOverflow baseline.

As illustrated in Figure 3, our SoftNER model reduced the errors in both categories by incorporating the auxiliary outputs from segmenter and code recognizer model.

Domain Adaptation to GitHub data

To understand the domain adaptability of our StackOverflow based SoftNER, we evaluate its performance on readme files and issue reports from 143 randomly sampled repositories in the GitHub dump Gousios and Spinellis (2012). We also trained ELMo embeddings (ELMoGithub) on 4 million sentences from randomly sampled 5,000 GitHub repositories.

Table 8 shows that the performance of our SoftNER model using StackOverflow ELMo embeddings is similar to the top performing BiLSTM-CRF model using GitHub ELMo embeddings with a difference of only 1.61 points in F1. We also did not observe any significant gain after adding the code recognizer and segmenter vectors to the Github ELMo embeddings. We think one likely explanation is that GitHub data contains less code-related tokens when compared to StackOverflow. The percentage of code-related entity tokens is 63.20% in GitHub and 77.21% in StackOverflow. Overall, we observe a drop of our SoftNER tagger from 79.10 F1 on StackOverflow (Table 2) to 61.08 F1 on GitHub data (Table 8) in F1 due to domain mismatch. However, we believe that our NER tagger still achieves sufficient performance to be useful for applications on GitHub.As a reference, the state-of-the-art performance for 10-class Twitter NER is 70.69 F1Zhang et al. (2018). We leave investigation of semi-supervised learning and other domain adaptation approaches for future work.

Related Work

The CoNLL 2003 dataset Sang and De Meulder (2003) is a widely used benchmark for named entity recognition, which contains annotated newswire text from the Reuters RCV1 corpus. State-of-the-art approaches on this dataset Baevski et al. (2019) use a bidirectional LSTM (Lample et al., 2016; Ma and Hovy, 2016) with conditional random field Collobert et al. (2011) and contextualized word representations McCann et al. (2017); Peters et al. (2018); Devlin et al. (2019).

Named entity recognition has been explored for new domains and languages, such as social media Finin et al. (2010); Ritter et al. (2011); Plank et al. (2014); Derczynski et al. (2015); Limsopatham and Collier (2016); Aguilar et al. (2017), biomedical texts Collier and Kim (2004); Greenberg et al. (2018); Kulkarni et al. (2018), multilingual texts Benajiba et al. (2008); Xie et al. (2018) and code-switched corpora Aguilar et al. (2018); Ball and Garrette (2018). Various methods have been investigated for handling rare entities, for example incorporating external context Long et al. (2017) or approaches that make use of distant supervision Choi et al. (2018); Yang et al. (2018); Onoe and Durrett (2019).

There has been relatively little prior work on named entity recognition in the software engineering domain. Ye et al. (2016) annotated 4,646 sentences from StackOverflow with five named entity types (Programming Language, Platform, API, Tool-Library-Framework and Software Standard). The authors used a traditional feature-based CRF to recognize these entities. In contrast, we present a much larger annotated corpus consisting of 15,372 sentences labeled with 20 fine-grained entity types. We also develop a novel attention based neural NER model to extract those fine-grained entities.

Conclusion

In this work, we investigated the task of named entity recognition in the social computer programming domain. We developed a new NER corpus of 15,372 sentences from StackOverflow and 6,510 sentences from GitHub annotated with 20 fine-grained named entities. We demonstrate that this new corpus is an ideal benchmark dataset for contextual word representations, as there are many challenging ambiguities that often require long-distance context to resolve. We also proposed a novel attention based model, named SoftNER, that outperforms the state-of-the-art NER models on this dataset. Furthermore, we investigated the important sub-task of code recognition. Our code recognition model captures additional spelling information beyond then contextual word representations and consistently helps to improve the NER performance. We believe our corpus, StackOverflow-specific BERT embeddings and named entity tagger will be useful for various language-and-code tasks, such as code retrieval, software knowledge base extraction and automated question-answering.

Acknowledgement

We thank anonymous reviewers for their thoughtful comments. We also thank NVIDIA, Google, and Ohio Supercomputer Center Center (2012) for providing GPU/TPU computing resources; Wuwei Lan for kindly helping to train in-domain BERT on StackOverflow data; Sydney Lee, Rita Tong, Lillian Chow, and Raleigh Potluri for help with data annotation. This research is supported in part by the NSF awards IIS-1822754 and IIS-1845670, ODNI and IARPA via the BETTER program contract 19051600004, ARO and DARPA via the SocialSim program contract W911NF-17-C-0095, Criteo Faculty Research Award to Wei Xu, and Amazon Faculty Research Award to Alan Ritter. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, ARO, DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

References

Appendix A Feature-Based CRF Baseline

We implemented a CRF baseline model using CRFsuitehttp://www.chokkan.org/software/crfsuite/ to extract the software entities. This model uses standard orthographic, contextual and gazetteer features. It also includes the code markdown tags (§3.1.3) and a set of regular expression features. The regular expressions are developed to recognize specific categories of code-related entities. Feature ablation experiments on this CRF model are presented in Table 9. One noticeable distinction from the named entity recognizer in many other domains is that the contextual features are not as helpful in feature-based CRFs for classifying software entities. This is because, in the StackOverflow NER corpus a significant number of neighbouring words are shared among different software entities. As an example, the bigram ‘in the’ frequently appears as the left context of the following types: Application, Class, Function, File Type, UI Element, Library, Data Structure and Language.

Appendix B Attentive BiLSTM CRF with ELMoVerflow

We propose a baseline Attentive NER model that utilizes a BiLSTM-CRF network to predict the entity type of each word from its weighted representations. The weighted word representations are extracted by a multi-level attention network, similar to Yang et al.Yang et al. (2016), that combines the contextualized ELMo embeddings with the code recognizer (§3.1.2) and segmenter vector (§C). These three input embeddings are merged together in the first attention layer and then their corresponding weights are calculated using the second layer. Although such multi-level attention is not commonly used in NER, we found it empirically helpful for the software domain (see Table 10).

uses three embeddings, ELMo ($w_{i1}$), Code Recognizer ($w_{i2}$), and Entity Segmenter ($w_{i3}$), for each word $w_{i}$ in the input sentence. The embedding-level attention ${\alpha}_{it}$ ($t\in\{1,2,3\}$) to captures each embedding’s contribution towards the meaning of the word. To compute ${\alpha}_{it}$, it pass the input embeddings through a bidirectional GRU and generate their corresponding hidden representations $h_{it}=\overleftrightarrow{GRU}(w_{it})$. These vectors are then passed through a non-linear layer, which outputs $u_{it}=tanh(W_{e}h_{it}+b_{e})$. It uses an embedding-level context vector, $u_{e}$, which is learned during the training process. This context vector is combined with the hidden embedding representation using a softmax function to extract weight of the embeddings, ${\alpha}_{it}=\frac{exp({u_{it}}^{T}u_{e})}{{\sum}_{t}exp({u_{it}}^{T}u_{e})}$. Finally, the word vector is created by a weighted sum of all the information from different embeddings as $word_{i}={\sum}_{t}{\alpha}_{it}h_{it}$.

uses a word-level weighting factor $\alpha_{i}$ to emphasize the importance of each word $w_{i}$ for the NER task. Similar to the embedding-level attention, it calculates $\alpha_{i}$ from the weighted word vectors $word_{i}$. A bidirectional GRU is used to encode the summarized information from neighbouring words and thus it get $h_{i}=\overleftrightarrow{GRU}(word_{i})$. This is then passed through a hidden layer which outputs $u_{i}=tanh(W_{w}h_{i}+b_{w})$. Then the normalized weight for each word vector is extracted by ${\alpha}_{i}=\frac{exp({u_{i}}^{T}u_{w})}{{\sum}_{t}exp({u_{i}}^{T}u_{w})}$, where $u_{w}$ is another word-level context vector that is learned during training. The final weighted word representation is computed by $word_{i}^{\prime}={\alpha}_{i}h_{i}$.

Subsequently, the aggregated word vector $word_{i}^{\prime}$ is fed into a BiLSTM-CRF network, which predicts the entity category for each word. The complete architecture of the Attentive BiLSTM CRF model is illustrated in Figure 4. Compared to BiLSTM-CRF, our proposed Attentive BiLSTM-CRF demonstrates a 9.7 increase in F1 on the test set (Table 2) and reduces the segmentation errors and entity type errors by 80.33% 23.34% respectively.

Appendix C Entity Segmentation with ELMoVerflow

The Attentive-NER tagger utilizes the outputs from an auxiliary segmentation module which consists of a BiLSTM encoder and a CRF decoder. This model concatenates ELMo embeddings with two hand-crafted features- word frequency and code markdown (§3.1.3).

The segmentation model follows the same architecture and training setup as the Attentive-NER model except for the input, where ELMo embeddings are concatenated with 100-dimensional code markdown and 10-dimensional word frequency features. The binary output from this entity segmenter model is later passed as through an embedding layer and used as one of the auxiliary inputs of the Attentive NER model.

Table 11 shows the performance of this segmentation model with ELMoVerflow on the dev set. This model achieves an F1 score of 84.3 and an accuracy of 97.4%. The ablation study in Table 11 depicts the importance of the hand-crafted frequency and markdown features for this segmenter model by providing an increment of 1.2 and 2.1 points in the F1 score respectively.