A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes

Introduction

Source code, while conceived as a set of structured and sequential instructions, inherently reflects human intent: it encodes the way we command a machine to perform a task. In that sense, it is expected that it follows to some extent the same distributional regularities that a proper natural language manifests Hindle et al. (2012). Moreover, the unambiguous nature of source code, comprised in plain and human-readable format, allows an indirect way of communication between developers, a phenomenon boosted in recent years given the current software development paradigm, where billions of lines code are written in a distributed and asynchronous way Gousios et al. (2014).

The scale and complexity of software systems these days has naturally led to explore automated ways to support developers’ code comprehension Letovsky (1987) from a linguistic perspective. One of these attempts is automatic summarization, which aims to generate a compact representation of the source code in a portion of natural language Haiduc et al. (2010).

While existing code summarization methods are able to provide relevant insights about the purpose and functional features of the code, their scope is inherently static. In contrast, software development can be seen as a sequence of incremental changes, intended to either generate a new functionality or to repair an existing one. Source code changes are critical for understanding program evolution, which motivated us to explore if it is possible to extend the notion of summarization to encode code changes into natural language representations, i.e., develop a model able to explain a source code level modification. With this, we envision a tool for developers that is able to i) ease the comprehension of the dynamics of the system, which could be useful for debugging and repairing purposes and ii) automate the documentation of source code changes.

To this end, we rely on the concept of code commit, the standard contribution procedure implemented in modern subversion systems Gousios et al. (2014), which provides both the actual change and a short explanatory paragraph. Our model consists of an encoder-decoder architecture which is trained on a set of triples conformed by the version of a system before and after the change, along with the comment. Given the high heterogeneity of the modalities involved, we rely on an attention mechanism to efficiently learn the parts of the sequences that are more expressive and have more explanatory power.

We performed an empirical study on twelve real world software systems, from which we obtained the commit activity to evaluate our model. Our experiments explored in-project and cross-project scenarios, and our results showed that the proposed model is able to generate semantically sound descriptions.

Related Work

The use natural language processing to support software engineering tasks has increased consistently over the years, mainly in terms of source code search, traceability and program feature location Panichella et al. (2013); Asuncion et al. (2010).

The emergence of unifying paradigms that explicitly relate programming and natural languages in distributional terms Hindle et al. (2012) and the availability of large corpus mainly from open source software opened the door for the use of language modeling for several tasks Raychev et al. (2015). Examples of this are approaches for learning program representations Mou et al. (2016), bug localization Huo et al. , API suggestion Gu et al. (2016) and code completion Raychev et al. (2014).

Source code summarization has received special attention, ranging from the use of information retrieval techniques to the addition of physiological features such as eye tracking Rodeghero et al. (2014). In recent years several representation learning approaches have been proposed, such as Allamanis et al. (2016), where the authors employ a convolutional architecture embedded inside an attention mechanism to learn an efficient mapping between source code tokens and natural language keywords.

More recently, Iyer et al. (2016) proposed a encoder-decoder model that learns to summarize from Stackoverflow data, which contains snippet of code along with descriptions. Both approaches share the use of attention mechanisms Bahdanau et al. (2014) to overcome the natural disparity between the modalities when finding relevant token alignments. Although we also use an attention mechanism, we differ from them in the sense we are targeting the changes in the code rather than the description of a file.

In terms of specifically working on code change summarization, Cortés-Coy et al. (2014); Linares-Vásquez et al. (2015) propose a method based on a set of rules that considers the type and impact of the changes, and Buse and Weimer (2010) combines summarization with symbolic execution. To the best of our knowledge, our approach represents the first attempt to generate natural language descriptions from code changes without the use of hand-crafted features, a desirable setting given the heterogeneity of the data involved.

Proposed Model

Concretely, similarly to Iyer et al. (2016), we use an attention-augmented encoder-decoder architecture. The encoder can be seen as a lookup layer, which simply reads through the source input sequence and returns the embedded tokens. The decoder is a RNN that reads this representation and generates NL words one at a time based on its current hidden state and guided by a global attention model Luong et al. (2015). We model the probability of a description as a product of the conditional next-word probabilities. More formally, for each NL token $n_{i}\in N_{t}$ we define,

where $E$ is the embedding matrix for NL tokens, $\propto$ denotes a softmax operation, $h_{i}$ represents the hidden state and $a_{i}$ is the contribution from the attention model on the source code. $W$, $W_{1}$ and $W_{2}$ are trainable combination matrices. The decoder repeats the recurrence until a fixed number of words or a special END token is generated. The attention contribution $a_{i}$ is defined as $a_{i}=\sum_{j=1}^{k}\alpha_{i,j}\cdot c_{j}F$, where $c_{j}\in C_{t}$ is a source code token, $F$ is the source code token embedding matrix and $\alpha_{i,j}$ is:

We use a dropout-regularized LSTM cell for the decoder Zaremba et al. (2015) and also add dropout at the NL embeddings and at the output softmax layer, to prevent over-fitting. We added special START and END tokens to our training sequences and replaced all tokens and output words occurring less than 2 and 3 times, respectively, with a special UNK token. We set the maximum code and NL length to be 100 tokens. For decoding, we approximate $N^{\star}$ by performing a beam search on the space of all possible summaries using the model output, with a beam size of 10 and a maximum summary length of 20 words.

To evaluate the quality of our generated descriptions we use both METEOR Lavie and Agarwal (2007) and sentence level BLEU-4 Papineni et al. (2002). Since the training objective does not directly optimize for these scores, we compute METEOR on our validation set after every epoch and save the intermediate model that gives the maximum score as the final model. For evaluation on our test set we used the BLEU-4 score.

Empirical Study

Data and pre-processing: We captured historical data from twelve open source projects hosted on Github based on their popularity and maturity, selecting 3 projects for each of the following languages: python, java, javascript and c++. For each project, we downloaded diff files and metadata of the full commit history. Diff files encode per-line differences between two files or sets of files in a standard format, allowing us to recover source code changes in each commit at the line level. On the other hand, medatada allows us to recover information such as the author and message of each commit.

The extracted commit messages were processed using the Penn Treebank tokenizer Marcus et al. (1993), which nicely deals with punctuation and other text marks. To obtain a source code representation of each commit, we parsed the diff files and used a lexer Brandl (2016) to tokenize their contents in a per-line fashion allowing us to maximize the amount of source code recovered from the diff files. Data and source code availablehttp://github.com/epochx/commitgen.

Experimental Setup: Given the flat structure of the diff file, source code in contiguous lines might not necessarily correspond to originally neighboring code lines. Moreover, they might come from different files in the project. To deal with this issue, we first worked only with those commits that modify a single file in the project; we call this the atomicity assumption. By using only atomic commits we reduced our training data by an average of roughly 50%, but in exchange we made sure all the extracted code lines came from the same file. At the same time, we expect to maximize the likelihood of observing a direct relation between the commit message and the lines altered.

We then relaxed our atomicity assumption and experimented with the full commit history. Given our maximum sequence length constrain of 100 tokens, we only observed an average of 1,97% extra data on each project. Since source code lines may come from different files, we added a delimiting token NEW_FILE when corresponding.

We were also interested in studying the performance of the model in a cross-project setting. Given the additional challenges that this involves, we designed a more controlled experiment. Starting from the atomic dataset, we selected commits that only add or only remove code lines, conforming a derived dataset that we call uni-action. We chose the python language to maximize the available data. See Table 1.

Results and Discussion: We begin by training our model on the atomic dataset. As baseline we used MOSES Koehn et al. (2007) which although is designed as a phrase-based machine translation system, was previously used by Iyer et al. (2016) to generate text from source code. Concretely, we treated the tokenized code snippet as the source language and the NL description as the target. We trained a 3-gram language model using KenLM Heafield et al. (2013) and used mGiza to obtain alignments. For validation, we use minimum error rate training Bertoldi et al. (2009); Och (2003) in our validation set.

As Table 3 shows, our model trained on atomic data outperforms the baseline in all but one project with an average gain of 5 BLEU points. In particular, we observe bigger gains for java projects such as CoreNLP and guava. We hypothesize this is because program differences in Java tend to be longer than the rest. While this impacts on training time, at the same time it allows the model to work with a larger vocabulary space. On the other hand, our model performs similarly to MOSES for the node and slightly worse for the youtube-dl. A detailed inspection of the NL messages for node showed that many of them exhibit a fixed pattern in their structure. We believe this rigidity restrains the generation capabilities of the decoder, making it more prone to memorization.

Table 2 shows examples of generated descriptions for real changes and their references. Results suggest that our model is able to generate semantically sound descriptions for the changes. We can also visualize the summarizing power of the model, as seen in the Theano and bitcoin examples. We observe a tendency to choose more general terms over too specific ones meanwhile also avoiding irrelevant words such as numbers or names. Results also suggest the emergence of rephrasing capabilities, specifically in the second example from Theano. Finally, our generated descriptions are, in most cases, semantically well correlated to the reference descriptions. We also report not so successful results, such as case of youtube-dl, where we can see signs of memorization on the generated descriptions.

Regarding the cross-project setting experiments on python, we obtained BLEU scores of 14.6 and 18.9 for only-adding and only-removing instances in the uni-action dataset, respectively. We also obtained validation accuracies up to 43.94%, suggesting feasibility in this more challenging scenario. Moreover, as the generated descriptions from the keras project in Table 2 show, the model is still able to generate semantically sound descriptions.

Despite the small data increase, we also trained our model on full datasets as a way to confirm the generative power of our model. In particular, we wanted to test the model is able leverage on atomic data to also capture and compress multi-file changes. As shown in Table 3, results in terms of BLEU and validation accuracy manifest reasonable consistency, despite the higher disparity between source code and natural language on this dataset, which means the model was able to learn representations with more compressive power.

Soft alignments derived from Figure 1, which shows examples of attention heatmaps, illustrate how the model effectively associates source code tokens with meaningful words.

Conclusion and Future work

We proposed an encoder-decoder model for automatically generating natural descriptions from source code changes. We believe our current results suggest that the idea is feasible and, if improved, could represent a contribution for the understanding of software evolution from a linguistic perspective. As future work, we will consider improving the model by allowing feature learning from richer inputs, such as abstract syntax trees and also functional data, such as execution traces.

References