Large Language Models for Software Engineering: A Systematic Literature Review

Introduction

In the field of language processing, traditional Language Models (LMs) have been foundational elements, establishing a basis for text generation and understanding (Moore and Lewis, 2010). Increased computational power, advanced machine learning techniques, and access to very large-scale data have led to a significant transition into the emergence of Large Language Models (LLMs) (Zan et al., 2023b; Zhao et al., 2023c). Equipped with expansive and diverse training data, these models have demonstrated an impressive ability to simulate human linguistic capabilities, leading to a sea of changes across multiple domains. With their capacity to learn from massive corpora and generate plausible text, LLMs are blurring the line between human and machine-produced language. They have provided researchers and engineers alike with a powerful tool to explore the complexity and richness of human communication, consequently sparking a transformational period in the field of language processing and beyond.

Software Engineering (SE) – a discipline focused on the development, implementation, and maintenance of software systems – is one of those areas reaping the benefits of the LLM revolution (Ma et al., 2023a). The utilization of LLMs in SE primarily emerges from an innovative perspective where numerous SE challenges can be effectively reframed into data, code, or text analysis tasks (Wang et al., 2022a). Using LLMs to address these SE tasks has shown a wealth of potential breakthroughs (Xia and Zhang, 2023b; Tian et al., 2023b; Xia and Zhang, 2023a; Lajkó et al., 2022; Charalambous et al., 2023; Sobania et al., 2023; Cao et al., 2023; Zhang et al., 2020a). The applicability of LLMs is particularly pronounced in tasks such as code summarization (Wan et al., 2018), which involves yielding an abstract natural language depiction of a code’s functionality, as well as the generation of well-structured code (Yin and Neubig, 2017) and code artifacts like annotations (Liang and Zhu, 2018). Codex, an LLM with 12 billion parameters, has demonstrated the ability to solve 72.31% of complex Python programming challenges posed by humans (Chen et al., 2021b). GPT-4 (OpenAI, 2023b), an LLM from OpenAI, has been used with a strong performance in several SE tasks, encompassing code writing, understanding, execution, and reasoning. It not only handles real-world applications and diverse coding challenges but also shows the ability to explain results in natural language and generate code from pseudocode (Bubeck et al., 2023).

Simultaneously, researchers have embarked on a series of research activities regarding LLM-related works, where a number of literature reviews or survey papers have been produced (Fan et al., 2023c; Chang et al., 2023; Fan et al., 2023b; Zhao et al., 2023c; Yang et al., 2023a). Table 1 summarises some of these. However, these related studies have limitations. They either focus narrowly on a single SE scope, such as the application of LLMs in software testing (Wang et al., 2023a) and natural-language-to-code (NL2Code) tasks (Zan et al., 2023b), or they are primarily centered on Machine Learning (ML) or Deep Learning (DL) models (Wang et al., 2022a; Yang et al., 2022b), overlooking more advanced and recently emerged LLM applications, such as ChatGPT (OpenAI, 2022a), which are increasingly finding applications within the SE field (Tian et al., 2023b; White et al., 2023b; Lubowitz, 2023; Sridhara et al., 2023). Alternatively, they merely offer a preliminary exploration of the performance of LLMs in various SE tasks through empirical experiments, without conducting a systematic literature survey (Zhao et al., 2023c; Ma et al., 2023a; Sridhara et al., 2023; Xu et al., 2022; Dou et al., 2023; Yuan et al., 2023a). The integration of LLMs within SE is undoubtedly a complex endeavor, requiring key considerations including the choice of the right model, comprehension of the unique features of different LLMs, devising pre-training and fine-tuning strategies, handling of data, evaluation of outcomes, and surmounting implementation challenges (Zan et al., 2023b). Despite the burgeoning interest and ongoing explorations in the field, a detailed and systematic review of LLMs’ application in SE has been notably absent in the current literature. This gap signifies a need for understanding the relationship between LLMs and SE. In response, our research aims to bridge this gap, providing valuable insights to the community.

In this paper, we conduct a systematic literature review on the utilization of LLMs in SE (LLM4SE). By mapping the current state-of-the-art, pinpointing the key strengths, weaknesses, and gaps in the existing LLM4SE literature, and proposing potential avenues for future research, our review aims to provide researchers and practitioners with a thorough guide to the convergence of LLMs and SE. We anticipate that our findings will be instrumental in guiding future inquiries and advancements in this rapidly evolving field. This work makes the following key contributions:

We are the first to present a comprehensive systematic literature review based on 229 papers published between 2017 and 2023 that focus on the use of LLM-based solutions to address SE challenges. We conducted a detailed analysis of the selected papers based on publication trends, distribution of publication venues, etc.

We have classified the LLMs utilized for the reported SE tasks and have provided a summary of the usage and trends of different LLM categories within the SE domain.

We describe the reported data processing stages, encompassing data collection, categorization, preprocessing, and representation.

We discuss optimizers used for LLM4SE tasks, including parameter and learning rate optimization, prevalent prompt engineering techniques, and commonly employed evaluation metrics.

We describe the key applications of LLM4SE encompassing a diverse range of 55 specific SE tasks, grouped into six core SE activities – requirements engineering, software design, software development, software quality assurance, software maintenance, and software management.

We have summarised key challenges that using LLMs encounters within the SE field and have suggested several potential research directions for LLM4SE.

Section 2 presents our research questions (RQs) and elaborates on our systematic literature review (SLR) methodology. The succeeding Sections 3 to 6 are devoted to answering each of these RQs individually. Section 7 discloses the potential threats to the validity of our study. Section 8 discusses the challenges yet to be overcome when employing LLMs to solve SE tasks and highlights promising opportunities and directions for future research. Section 9 concludes the whole paper.

Approach

This systematic literature review (SLR) follows the methodology proposed by Kitchenham et al. (Kitchenham et al., 2007, 2022), used in most other SE-related SLRs (Li et al., 2017; Wang et al., 2022a; Ramirez et al., 2018; Liu et al., 2022). Following the guidelines provided by Kitchenham et al., our methodology included three main steps: planning the review (i.e., Section 2.1, 2.2), conducting the review (i.e., Section 2.3, 2.4), and analyzing the basic review results (i.e, Section 2.5).

To provide a comprehensive overview of the LLM4SE field, it is important to fully comprehend how these models are currently being applied in SE, the challenges they face, and their potential future research directions in SE. Thus, we aim to provide a systematic literature review of the application of LLMs to software engineering. This study thus aims to answer the following research questions:

RQ1: What LLMs have been employed to date to solve SE tasks?

RQ2: How are SE-related datasets collected, preprocessed, and used in LLMs?

RQ3: What techniques are used to optimize and evaluate LLM4SE?

RQ4: What SE tasks have been effectively addressed to date using LLM4SE?

2. Search Strategy

As shown in Fig.1, we employed the “Quasi-Gold Standard” (QGS) (Zhang et al., 2011) approach for paper search. We conducted a manual search to identify a set of relevant studies and extracted a search string from them. This search string was then used to perform an automated search, and subsequently, a snowballing search was employed to further supplement the search results. This approach ensures both search efficiency and maximum coverage, minimizing the risk of omission. Subsequently, we employed a series of relatively strict filtering steps to obtain the most relevant studies. Specifically, we followed five steps to determine the relevance of the studies:

Select publication venues for manual search and select digital databases for automated search to ensure coverage of all the selected venues.

Establish QGS: Screen all papers for manual search and filter by inclusion/exclusion criteria (defined in Table 3).

Subjectively define the search string based on domain knowledge.

Conduct an automated search using the search string defined in Step (3).

Conduct snowballing search after performing study selection on the results of manual search and automated search.

During the manual search, we selected six of the top SE conferences and journals (i.e., ICSE, ESEC/FSE, ASE, ISSTA, TOSEM, and TSE, as shown in Table 2) and searched for papers that applied LLM4SE. We systematically crawled a list comprising 4,618 published papers from the top venues. Following automated scanning via scripts, we manually verified and identified 51 papers that were relevant to our research objectives. These 51 relevant papers formed the basis for constructing the Quasi-Gold Standard (QGS). Our search string should combine two sets of keywords: one pertaining to SE tasks, and the other related to LLMs. Only if the paper contains both types of keywords there is a higher probability that it is the paper we need. The complete set of search keywords is as follows:

Keywords related to SE tasks: Software Engineering, Software Development, Program*, Software Testing, Software Mainten*, SE, Software Lifecycle, Software Design*, Code representation, Code generation, Code comment generation, Code search, Code localization, Code completion, Code summarization, Method name generation, Bug detection, Bug localization, Vulnerability detection, Testing techniques, Test case generation, Program analysis, Bug classification, Defect prediction, Program repair, Code clone detection, Bug report, Software quality evaluation, SATD detection, Code smell detection, Compiled-related, Code review, Software classification, Code classification, Code change, Incident detection, Requirement extraction, Requirement traceability, Requirement validation, Effort cost prediction, Mining GitHub/Github mining, Mining SO (Stack Overflow)/SO mining, Mining app/App mining, Mining tag/Tag mining, Developer-based mining

Keywords related to LLMs: LLM, Large Language Model*, Language Model*, LM, PLM, Pre-trained, Pre-training, Natural Language Processing, NLP, Machine Learning, ML, Deep Learning, DL, Artificial Intelligence, AI, Transformer, BERT, Codex, GPT, T5, Sequence Model*, Attention Model*, Transfer Learning, Neural Network*, ChatGPT, GPT-*

It is important to note that the list of keywords related to LLMs that we set up includes Machine Learning, Deep Learning, and other such terms that do not seem to be necessarily related to LLMs. The reason for this is that we want to avoid omitting papers related to our research as much as possible, so the process of performing automated searches expands our search scope.

2.2. Search Datasets

After determining the search string, we conducted an automated search across seven widely used databases, which are capable of covering all published or latest papers. Given that the first paper about the Transformer architecture (Vaswani et al., 2017), which forms the basis for LLMs, was published in 2017, we focused our search on papers published from that year onwardThe cut-off date for the paper collection process of this version is August 1st, 2023.. Two authors independently performed the automated search, and the search results from each database were merged and deduplicated. Specifically, we obtained 1,089 papers from IEEE Xplore, 5,433 papers from the ACM Digital Library, 44,158 papers from ScienceDirect, 31,894 papers from Web of Science, 70,946 papers from Springer, 8,254 papers from arXiv, and 2,592 papers from DBLP.

3. Study Selection

Based on our search strategy, we initially obtained 164,366 papers that potentially relate to our research. Next, we needed to further evaluate the relevance of these papers based on inclusion and exclusion criteria, as shown in Table 3, so that the selected papers can directly address our research questions. The paper selection process, as illustrated in Fig. 1, consists of six phases.

In the first phase, we conducted automated filtering to exclude papers with less than 8 pages (Bashroush et al., 2017; Wang et al., 2022a) (Exclusion criteria 1), reducing the number of papers to 63,404. In the second phase, we examined the titles, abstracts, and keywords of the papers to identify those that include relevant LLM-related keywords. We then expanded the search scope to avoid missing relevant papers, including ML, DL, and other related keywords that may not directly correspond to LLM. The purpose of this phase is to narrow down the scope and filter out papers directly related to LLM (Inclusion criteria 1). Papers that are filtered out in this phase are then manually reviewed in the fifth phase. Additionally, we excluded 235 non-English written literature (Exclusion criteria 7). After the second phase, the number of papers was reduced to 4,341.

The third phase involves identifying the venues of the papers (Exclusion criteria 3). We extracted publication information such as “journal”, “URL”, “DOI”, and “series” to determine the publication sources. For papers from arXiv in 2022 and 2023, we chose to retain them, considering that this field is emerging and many works are in the process of submission. Although these papers did not undergo peer review, we have a quality assessment process to eliminate papers with low quality, ensuring the overall quality of this systematic literature review (SLR). This step resulted in 632 papers.

In the fourth phase, we merged and deduplicated the remaining papers from the seven databases and the manually searched paper list (Exclusion criteria 2), resulting in 548 papers. We then reviewed the full texts of the papers and excluded 176 papers that were grey publications or were published in workshops or doctoral symposiums (Exclusion criteria 4, 5, 6). By further assessing the quality of the papers, we identified 218 papers directly relevant to our research. This phase primarily involved excluding papers that mentioned LLMs but did not directly apply them, such as papers that only discussed LLMs in future work or focused on evaluating the performance of LLM-enabled tools (Wang et al., 2023a) (Exclusion criteria 8). For SLR, survey, and review papers, we have retained them and will assess their content during the quality assessment phase to determine their relevance to our research.

3.2. Study Quality Assessment

A well-crafted quality assessment can help to prevent biases introduced by low-quality studies and can indicate to readers where caution about conclusions should be drawn (Yang et al., 2021). We formulated ten Quality Assessment Criteria (QAC), as shown in Table 4. These aim to assess the relevance, clarity, validity, and significance of included papers. We used a scoring system of -1, 0, 1 (irrelevant/unmet, partially relevant/met, relevant/fully met). The first three questions were designed for the remaining 382 papers in the fifth stage. If QAC1, QAC2, or QAC3 received a score of -1, there is no need to proceed with QAC4-QAC10, and the paper can be excluded directly. QAC4-QAC10 involved assessing the content of the papers using a scoring system of 0, 1, 2, 3 (poor, fair, good, excellent). Finally, we calculated the total score of QAC4-QAC10 for each paper. For published papers, the maximum score for QAC4-QAC10 should be 21 (3 $\times$ 7). We retained papers with a score of 16.8 (21 $\times$ 0.8) or above. For unpublished papers on arXiv, the score for QAC4 is always 0, and the maximum score for QAC5-QAC10 should be 18 (3 $\times$ 6). We retained papers with a score of 14.4 (18 $\times$ 0.8) or above. After this quality assessment, we obtained a final set of 218 papers.

4. Snowballing Search

To identify any additional possibly relevant primary studies, we conducted a snowballing search. Snowballing refers to using the reference list of a paper or the citations to the paper to identify additional papers. Snowballing could benefit from not only looking at the reference lists and citations but also complementing them with a systematic way of looking at where papers are actually referenced and where papers are cited. Using the references and the citations respectively is referred to as backward and forward snowballing.

Before conducting snowballing, a set of initial papers needs to be prepared. In this study, the initial paper list consists of the remaining 218 papers after the quality assessment. We performed forward and backward snowballing, which resulted in the collection of 2,034 and 6,829 papers, respectively. After initial deduplication, we were left with 3,350 papers. We then conducted the full study selection process on these 3,350 papers, including deduplicating them with the 218 papers from performing snowballing on the initial list. As a result, we obtained an additional 11 papers.

5. Data Extraction and Analysis

We finally obtained 229 relevant research papers after searching and snowballing. Fig. 2 presents an overview of the distribution of the included papers. As shown in Fig. 2 (a), 38% of papers are published in peer-reviewed venues. ICSE is the most common of these venues, with a contribution of 9% of the total. Other venues with noteworthy contributions include TSE, ICSME, and SANER, contributing 5%, 3%, and 3% respectively. Meanwhile, the remaining 62% of papers are published on arXiv, an open-access platform that serves as a repository for scholarly articles. This finding is not surprising since much new LLM4SE research is rapidly emerging and thus many works are just completed and are likely in the peer review process. Despite the non-peer-reviewed nature of these papers, we have performed a rigorous quality assessment process on all collected papers, to ensure the quality of validity of our findings. This approach allows us to include all high-quality and relevant publications while maintaining high research standards.

Fig. 2 (b) shows the temporal distribution of the included papers. The number of publications has seen a rapidly growing trend since 2020. In 2020 and 2021, there are only 7 and 11 relevant papers, respectively. However, by 2022, the number of papers increases dramatically to 51. What’s surprising is that, in the first half of 2023 alone, the number of published papers has already reached 160. This rapid growth trend demonstrates that there is a growing research interest in the domain of LLM4SE.

In order to visualize the main content of our collection of papers, we generated a word cloud based on the abstracts of 229 papers as shown in Fig. 3. The most frequently occurring words include “code”, “LLM”, “task”, “generation”, “performance”, and “program”, clearly indicating the main themes explored in these papers. The term “code” emphasizes the core elements of software engineering, while “LLM” denotes the use of large language models in a variety of tasks. The terms “generation” and “task” emphasize the use of the LLM for automatic code generation and other SE tasks. In addition, “performance” reflects the evaluation and assessment of the effectiveness of LLM in SE applications. The word cloud provides further visual evidence that the literature we have collected is closely related to our research topic, which is to investigate the application of LLM in SE tasks.

We then conducted data extraction during the full-text review. This extraction phase collected all relevant data that would facilitate a comprehensive and insightful response to the RQs outlined in Section 2.1. As depicted in Table 5, we extracted data including the classification of SE tasks, their corresponding activities, as well as the category, characteristics, and applicability of the LLMs. With this collected data, we systematically analyzed the relevant aspects of LLM application in the SE domain.

RQ1: What LLMs have been employed to date to solve SE tasks?

Pre-trained language models (PLMs) have demonstrated impressive capabilities in solving various NLP tasks (Kojima et al., 2022; Shanahan, 2022; Wei et al., 2022b; Zhao et al., 2023c). Researchers have observed that scaling up the model sizes significantly enhances their capacity, leading to remarkable performance improvements when the parameter scale surpasses a certain threshold (Shanahan, 2022; Hoffmann et al., 2022; Taylor et al., 2022). The term “Large Language Model” (LLM) was introduced to distinguish language models based on their parameter size, specifically referring to large-sized PLMs (Zhao et al., 2023c). However, we note that the literature lacks a formal consensus on the minimum parameter scale for LLMs, as the model’s capacity is intertwined with both data size and total compute (Wang et al., 2023a). In this paper, we adopt the LLM scope division and taxonomy introduced by Pan et al.(Pan et al., 2023b) and categorize the mainstream LLMs investigated in this study into three groups according to their architectures: encoder-only, encoder-decoder, and decoder-only LLMs. This taxonomy and relevant models are shown in Fig. 4.

Encoder-only LLMs. Encoder-only LLMs are a type of neural network architecture that utilizes only the encoder component of the model (Devlin et al., 2018). The encoder’s function is to process and encode the input sentence into a hidden representation, capturing the relationships between words and the overall context of the sentence. Notable instances of encoder-only LLMs include BERT (Devlin et al., 2018) and its variants (Feng et al., 2020; Guo et al., 2020; Liu et al., 2019; Lan et al., 2019). As an example, BERT’s structure, based on the Transformer’s encoder architecture, has been referenced in 41 of the papers in this study. Its distinctive bidirectional attention mechanism simultaneously considers the left and right context of each word during training. In the SE domain, other prominent models like CodeBERT (Feng et al., 2020), GraphCodeBERT (Guo et al., 2020), RoBERTa (Liu et al., 2019), and ALBERT (Lan et al., 2019) have been widely employed. Specialized models such as BERTOverflow (Tabassum et al., 2020) and CodeRetriever (Li et al., 2022b) have been specifically developed for SE applications. These models’ innovations differ from BERT by leveraging the program structure, introducing new pre-training tasks, or engaging new modalities, thereby improving the architecture’s application to code-related tasks. For example, CodeBERT integrates a token prediction scheme to comprehend code by predicting subsequent tokens, enhancing its understanding of programming languages for tasks like code completion and bug detection (Feng et al., 2020). GraphCodeBERT introduces edge-type prediction, recognizing relationships between code elements as a graph. This enables GraphCoderBERT to leverage code structure, improving its effectiveness in tasks like code summarization and program analysis (Guo et al., 2020). These models have shown efficacy in tasks requiring a nuanced understanding of the entire sentence or code snippet. Examples include code review, bug report understanding, and named entity recognition pertaining to code entities (Pudari and Ernst, 2023; Sghaier and Sahraoui, 2023; Yang et al., 2022c; Arakelyan et al., 2023; Li et al., 2023c; Mukherjee and Hellendoorn, 2023).

Encoder-decoder LLMs. Encoder-decoder LLMs incorporate both encoder and decoder modules (Vaswani et al., 2017). The encoder ingests the input sentence and encodes it into a hidden space, effectively capturing the underlying structure and semantics. This hidden representation serves as an intermediary language, bridging the gap between diverse input and output formats. Conversely, the decoder utilizes this hidden space to generate the target output text, translating the abstract representation into concrete and contextually relevant expressions. Models such as PLBART (Ahmad et al., 2021), T5 (Raffel et al., 2020), and CodeT5 (Wang et al., 2021a) embody this architecture. Further advancements are evident in CodeT5+ (Wang et al., 2023c), while AlphaCode (Li et al., 2022a) and CoTexT (Phan et al., 2021) showcase the architecture’s adaptability to various SE tasks. The encoder-decoder design offers flexible training strategies and is proficient in handling multifaceted tasks such as summarization, translation, and question-answering. Within the field of SE, this ability has been successfully applied to tasks like code summarization (Al-Kaswan et al., 2023; Gu et al., 2022; Mastropaolo et al., 2021b). The encoder module’s capacity to understand and represent both the structure and semantics of code is pivotal, allowing the decoder to translate this comprehension into concise, human-readable summaries.

Decoder-only LLMs. Decoder-only LLMs exclusively utilize the decoder module to generate the target output text, following a distinct training paradigm that emphasizes sequential prediction (Radford et al., 2018). Unlike the encoder-decoder architecture, where the encoder processes input text, the decoder-only architecture begins with an initial state and predicts subsequent tokens, gradually building the output text. This approach relies heavily on the model’s ability to understand and anticipate language structure, syntax, and context. GPT-series models, such as GPT-1 (Radford et al., 2018), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), GPT-3.5 (OpenAI, 2022b), GPT-4 (OpenAI, 2023b), as well as their notable derivative, ChatGPT (OpenAI, 2022a)ChatGPT is a conversational agent built upon the GPT architecture, with GPT-3.5 and GPT-4 being specific versions of the architecture, each representing successive advancements., represent their major implementations. More specialized versions like CodeGPT (Lu et al., 2021), InstructGPT (Ouyang et al., 2022), Codex (Chen et al., 2021b), Copilot (GitHub, 2023)Copilot is an application built upon LLMs tailored for coding tasks. For convenience, all subsequent references in this paper to LLMs and their applications, such as ChatGPT and Copilot, will collectively be referred to as LLMs., and others have been fine-tuned for specific tasks in SE. Open-source models like GPT-J (Wang and Komatsuzaki, 2021), GPT-Neo (Black et al., 2021), GPT-NeoX (Black et al., 2022), LLaMA (Touvron et al., 2023a), and Vicuna (Chiang et al., 2023) also follow this architecture. These models can generally perform downstream tasks from a few examples or simple instructions without adding prediction heads or fine-tuning, making them valuable tools in SE research. The year 2022 marked a surge in the development of decoder-only LLMs, a trend that gained further momentum in 2023, notably with the launch of commercial products by leading Internet companies. For example, Google launched Bard (Google, 2023), Meta introduced LLaMA (Touvron et al., 2023a) and Llama 2 (Touvron et al., 2023b), Microsoft unveiled Bing Chat (Microsoft, 2023), etc. Contrary to LLMs such as GPT-4 and its derivative application, ChatGPT, released by OpenAI, which were promptly integrated into SE tasks, these new additions have not yet found widespread application within the SE field. Their potential remains largely unexplored, with opportunities for further assessment and utilization in specific tasks and challenges. The continued advancement of these models emphasizes the active exploration and innovation within decoder-only architectures.

2. Trend Analysis

As shown in Fig. 5, in the span from 2020 to 2023, the architecture of LLMs has witnessed notable shifts in preference and application within SE tasks. The specific choices between decoder-only, encoder-decoder, and encoder-only structures have shaped the direction of research and solutions in the SE domain (Wong et al., 2023). This analysis explores trends in the adoption of these architectures over the years, reflecting the evolving dynamics of LLM for SE tasks.

Evolution of LLM architectures in 2021. The year 2020 saw research papers predominantly concentrating on encoder-only LLMs for SE tasks, evidenced by a total of eight papers. Decoder-only LLMs or encoder-decoder LLMs were not featured in that year’s research. A marked change occurred in 2021. Out of 15 papers in 2021, five were dedicated to decoder-only LLMs, constituting 33.33% of the research. Additionally, three papers, or 20%, focused on encoder-decoder LLMs. Encoder-only LLMs witnessed a slight decline, representing 46.67% of the field with seven papers. This rapid transition can be linked to the generative capability of decoder-only LLMs. Researchers (Laskar et al., 2023; Sadik et al., 2023; Sridhara et al., 2023) found that these models, e.g., GPT series, requiring minimal fine-tuning, could produce not only syntactically correct but also functionally relevant code snippets. Their proficiency in grasping the context of code quickly made them a preferred choice.

Diversity of LLM architectures in 2022. 2022 experienced a significant increase in diversity, with more varied LLM architectures finding representation. Out of a total of 112 papers, 47 were centered around decoder-only LLMs, comprising 41.96% of the studies. Encoder-decoder LLMs made their presence known in 16 papers, accounting for 14.29%. Meanwhile, encoder-only LLMs led the field slightly with 49 papers, capturing 43.75% of the research interest. This diverse distribution suggests an exploration phase where researchers were actively assessing and leveraging different architectures to suit varied needs and challenges. The near-equal interest across different architectures underscores the field’s richness, indicating that no single approach had become the definitive choice.

Dominance of the decoder-only architecture in 2023. 2023 signaled a strong shift towards decoder-only LLMs. An impressive 258 instances of utilizing decoder-only LLMs were recorded across 138 unique papers, reflecting that a single paper might employ multiple such models. These papers focusing on decoder-only LLMs constituted a significant 73.09% of the total research this year. In comparison, encoder-decoder LLMs were the subject of 40 papers, contributing 11.33%, while encoder-only LLMs appeared to stabilize, with 55 papers, representing 15.58% of the 2023 research landscape. This trend signifies a shift in focus and resources toward exploring and harnessing the decoder-only architecture as the primary approach in many current and future LLM4SE research and applications.

Criteria for LLM selection in SE tasks. The selection of an LLM for SE tasks should involve careful consideration rather than arbitrary choice. Key factors guiding this selection encompass the model’s proficiency in understanding the context of code, its ability to generate relevant content, responsiveness to fine-tuning, and demonstrated performance on SE-specific benchmarks (Xie et al., 2023; Li et al., 2023b, a). Given the stringent syntactical rules and functional requirements inherent to SE tasks, models capable of seamlessly integrating these complex aspects were typically favored.

Task-specific fine-tuning. A notable trend is the customization of LLMs for precise SE tasks (Izadi et al., 2022; Li et al., 2023c; Zhang et al., 2022c). By fine-tuning models with datasets tailored to specific functions such as bug detection or code review, researchers were able to achieve marked performance improvements (Ciborowska and Damevski, 2023; Kou et al., 2023a).

In conclusion, the evolution of LLMs for SE, transitioning from encoder-only to decoder-only architectures, highlights the field’s vibrancy and adaptability. This shift has fundamentally altered the approach to SE tasks, reflecting the ongoing innovation within the discipline.

RQ2: How are SE-related datasets collected, preprocessed, and used in LLMs?

Data plays a crucial role in the model training phase (Sun et al., 2022). First, data is collected to obtain diversity and richness to ensure that the model can cope with different scenarios and situations. Second, data is classified to clarify the training objectives of the model and avoid confusion and misinformation. The preprocessing of data is indispensable to clean and transform the data to improve its quality. Finally, data is formatted into a structure suitable for model processing, allowing the LLM to effectively learn the data’s features and patterns. We analyze the reported processes of data collection, data classification, data preprocessing, and data representation in our selected primary studies on LLM4SE.

Data is an indispensable and critical factor in the training of LLMs, which determines the generalization ability, effectiveness, and performance of the models (Sun et al., 2022). Adequate, high-quality, and diverse data is critical to allow models to fully learn features and patterns, optimize parameters, and ensure reliability in validation and testing. We first investigate the methods used to obtain the dataset. By analyzing the methods of data collection, we divided the data sources into four categories: open-source datasets, collected datasets, constructed datasets, and industrial datasets. Open-source datasets (Chen et al., 2023c; Khakhar et al., 2023; Wang et al., 2023d; Zeng et al., 2022) refer to publicly accessible collections of data that are often disseminated through open-source platforms or repositories. For example, datasets like HumanEval (Chen et al., 2021b), which consists of 164 manually crafted Python problems, each accompanied by its respective unit tests. The open-source nature of these datasets ensures their credibility and allows for community-driven updates, making them a reliable resource for academic research. Collected datasets (Huang et al., 2018; Tian et al., 2023b; Sghaier and Sahraoui, 2023; Mastropaolo et al., 2022b) are those that researchers compile directly from a multitude of sources, including but not limited to, major websites, forums, blogs, and social media platforms. For instance, researchers (Chan et al., 2023; Salza et al., 2022; Weyssow et al., 2023; Yang et al., 2022c) often scrape data from Stack Overflow (Overflow, 2023) threads or GitHub (Github, 2023) issue comments to create a dataset tailored to their specific research questions. Constructed datasets (Ezzini et al., 2022; Koide et al., 2023; Kang et al., 2022; Zhang et al., 2022a) are specialized datasets that researchers create by modifying or augmenting collected datasets to better align with their specific research objectives. These modifications can be carried out through manual or semi-automatic methods and may include the generation of domain-specific test sets, annotated datasets, or synthetic data. For example, researchers often take a collected dataset of code snippets and manually annotate them with bug types to create a constructed dataset for studying automated program repair techniques (Fan et al., 2023a; Jin et al., 2023; Wu et al., 2023a). Industrial datasets (Alhamed and Storer, 2022; Moharil and Sharma, 2022; Wang et al., 2020c) are those obtained from commercial or industrial entities and often contain proprietary business data, user behavior logs, and other sensitive information. These datasets are particularly valuable for research that aims to address real-world business scenarios. However, the acquisition of such datasets is often complicated by issues related to business confidentiality and data privacy. For example, in a collaborative effort with China Merchants Bank (CMB), Wang et al. (Wang et al., 2020c) were able to access 21 projects from CMB’s repositories. Access to such data would likely require non-disclosure agreements and other legal safeguards to protect business interests. Each of these dataset types offers unique advantages and challenges, and the choice between them should be guided by the specific requirements and constraints of the research project at hand.

Fig. 6 shows the collection strategies of LLM-related datasets. As can be seen from the data in the figure, 127 studies used open-source datasets for training large models. One of the main reasons for using open-source datasets in LLM training is their authenticity and credibility. Open-source datasets usually contain real-world data collected from various sources (such as relevant studies that have been conducted), which makes them highly reliable and representative of real-world scenarios. This helps LLMs learn from real examples to better understand real-world applications and improve their performance. Second, since LLMs are a topic that has just recently emerged, a lack of suitable training sets does exist. Therefore, researchers often collect data from sites such as Stack Overflow and GitHub and build datasets to make the data more composite for SE tasks. Out of the 229 papers we studied, we found that only four of these studies were using industrial datasets. This suggests a potential misalignment between the properties of datasets used in academic research and those encountered in real-world industrial contexts. This divergence underscores the need for future research to investigate industrial datasets, thereby ensuring that LLMs are applicable and robust across both academic and industrial scenarios.

Note that some papers use multiple datasets that span different categories, e.g., Xu et al. (Xu et al., 2022) evaluated the performance of Codex, GPT-J, GPT-Neo, and other LLMs on SE tasks, and Mastropaolo et al. (Mastropaolo et al., 2021b) investigated the use of T5 in several code-related tasks such as fixing bugs and generating code comments. For different LLMs or different SE tasks, researchers may use different training datasets. On the other hand, some papers focus on exploring how existing LLMs (e.g., ChatGPT) are used in SE tasks (White et al., 2023b) and do not specify the dataset used for model training, as these LLMs like ChatGPT often do not require users to prepare training data themselves for general usage scenarios.

2. What types of SE datasets have been used in existing LLM4SE studies?

Data types play a pivotal role in shaping the architecture and selection of LLMs, as they directly influence the extraction of implicit features and subsequent model decisions(Chan et al., 2023; Ghadhab et al., 2021; Yang et al., 2023c; Shi et al., 2022). The choice of data types can significantly impact the overall performance and generalization ability of the LLMs. We examine and classify the types of SE datasets employed in LLM4SE studies. By investigating the relationship between data types, model architectures, and performance, we seek to shed light on the critical role of data types in the success of LLM4SE applications.

Data type categorization. We classified the data types of all datasets into five categories: code-based, text-based, graph-based, software repository-based, and combined data types. Table 6 describes the specific data included in the data types corresponding to the datasets we summarized from the 229 studies. We can find that most of the studies used text-based datasets, accounting for a total of 104. The dominance of text-based datasets in training LLMs for SE tasks highlights the models’ exceptional natural language processing capabilities. These LLMs excel in understanding and processing textual data, making them an ideal choice for tasks that involve code comprehension, bug fixing, code generation, and other text-oriented SE challenges. Their ability to process and learn from vast amounts of text data enables them to provide powerful insights and solutions for various SE applications. Text-based datasets with a large number of prompts (28) are commonly used in training LLMs for SE tasks to guide their behavior effectively. While understanding the training data may not be essential for closed-source LLMs like ChatGPT, insights into the data handling techniques of other models remain valuable. This is particularly true as black-box models can be fine-tuned with small-sized data inputs during usage. Among the 229 surveyed papers, this understanding is reinforced by the fact that text-based datasets with a large number of prompts are the most frequently used data types for training LLMs in SE tasks. programming problems (14) are also essential as they provide diverse and challenging tasks, allowing models to generalize knowledge and skills for various SE challenges. This combination helps the models develop a robust understanding of software concepts and perform well in a wide range of tasks. There are also SO (i.e., Stack Overflow) posts (9), bug reports (9), programming tasks (and solutions) (7), etc., which are among the more numerous data types in text-based datasets.

The predominance of source code (44) as the most abundant data type in code-based datasets can be attributed to its fundamental role in SE. Source code serves as the foundation of any software project, containing the logic and instructions that define the program’s behavior. Therefore, having a large volume of source code data is crucial for training LLMs to understand the intricacies of software development, enabling them to effectively generate, analyze, and comprehend code in various SE tasks. There are also common data types such as bugs (4) and patches (4) for program repair tasks. Graph-based datasets are used in some research studies for SE tasks, e.g., Kolthoff et al. (Kolthoff et al., 2023) used a dataset composed of screenshots from Google Play Android applications to construct a graphical user interface (GUI) repository in their study on LLM for the rapid prototyping task. These datasets represent code using graph structures, capturing relationships and dependencies between code components.

Software repository-based datasets usually mean data collected from software version control systems (e.g., Git) and issue tracking systems (e.g., GitHub, Jira, etc.). This data includes issues and commits (3), pull requests (2), and so on. The data in software repositories can provide a wealth of information covering all aspects of the software development process, including code evolution history, records of issue fixes and feature improvements, code quality assessments, and so on. These data are valuable for studying behaviors and trends in the software development process, improving software quality and development efficiency, and evaluating the performance of software engineering techniques. Therefore, many studies have used software repository-based datasets for empirical analysis and model training.

Some studies employed combined datasets containing multiple datatypes. Among them, the most common type is “source code and comments”. For instance, Tufano et al. (Tufano et al., 2022) used a dataset comprising source code and comments to train a model and showed that a pre-trained text-to-text converter (T5) model outperforms a previous deep learning model in automating the code review task. Other combinations of data types include “binary code and related annotations”, “failing test code and error messages”, “source code and Q&A pairs”, “source code, description, and code environment”, etc.

3. How do data types influence the selection of data-preprocessing techniques?

For the training and application of LLMs, the raw dataset needs to be subjected to data processing to obtain a clean and suitable dataset for model training. The data processing steps (Manh et al., 2023; Lee et al., 2022) involve operations such as data cleaning, noise removal, normalization, etc. To ensure consistency and quality of the data, different data types may require different processing methods to improve the performance and effectiveness of LLMs in SE tasks. In this section, we aim to detail the data preprocessing procedures for the two most used types of datasets, i.e., code-based datasets and text-based datasets.

The data preprocessing procedure for code-based datasets. We now summarize the process of preprocessing a code-based dataset, which consists of seven steps. Table 7 describes the individual data processing steps in detail and gives examples. The first step is data extraction, which involves retrieving relevant code segments from different sources such as software repositories or version control systems (Kang et al., 2023; Yang et al., 2023c). Depending on the requirements of the research task (Mastropaolo et al., 2021b; Yuan et al., 2023b), code segments can be extracted at different levels of granularity, ranging from individual methods and functions to entire source code files or even complete software projects. The next step is to remove any code segments that do not meet predefined criteria or quality standards (Li et al., 2021; Shi et al., 2022; Prenner and Robbes, 2021). This filtering process ensures that the extracted code is relevant to the specific SE task under study, thus eliminating incomplete or irrelevant code snippets. To avoid introducing bias and redundancy during model training, the third step involves removing duplicate instances (Zhao et al., 2021; Ciniselli et al., 2021; Xu et al., 2022). Any duplicate code instances are identified and removed from the dataset, thus increasing the diversity and uniqueness of the data. After the data extraction and filtering steps, the fourth step, data compilation, comes into play. The extracted and filtered code segments are merged and compiled into a unified code dataset. This compilation process simplifies data storage and access and facilitates subsequent analysis and model training (Chan et al., 2023; Mastropaolo et al., 2022a). In the fifth step, the problem of invalid or non-executable code is solved by removing data that cannot be compiled. Any code segments that cannot be compiled or executed are removed from the dataset to ensure that the remaining code instances are valid and usable during model training and evaluation. The sixth step is code representation, which consists of converting the code segments into a suitable representation that can be processed by the LLMs. This conversion can take different forms: token-based representation involves tokenizing the source or binary code into distinct tokens; tree-based representation parses the code into Abstract Syntax Trees (AST); and graph-based representation generates a Program Dependence Graph (PDG), encompassing Control Flow Graphs (CFG) and Call Graphs (CG). Finally, in the “data segmentation” step, the preprocessed dataset is partitioned into different subsets for training, validation, and testing (Ciniselli et al., 2021; Weyssow et al., 2023). The training set is used to train the LLM, the validation set helps to tune the hyperparameters and optimize the model performance, and the testing set evaluates the model’s ability on unseen data. By strictly adhering to these seven preprocessing steps, researchers can create structured and standardized code-based datasets, thus facilitating the effective application of LLMs for a variety of SE tasks such as code completion, error detection, and code summarization.

The data preprocessing procedure for text-based datasets. As displayed in Table 8, the steps of text-based dataset preprocessing consist of a total of seven steps, but there are some differences from the code-based dataset preprocessing steps. The process begins with data extraction (Yang et al., 2023c; Ciborowska and Damevski, 2022; Ezzini et al., 2022; Ciborowska and Damevski, 2023), where relevant text is carefully extracted from SE documentation from a variety of sources, including bug reports (Ciborowska and Damevski, 2023), requirements documents (Kolthoff et al., 2023), code comments (Prenner and Robbes, 2021), and API documentation (Khan et al., 2021). This step ensures that the dataset captures diverse, task-specific textual information. After data extraction, the text is initially segmented and categorized according to the specific requirements of the research task. For example, the text can be segmented into sentences or further broken down into individual words as needed for analysis (He et al., 2023; Kou et al., 2023a). To ensure the quality and relevance of the dataset, substandard data deletion is performed to eliminate any invalid or irrelevant text. For example, the dataset used by Lee et al. (Lee et al., 2022) was constructed from bug reports, and in the “unqualified data deletion” process the researchers filtered out bug reports with fewer than 15 words because the text was too short to contain contextual information. Next, preprocessing operations are performed on the text to standardize and clean it. Common preprocessing steps include removing certain symbols, stop words, and special characters (Rahmani et al., 2023; Wang et al., 2020c). This standardized form of text facilitates the efficient processing of LLMs. To avoid introducing bias and redundancy in the dataset, we eliminated duplicate instances by removing any duplicate text samples (Xu et al., 2022). This step enhances the diversity of the dataset and helps the model to generalize better to new inputs. “Data tokenization” is a key step in preparing the text for LLMs (Luo et al., 2022). Text is labeled into smaller units, such as words or subwords, so that LLMs are easier to manage and process efficiently. Finally, the preprocessed dataset is partitioned into different subsets, usually including a training set, a validation set, and a test set.

4. What input formats are the datasets for LLM training converted to?

Once suitable datasets have been carefully chosen and clean data has been achieved through the preprocessing steps, the next critical aspect is the transformation of the data into appropriate formats that can effectively serve as inputs for LLMs. Table 9 shows four distinct data input types that emerged during the research: Token-based input, Tree/Graph-based input, Pixel-based input, and Hybrid-based input.

Token-based input. Token-based input (Ahmed et al., 2023; Al-Kaswan et al., 2023; Alqarni and Azim, 2022; Arakelyan et al., 2023) involves representing code and text as sequences of tokens, which are smaller units like words or subwords. Code in tokens refers to the representation of code snippets broken down into meaningful tokens, allowing the LLMs to understand programming language syntax and semantics at a fine-grained level. Text in tokens refers to the tokenization of textual data, such as documentation, bug reports, or requirements, enabling the LLMs to process and analyze natural language descriptions effectively. Code and text in tokens combine both code and its associated textual context, allowing the model to capture the relationships between code elements and their descriptions.

Tree/Graph-based input. Tree-based input (Ma et al., 2023a; Ochs et al., 2023; Zhang et al., 2023c) represents code as hierarchical tree structures, capturing the syntactic relationships between code elements. Each node in the tree represents a code element, and the edges represent the hierarchical nesting of control flow statements and other code structures. This form of input allows the LLMs to understand the code’s hierarchical structure and perform tasks like code completion and bug fixing. Graph-based input represents code as a graph structure, where nodes represent code elements and edges represent the relationships between them. Unlike trees, graphs allow more flexible and complex relationships between code elements, enabling the model to capture non-linear dependencies in the code. This form of input is used in tasks like code summarization and vulnerability detection by considering the code’s intricate relationships.

Pixel-based input. Pixel-based input (Nasir et al., 2023) visualizes code as images, where each pixel represents a code element or token. This visual representation allows the LLMs to process and understand code through image-based learning. In this input form, LLMs learn from the visual patterns and structures in the code to perform tasks like code translation or generating code visualizations.

Hybrid-based input. Hybrid-based input (Niu et al., 2022) combines multiple modalities to provide LLMs with diverse perspectives for better code comprehension. For example, a hybrid input may combine code in tokens with visual representations of code, allowing the model to learn both from the fine-grained details in the tokenized code and from the overall visual structure of the code. This approach enhances the model’s ability to understand complex code patterns and improve performance in tasks such as code comprehension and code generation.

During our investigation of LLM-based models for SE tasks, we observed distinct trends in the usage of different input forms during the training process. Token-based input forms, namely code in tokens and text in tokens were the most prevalent, collectively constituting approximately 95.52% of the studiesThis refers to studies that explicitly state input forms of LLMs, i.e., a total of 201 papers as shown in Table 9.. Specifically, code in tokens was widely adopted in 64 studies, accounting for approximately 31.84% of the total studies, demonstrating its popularity as a primary choice for representing code snippets. This approach allowed LLMs to grasp programming language syntax and semantics effectively, making it suitable for a wide range of code-related tasks. Similarly, text in tokens was utilized in 99 studies, comprising around 49.25% of the total studies. This input form allowed LLMs to process natural language descriptions, bug reports, and documentation with greater efficiency and accuracy. The popularity of token-based input forms underscores their significance in leveraging the power of LLMs for software engineering applications.

In contrast, tree/graph-based input forms, such as code in tree-structure, were used in only seven studies, making up approximately 3.48% of the total. Although less prevalent, this input type emerged as a promising choice to represent the hierarchical structure and syntactic relationships within code. Its adoption indicated an ongoing exploration of tree-based representations in specialized tasks, such as code completion and bug fixing.

Pixel-based input and hybrid-based input were relatively less common, each found in one study, contributing approximately 0.5% of the total studies each. While their adoption rates were lower, these input forms presented intriguing possibilities for specific applications. Pixel-based input offered a unique visual representation of code, potentially advantageous for code translation tasks. Meanwhile, hybrid-based input, combining multiple modalities (e.g., code in tree structure and text in tokens in Niu et al.’s work (Niu et al., 2022)), showcased the potential for enhancing code comprehension tasks by offering diverse perspectives for the models to learn from.

In summary, the trends in input form usage reveal a strong preference for token-based input, demonstrating its versatility and effectiveness in various SE tasks. However, ongoing exploration of other input forms, such as tree/graph-based, pixel-based, and hybrid-based, suggests a dynamic and evolving landscape in the application of LLMs for SE, with potential for further innovation and improvement in specialized domains. Each of these input forms caters to specific characteristics of the SE tasks being addressed, enabling LLMs to perform effectively across a wide range of code-related applications with a more comprehensive understanding of the input data.

RQ3: What techniques are used to optimize and evaluate LLM4SE?

We examined the parameter and learning rate optimizers reported in our selected primary studies. As shown in Fig. 7 (a), fine-tuning emerges as the most widely used optimization algorithm in LLM studies, appearing in 87 research works (Niu et al., 2022; Thapa et al., 2022; Von der Mosel et al., 2022; Wang et al., 2023d; Zheng et al., 2023b; Al-Kaswan et al., 2023; Deng et al., 2023d; Döderlein et al., 2022; Jiang et al., 2023b). This signifies the dominance of fine-tuning in adapting pre-trained models to specific tasks, resulting in enhanced performance across various natural language processing tasks (Cassano et al., 2023; Chen et al., 2023c; Ciniselli et al., 2021). Hyperparameter optimization is another prominent approach, found in 55 studies (Ochs et al., 2023; Patil et al., 2023; Saieva et al., 2023; Tufano et al., 2022; Wan et al., 2022b), highlighting its significance in fine-tuning the hyperparameters of language models to achieve optimal performance on specific tasks (Zhang et al., 2023c; Zan et al., 2022b; Yang et al., 2022c). Of the data described above, 35 of these studies (Zhang et al., 2023c; Zhu et al., 2023) used both fine-tuning and hyperparameter parameter optimization algorithms. The limited occurrences of Bayesian (Prenner and Robbes, 2021) and Stochastic Gradient Descent (SGD) (Nasir et al., 2023) optimization (one study each) suggest that they are less frequently employed in LLM research.

Among the learning rate optimization algorithms illustrated in Fig. 7 (b), Adam stands out with 25 occurrences in the studies (Chen et al., 2023a; Ciborowska and Damevski, 2023; He et al., 2023; Huang et al., 2023b; Kolthoff et al., 2023; Kou et al., 2023a). Adam is an adaptive optimization algorithm that combines adaptive learning rates with momentum, facilitating faster convergence and reducing the risk of getting stuck in local minima during training (Mirzadeh et al., 2020). Similarly, AdamW appears in 21 studies (Deng et al., 2023c; Fatima et al., 2022; Fu and Tantithamthavorn, 2022; Hey et al., 2020; Khan et al., 2021), demonstrating its significance in improving generalization by adding weight decay regularization to Adam (Yuan et al., 2022). ZeRO (Hendrycks et al., 2021; Thapa et al., 2022) and Adafactor (Gupta et al., 2023; Ye et al., 2023) are relatively less explored, mentioned in three and two studies respectively. NVLAMB is found in one study (Von der Mosel et al., 2022), indicating limited exploration of this specific learning rate optimization algorithm in LLM research.

2. What prompt engineering techniques are applied to improve the performance of LLMs in SE tasks?

Large-scale pre-trained models have demonstrated effectiveness across numerous code intelligence tasks. These models are initially pre-trained on extensive unlabeled corpora and subsequently fine-tuned on downstream tasks. However, the disparity in input formats between pre-training (Ahmad et al., 2021) and downstream tasks (Devlin et al., 2018; Liu et al., 2016) poses challenges in fully harnessing the knowledge embedded in pre-trained models. Furthermore, the efficacy of fine-tuning (Kanade et al., 2020) strongly hinges on the volume of downstream data (Gu et al., 2021; Han et al., 2022; Lester et al., 2021), a circumstance frequently characterized by data scarcity (Feng et al., 2020; Guo et al., 2020; Wang et al., 2021a).

Recent research in the domain of Natural Language Processing (NLP) underscores that prompt engineering (Feng and Chen, 2023; Liu et al., 2023f), as an emerging fine-tuning (Lester et al., 2021; Li and Liang, 2021) paradigm, holds the potential to mitigate the aforementioned issues, yielding commendable outcomes across diverse NLP tasks. In the study conducted by Wang et al. (Wang et al., 2022d), they delve into the application of prompt engineering techniques to enhance the performance of LLMs in SE tasks. The research chiefly explores two prompt types: hard prompts (Gu et al., 2021; Han et al., 2022; Iyer et al., 2016) and soft prompts (Li and Liang, 2021; Han et al., 2022; Tsimpoukelli et al., 2021). Hard prompts entail manually predefined natural language instructions, while soft prompts, including plain soft prompts, replace natural language tokens in hard prompts with surrogate tokens. A variant known as prefix soft prompts adds a few surrogate tokens before the original input. In the context of prompt engineering, the task-specific knowledge imparted by inserted prompts is particularly advantageous for tasks characterized by data scarcity.

Prompt engineering, characterized by the careful design of specialized prompts, has become a fundamental technique for enhancing interactions with LLMs such as ChatGPT (Dong et al., 2023; White et al., 2023b) and WizardCoder (Luo et al., 2023). These tailored prompts serve dual purposes: they direct the LLMs toward generating specific outputs and also function as an interface to access the extensive knowledge embedded in these models. This becomes particularly important in scenarios where traditional datasets, such as those derived from software repositories or standard benchmarks, are either limited or lack the granularity required for specific tasks. In the context of LLM4SE — which encompasses a range of tasks such as code generation (Bareiß et al., 2022; Li et al., 2023c; Tan et al., 2023; Tian et al., 2020), code summarization (Gao et al., 2023; Ahmed et al., 2023), program repair (Huang et al., 2023c; Paul et al., 2023b; Xia and Zhang, 2023a; Yuan et al., 2022), and test generation (Siddiq et al., 2023b; Vikram et al., 2023; Xie et al., 2023) — the role of prompt engineering is indispensable. In summary, recent research highlights the critical role of prompt engineering in enhancing the performance of LLMs for targeted SE tasks, thereby contributing to the evolution of automated software development methodologies.

3. How are evaluation metrics utilized to assess the performance of LLM4SE tasks?

Evaluating the performance of LLM4SE is a crucial aspect of their development and deployment (Kang et al., 2022). Benchmarking against existing datasets and using baselines are common practices to evaluate the effectiveness of LLMs (Cassano et al., 2023). However, given the diversity of SE tasks, a single evaluation metric may not suffice to capture the model’s performance comprehensively. Thus, researchers often employ a range of evaluation metrics tailored to specific problem types (Mastropaolo et al., 2021b; Niu et al., 2022; Salza et al., 2022). We categorize the SE tasks summarized from 229 papers into four categories according to their addressed problem types, i.e., regression, classification, recommendation, and generation tasks, as displayed in Fig. 8 (b). The selection of evaluation metrics depends on the target problem types. For example, MAE (Mean Absolute Error) has been used for regression tasks (Fu and Tantithamthavorn, 2022). We summarize the most frequently used evaluation metrics for each task type.

For classification tasks, the most commonly used metrics are F1-score (Alhamed and Storer, 2022; Biswas et al., 2020; Chen et al., 2023a; Ezzini et al., 2022; Fatima et al., 2022; He et al., 2022), Precision (Biswas et al., 2020; Chen et al., 2023a; Ezzini et al., 2022; Fatima et al., 2022; He et al., 2022), and Recall (Biswas et al., 2020; Chen et al., 2023a; Ezzini et al., 2022; Fatima et al., 2022; He et al., 2022; Hey et al., 2020), with 21, 20, and 18 studies, respectively, employing these metrics. For example, in the study conducted by Khan et al. (Khan et al., 2021), F1-score is utilized to evaluate the performance of an automatic bug-fixing model. Similarly, Sharma et al. (Sharma et al., 2022) use Precision and Recall to assess the effectiveness of a transformer-based model for code summarization. These metrics are essential for evaluating the model’s ability to correctly classify code snippets (Fatima et al., 2022) or identify specific SE properties (Chen et al., 2023a).

For recommendation tasks, MRR (Mean Reciprocal Rank) is the most frequent metric, used in 9 studies (Ciborowska and Damevski, 2022; Izadi et al., 2022; Li et al., 2021; Lin et al., 2021; Rahmani et al., 2023; Salza et al., 2022; Shi et al., 2022; Wei et al., 2022a). MRR is employed to measure the effectiveness of recommendation systems for code completion, as demonstrated in the study by Ciborowska et al. (Ciborowska and Damevski, 2022). Precision@k (He et al., 2023; Ciborowska and Damevski, 2022; Lin et al., 2021; Zhu et al., 2023) and F1-score@k (He et al., 2023; Lin et al., 2021; Zhu et al., 2022, 2023) are also utilized in recommendation tasks, with 4 studies each. These metrics are used to evaluate the precision and F1-score of the recommended code snippets or code completions.

In generation tasks, metrics like BLEU, along with its variants BLEU-4 and BLEU-DC (Ahmed et al., 2023; Al-Kaswan et al., 2023; Arakelyan et al., 2023; Chen et al., 2022; Ciniselli et al., 2021), and Pass@k (Bui et al., 2023; Cassano et al., 2023; Chen et al., 2023c, 2021b; Dibia et al., 2022; Döderlein et al., 2022) are the most commonly used, appearing in 29 and 28 studies, respectively. For instance, Wang et al. (Wang et al., 2023c) employed BLEU to evaluate a code-to-code translation model. Pass@k is used in the research by Jiang et al. (Jiang et al., 2023a) to assess code generation models, measuring the proportion of generated code snippets that match the reference solutions. Additionally, ROUGE/ROUGE-L (Ahmed et al., 2023; Al-Kaswan et al., 2023; Gao et al., 2023; Geng et al., 2024; Li et al., 2022f; Mastropaolo et al., 2021b, 2022b; Niu et al., 2022; Zan et al., 2023a; Li et al., 2023a), METEOR (Ahmed et al., 2023; Al-Kaswan et al., 2023; Chen et al., 2022; Gao et al., 2023; Niu et al., 2022; Geng et al., 2024), EM (Exact Match) (Al-Kaswan et al., 2023; Gao et al., 2023; Gupta et al., 2023; Murali et al., 2023; Wang et al., 2023c; Weyssow et al., 2023; Ye et al., 2023; Zhang et al., 2023c), and ES (Edit Similarity) (Liu et al., 2023e) are used in specific studies to evaluate the quality and accuracy of generated code or natural language code descriptions.

RQ4: What SE tasks have been effectively addressed to date using LLM4SE?

In this section, we provide a detailed analysis of the use of LLMs in different SE tasks. We summarise reported SE tasks (Yang et al., 2022b) addressed with LLMs, following the six phases of the Software Development Life Cycle (SDLC) (i.e., requirements engineering, software design, software development, software quality assurance, software maintenance, and software management). Fig.8 (a) describes the distribution of LLMs in these six activities. Table 11 shows a detailed count of studies reporting specific SE tasks addressed with LLMs.

The highest number of studies is observed in the software development domain, constituting approximately 58.37% of the total research volume. This underscores the primary focus to date on utilizing LLMs to enhance coding and development processes. Software maintenance tasks account for about 24.89% of the research share, highlighting the significance of LLMs in aiding software updates and improvements. The software quality assurance domain holds approximately 10.3% of the research proportion, indicating a growing interest in automating testing procedures. In contrast, requirements engineering and software design activities represent approximately 4.72% and 1.29% of the research share, respectively, suggesting relatively limited exploration so far in these areas. The software management domain has the least research representation, accounting for a tiny 0.43% proportion. This distribution underscores the vital focus on development and maintenance tasks while also indicating potential avenues for further research in testing, design, and management domains.

In our collection of LLM studies for SE tasks, we’ve classified them based on the type of problems they address (shown in Fig.8 (b)). The distribution reveals that the majority of studies, about 64.34%, center around generation tasks, showcasing the significance of LLMs in producing code or text. Following this, around 24.48% of studies fall under classification tasks, indicating the relevance of LLMs in categorizing software elements. Additionally, roughly 9.79% of studies are related to recommendation tasks, demonstrating the utility of LLMs in suggesting solutions. Lastly, a smaller portion, around 1.4%, is allocated to regression tasks, reflecting the limited exploration of LLMs for predictive modeling. This distribution underscores the broad applicability of LLMs across different SE challenges, with a notable emphasis on code generation and classification tasks.

2. How are LLMs used in requirements engineering?

This section explores the utilization of LLMs in the domain of requirements engineering. It encompasses tasks such as anaphoric ambiguity treatment, requirements classification, coreference detection, requirements elicitation, and software traceability.

Anaphoric ambiguity treatment. Ambiguity in software requirements arises when a single reader can interpret a natural language (NL) requirement in multiple ways, or different readers have varying understandings of the same requirement. Unclear and ambiguous NL software requirements can lead to suboptimal software artifacts during later development stages. Moharil et al. (Moharil and Sharma, 2023) and Ezzini et al. (Ezzini et al., 2022) have empirically demonstrated the significant role of LLMs such as BERT and SpanBERT in effectively addressing anaphoric ambiguity. Sridhara et al. (Sridhara et al., 2023) revealed that ChatGPT excels in addressing anaphoric ambiguity in software requirements. Through researchers’ analysis of ten English requirement specifications (Ezzini et al., 2022) containing anaphora-related challenges, ChatGPT consistently demonstrated its remarkable capability to accurately identify antecedents. This empirical evidence emphasizes the valuable role ChatGPT can play in enhancing the clarity and precision of software requirements, ultimately contributing to more effective software development processes by reducing interpretational uncertainties.

Requirements classification. Originating in NL documents, requirements demand effective classification, especially for early-stage project discernment, like security-related ones (Knauss et al., 2011; Li et al., 2014). Automated processing hinges on identifying these requisites. Categorizing into functional (FR) or non-functional (NFR) requirements, with quality constraints, benefits automated approaches (Li et al., 2014). Hey et al.(Hey et al., 2020) employ BERT for requirement classification, where it excels in categorizing both FR and NFR requirements using a fine-tuning transfer learning technique, outstripping traditional methods. Luo et al.(Luo et al., 2022) introduce a BERT-based software requirement classification method, demonstrating remarkable transferability and generalization, especially in zero-shot scenarios.

Requirements term identification. Moharil et al. (Moharil and Sharma, 2022) propose a technique for identifying terms used in different contexts within the same domain or in interdisciplinary projects. Using BERT, which reads entire word sequences for deeper language understanding, and K-means clustering, they create and group vectors for each term in the corpora. The method has been validated on large Computer Science and multi-domain corpora comprising eight different fields.

Coreference detection. Requirements, authored by diverse stakeholders, continually evolve, leading to terminology differences and inconsistencies across domains. Entity coreference in Requirement Engineering (RE), where various expressions refer to the same real-world entity, can cause confusion and affect comprehensibility. Wang et al. (Wang et al., 2020c) offer a novel application of the BERT model for coreference detection.

Traceability automation. Software and system traceability refers to the ability to establish and maintain relationships between software artifacts, such as requirements, design definitions, code, and test cases, for product querying and development support (Rierson, 2017). Lin et al. (Lin et al., 2021) found that T-BERT can effectively migrate knowledge from code search to NLA-PLA (i.e., Natural Language Artifacts to Programming Language Artifacts) traceability, even with limited training instances. It outperforms existing techniques in accuracy and can be adapted to different domains without intermediate training for each project, offering a promising step toward practical, trustworthy traceability.

3. How are LLMs used in software design?

GUI (Graphical User Interface) retrieval. Kolthoff et al. (Kolthoff et al., 2023) present the application of BERT in the task of GUI retrieval in SE. The authors fine-tune a BERT-based learning-to-rank (LTR) model for this task. GUIs, which are not standard well-structured text documents, present unique challenges for text-based ranking tasks. The BERT model is prepared by concatenating the natural language query and the GUI document text, and then this input is used to train different BERT-LTR models. The models are evaluated based on their performance in NL-based GUI ranking.

Rapid prototyping. Rapid prototyping enables developers to quickly visualize and iterate on software designs, thereby accelerating the development process and ensuring alignment with user needs. White et al. (White et al., 2023b) investigate the role of LLMs in augmenting this process. The study introduces prompt design techniques, organized into patterns, providing a structured methodology to tackle prevalent challenges in LLM4SE. This research indicates that the realm of rapid prototyping stands to benefit from deeper integration with advanced machine learning techniques, thereby creating opportunities for additional research and refinement aimed at producing more intuitive and user-centric software designs.

Software specification synthesis. Software configuration is vital for system behavior, but managing configurations and specifications becomes complex with larger systems. Mandal et al. (Mandal et al., 2023) introduce SpecSyn, a framework using an LLM for automatic software specification synthesis from natural language sources. This end-to-end approach treats the task as a sequence-to-sequence learning problem, surpassing the previous state-of-the-art tool by 21% in F1 score, and can find specifications from both single and multiple sentences.

4. How are LLMs used in software development?

Our analysis identifies wide-ranging applications of LLMs for software development, encompassing tasks such as code generation, code completion, and code summarization.

Code generation. Code generation has long been a task of interest: there is extensive work on program synthesis using symbolic and neural-semiotic approaches (Alur et al., 2013; Wu et al., 2023b). Recently, LLMs trained for text generation have demonstrated the ability to complete programs (Brown et al., 2020; Black et al., 2022). Since 2020, several code generation models have been trained or fine-tuned on programming language text (Nijkamp et al., 2022b; Chen et al., 2021b; Fried et al., 2022; Xu et al., 2022; Feng et al., 2020; Clement et al., 2020). Unlike traditional program synthesis techniques, neurolinguistic models can be conditioned on natural language (e.g., code annotations) as well as generate programming language text. Researchers have experimentally demonstrated that LLMs like GPT-4 (Bareiß et al., 2022; Liu et al., 2023d; Jiang et al., 2023b; Gilbert et al., 2023), GPT-2/GPT-3/GPT-3.5 (Azaria et al., 2023; Yetiştiren et al., 2023; Ke et al., 2023; Liu et al., 2023d; Nascimento et al., 2023; Li et al., 2023b; Wang et al., 2023d; Liu et al., 2023a; Dong et al., 2023), BERT series (Zeng et al., 2022; Lai et al., 2023), Codex (Dibia et al., 2022; Bareiß et al., 2022; Yu et al., 2023; Chen et al., 2021b; Gupta et al., 2023; Madaan et al., 2022; Kuznia et al., 2022), CodeGen (Dibia et al., 2022; Jones and Steinhardt, 2022; Zan et al., 2022a), InCoder (Murali et al., 2023; Kou et al., 2023b; Liu et al., 2023d; Wang et al., 2022b), Copilot (Wu et al., 2023b) and CodeGeeX (Zheng et al., 2023b), play a key role in code generation. By pre-training on large-scale text data, these models learn rich linguistic knowledge and semantic representations that enable them to understand the meaning and structure of natural language. LLMs can automate code generation by converting natural language descriptions into code (Jiang et al., 2023a). These models generate program code from natural language descriptions, enhancing code-writing efficiency and accuracy. They show excellent performance in code completion, automatic code generation, and conversion of natural language annotations to code, providing software developers with powerful auxiliary tools and promoting further automation and intelligence in the code writing and development process.

Within the domain of LLMs applied to software development tasks, studies centered on code generation distinctly dominate the academic landscape. As reflected in Table 12, the GPT series, particularly GPT-4, emerge as a key focus, with many more studies using them in the realm of code generation (Dong et al., 2023; Du et al., 2023; Li et al., 2023b; Liu et al., 2023d). Analysing these studies, several noteworthy findings surface:

Programming thinking in LLMs. Techniques that evoke “programming thinking” within LLMs, such as the TIP (i.e., Thinking in Programming) (Li et al., 2023b) methodology, have shown promising strides. By guiding LLMs to first craft a high-level code sketch before delving into detailed implementations, the synthesized code exhibits higher accuracy and robustness.

Class-level vs. Method-level generation. LLMs, while adept at method-level code generation, present varied performance metrics when tasked with class-level generation (Du et al., 2023). This divergence underscores the evolving nature of challenges as the granularity of code synthesis shifts.

Expanding LLM capabilities. The next frontier in this discipline seems to lie in harmoniously integrating LLMs with established SE tools and practices. The emergence of frameworks like EvalPlus (Dong et al., 2023) indicates a trend towards enhancing the evaluation and accuracy of LLM-generated code, possibly ushering in an era where human developers and LLMs collaboratively craft software solutions.

Code completion. Code completion is an assistive feature provided by many integrated development environments (IDEs) and code editors. Its purpose is to automatically display possible code suggestions or options as developers write code (Amann et al., 2016). This innovation has been advanced by Language Models (LMs), evolving from n-gram and RNN models to transformer-based models like Copilot (GitHub, 2023) and CodeGPT (Judini, 2023), pre-trained on extensive code datasets. Recent LLMs equipped with billions of parameters, excel in generating code snippets. These models are trained on vast amounts of natural language text, equipping them with powerful semantic understanding capabilities. In the context of code completion, LLMs such as Codex (Li et al., 2022e; Pearce et al., 2021; Döderlein et al., 2022; Chen et al., 2021b), BERT series (Khan and Uddin, 2022), Github Copilot (Li et al., 2022e; Pudari and Ernst, 2023; Döderlein et al., 2022), CodeParrot (Li et al., 2022e; Xu et al., 2022), GPT series (Xu et al., 2022; Ochs et al., 2023), T5 (Ciniselli et al., 2021), InCoder (Fried et al., 2022), PolyCoder (Xu et al., 2022), CodeGen (Ding et al., 2023; Dinh et al., 2023; Li et al., 2022e; Nijkamp et al., 2022a), and other LLMs (Izadi et al., 2022; Ochs et al., 2023), can generate accurate and intelligent code suggestions based on code context and syntax structures. They comprehend the developer’s intent, predict the next possible code snippet, and provide appropriate recommendations based on the context.

With the support of LLMs, code completion achieves significant improvements in efficiency and accuracy. Developers can save time by avoiding manual input of lengthy code and reducing the risk of code errors. LLMs also learn from extensive code repositories, acquiring knowledge and best practices to offer more intelligent and precise suggestions, aiding developers in better understanding and utilizing code (Ciniselli et al., 2021). Additionally, these models can provide personalized code recommendations based on developers’ coding styles and preferences, further enhancing the effectiveness and user experience of code completion (Liu et al., 2023e).

Code summarization. Code summarization is a task that attempts to understand the code and automatically generate descriptions directly from the source code. It can also be viewed as an extended form of documentation. Successful code summarization not only facilitates the maintenance of source code (Iyer et al., 2016; Nguyen and Nguyen, 2017) but can also be used to improve the performance of code search using natural language queries (Nie et al., 2016; Yang et al., 2016) and code classification (Nguyen and Nguyen, 2017). LLMs play a significant role in code summarization by analyzing code structures and contexts to generate informative natural language summaries. Specifically, LLMs such as Codex (Gao et al., 2023; Arakelyan et al., 2023; Ahmed et al., 2023), CodeBERT (Chen et al., 2022; Gu et al., 2022; Gao et al., 2023), and T5 (Mastropaolo et al., 2021b, 2022b) comprehend the functionality and logic of the code, producing easily understandable human language descriptions. For example, Arakelyan et al. (Arakelyan et al., 2023) rigorously evaluate the efficacy of CodeT5 and Codex across code generation and summarization tasks, shedding light on their performance under distribution shifts. It unveils practical adaptation techniques, underscoring Codex’s commendable performance. Additionally, the study demonstrates that while adapted models exhibit proficiency in code generation, their generality can present trade-offs in the context of code summarization. As a result, code summarization with the support of LLMs enhances code readability, improves software documentation quality, and accelerates code comprehension and collaboration among developers. This advanced approach to code summarization demonstrates great potential for automating and streamlining various aspects of software development in modern SE practices with the employment of LLMs.

Code understanding. Code Understanding refers to the process of deeply comprehending and analyzing source code. It involves gaining insights into the logic, structure, functionality, and dependencies of the code (Shen et al., 2022), as well as understanding the programming languages, frameworks, and libraries used. LLMs can assist in code understanding by leveraging their powerful natural language processing capabilities to interpret code-related text, such as comments and documentation (Wang et al., 2023c; Kanade et al., 2020). They aid developers in grasping code functionality, identifying dependencies, and generating relevant code documentation (Shen et al., 2022; Ma et al., 2023a). Through their ability to comprehend both code and natural language, LLMs enhance the efficiency and accuracy of code understanding, empowering developers to maintain, optimize, and integrate code effectively (Kanade et al., 2020).

Code search. Code search, or code retrieval, is the task of retrieving source code from a large code base, usually based on a user’s natural language query. Despite the success of neural models in code search, such models are relatively shallow and are not capable of learning large amounts of data (Salza et al., 2022). In recent years, some bimodal pre-training models based on the BERT neural architecture have been proposed to capture semantic links between natural and programming languages (Feng et al., 2020; Guo et al., 2020; Roziere et al., 2021; Wang et al., 2021b), such as CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al., 2020). Bimodal pre-training models learn generic representations from large amounts of data in an unsupervised manner by designing pre-training goals. Salza et al. (Salza et al., 2022) explored the effectiveness of LLMs such as BERT (Salza et al., 2022) and RoBERTa (Chen et al., 2022) in understanding natural language and code semantics and enhancing code search and retrieval. These studies show that pre-training tasks alone may not be sufficient for code search, which emphasizes the need for a multimodal understanding of data (Shi et al., 2022), including both natural language and code. In addition, research has shown that the use of code generation models such as Codex (Li et al., 2022d) can enhance code retrieval by generating code snippets from natural language documents, thereby improving semantic similarity and obtaining state-of-the-art results on benchmark datasets.

Program synthesis. Program synthesis is the automated process of generating code that satisfies a given specification or set of constraints, emphasizing the derivation of functional properties of the code (Chen et al., 2017, 2021a; Manna and Waldinger, 1980; Srivastava et al., 2010; Parisotto et al., 2016). It differs from code generation, which primarily translates higher-level representations into target code without necessarily deriving its functionality from scratch (Siddiq et al., 2023a; Zhang et al., 2023e; Zheng et al., 2023b). Several studies have demonstrated that LLMs can be used for program synthesis tasks. LLMs have a significant impact on program synthesis due to their advanced language understanding and generation capabilities. LLMs can effectively interpret natural language descriptions, code comments, and requirements, and then generate corresponding code snippets that fulfill the given specifications. This helps developers in rapidly prototyping code and automating repetitive coding tasks (Kuznia et al., 2022; Gandhi et al., 2023). When applied to program synthesis, LLMs enhance productivity and reduce the burden on developers by automating the code-writing process based on high-level input (Jain et al., 2022). Their ability to understand the nuances of both natural language and programming languages makes them valuable tools in advancing the field of SE and streamlining the development lifecycle.

API recommendation. Several methods have been proposed to automate API (Application Programming Interface) recommendations (Gu et al., 2016; Huang et al., 2018; Liu et al., 2018; Nguyen et al., 2016), falling into two orthogonal approaches: information retrieval-based (IR-based) and neural-based. In this context, our focus is on the latter. Wei et al. (Wei et al., 2022a) introduced CLEAR, an API recommendation method that employs the BERT sentence embedding model to represent queries, capturing continuous semantic information. Through contrast training, CLEAR enables BERT to learn precise semantic representations of queries, independent of their lexical content. Recently, Zhang et al. (Zhang et al., 2023d) developed ToolCoder, which combines API search tools with existing models to aid in code generation and API selection. This approach involves an automated data annotation method using ChatGPT, adding tool usage information to the source code data, followed by fine-tuning the code generation model. During inference, an API search tool is integrated into the generation process, allowing the model to automatically utilize the tool for suggestions when selecting APIs.

API synthesis. The automated generation of application programming interface calls, known as API synthesis, plays a crucial role in bridging human intent with machine execution. In recent studies, Wang et al. (Wang et al., 2023b) and Patil et al. (Patil et al., 2023) have both explored the potential of LLMs in this realm. Utilizing models like GPT-4 and LLaMA-based architectures, these researchers showcase the prowess of LLMs in generating accurate API calls and adapting to real-time documentation changes, effectively addressing challenges like hallucination and inaccurate input arguments. The integration of LLMs in API synthesis signifies a paradigm shift, promising enhanced accuracy, adaptability, and reliability in code generation. As illuminated by these studies, the future of API synthesis may be deeply anchored in advanced machine learning, heralding new research avenues and refinements for more seamless human-machine interactions.

Code comment generation. Code comment generation, the automatic creation of comments for source code, serves to elucidate code functionality, implementation logic, and input-output details, thereby enhancing readability and maintainability (Geng et al., 2024). As code complexity grows, manually crafting these comprehensive and accurate comments can become burdensome and prone to errors. Automation in this domain can markedly enhance the efficiency and quality of code documentation. LLMs such as Codex (Geng et al., 2024) and T5 (Mastropaolo et al., 2021a) have been effectively applied to code comment generation. These models are pre-trained on vast amounts of data and possess powerful natural language processing and semantic understanding capabilities. During comment generation, LLMs analyze the structure, semantics, and context of the source code to automatically generate high-quality comments that correspond to the code’s functionality and logic. Addressing the often observed disconnect between code evolution and its accompanying documentation, Mastropaolo et al. (Mastropaolo et al., 2021a) explore the potential of LLMs, particularly the T5 architecture, in assisting developers with code comment completion. Their empirical study juxtaposes the performance of the T5 model against an n-gram model, revealing T5’s superior capabilities, though the n-gram model remains a competitive alternative. The research underscores the significance of open-source datasets for training and highlights the scant use of industrial datasets in current studies.

Code representation. Code representation learning (also known as code embedding) aims to encode the code semantics into distributed vector representations and plays a key role in recent deep-learning-based models for code intelligence. Code representation can be used to support a variety of downstream tasks, such as code completion (Raychev et al., 2014), code search (Gu et al., 2018; Wan et al., 2019), and code summarization (Wan et al., 2018; Zhang et al., 2020a). Niu et al. (Niu et al., 2022) propose a novel sequence-to-sequence pre-training model that utilizes structural information from source code to enhance its representation learning. The model is trained on a large corpus of source code, which enables it to capture the complex patterns and dependencies inherent in programming languages. Wan et al. (Wan et al., 2022b) show through their research that attention is highly consistent with the syntactic structure of the code, that pre-trained code language models can preserve the syntactic structure of the code in the intermediate representations of each converter layer, and that pre-trained code models have the ability to induce a syntactic tree of the code. These revelations suggest that incorporating the syntactic structure of the code into the pre-training process results in better code representations.

Method name generation. Method names significantly affect program comprehensibility, serving as a brief summary of the source code and indicating the developer’s intent (Ko et al., 2006). The importance of method names in program comprehension is further evidenced by recent studies showing that some programmers even write down important method names to help them figure out the procedures of an application (Roehm et al., 2012). Zhu et al. (Zhu et al., 2023) present AUMENA, a novel approach using the CodeT5 model for context-aware method naming in SE. AUMENA first learns the contextualized representation of programming and natural language, then leverages LLMs with prompt tuning to detect inconsistent method names and suggest accurate alternatives. This method avoids previous generate-then-compare consistency checking limitations, modeling the task as a two-class classification problem.

Agile story point estimation. Agile story point estimation, representing the total work needed to implement a product backlog item, is a complex task in agility. Story points are typically estimated by team consensus, using methods like plan poker and expert judgment, and considering factors like workload and complexity. However, subjective estimates may introduce uncertainty. Fu et al. (Fu and Tantithamthavorn, 2022) present GPT2SP, a Transformer-based approach that overcomes limitations of a previous method called Deep-SE. Unlike Deep-SE, which restricts language models to known words within a trained project, GPT2SP employs a broader context, making it transferable across projects. GPT2SP’s performance is comparable to Deep-SE in within-repository evaluations and surpasses it in 62.5% of cases, with improvements ranging from 3% to 46% across various projects.

API documentation smell detection. APIs, vital for modern software development, are often accompanied by official documentation. Good documentation is key to proper API use, while poor quality can hinder adoption and negatively impact developers’ productivity (Aghajani et al., 2020; Robillard, 2009; Robillard and DeLine, 2011). Khan et al. (Khan et al., 2021) identified five API documentation smells and presented a benchmark of 1,000 API documentation units containing the five smells found in the official API documentation. The authors developed classifiers to detect these odors, with BERT showing the best performance, demonstrating the potential of LLMs in automatically monitoring and warning about API documentation quality.

API entity and relation extraction. Extracting APIs and their semantic relationships from unstructured text (e.g., data from Stack Overflow) is a fundamental task in SE, but existing methods require labor-intensive manual rule creation or data labeling. Huang et al. (Huang et al., 2023b) present an innovative approach, AERJE, that leverages LLMs for this task. AERJE consists of a BERT-based dynamic hint generator and a T5-based joint entity-relationship extractor, which together enable efficient extraction of API entities and relationships without manual effort. The approach achieved an F1 score of 96.51% for API entity extraction and 81.2% for API relationship extraction, offering a significant advancement over traditional methods.

Code optimization. Efficiency in programming is vital, particularly in resource-limited or large-scale applications. Traditional optimizing compilers enhance efficiency through various considerations like algorithm and data structure selection (Aho et al., 2007). Madaan et al.(Madaan et al., 2023) explore the use of LLMs in suggesting performance-enhancing code edits. They curate a dataset of Performance-Improving Edits (PIE), showing how Codex and CodeGen can generate these edits, resulting in over 2.5x speedups for more than 25% of the C++ and Python programs, even after C++ code was compiled using the O3 optimization level.

Code example recommendation. Zhou et al. (Zhou et al., 2019) pointed out that software developers tend to write similar code examples several times due to the need to implement similar features in different projects. Therefore, during the software development process, recommender systems can provide programmers with the most pertinent and high-quality examples written by other programmers, thus helping them to complete their tasks quickly and efficiently (Di Rocco et al., 2021). Open-source projects and informal documentation are the two main sources of information that developers rely on to perform programming tasks. For example, open-source projects on GitHub provide code examples and code resources for various tasks. Rahmani et al. (Rahmani et al., 2023) introduce a methodology to improve code example recommendations for Java programming language on Stack Overflow using BERT and Query-Aware Locality-Sensitive Hashing (LSH). They employ BERT to convert code into numerical vectors and then apply two LSH variants, Random Hyperplane-based, and Query-Aware, to identify Approximate Nearest Neighbors (ANN).

Control flow graph generation. Control Flow Graphs (CFGs) are a cornerstone of SE that illustrate program behavior by showing sequences of statements and their execution order conditions (Allen, 1970). As a graphical representation of program behavior, CFGs are critical in many SE tasks, including code search (Guo et al., 2020; Chen et al., 2019b), code clone detection (Wang et al., 2020a; Hu et al., 2018; Wei and Li, 2017) and code classification (Wang et al., 2020b; Zhang et al., 2019). Huang et al. (Huang et al., 2023d) presented a novel approach for generating behaviorally correct CFGs of statically typed partial code by leveraging the error-tolerant and understanding ability of LLMs. The approach involves a Chain of Thoughts (CoT) with four steps: structure hierarchy extraction, nested code block extraction, CFG generation of nested code blocks, and fusion of all nested code blocks’ CFGs (Le-Cong et al., 2022). The CoT is broken down into an AI chain according to the single responsibility principle, along with effective prompt instructions. This results in superior node and edge coverage compared to traditional program analysis-based methods and the original CoT method.

Identifier normalization. Identifiers usually consist of multiple words, and a certain number of identifiers contain abbreviations (Jiang et al., 2020). Consequently, the lexical meaning of identifiers and the overall functionality of source code written by one developer may be challenging for other developers to comprehend. In addition, the source code cannot match the vocabulary in other software artifacts described in natural language, thus invalidating some automated algorithms. Therefore, there is a strong need to normalize identifiers with the aim of aligning the vocabulary in identifiers with the natural language vocabulary in other software artifacts. Zhang et al. (Zhang et al., 2022a) addressed this by introducing BEQAIN, an approach for identifier normalization. BEQAIN combines BERT with a Question and Answering (Q&A) system and Conditional Random Fields (CRF), treating identifier splitting as sequence labeling and abbreviation expansion as a Q&A task. It uses programming context to refine expansion results when multiple expansions are possible, aligning identifier vocabulary with natural language and enhancing software development comprehension and automation.

Type inference. Type inference, the automated process of determining data types in programming, plays a crucial role in enhancing readability, maintainability, and reducing runtime errors (Hellendoorn et al., 2018; Pierce and Turner, 2000). TypeScript, with its unique blend of optional typing, presents a nuanced challenge, especially when navigating the vast landscape of user-defined types. Addressing this complexity, Jesse et al. (Jesse et al., 2022) introduced an approach that leverages the capabilities of a BERT-style pre-trained model. Their solution, DIVERSETYPER, adeptly infers types for user-defined classes and interfaces by uniquely correlating class and interface declarations with their respective usage contexts. Beyond merely filling the gaps of previous methodologies, DIVERSETYPER sets a new benchmark in type inference, especially for user-defined types.

5. How are LLMs used in software quality assurance?

Within the domain of software quality assurance, LLMs have emerged as valuable tools with diverse applications for various tasks, including vulnerability detection, test generation, bug localization, etc.

Test generation. Test generation involves automating the process of creating test cases to evaluate the correctness and functionality of software applications. It encompasses various aspects, including test case generation (Zhang et al., 2023g), unit test generation (Tang et al., 2023a; Yuan et al., 2023b; Schäfer et al., 2023; Xie et al., 2023; Siddiq et al., 2023b), etc. LLM application in test generation offers several advantages, including the ability to automatically generate diverse test cases, improving test coverage (Schäfer et al., 2023; Siddiq et al., 2023b) and identifying potential defects (Xie et al., 2023). LLMs can also assist in generating test cases based on natural language descriptions, fostering better collaboration between developers and testers. Additionally, they help identify areas lacking test coverage and suggest relevant test cases, ensuring comprehensive testing and reducing the risk of undiscovered issues (Zhang et al., 2023g). By enhancing test efficiency and effectiveness, LLMs contribute to producing more reliable and high-quality software products.

Vulnerability detection. The number of software vulnerabilities is rapidly increasing, as shown by the vulnerability reports from Common Vulnerabilities and Exposures (CVEs) (Anon, 2022) in recent years. As the number of vulnerabilities increases, there will be more possibilities for cybersecurity attacks, which can cause serious economic and social harm. Therefore, vulnerability detection is crucial to ensure the security of software systems and protect social and economic stability. Traditional static detection methods are based on static analysis and predefined matching rules, which rely on developers’ expertise and make it difficult to detect unknown vulnerabilities. With the assistance of LLMs (Thapa et al., 2022; Chan et al., 2023; Chen et al., 2023a), Alqarni et al. (Alqarni and Azim, 2022) present an updated BERT model fine-tuned for vulnerability detection. Additionally, Tang et al. (Tang et al., 2023b) introduced novel approaches using LLMs to enhance vulnerability detection. One of their proposed models, CSGVD, combines sequence and graph embedding for function-level vulnerability detection, outperforming other deep learning-based models on a real-world benchmark dataset. Their study also explores the application of CodeT5 for vulnerability detection, highlighting the importance of code-specific pre-training tasks.

Test automation. Automated testing methodologies offer a comprehensive array of tools and strategies designed for the evaluation of software applications’ accuracy, reliability, and performance. These methodologies encompass various techniques, such as mutation testing (Khanfir et al., 2023) and fuzzing (Deng et al., 2023c, d). LLMs have been used for mutation testing, introducing faults to the codebase to assess the effectiveness of test suites in identifying and detecting errors (Khanfir et al., 2023). Furthermore, LLMs can aid in fuzzing, generating valid and diverse input programs that help identify vulnerabilities and bugs, particularly in challenging domains like deep learning libraries (Deng et al., 2023c). By incorporating LLMs into test techniques, software engineers benefit from improved test coverage, reduced manual effort, and enhanced bug detection (Deng et al., 2023d), leading to more robust and reliable software systems.

Verification. Verification techniques, including prominent methods such as formal verification, hold a pivotal role in the domain of software quality assurance (Charalambous et al., 2023; Tihanyi et al., 2023). These techniques validate the correctness of software systems, improving their reliability and security against potential threats. Utilizing mathematical and logical principles in the verification process facilitates thorough error detection and correction before deployment, ensuring stable and secure performance in different operational contexts. Charalambous et al. (Charalambous et al., 2023) leverage LLMs, particularly the GPT-3.5, in the realm of formal verification. Their approach combines LLMs with bounded model checking (BMC) to automatically repair software based on formal methods, showcasing the model’s capability to understand intricate software structures and generate accurate repairs.

Bug localization. Bug localization refers to the process of identifying the specific source code files, functions, or lines of code that are responsible for a reported bug or software defect. Bug localization typically involves analyzing bug reports or issue descriptions provided by users or testers and correlating them with the relevant portions of the source code. This process can be challenging, especially in large and complex software projects, where codebases can contain thousands or even millions of lines of code. Traditional bug localization methods often rely on heuristics, code metrics, or stack trace analysis, which may not always provide precise results. Ciborowska et al. (Ciborowska and Damevski, 2023) investigated data augmentation techniques to enhance bug localization models. They introduce a pipeline applying token-level operations such as dictionary replacement, insertion, random swapping, and deletion, along with paragraph-level back-translation to bug reports. By employing augmented data to train BERT-based models for bug localization, they demonstrate that these techniques can substantially expand the training data and boost the models’ performance.

Failure-inducing test identification. Test suites typically include two types of test cases: pass-through test cases and fault-inducing test cases (Li et al., 2023f). In practice, there are far more pass test cases for faults than fault-inducing test cases, which hinders the effectiveness of program debugging. However, in practice, it is difficult to find fault-inducing test cases. This is because developers first need to find test inputs that trigger program faults, and the search space for such test inputs is huge (Fraser et al., 2015). Moreover, developers need to build a test oracle to automatically detect program faults, and building a test oracle is often an undecidable problem (Ibrahimzada et al., 2022). Li et al. (Li et al., 2023f) investigated the application of ChatGPT to the task of finding fault-inducing test cases in SE. While recognizing ChatGPT’s potential, they initially observed suboptimal performance in pinpointing these cases, particularly when two versions of a program had similar syntax. The authors identified this as a weakness in ChatGPT’s ability to discern subtle code differences. To enhance its performance, they devised a novel approach blending ChatGPT with difference testing. Leveraging ChatGPT’s strength in inferring expected behavior from erroneous programs, they synthesized programs that amplified subtle code differences. The experimental results reveal that this approach greatly increases the probability of finding the correct fault-inducing test case.

Flaky test prediction. In many environments, it has been found that test cases can be non-deterministic, with test cases passing and failing in different executions, even for the same version of the source code. These test cases are called piecewise test cases (Zolfaghari et al., 2021; Luo et al., 2014; Eck et al., 2019; Fatima et al., 2022). Fatima et al. (Fatima et al., 2022) propose a black-box approach named Flakify that uses CodeBERT to predict flaky tests. The model is trained on a dataset of test cases labeled as flaky or non-flaky. The model’s predictions can help developers focus their debugging efforts on a subset of test cases that are most likely to be flaky, thereby reducing the cost of debugging in terms of both human effort and execution time.

6. How are LLMs used in software maintenance?

Within the context of software maintenance, LLMs have been leveraged for bug prediction, program repair, code review, debugging, and an array of other activities.

Program repair. The goal of automated program repair (APR) is to automatically identify and fix bugs or defects in software (Zhang et al., 2023c). It involves leveraging automated techniques to analyze buggy code and generate correct patches to address the identified issues. LLMs, such as BERT (Zhang et al., 2023a; Tian et al., 2023a), CodeBERT (Le-Cong et al., 2023), CodeT5 (Paul et al., 2023a), Codex (Fan et al., 2022; Jin et al., 2023; Wu et al., 2023a), PLBART (Paul et al., 2023a; Wu et al., 2023a), T5 (Yuan et al., 2022; Mastropaolo et al., 2022b) and GPT series (Xia and Zhang, 2023b; Tian et al., 2023b; Xia and Zhang, 2023a; Lajkó et al., 2022; Charalambous et al., 2023; Sobania et al., 2023; Cao et al., 2023), have shown effectiveness in generating syntactically correct and contextually relevant code. Leveraging LLMs for program repair can achieve competitive performance in generating patches for various types of bugs and defects (Xia and Zhang, 2023b). These models can effectively capture the underlying semantics and dependencies in the code (Charalambous et al., 2023), leading to the production of accurate and effective patches (Zhang et al., 2023a; Xia and Zhang, 2023a). Moreover, LLMs can be fine-tuned on specific code repair datasets (Mastropaolo et al., 2022b), further improving their ability to generate high-quality patches for real-world software projects. The application of LLMs in program repair not only accelerates the bug-fixing process but also enables software developers to focus on more complex tasks, leading to enhanced software reliability and maintainability.

In recent research, program repair has emerged as a prevalent application. Among the LLMs, as shown in Table 13, Codex (Wu et al., 2023a; Xia et al., 2023) and ChatGPT (Xia and Zhang, 2023a) have particularly distinguished themselves in the program repair domain. ChatGPT edges ahead due to its inherent interactive design, enabling a continuous feedback loop that yields refined and contextually apt patches (Xia and Zhang, 2023a, b). Such conversational dynamics, coupled with rigorous comparisons across diverse baselines, underscore its superior adaptability and efficiency.

Summarising several key findings from research on LLMs for program repair:

Interactive feedback. Incorporating an interactive feedback loop, as observed with ChatGPT, significantly augments the accuracy of program repair (Xia and Zhang, 2023a). This dynamic interplay between patch generation and validation fosters a deeper understanding of the software’s semantics, leading to more effective repairs.

Domain-specific integration. Merging the capabilities of LLMs with domain-specific knowledge and techniques further enhances their performance. Customized prompts, project-specific fine-tuning, and leveraging SE techniques (Xia et al., 2023; Wang et al., 2023a) can dramatically elevate the efficacy of LLM-driven program repairs.

Comparative analysis. Rigorous evaluation against diverse baselines reveals the versatility and adaptability of LLMs, especially ChatGPT. This wide-ranging comparison not only establishes their superiority but also underscores areas for potential improvement (Xia and Zhang, 2023b).

Code review. Code review is a critical quality assurance practice used to inspect, assess, and validate the quality and consistency of software code (Sghaier and Sahraoui, 2023). Code review aims to identify potential errors, vulnerabilities, and code quality issues, while also improving code maintainability, readability, and scalability. LLMs like BERT (Sghaier and Sahraoui, 2023), ChatGPT (Sridhara et al., 2023), and T5 (Tufano et al., 2022; Li et al., 2022f), trained on massive code repositories, possess the ability to understand and learn the semantics, structures, and contextual information of code (Zhang et al., 2022c). In the code review process, LLMs assist reviewers in comprehensively understanding code intent and implementation details, enabling more accurate detection of potential issues and errors. Moreover, these models can generate suggestions for code improvements and optimizations, providing valuable insights and guidance to reviewers. By combining the intelligence of LLMs with the expertise of human reviewers, code review becomes more efficient and precise, further enhancing software quality and reliability.

Debugging. Debugging targets identifying, locating, and resolving software defects or errors, commonly known as bugs. The debugging process involves scrutinizing the code, tracing the execution flow, and isolating the root cause of the problem to effectively correct the error. LLMs, such as BERT and other converter-based architectures, excel at utilizing contextual information and natural language understanding. In terms of debugging, LLMs can be used to simulate the scientific debugging process, such as AutoSD proposed by Kang et al. (Kang et al., 2023). This model generates hypotheses about code problems and extracts relevant values to identify potential problems. In addition, the SELF-DEBUGGING method proposed by Chen et al. (Chen et al., 2023b) enables LLM to debug its own generated code by learning a small number of presentations and explanations, which effectively improves the accuracy and sampling efficiency of code generation. Using LLMs in debugging not only improves fixing performance by generating competitive fixes but also provides insights into and explanations of the model’s decision-making process, making it an important tool for improving software quality and developer productivity.

Bug report analysis. LLMs such as Codex (Kang et al., 2022) and BERT (Ciborowska and Damevski, 2022) comprehensively analyze natural language text, code snippets, and contextual information within bug reports to generate precise code repair suggestions, test cases, or steps for reproducing errors. By deeply understanding the semantics and context of the issues, LLMs offer developers more intelligent solutions, expediting the error-fixing process and alleviating development burdens (Lee et al., 2022; Gomes et al., 2023). These models excel not only in code generation but also in identifying and interpreting crucial information within error reports, aiding developers in better comprehending the underlying causes (Li et al., 2022g). With the integration of LLMs, bug report analysis tasks are conducted more efficiently and accurately advancing optimization and enhancement of the maintenance workflow.

Code clone detection. Code clones are code samples that are identical to each other (Baxter et al., 1998; Karampatsis and Sutton, 2020). These code samples can have structural or semantic equivalence (Svajlenko et al., 2014). Sharma et al. (Sharma et al., 2022) investigate BERT’s application in code clone detection through an exploratory study. Analyzing BERT’s attention to code markers, they found that identifiers received higher attention, advocating their use in clone detection. This insight enhanced clone detection across all layers, and the implications extended beyond BERT. The researchers suggest that these findings could lead to the development of smaller models with performance akin to larger ones, thus mitigating computational accessibility issues.

Logging. Logging involves the systematic recording of events, messages, or information during the operation of a software application. It provides valuable information for understanding the behavior, performance, and potential problems of an application. Developers strategically insert logging statements throughout the code base to capture relevant data such as variable values, function calls, and error messages. These logs are an important tool for testing (Chen et al., 2018, 2019a), debugging (Satyanarayanan et al., 1992), monitoring (Harty et al., 2021; Hasselbring and van Hoorn, 2020), and analyzing the behavior of software operations, helping developers identify and diagnose bugs, performance bottlenecks, and other critical issues. Mastropaolo et al. (Mastropaolo et al., 2022b) introduce LANCE, a system for automatically generating and injecting full log statements into Java code using the T5 model. Sridhara et al. (Sridhara et al., 2023) present that ChatGPT performs well in the log summarization task, generating aggregated results that are better than the current state of the art.

Bug prediction. Gomes et al. (Gomes et al., 2023) conduct a BERT and TF-IDF (Term Frequency-Inverted Document Frequency) application for long-lived bug prediction in Free/Libre Open-Source Software (FLOSS) study to compare their accuracy in predicting long-lived errors. The results show that BERT-based feature extraction consistently outperforms TF-IDF, demonstrating BERT’s ability to capture the semantic context in error reports. In addition, smaller BERT architectures also show competitive results, highlighting the effectiveness of LLMs in bug prediction. This approach promises to enable more accurate error detection in FLOSS projects and improve software quality and maintenance.

Bug triage. Bug triage is pivotal for effective issue management in large projects. It entails prioritizing bugs and assigning appropriate developers for resolution. While bug triage is straightforward for smaller projects, scalability brings complexity. Finding the right developers with the needed skills becomes intricate as bugs vary in expertise requirements. Some even demand combined skills, amplifying the intricacy. Lee et al. (Lee et al., 2022) introduce the Light Bug Triage framework (LBT-P). This innovative approach employs BERT to extract semantic information from bug reports. To surmount challenges with LLMs in bug triage, the researchers employ techniques like model compression, knowledge preservation fine-tuning, and a new loss function.

Bug report replay. Bug reports are crucial for software maintenance, allowing users to inform developers of problems encountered while using the software. Therefore, researchers have invested significant resources in automating error playback to speed up the software maintenance process. The success of current automated approaches depends heavily on the characteristics and quality of error reports, as they are limited by manually created schemas and predefined vocabularies. Inspired by the success of the LLMs in natural language understanding, Feng et al. (Feng and Chen, 2023) propose AdbGPT, which utilizes natural language understanding and logical reasoning capabilities of the LLM to extract Steps to Reproduce (S2R) entities from bug reports and guide the bug replay process based on the current graphical user interface (GUI) state. The researchers describe how cue engineering, a small amount of learning, and thought chain reasoning can be utilized to leverage the knowledge of the LLM for automated error replay. This approach is significantly lightweight compared to traditional approaches, which utilize a single LLM to address both phases of S2R entity extraction and guided replay through novel hint engineering.

Duplicate bug report detection. In large software projects, multiple users may encounter and report the same or similar bugs independently, resulting in a proliferation of duplicate bug reports (Isotani et al., 2021). Duplicate bug report detection involves analyzing the textual content of bug reports and comparing them to find similarities and redundancies. LLM models, such as BERT (Isotani et al., 2021), ChatGPT (Sridhara et al., 2023), and other transformer-based architectures, are well-suited for natural language understanding and contextual representation. When applied to this task, LLMs can effectively capture the semantic similarities between bug reports, even in cases with slight variations in language or phrasing. The utilization of LLMs in this context not only enhances efficiency in managing bug reports but also contributes to improving the overall software development and maintenance workflow, reducing redundancy, and ensuring prompt bug resolution (Zhang et al., 2023b).

Decompilation. Decompilation is crucial in many security and SE tasks. For example, decompilation is often the first step in malware analysis (Nafisi, 2021), where human analysts examine malware code to understand its behavior. It is also important for binary vulnerability analysis (where analysts want to identify critical vulnerabilities in executables) (Dinesh et al., 2020; Nagy et al., 2021), software supply chain analysis (Hemel et al., 2011; Ombredanne, 2020), and code reuse (where legacy executables may need to be ported or hardened) (Ding et al., 2019; Marcelli et al., 2022; Pei et al., 2020). Decompilation tools, such as IDA and Ghidra, have been useful in security threat analysis (Nafisi and Lelli, 2021) proves its importance. Xu et al. (Xu et al., 2023) propose a new technique for recovering symbolic names during decompilation that leverages the synergy between LLMs (especially ChatGPT) and program analysis. The method employs an iterative algorithm to propagate ChatGPT query results based on program semantics. This propagation in turn provides better context for ChatGPT. The results show that 75% of the recovered names are perceived as good by the users and that the technique outperforms the state-of-the-art by 16.5% and 20.23% in terms of precision and recall, respectively.

Program merge conflicts repair. Program merge conflicts repair addresses the challenges faced when integrating individual code changes, which can lead to textual or semantic inconsistencies. Zhang et al. (Zhang et al., 2022b) explored the potential of using k-shot learning with LLMs like GPT-3 to automate this repair process. While these models showed promise in resolving semantic conflicts for Microsoft Edge, they didn’t fully replace the benefits of domain-specific languages for certain synthesis patterns.

Sentiment analysis. Sentiment analysis involves determining emotions in text data related to software products, such as user feedback or comments (Guzman et al., 2014; Jongeling et al., 2015; Islam and Zibran, 2017). The goal of sentiment analysis is to automatically classify the sentiment of the text as positive, negative, or neutral, providing valuable insights into how users perceive and react to software applications. Zhang et al. (Zhang et al., 2020b) conducted a study comparing pre-trained Transformer models like BERT, RoBERTa, XLNet, and ALBERT with existing SA4SE tools across six datasets. The results show that the Transformer models outperformed previous tools by 6.5% to 35.6% in macro/micro-averaged F1-scores, albeit with a trade-off in runtime efficiency. However, this accuracy boost comes with some runtime costs, indicating that while Transformer models are less efficient than existing SA4SE approaches, their runtime cost is not prohibitively high.

Tag recommendation. Improper tagging in software Q&A sites can lead to redundancy and other issues such as tag explosion. He et al. (He et al., 2022) introduced PTM4Tag, a framework utilizing PLMs with a triplet architecture to recommend tags for posts. By separately modeling the title, description, and code snippets of posts, PTM4Tag was compared using five popular PLMs, including BERT, CodeBERT, etc. The SE-specialized CodeBERT showed the best performance, notably surpassing CNN-based methods. An ablation study revealed that while the title was crucial in tag prediction, using all post components achieved the optimal result.

Vulnerability repair. Vulnerability repair is the process of identifying and fixing security holes or weaknesses in software applications. Pearce et al. (Pearce et al., 2021) investigate how to use LLMs for software zero-point vulnerability remediation. The authors explore the challenges faced in designing hints to induce LLMs to generate fixed versions of insecure code. It shows that while the approach is promising, with LLMs capable of fixing 100% of synthetic and hand-created scenarios, a qualitative assessment of the model’s performance on a corpus of historical real-life examples reveals challenges in generating functionally correct code. It is concluded that despite the potential for future targeted LLM applications in this area, challenges remain. For a complete end-to-end system, the full system needs to be evaluated in conjunction with error localization and an improved testbed.

Traceability recovery. Traceability recovery focuses on re-establishing lost or unclear connections between related software artifacts, thereby facilitating coherent software evolution and maintenance (Gethers et al., 2011). While traditional methods have offered some solutions, the integration of LLMs has recently emerged as a promising avenue for enhancing the accuracy and efficiency of this task. Zhu et al. (Zhu et al., 2022) present TRACEFUN, a traceability link recovery framework enhanced with unlabeled data, serves as a testament to this potential, leveraging LLMs to bridge the gap between labeled and unlabeled data, thereby refining traceability link predictions.

7. How are LLMs used in software management?

Research papers describing the utilization of LLMs in software management are still limited.

Effort estimation. Effort estimation refers to the process of predicting the amount of time, resources, and manpower required to complete a software development project. Alhamed et al. (Alhamed and Storer, 2022) conduct an evaluation of the application of BERT in the task of effort estimation for software maintenance. Their study underscores BERT’s potential to offer valuable insights and aid in the decision-making process while also highlighting the associated challenges and need for further investigation.

Threats to Validity

Paper search omission. One key limitation is the possibility of omitting relevant papers during the search process. When gathering papers related to LLM4SE tasks from various publishers, it is possible to miss some papers due to incomplete summarization of keywords for software engineering tasks or LLMs. To address this concern, we adopted a comprehensive approach, combining manual search, automated search, and snowballing techniques, to minimize the risk of missing relevant papers. For the manual search, we diligently searched for LLM papers related to SE tasks in six top-tier SE venues and extracted authoritative and comprehensive SE tasks and LLM keywords from these sources. With these numbered keyword search strings in place, we conducted automated searches on seven widely used publisher platforms. Additionally, to further augment our search results, we employed both forward and backward snowballing.

Study selection bias. Another limitation is the potential study selection bias. We established inclusion and exclusion criteria to perform the initial selection of papers, followed by manual verification based on quality assessment criteria (QAC). This process involves a combination of automated and manual procedures. The automated selection process may result in mislabeling of papers due to incomplete or ambiguous information in their corresponding BibTeX records. To mitigate this issue, any papers that cannot be confidently excluded are temporarily retained for manual verification. However, the manual verification stage could be influenced by the subjective judgment biases of the researchers, affecting the accuracy of the quality assessment of papers. To address these concerns, we invited two experienced reviewers in the fields of SE and LLM research to conduct a secondary review of the study selection results. This step aims to enhance the accuracy of our paper selection and minimize the likelihood of omission or misclassification. By implementing these measures, we strive to ensure that the selected papers are accurate and comprehensive, minimizing the impact of study selection bias and enhancing the reliability of our systematic literature review. We additionally provide a replication packagehttps://docs.google.com/spreadsheets/d/1iomMvoDL2znNDQ_J4aGnqb3BhZpEMlfz for others to view.

Challenges and Opportunities

Model size and deployment. The size of LLMs has seen a marked increase over time, moving from GPT-1’s 117M parameters to GPT-2’s 1.5B, and further to GPT-3’s 175B parameters (Yang et al., 2023a). The billions and even trillions (Moss, 2021) of parameters pose significant storage, memory, and computational challenges, which can hinder LLMs in resource-limited and real-time scenarios, especially when developers lack access to powerful GPUs or TPUs. CodeBERT (Feng et al., 2020), a pre-trained model proposed in 2019, has a total of 125M parameters, resulting in a large model size of 476 MB. Recently proposed models like Codex (Chen et al., 2021b) and CodeGen (Nijkamp et al., 2022a), have over 100 billion parameters and over 100 GB in size. The large sizes also require more computational resources. As pointed out by Hugging Face team (Bekman, 2022), training a 176B model (i.e., BLOOM (Scao et al., 2022)) on 1.5 TB datasets consumes an estimated 1,082,880 GPU hours. Similarly, the training of the GPT-NeoX-20B model (Black et al., 2022) on the Pile dataset (Gao et al., 2020), encompassing over 825 GiB of raw text data, requires the deployment of eight NVIDIA A100-SXM4-40GB GPUs. Each of these GPUs comes with a price tag of over 6,000 dollars (Amazon, 2023b), and the training extends to 1,830 hours or approximately 76 days. Moreover, even training a relatively smaller model like the PolyCoder (2.7B) (Xu et al., 2022), employing eight NVIDIA RTX 8000 GPUs on a single machine, demands a commitment of around 6 weeks. These examples illustrate the significant computational costs associated with training LLMs. These also have significant energy costs with predictions of massively increased energy usage by LLM-based platforms (Rillig et al., 2023). Fortunately, there are preliminary studies on reducing code models’ size and improving their efficiency. Shi et al. (Shi et al., 2023) use a genetic algorithm to compress CodeBERT into only 3 MB and reduce its response latency by more than 70%. Overall, the challenge of increasing model sizes and efficient deployment requires further attention from the communities.

Data dependency. In Section 4, we provide a detailed analysis of the datasets used in 229 studies and the data preprocessing process, finding that LLMs rely heavily on a large number of different datasets for training and fine-tuning, posing the data dependency challenge. The quality, diversity, and quantity of data directly affect the performance and generalizability of the models. Given their size, LLMs often require large amounts of data to capture nuances, but obtaining such data can be challenging. Relying on limited or biased datasets may cause the model to inherit these biases, resulting in biased or inaccurate predictions. In addition, the domain-specific data required for fine-tuning can be a bottleneck. Due to the relatively short period of time since the emergence of LLM, such large-scale datasets are still relatively rare, especially in the SE domain. Another issue is the risk of benchmark data contamination, where training and test data overlaps could lead to inflated performance metrics (Zhao et al., 2021). For instance, Brown et al. (Brown et al., 2020) discovered a code bug that prevented them from fully removing all overlapping data. They were unable to afford retraining and resorted to using “cleaned” variants of the benchmarks to mitigate the issue. Moreover, there are grave concerns around the inclusion of Personally Identifiable Information (PII) in pre-training corpora. Instances of PII, such as phone numbers and email addresses, have led to privacy leaks during the prompting process (Kulkarni, 2021; El-Mhamdi et al., 2023).

Ambiguity in code generation. Ambiguity in code generation poses a significant challenge for LLMs in SE tasks. When code intent is unclear (e.g., multiple valid solutions exist), LLMs may struggle to produce accurate and contextually appropriate code. This can lead to syntactically correct but functionally incorrect code, impacting the reliability and effectiveness of LLM-based code generation. Addressing this issue requires exploring techniques to incorporate additional context, domain-specific knowledge, or multi-model ensembles to improve LLMs’ ability to handle ambiguity and generate precise code, ensuring their successful integration into real-world software development processes.

1.2. Challenges in LLM Generalizability

The generalizability of LLMs refers to the ability of these models to consistently and accurately perform tasks in different tasks, datasets, or domains outside their training environment. While LLMs are trained on massive amounts of data, ensuring extensive knowledge capture, their performance is sometimes problematic when confronted with specific or idiosyncratic tasks outside the scope of their training. This challenge is particularly evident in the SE domain, where we present the application of LLMs to 55 SE tasks in Section 6. We observed that the context and semantics of code or documents vary greatly across projects, languages, or domains. Ensuring that the LLM generalizes well requires careful fine-tuning, validation on different datasets, and continuous feedback loops. Without these measures, models run the risk of over-adapting their training data, thus limiting their usefulness in a variety of real-world applications. Recent studies have shown that the LLMs cannot generalize their good performance to inputs after semantic-preserving transformations. For example, Yang et al. (Yang et al., 2022a) show that the performance of CodeBERT on different tasks decreases significantly after substituting the variables’ names in the input.

1.3. Challenges in LLM Evaluation

We summarized key evaluation metrics used in different types of SE tasks according to four task types: regression, classification, recommendation, and generation (Section 6). We found that when applying LLMs in the software engineering domain, the methodology for evaluating the performance of the models is usually based on a set of predefined metrics. Unfortunately, these metrics (e.g., Accuracy, Recall, or F1-score), while useful in some cases, may not fully capture all the effects and impacts of a model in a given SE task. For example, a model may perform well in terms of accuracy but may fail in processing specific types of inputs or in some specific situations. In addition, these metrics may not capture certain qualitative aspects of the model, such as its interpretability, robustness, or sensitivity to specific types of errors. Some of the most recent studies on LLM4SE tasks (Hu et al., 2023; Singla, 2023; Xu et al., 2023; Yuan et al., 2023a; Zhang et al., 2023e), in which researchers customized some evaluation metrics to assess the performance of models, also further illustrate the limitations of some of the widely used evaluation metrics in the field of LLM.

1.4. Challenges in LLM Interpretability, Trustworthiness, and Ethical Usage

Interpretability and trustworthiness are crucial aspects in the adoption of LLMs for SE tasks. The challenge lies in understanding the decision-making process of these models, as their black-box nature often makes it difficult to explain why or how a particular code snippet or recommendation is generated. Recent studies (Yang et al., 2023b; Wan et al., 2022a; Li et al., 2022c) also show that LLM of code trained on low-quality datasets can have vulnerabilities (e.g., generating insecure code). The lack of interpretability and trustworthiness can lead to uncertainty and hesitation among developers, who may be hesitant to rely on LLM-generated code without a clear understanding of how it was derived. Establishing trust in LLMs requires efforts to develop techniques and tools that provide insights into the model’s internal workings and enable developers to comprehend the reasoning behind the generated outputs. Enhancing interpretability and trustworthiness can ultimately promote the widespread adoption of LLMs in SE, leading to more efficient and effective development practices. Many LLMs are not open and it is unclear what data they have been trained on, both quality and representativeness but also ownership of the source training data. This brings into question ownership of the derivative data, e.g., generated designs, code, or test cases. There is also potential for various adversarial attacks e.g. deliberately seeding LLMs with code vulnerabilities so that automatically generated code snippets have subtle but vulnerable aspects.

2. Opportunities

The advent of code-specialized LLMs in SE. The recent emergence of code-specialized LLMs, such as GitHub Copilot (GitHub, 2023), Amazon’s CodeWhisperer (Amazon, 2023a), OpenAI Code Interpreter (OpenAI, 2023a) integrated into ChatGPT, and Code Llama (Meta, 2023) from Meta’s Llama family, signals a transformative phase in LLM4SE. These specialized LLMs, fine-tuned on code-specific datasets, are not merely incremental improvements but paradigm shifts in code understanding, generation, and efficiency. They offer new avenues for automated coding, personalized developer assistance, enhanced code review, and quality assurance, among other tasks, setting the stage for groundbreaking advancements in the SE domain.

Influence and applications of ChatGPT. ChatGPT’s popularity in recent academic research, as evidenced by its large presence in our 229 analyzed papers, emphasizes its escalating influence and acceptance within academia. Researchers’ preference for ChatGPT over other LLMs and LLM-based applications since its release can be attributed to its computational efficiency, adaptability to various tasks, and potential cost-effectiveness (Laskar et al., 2023; Li et al., 2023b; Xia and Zhang, 2023a). Its applications extend beyond mere code efficiency and debugging, fostering a collaborative era in development. This paradigm shift signifies a broader move towards integrating advanced natural language understanding into conventional coding practices (Laskar et al., 2023; Ma et al., 2023a; Sadik et al., 2023). By thoughtfully analyzing these dynamics and trends, we can foresee the potential pathways for LLMs and LLM applications like ChatGPT in shaping more robust, efficient, and collaborative software development procedures. Such insights stand as a promising indication of the future revolutionary impact of LLMs on SE.

Performance enhancement from task-specific model training. The choice between leveraging commercially available pre-trained models like GPT-4 and building upon open-source frameworks such as LLaMA (Touvron et al., 2023a), Llama 2 (Touvron et al., 2023b), and Alpaca (Alpaca, 2023) (fine-tuned from LLaMA 7B on 52K instruction-following demonstrations) provides a nuanced set of options for individual or organizational customization in specialized tasks. The distinction between these two approaches lies in the degree of control and customization. Pre-trained models like GPT-4 are generally not designed for large-scale retraining due to their proprietary nature, but they allow quick task-specific adaptations with limited data, thereby minimizing computational overhead. On the other hand, frameworks like LLaMA offer an open-source foundation for more extensive customization. While they come pre-trained, organizations often modify the source code and retrain these models on their own large-scale datasets to meet specialized requirements (ymcui, 2023; hiyouga, 2023). This process is computationally intensive, leading to greater resource allocation and cost, but affords the advantage of creating highly domain-specific models. Hence, the primary trade-off is between the ease of use and quick deployment offered by models like GPT-4, and the deep customization capabilities but higher computational demands associated with open-source frameworks like LLaMA.

Collaborative LLMs. From our review it is evident that LLMs have made significant strides in addressing various SE challenges. However, as the complexity of SE tasks continues to grow, there’s an emerging need for more sophisticated and tailored solutions. One promising direction is the concept of Collaborative LLMs. This approach involves integrating multiple LLMs (Dong et al., 2023; Zhao et al., 2023b) or combining LLMs with specialized machine-learning models (Ezzini et al., 2022; Zhang et al., 2022a) to enhance their efficacy for SE tasks. By harnessing the collective strengths of different models, we believe that the SE community can achieve more precise and efficient outcomes, from code completion to bug detection.

2.2. Expanding LLM’s NLP Capabilities in More SE Phases.

Integration of new input forms. In our analysis we observed that the predominant input forms were code-based datasets and text-based datasets. However, there was a noticeable scarcity of graph-based datasets (Kolthoff et al., 2023) (Section 4). Leveraging new input forms of natural language, such as spoken language, diagrams, and multimodal inputs, presents an opportunity to enhance the LLMs’ ability to understand and process diverse user requirements. Integrating spoken language could improve interactions between developers and models, enabling more natural and context-rich communication. Diagrams can facilitate visual representations of code and requirements, offering a complementary perspective for code generation. Furthermore, multimodal inputs that combine text, audio, and visual cues could offer a more comprehensive context understanding, leading to more accurate and contextually appropriate code generation. Additionally, exploring graph-based datasets could be crucial for addressing complex code scenarios, as graphs capture the structural relationships and dependencies in code, allowing LLMs to better comprehend code interactions and dependencies.

Widening LLM applications across SE phases. We observed a pronounced emphasis on the application of LLMs in software development and maintenance. These areas have undoubtedly benefited from the capabilities of LLMs, leading to enhanced code completion (Izadi et al., 2022; Li et al., 2022e; Liu et al., 2023e), bug detection (Ciborowska and Damevski, 2023; Feng and Chen, 2023; Kang et al., 2023), and other related tasks. The current application of LLMs in requirements engineering, software design, and software management remains relatively sparse. This presents a significant opportunity: by expanding the use of LLMs to these under-explored areas, we can potentially improve how requirements are elicited, how software designs are conceptualized, and how projects are managed.

2.3. Enhancing LLM’s Performance in Existing SE Tasks

Tackling domain-specific challenges. Many SE domains, including safety-critical systems and specific industries, suffer from a scarcity of open-source datasets, hindering the application of LLMs in these specialized areas. Future research can focus on creating domain-specific datasets and fine-tuning LLMs to cater to the unique challenges and intricacies of these fields (Biswas et al., 2020; Sun et al., 2023). Collaboration with domain experts and practitioners is vital to curate relevant data, and fine-tuning LLMs on this data can enhance their effectiveness and ensure better alignment with the specific requirements of each domain, paving the way for LLMs to address real-world challenges (Bubeck et al., 2023) in diverse software engineering domains (Li et al., 2023f).

Establishing a comprehensive evaluation framework for LLM4SE. The necessity for a universal, yet adaptable, evaluation framework for LLM4SE is pressing for both academic and industrial sectors. In academia, such a framework enables streamlined assessments of LLM performance, efficacy, and limitations, serving as a benchmark to verify the models’ practical readiness. On the industrial side, collaborations with real-world development teams using this framework yield empirical insights into LLMs’ utility, including their impacts on productivity, code quality, and team collaboration, while also revealing challenges like model biases, misinterpretation of code semantics, and context-specific limitations. Establishing this framework is critical for standardizing assessments and facilitating responsible LLM adoption in both academic research and practical applications (Biswas et al., 2020; Gong et al., 2023).

3. Roadmap

We provide a roadmap for future development in leveraging Large Language Models for Software Engineering (LLM4SE), with an additional high-level perspective that acknowledges the reciprocal relationship and emerging exploration of Software Engineering for Large Language Models (SE4LLM).

Automated coding, development and personalized developer assistance. The pursuit of automation in coding encompasses the auto-generation of code snippets, bug fixes, system optimization, and the creation of intelligent, personalized assistance for developers that is context-aware and adaptable to individual needs. LLM’s generative capabilities can be leveraged to help developers better understand requirements and generate syntactically and semantically correct code, thereby accelerating development cycles and improving software quality. Leveraging LLM’s natural language processing to develop context-aware tools allows for interaction with developers in a more intuitive and responsive manner. Additionally, fine-tuning LLMs for specific coding tasks and developer assistance can further enhance their accuracy and efficiency, customizing the automation process to suit the unique demands of different projects and individuals.

Advancing testing and analysis. The inclusion of LLMs in software testing methods opens up avenues for enhanced test case generation, bug classification, and defect prediction, thereby improving the precision and efficiency of the software testing process. For instance, LLMs show potential to be fine-tuned to a project’s specific requirements to generate customized test cases, which elevates the likelihood of early detection of subtle bugs or security vulnerabilities. Furthermore, the integration of LLMs with traditional SE techniques, including both static and dynamic program analysis presents a compelling direction for more rigorous code analysis. The potential for utilizing LLMs in formal analysis methodologies, including formal verification, is another area that merits investigation (Charalambous et al., 2023). These advancements not only facilitate the early discovery of complex errors but also lead to reduced development costs and quicker time-to-market, ultimately contributing to the robustness and reliability of the software products.

Integrating programming knowledge into LLMs. One critical future direction lies in the integration of specialized code representation methods and programming domain knowledge into LLM4SE (Wan et al., 2022b; Ma et al., 2023b). This integration aims to enhance the capability of LLMs to generate code that is not only functionally accurate but also secure and compliant with programming standards. Leveraging advanced techniques in code embedding, syntax tree parsing, and semantic analysis could significantly refine the generation capabilities of LLMs. Moreover, embedding domain-specific rules and best practices into these models would enable them to auto-generate code that adheres to industry or language-specific guidelines for security and style.

Enhanced code review and quality assurance. The transformation of the code review process can be supported by employing LLMs to analyze code context, perform intelligent comparisons, and offer insights that go beyond traditional automated review systems. The application of fine-tuned LLMs for code review can allow for more precise error detection and tailored feedback, offering a more nuanced understanding of code quality and potential improvements.

Extracting insights from data mining. LLMs can play a critical role in mining insights from platforms like GitHub, StackOverflow, and app stores. Through the application in tasks such as requirement extraction, traceability, validation, and various types of mining (tag, app, developer-based), LLMs can provide valuable insights that inform development strategies and decision-making. By automating and enhancing these mining tasks, LLMs contribute to a deeper understanding of user needs, emerging trends, and the efficiency of development practices.

Empowering predictive analytics and decision support. Leveraging LLMs for effort cost prediction, software classification, code classification, incident detection, and software quality evaluation may support better data-driven insights and predictive analytics. This empowers organizations to make informed decisions throughout the development lifecycle. LLMs’ ability to model and analyze vast amounts of data enables more accurate forecasts of project timelines, resource needs, and potential risks.

LLMs in software security. The growing impact of LLM4SE offers both unparalleled opportunities and challenges in the domain of software security. On the one hand, LLMs offer promising solutions for automated security audits, compliance verifications, and vulnerability detection. These models can potentially be leveraged for automated code reviews to ensure compliance with industry standards and legal regulations, while also identifying potential security vulnerabilities (Ferrag et al., 2023; Ahmad et al., 2023; Feng and Chen, 2023; Pearce et al., 2023; Deng et al., 2023b; Happe and Cito, 2023). For instance, Ferrag et al. (Ferrag et al., 2023) showcased the efficacy of LLMs in cyber reasoning tasks related to software security. On the other hand, the usage of LLMs introduces novel security concerns. Their complexity makes them susceptible to attacks, demanding novel strategies to fortify the models themselves (Wu et al., 2023c; Rao et al., 2023b; Elizondo, 2023; Ramly, 2023; Deng et al., 2023a; Liu et al., 2023b). As an example, Wu et al. (Wu et al., 2023c) delve into methods to secure LLMs against jailbreak attacks. An intriguing direction for future research lies in enabling LLMs to automatically identify and rectify their own vulnerabilities. Specifically, the focus could be on equipping LLMs to generate self-applied patches to their underlying code, thereby enhancing their inherent security, as opposed to merely implementing application-layer restrictions. Given this landscape, future research should adopt a balanced approach, aiming to exploit LLMs for automating and enhancing existing software security protocols while concurrently developing techniques to secure the LLMs themselves. This dual focus is crucial for fully realizing the potential of LLMs in enhancing the security and compliance assurance of software systems.

Software Engineering for Large Language Models (SE4LLM). As the capabilities and complexities of LLMs continue to expand, there arises a reciprocal need for specialized SE practices tailored for the development, optimization, and maintenance of these models. SE4LLM encompasses a range of challenges and opportunities, including the design of scalable and maintainable architectures, the creation of efficient training algorithms, the development of rigorous testing frameworks for model robustness and fairness, and the implementation of ethical guidelines and compliance mechanisms. The convergence of SE with LLMs not only facilitates the growth of more sophisticated and adaptable models but also opens up new avenues for interdisciplinary research and innovation, bringing together the expertise of both the AI and SE communities. This aligns with a broader vision where SE practices become an integral part of the lifecycle of LLMs, ensuring their robustness, efficiency, and ethical alignment with societal values.

Conclusion

LLMs are bringing significant changes to the field of SE. The potential of these models to handle complex tasks can fundamentally reshape many SE practices and tools. In this systematic literature review, we analyzed the emerging utilization of LLMs for software engineering, encompassing papers published since the inception of the first LLM (BERT). We examined the diverse LLMs that have been employed in SE tasks and explored their distinct features and applications (RQ1). We then investigated the processes involved in data collection, preprocessing, and usage, emphasizing the significant role well-curated datasets play in the successful application of LLMs to solve SE tasks (RQ2). Following this, we investigated the various strategies utilized to optimize and assess the performance of LLMs for SE tasks (RQ3). Lastly, we reviewed the wide range of SE tasks where LLMs have been applied to date, shedding light on the practical contributions LLMs have made (RQ4). We summarised some key existing challenges of LLM4SE and provided a research roadmap, outlining promising future research directions.

References