No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation

I Introduction

Unit testing validates whether a functionally-discrete program unit (e.g., a method) under test behaves correctly. As the primary stage in the software development procedure, unit testing plays an essential role in detecting and diagnosing bugs in a nascent stage and prevents their further propagation in the development cycle. Therefore, writing high-quality unit tests is crucial for ensuring software quality.

For a method under test (i.e., often called as the focal method), its corresponding unit test consists of a test prefix and a test oracle . In particular, the test prefix is typically a series of method invocation statements or assignment statements, which aim at driving the focal method to a testable state; and then the test oracle serves as the specification to check whether the current behavior of the focal method satisfies the expected one. For example, the assertion is one of the common test oracles in unit tests.

Manually writing and maintaining high-quality unit tests can be very time-consuming and laborious . To alleviate manual efforts in writing unit tests, researchers have proposed various techniques to facilitate automated unit test generation. Traditional unit test generation techniques leverage search-based , constraint-based , or random-based strategies to generate a suite of unit tests with the main goal of maximizing the coverage in the software under test. Although achieving reasonable coverage, these automatically-generated tests exhibit a large gap to manual-written ones in terms of readability and meaningfulness, and thus developers are mostly unwilling to directly adopt them in practice .

To address these issues, recent work has leveraged advanced deep learning (DL) techniques, especially large language models (LLMs), to generate unit tests. These techniques mostly formulate unit test generation as a neural machine translation problem by translating a given focal method into the corresponding test prefix and the test assertion. In particular, they incorporate the power of LLMs by fine-tuning these pre-trained models on the test generation task. Owing to being extensively pre-trained on a massive developer-written code corpus and then being specifically fine-tuned on the test generation task, these models are capable of generating more human-like and meaningful test code, showing a large potential of LLMs in unit test generation.

ChatGPT , a very recent LLM developed by OpenAI based on the generative pre-trained transformer architecture, has attracted wide attention by showing the outstanding capability of solving various tasks. Different from the LLMs (e.g., BART , BERT , and T5 ) used in existing learning-based test generation techniques , ChatGPT further incorporates RLHF (Reinforcement Learning from Human Feedback) and a significantly-larger model scale, which thus exhibits better generalization and higher alignment with human intention in various domains. However, it still remains unclear how effective ChatGPT is in generating unit tests.

Study. In this work, we perform the first empirical study to evaluate ChatGPT’s capability of unit test generation. We first construct a dataset of 1,000 Java focal methods, each along with a complete and executable project environment. In addition, based on the common practice in previous test generation work and widely-acknowledged ChatGPT-relevant experience, we design a basic prompt including both (i) a natural language description about the task and (ii) a code context of the focal method and other relevant contexts. We then query the ChatGPT API with our basic prompt, and analyze the quality of the returned tests to answer the following four research questions.

RQ1 (Correctness): How is the correctness of the unit tests generated by ChatGPT? What are the common errors in the incorrect tests? We first measure the syntactic correctness, compilation correctness, and execution correctness of the generated tests; and then further build a breakdown of the error types in the incorrect tests.

RQ2 (Sufficiency): How is the sufficiency of the unit tests generated by ChatGPT? For those correct tests generated by ChatGPT, we investigate the adequacy of coverage and assertions.

RQ3 (Readability): How is the readability of the unit tests generated by ChatGPT? For those correct tests generated by ChatGPT, we perform a user study to assess their readability along with the manually-written tests as reference.

RQ4 (Usability): How can the tests generated by ChatGPT be used by developers? For those correct tests generated by ChatGPT, we perform a user study to investigate whether developers are willing to directly adopt them.

Based on our results, we have the following main findings. On the bad side, we find that only a portion (24.8%) of tests generated by ChatGPT can pass the execution and the remaining tests suffer from diverse correctness issues. In particular, 57.9% ChatGPT-generated tests encounter compilation errors, such as using undefined symbols, violating type constraints, or accessing private fields; and 17.3% tests are compilable but fail during execution, which mostly results from the incorrect assertions generated by ChatGPT. On the good side, we find the passing tests generated by ChatGPT actually resemble manually-written ones by achieving comparable coverage, readability, and sometimes even developers’ preference compared to manually-written ones. Overall, our results indicate that ChatGPT-based test generation could be very promising if the correctness issues in the generated tests could be further addressed. To this end, we further distill two potential guidelines, i.e., providing ChatGPT with deep knowledge about the code and helping ChatGPT better understand the intention of the focal method, so as to reduce the compilation errors and assertion errors, respectively.

Technique. Inspired by our findings above, we further propose ChatTester, a novel ChatGPT-based unit test generation approach, which leverages ChatGPT itself to improve the correctness of its generated tests. ChatTester includes an initial test generator and an iterative test refiner. The initial test generator decomposes the test generation task into two sub-tasks by (i) first leveraging ChatGPT to understand the focal method via the intention prompt and (ii) then leveraging ChatGPT to generate a test for the focal method along with the generated intention via the generation prompt. The iterative test refiner then iteratively fixes the compilation errors in the tests generated by the initial test generator, which follows a validate-and-fix paradigm to prompt ChatGPT based on the compilation error messages and additional code context.

To evaluate the effectiveness of ChatTester, we further apply ChatTester on an evaluation dataset of 100 additional focal methods (to avoid using the same dataset that has been extensively analyzed in our study part), and compare the tests generated by ChatTester with the default ChatGPT to answer the following research question.

RQ5 (Improvement): How effective is ChatTester in generating correct tests compared to ChatGPT? How effective is each component in ChatTester? We compare the number of compilation errors and execution failures between the tests generated by ChatTester and the default ChatGPT. Moreover, we further investigate the contribution of each component in ChatTester.

Our results show that ChatTester substantially improves the correctness of the ChatGPT-generated tests with 34.3% and 18.7% improvement in terms of the compilable rate and execution passing rate. In addition, our results further confirm the contribution of both components in ChatTester, i.e., the initial test generator is capable to generate more tests with correct assertions while the iterative test refiner is capable to fix the compilation errors iteratively.

In summary, this paper makes the following contributions:

The first study that extensively investigates the correctness, sufficiency, readability, and usability of ChatGPT-generated tests via both quantitative analysis and user study;

Findings and practical implications that point out the limitations and prospects of ChatGPT-based unit test generation;

The first technique ChatTester including a novel initial test generator and iterative test refiner, which leverages ChatGPT itself to improve the correctness of its generated tests;

An extensive evaluation that demonstrates the effectiveness of ChatTester by substantially reducing the compilation errors and incorrect assertions in ChatGPT-generated tests.

II Background

Large language models (LLMs) are a category of large-scale models that have been pre-trained on a massive textual corpus . In order to fully utilize the massive unlabeled training data, LLMs are often pre-trained with self-supervised pre-training objectives , such as Masked Language Modeling , Masked Span Prediction , and Causal Language Modeling . Most LLMs are designed on a Transformer , which contains an encoder for input representation and a decoder for output generation. Existing LLMs can be grouped into three categories, including encoder-only models (e.g., CodeBERT ), decoder-only models (e.g., CodeGen ), and encoder-decoder models (e.g., CodeT5 ). To date, LLMs have been applied to various domains and achieved a great success .

More recent work leverages reinforcement learning to further align LLMs with human intent . For example, ChatGPT , a very recent LLM developed by OpenAI based on the generative pre-trained transformer (GPT) architecture, first tunes the GPT model with supervised learning and then updates the model with RLHF (i.e., reinforcement learning from human feedback). ChatGPT has attracted wide attention due to its outstanding capability of solving various tasks . In particular, there also emerge several recent works exploring ChatGPT’s potential in software engineering tasks , e.g., program repair , code generation . To date, it still remains unclear how effective ChatGPT is in generating unit tests. To fill this knowledge gap, in this work, we perform the first study to explore the ChatGPT’s ability for unit test generation; and then propose ChatTester, a novel ChatGPT-based unit test generation approach, which improves the quality of the tests generated by ChatGPT.

II-B Unit Test Generation

Traditional Techniques Traditional techniques incorporate search-based , random-based , or constraint-based strategies to generate unit tests automatically. For example, Evosuite , one of the most representative search-based techniques, leverages the evolutionary algorithm to automatically generate test suites for the given Java classes with the goal of maximizing the coverage. More recently, Lemieux et al. enhance search-based techniques by using tests generated by LLMs to escape from the “plateaus” during the search procedure. Although achieving reasonable coverage, unit tests generated by traditional techniques exhibit low readability and meaningfulness compared to manually-written ones, which thus cannot be directly adopted by developers in practice .

Learning-based Techniques. Recent work leverages advanced deep learning (DL) techniques, especially large language models (LLMs), to generate unit tests. Learning-based techniques often regard test generation as a neural machine translation problem, which trains a model to translate the focal method into the corresponding test prefix or the test assertion. For example, studies focus on generating assertions by training DL models or fine-tuning LLMs on the assertion generation dataset, where the input is a given test prefix along with the focal method and the output is the assertion. In addition to generating only assertions, recent work further fine-tunes LLMs to generate a complete test case for a given focal method. For example, AthenaTest fine-tunes the LLM BART on a test generation dataset where the input is the focal method with the relevant code context while the output is the complete test case. Similarly, Teco fine-tunes the LLM CodeT5 on a test completion dataset where the input is an incomplete test case along with the focal method and the relevant static code features while the output is the next statement in the given test. Due to being extensively trained on a massive developer-written code corpus, learning-based test generation techniques are capable of generating more human-like and meaningful test code , showing a large potential of LLMs in unit test generation.

In this work, we evaluate the capability of ChatGPT in unit test generation. As a newly-released LLM, ChatGPT is different from the ones (e.g., BART , BERT , and T5 ) adopted in existing learning-based test generation techniques by incorporating RLHF (reinforcement learning from human feedback) and a significantly-larger model scale, which has shown better alignment with human intent and outstanding performance in various tasks. Therefore, it is worthwhile to systematically explore ChatGPT’s potential in unit test generation. In fact, our study results further indicate that ChatGPT substantially outperforms state-of-the-art learning-based unit test generation techniques by generating more correct tests also with higher coverage.

III Study Setup

Existing benchmarks on unit test generation only include a limited code context (e.g., the focal method alone) rather than a complete and executable project, and it is hard to directly compile and execute the generated tests with the existing datasets. Therefore, to comprehensively evaluate the quality of ChatGPT-generated tests, we construct a new benchmark of not only focal methods but also complete and executable projects. Specifically, we construct our benchmark as follows.

Project Collection. We use the 4,685 Java projects in the popular benchmark CodeSearchNet as the initial project list. For each project, we clone it from GitHub (as of March 25, 2023) and collect its relevant information (e.g., its creating time and its last commit time). To keep high-quality projects, we then filter the 4,685 Java projects according to the following criteria: (i) the project is under continuous maintenance (i.e.,, the project should have been updated as of January 1, 2023); (ii) the project has at least 100 stars; (iii) the project is built with Maven framework (for the ease of test executions) and it could be successfully compiled in our local environment. In this way, we obtain 185 Java projects.

Data Pair Collection. We then extract data pairs from the 185 Java projects. Each data pair here refers to the pair of the focal method information and its corresponding test method. In particular, in addition to the focal method itself, the focal method information also includes the focal class declaration, all the fields, and all the method signatures (i.e., the class constructor and the instance methods). For each Java project, we extract data pairs in the following steps. (i) Given a Java project, we first find all the test classes in the project. If a class contains at least one method annotated with @Test, we regard this class as a test class and collect all the test methods in this test class. (ii) We then find the corresponding focal method for each test method based on the file path and the class name matching. For example, for a test method “testFunction()” located in the path “src/test/java/FooTest.java”, we consider the method “Function()” located in the path “src/main/java/Foo.java” as its focal method. For the cases when there are multiple focal methods with the same name in the same class, we further filter them by the number and types of parameters so as to find the unique matching one.

With such strict mapping criteria, we extract 1,748 data pairs from the 185 Java projects. Considering the costs of using ChatGPT API and the manual efforts involved in the user study, we further sample 1,000 data pairs as our final benchmark for the empirical study. Table I presents statistical distribution of our benchmark, including the lines of code in each focal method and test method, the number of parameters in each focal method, and the number of assertions in each test methods. The table shows that our benchmark includes test methods and focal methods of diverse scales and structures.

III-B Basic Prompt Design

To avoid using too simple prompts that might lead to underestimation of ChatGPT’s capability or using too sophisticated prompts that are uncommon in practice, we design our basic prompt by carefully following the common practice in existing unit test generation work and widely-adopted experience on using ChatGPT . In particular, our basic prompt includes two part: (i) the natural language description part (i.e., NL part) that explains the task to ChatGPT, and (ii) the code context part (i.e., CC part) that contains the focal method and the other relevant code context. We then explain each part in detail.

CC Part. Following existing learning-based unit test generation work , we include following code context into the CC part: (i) the complete focal method, including the signature and body; (ii) the name of the focal class (i.e., the class that the focal method belongs to); (iii) the field in the focal class; and (iv) the signatures of all methods defined in the focal class.

NL Part. Based on the widely-acknowledged experience on using ChatGPT, we include the following contents in the NL part: (i) a role-playing instruction (i.e., “You are a professional who writes Java test methods.”) to inspire ChatGPT’s capability of test generation, which is a common prompt optimization strategy ; and (ii) a task-description instruction (i.e., “Please write a test method for the {focal method name} based on the given information using {Junit version}”) to explain the task.

The top half of Figure 1 shows an example of our basic prompt. After querying with the basic prompt, ChatGPT then returns a test as shown in the bottom half of Figure 1.

III-C Baselines

We further include two state-of-the-art traditional and learning-based unit test generation techniques as baselines. For traditional techniques, we include Evosuite as the baseline. In particular, we first apply Evosuite with the default setting to generate tests for the focal class of each data pair in our benchmark, and then keep the tests that invoke the focal method. Since Evosuite might generate more than one tests for a focal method, we only keep the first one for a fair comparison with other techniques (i.e., AthenaTest and ChatGPT). For learning-based techniques, we include AthenaTest as the baseline. Since AthenaTest has not released its pre-trained BART-based model or its fine-tuned model, we reproduce it according to its paper by fine-tuning the widely-used LLM CodeT5 on the same fine-tuning dataset used in AthenaTest. We choose CodeT5 since it has been pre-trained on both textual and code corpus, which is also the best pre-training setting shown in the AthenaTest paper . In addition, to avoid potential data leakage, the data that is duplicated between the fine-tuning dataset and our benchmark are further removed from the fine-tuning dataset.

III-D Experimental Procedure

Figure 2 shows the overview of our experimental procedure. For each data pair in the benchmark constructed in Section III-A, we query ChatGPT with our basic prompt designed in Section III-B and take the test generated by ChatGPT as the output. To automate our experiments, we use the official ChatGPT API with the default setting. In this work, we focus on the gpt-3.5-turbo model. We then put the generated test in the same directory of its focal class, and attempt to compile and execute it for further analysis. We then explain the detailed procedure in each RQ, respectively.

RQ1: Correctness. Following existing learning-based test generation work , we measure the correctness of the generated tests with three metrics, including (i) syntactic correctness (whether the test could pass the syntax checker), (ii) compilation correctness (whether the test could be successfully compiled), and (iii) execution correctness (whether the test could pass the execution). Here we leverage AST parser (such as JavaParser ) as syntax checker. In addition, we further investigate the common error types in those incorrect tests to better understand the limitations of ChatGPT in test generation. Specifically, we automatically extract the error messages thrown during the compilation and execution, including different compilation and execution error types.

RQ2: Sufficiency. In this RQ, we include three metrics to assess the sufficiency of the tests generated by ChatGPT: (i) the statement coverage of the test on the focal method; (ii) the branch coverage of the test on the focal method; (iii) the number of assertions in the test case. In particular, we leverage Jacoco to collect the coverage.

RQ3 & RQ4: User study for readability and usability. In these two RQs, we conduct a user study to investigate the readability and usability of the tests generated by ChatGPT. Here, we only focus on 248 passing tests generated by ChatGPT since it is less meaningful to recommend tests with compilation errors or execution errors to developers in practice. We invite five participants whose Java development experiences vary from 4 years to 5 years. Given the test generated by ChatGPT (denoted as X) and the manually-written test in the project (denoted as Y), we ask each participant about the following two questions. It is worth noting that participants are not informed which test is generated by ChatGPT or is manually written.

Question 1 (readability): “Please rate the readability of X and Y from 1 to 4.” (1: poor readability, 2: fair readability with major issues, 3: acceptable readability with minor issues, 4: great readability).

Question 2 (usability): “Which test (X or Y) do you prefer to directly use in the project? A: X, B:Y, C: No preferencce”.

IV Study Results

Table II presents the correctness of the 1,000 tests generated by ChatGPT and other techniques. Overall, we could observe that a large portion of tests generated by ChatGPT suffer from correctness issues, i.e.,, 42.1% of the generated tests are successfully compiled while only 24.8% of the generated tests are executed successfully without any execution errors. We further manually inspect the failed tests to check whether they actually reveal bugs in the focal method under test, but we find that all of them are caused by the improper test code itself.

As for the learning-based baseline AthenaTest, ChatGPT has a substantial improvement over AthenaTest in terms of the syntactic correctness, compilation correctness, and executable correctness. For example, almost all the tests generated by ChatGPT (except the one has an incorrect parenthesis generated) are syntactically correct, but nearly a half of the tests generated by AthenaTest are syntactically incorrect. The reason might be that the significantly-larger model scale in ChatGPT helps better capture syntactical rules in the massive pre-training code corpus. As for the traditional search-based baseline Evosuite, we could observe a higher compilable rate and passing rate in its generated tests. In fact, it is as expected since Evosuite prunes invalid test code during its search procedure and generates assertions exactly based on the dynamic execution values, while learning-based techniques (i.e., ChatGPT and AthenaTest) directly generate tests token by token without any post-generation validation or filtering. Therefore, we do not intend to conclude that Evosuite is better at generating more correct tests, since the correctness of tests generated by learning-based techniques could be further improved if they also incorporate similar post-generation validation to filter those incorrect tests.

Bad Case Breakdown. We further analyze the common error types in the failed tests generated by ChatGPT (i.e., those tests failed on the compilation or execution). In particular, we first automatically categorize each test based on the error message; and then we manually summarize and merge similar types into high-level categories. Table III shows the breakdown for tests with compilation errors, while Table IV shows the breakdown for tests with execution failures.

Failed Compilation. In Table III, the column “Frequency” shows the number of each compilation errors in 579 uncompilable tests. Note that there could be multiple compilation errors in one test and thus the sum is larger than 579. Due to space limits, we only present the frequent compilation errors observed more than 10 times. As shown in the table, the generated tests have diverse compilation errors. First, the most frequent compilation errors are caused by un-resolvable symbols, e.g., the generated tests include some undefined classes, methods, or variables. Second, another large category of compilation errors are related to type errors, e.g., the parameter type in the method invocation is inconsistent with the one defined in the method declaration. Third, ChatGPT also frequently generates test code that invalidly accesses private variables or methods (i.e., access errors.). In addition, some generated tests encounter compilation errors by invalidly instating abstract class or using unsupported operators.

Failed Execution. In Table IV, we group all the infrequent errors (i.e., less than three times) into the “others” category due to space limits. As shown in the table, the majority of failed executions (85.5%) are caused by assertion errors, i.e., the assertions generated by ChatGPT consider the behavior of the program under test violates the specification. As mentioned above, we manually inspect these assertion errors to identify whether they are caused by the bugs in the focal method or the incorrect assertions themselves, and we find all of them as a result of incorrect assertions generated by ChatGPT. It implies that ChatGPT might fail to precisely understand the focal method and the quality of the assertions generated by ChatGPT should be largely improved. In addition, we observe that the remaining execution errors are related to different runtime exceptions. For example, Figure 3 presents an example of the failed execution in the test generated by ChatGPT. In particular, the test throws NullpointerExeception during its execution at line 3. The error occurs because the created object “url” assesses an external resource “/test.jar” which does not exist (in line 2). It actually shows an inherent limitation in ChatGPT, i.e., the unawareness of external resources during test generation.

IV-B RQ2: Sufficiency

Table V presents the statement and branch coverage of generated tests that could pass the execution. We further include the coverage of the manually-written tests (i.e., the original test for the focal method in the project) for reference. As shown in the table, we could observe that the tests generated ChatGPT achieve the highest coverage compared to existing learning-based and search-based techniques and it also achieves comparable coverage as manually-written tests.

Figure 4 presents a distribution plot for the number of assertions in each test generated by different techniques. Interestingly, we observe that ChatGPT-generated tests exhibit most similar distribution as manully-written tests in the number of assertions per test. In particular, Evosuite tends to generate tests with less assertions while the learning-based technique AthenaTest would generate some tests with abnormally-larger number of assertions (i.e., more than 15 assertions per test) than manually-written ones. The potential reason might be that RLHF helps ChatGPT generate more human-like test code.

IV-C RQ3: Readability

Figure 5 reports the answers to the first survey question (i.e., readability) in a stacked bar chart, where the x-axis represents each participant (i.e., from A to E) and the y-axis represents the ratio of different scores. Overall, most ChatGPT-generated tests are assessed with decent readability, and they are also considered with comparable and sometimes even better readability compared to manually-written tests.

IV-D RQ4: Usability

Figure 6 reports the answers to the second survey question (i.e., usability), where the y-axis represents each participant and the x-axis shows the number of responses that prefer the manually-written tests, ChatGPT-generated tests, or no preference. Interestingly, we find the ChatGPT-generated tests can be very competitive and sometimes there are even a considerable portion of cases that participants prefer tests generated by ChatGPT. Based on the feedback from participants, we find that people often make the decision based on a comprehensive consideration of multiple factors, such as the code format, the comments, the way how the focal method is invoked, and the rationality of assertions. In other words, the participants’ preference implies that ChatGPT is able to generate tests in line with good manual coding practice, which makes the participants willing to directly use the generated tests.

IV-E Enlightenment

Based on our results above, we further discuss the implications on the strengths and the limitations of ChatGPT-based unit test generation.

Limitations. As shown in our results, a large portion of ChatGPT-generated tests fail in compilation or execution, which might be a result of two inherent limitations in such a generative language model like ChatGPT.

First, most compilation errors might be caused by ChatGPT’s unawareness of the “deep knowledge” in the code. Although being pre-trained on a massive code corpus could help ChatGPT capture the syntactical rules in the code, the nature of ChatGPT is still a probabilistic token-sequence generator and thus it is challenging for ChatGPT to be fully aware of the deep rules in the code, e.g., only the public fields could be accessed outside the class and the abstract classes cannot be instantiated. Therefore, to help ChatGPT overcome this limitation, it is important to remind ChatGPT of such deep knowledge during its generating tests.

Second, most execution errors (i.e., assertion errors) result from ChatGPT’s lack of understanding about the intention of the focal method. As a result, it is challenging for ChatGPT to write proper assertions as specification for the focal method under test. Therefore, to help ChatGPT overcome this limitation, it is essential to help ChatGPT to better understand the intention of the focal method.

Strengths. Although generating a lot of tests failed in compilation or execution, the good thing is that most of the passing tests generated by ChatGPT are often of high quality in terms of the sufficiency, the readability, and the usability in practice. These passing tests could mostly be put into direct use to alleviate manual test-writing efforts. Therefore, leveraging ChatGPT to generate unit tests is a promising direction if the correctness issues in its generated tests could be further addressed.

V Approach of ChatTester

Overview. Inspired by our findings and enlightenment above, we then propose ChatTester, a novel ChatGPT-based unit test generation approach, which improves the correctness of ChatGPT-generated tests by ChatGPT itself. In particular, ChatTester contains two components, i.e., an initial test generator and an iterative test refiner. Figure 7 shows the workflow of ChatTester.

Instead of directly asking ChatGPT to generate a test for the given focal method, the initial test generator decomposes the test generation task into two sub-tasks: (i) first understanding the intention of the focal method, and (ii) then generating a unit test for the focal method based on the intention. Compared to the basic prompt, the initial test generator aims to generate tests with higher-quality assertions based on the help of the intermediate step of intention generation.

The iterative test refiner iteratively fixes the compilation errors in the tests generated by the initial test generation. As mentioned in our enlightenment, the key to eliminating most uncompilable tests is to provide “deep knowledge” to ChatGPT during test generation. However, given the large number of such potential rules, it is infeasible to include all of them in the prompt in advance to the test generation. Therefore, we follow a validate-and-fix paradigm to iteratively refine the uncompilable test by prompting ChatGPT with compilation error messages and additional relevant code context. In other words, the iterative test refiner actually leverages the error messages from the compiler as the violation instances of the “deep knowledge”, so as to fix compilation errors in the generated tests.

The initial test generator decomposes test generation into two steps: (i) first leveraging ChatGPT to understand the focal method via the intention prompt, and (ii) then leveraging ChatGPT to generate a test for the focal method along with the generated intention via the generation prompt.

The intention prompt asks ChatGPT to return a natural language description of the intended functionality of the focal method under test. In particular, the code context part is similar to the basic prompt in Section III-B, including the class declaration, constructor signatures, relevant fields, and the focal method itself; and the natural language instruction is to ask ChatGPT to infer the intention of the focal method. Then, the generation prompt further includes the generated intention and asks ChatGPT to generate a unit test for the focal method.

Figure 8 presents an example comparing how the basic prompt and the initial test generator generate a test for the same given focal method “setCharAt()”. As shown in Figure 8 (a), given the basic prompt without any intention inference, ChatGPT generates a test with an incorrect assertion (i.e., “assertEquals(Hello-World, strBuilder.toString())”. However, in Figure 8 (b), with the intention prompt, ChatGPT first correctly generates the intention for the focal method “setCharAt()”; and then with the generation prompt, ChatGPT generates a test with a correct assertion (i.e., “assertEquals(Hello Jorld, strBuilder.toString())”. The additional intention inference is designed to enhance ChatGPT’s understanding about the focal method, which further leads to more accurate assertion generation.

V-B Iterative Test Refiner

The iterative test refiner iteratively fixes the compilation errors in the tests generated by the initial test generation. Each iteration successively leverages two steps: (i) first validating the generated test by compiling it in a validation environment; (ii) second constructing a prompt based on the error message during compilation and the extra code context related to the compilation error. The new prompt is then queried into ChatGPT to get a refined test. Such a procedure repeats until the generated test could be successfully compiled or the maximum number of iterations is reached. Note that currently we only focus on fixing compilation errors instead of execution errors, since in practice it is challenging to identify whether a test execution failure is caused by the incorrect test code or by the incorrect focal method. We then explain each step with the illustration example in Figure 9.

Validator. For ease of compiling the generated test, we directly create a test file in the same directory of the focal class. In particular, the generated test method is encapsulated in a test class with relevant import statements. Then, the test file is compiled with the Java compiler. A controller then decides the next step based on the compilation status:

Successful compilation: if there is no compilation error, the controller would terminate the iterative refinement procedure and return the final test;

Valid refinement: if the number of compilation errors is less than that in the last iteration, the current refinement is considered as a valid refinement. The controller then proceeds to the iterative prompt constructor so as to continue the refinement;

Invalid refinement: if the number of compilation errors is larger than or the same as that in the last iteration, the current refinement is considered an invalid refinement. The controller would terminate the refinement if the accumulated number of invalid refinements is larger than the maximum (e.g., 3 in our experiments); or proceeds to the iterative prompt constructor.

Iterative Prompt Constructor. The iterative prompt constructor is built on top of (i) an EM parser that analyzes the error message about the compilation error, and (ii) a code analyzer that extracts the additional code context related to the compilation error.

In particular, the EM parser collects three types of information by parsing the error message:

Error type: the high-level description about the error, which is often the first sentence in the error message. For example, “cannot find simple class …” and “cannot find symbol method …” are extracted error types in the illustration example.

Buggy location: the line number of the test code triggering compilation errors. With such location information, the prompt constructor is able to insert the relevant information around the buggy line, i.e., starting with the tag “$\langle$Buggy line$\rangle$” as shown in the example.

Buggy element: the objects or variables in the buggy location. For example, for the second iteration in Figure 9, we analyze the buggy line with the error message, and find that they are associated with the class “Xml” (which is the buggy element in this example).

With the buggy elements, the code analyzer is then able to extract additional code context from other Java files rather than the focal class. In particular, the code analyzer first parses the whole project to find the class file that the buggy element belongs to, and then extracts the class declaration and public method signature from the class file. This extracted class information would further be added to the prompt as additional information, e.g., “//Xml class …” highlighted in blue. In fact, such additional information from other classes could be very important for generating high-quality tests, since it is very common that the test code involves with not only the focal class but also other classes. However, given the limited input length for ChatGPT, it is infeasible to directly include the whole program into the prompt (which would also lead to bad performance since the focus of ChatGPT might be confused). Therefore, in ChatTester, we propose to append necessary additional code contexts in such an iterative way.

VI Evaluation of ChatTester

Evaluation Dataset. To evaluate the effectiveness of ChatTester, we further construct an additional evaluation dataset so as to avoid using the same benchmark that has been extensively analyzed in our previous study. Since our approach is inspired by the findings from our study, evaluating it on a separate dataset could eliminate the potential overfitting issues. In Section III-A, we collect 1,748 data pairs in total and 1,000 of them are included in the benchmark for the empirical study; and in this section, we further re-sample another 100 data pairs from the remaining 748 data pairs as our evaluation dataset to evaluate the effectiveness of ChatTester.

Studied Techniques. To evaluate the overall effectiveness of ChatTester and the individual contribution of each component (i.e., the initial test generator and the iterative test refiner) in ChatTester, we study three techniques:

ChatGPT: the default ChatGPT with the basic prompt, which is the one used in our empirical study.

ChatTester-: a variant of ChatTester without the iterative test refiner, which actually enhances the default ChatGPT with the initial test generator of ChatTester.

ChatTester: the complete ChatTester with both the initial test generator and the iterative test refiner.

To mitigate the randomness in ChatGPT, we repeat all experiments for three times and present the average results.

VI-B Evaluation Results

Table VI presents the correctness of the tests generated by ChatGPT and our approaches. Overall, we could observe a substantial improvement in both the compilation rate and passing rate of the tests generated by ChatTester compared to the default ChatGPT. For example, additional 34.3% tests (= 73.3% - 39.0%) can be successfully compiled and additional 18.7% tests (= 41.0% - 22.3%) can pass the execution. In summary, the proposed approach ChatTester effectively improves the correctness of the tests generated by ChatGPT.

In addition, we could observe that the variant ChatTester- outperforms the default ChatGPT by achieving 11.7% and 7.4% improvements in the compilation rate and passing rate. In particular, we find that among the ChatGPT-generated tests with incorrect assertions, 12.5% of them are fixed into correct assertions in ChatTester-, indicating the effectiveness of the initial test generator. Figure 8 is an example of how the intention prompt improves the correctness of assertions in our dataset. Moreover, we could observe a further improvement from ChatTester- to ChatTester, i.e., additional 22.6% tests and 11.3% tests are fixed by the iterative test refiner into compilable tests and passing tests. Figure 9 is an example of how the iterative test refiner fixes the compilation errors in two iterations. In summary, both components (i.e., the initial test generator and the iterative test refiner) positively contribute to the effectiveness of ChatTester.

VII Threats to Validity

One threat to validity lies in the randomness in ChatGPT. To alleviate this issue, we repeat our experiments for three times (given the costs in using ChatGPT API) and present the average results when automatically evaluating the effectiveness of ChatTester. We do not repeat our experiments in the empirical study, due to the large manual efforts involved in the user study. However, we actually observe similar correctness results of the tests generated by ChatGPT on two different datasets (Table II and Table VI), indicating the consistency of our results. Another threat to validity lies in the benchmarks used in this work. Our findings might not generalize to other datasets. To eliminate this issue, we construct our datasets to include more high-quality projects and diverse focal methods and test methods. In addition, we further evaluate the proposed approach ChatTester on a different evaluation dataset to avoid overfitting issues. Another threat lies in the potential data leakage of the manually-written tests being part of the training data in ChatGPT, which might lead to the overestimation of ChatGPT’s capability in test generation. Since ChatGPT has not released its training data, it is hard to precisely identify this issue. However, both the large portion of uncompilable and failed tests generated by ChatGPT and the improvement achieved by ChatTester over ChatGPT indicate that ChatGPT has not simply memorized the data used in our work.

VIII Conclusion

In this work, we perform the first empirical study to evaluate ChatGPT’s capability of unit test generation, by systematically investigating the correctness, sufficiency, readability, and usability of its generated tests. We find that the tests generated by ChatGPT still suffer from correctness issues, including diverse compilation errors and execution failures (mostly caused by incorrect assertions); but the passing tests resemble manually-written tests by achieving comparable coverage, readability, and even sometimes developers’ preference. Our findings indicate that ChatGPT-based unit test generation is very promising if the correctness of its generated tests could be further improved. Inspired by our findings above, we further propose ChatTester, which leverages ChatGPT itself to improve the quality of its generated tests. Our evaluation demonstrates the effectiveness of ChatTester by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT.

References