Automated Inline-Test Generation without Relying on Method-Level Unit Tests

This component extracts target statements from their original enclosing method (e.g., getVerificationDigit() in Figure 2) into a new method (e.g., targetStmtGenerated in Figure 4) in a new class (e.g., MDFGenerateKey_7 in Figure 4). Figure 1 shows the Extractor’s steps. For each target statement $𝗌$, the Extractor creates a new class ${𝖼}^{𝗌}$ containing only method ${𝗆}^{𝗌}$, whose arguments are the input variables used in $𝗌.\text{code}$. The argument types are obtained via analysis of $𝗌.\text{file}$ and its dependencies. The body of ${𝗆}^{𝗌}$ is the output of $\mathrm{𝗋𝗐}(𝗌)$ – line 7 in Figure 1; $\mathrm{𝗋𝗐}$ applies rewrite rules in Table 1 to increase the chance that the extracted code compiles. But, the Extractor cannot rewrite statements that call methods that return void. Also, return statements are not extracted: unit tests are better suited for testing them. In principle, the Extractor can rewrite all other statements, but Table 1 shows what kinds of rewriting Smack currently supports. The Extractor also records ${\delta }^{𝗌}$, the modifications that $\mathrm{𝗋𝗐}$ makes to $𝗌$, so that the Transplanter can later reverse them. In sum, the Extractor takes a set $S$ of target statements and produces the set $E^{\text{<a href="\#S3.E1" title="In 1 Extractor. ‣ 3.1 Smack ‣ 3 Technique and Implementation ‣ Automated Inline-Test Generation without Relying on Method-Level Unit Tests" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">1</span></a>}}$:

This component uses the given generators to obtain unit tests for each class ${𝖼}^{𝗌}$ that Extractor produces. More precisely, the Generator takes $E^{\text{<a href="\#S3.E1" title="In 1 Extractor. ‣ 3.1 Smack ‣ 3 Technique and Implementation ‣ Automated Inline-Test Generation without Relying on Method-Level Unit Tests" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">1</span></a>}}$ and a unit-test generator, and produces set $E^{\text{<a href="\#S3.E2" title="In 2 Generator. ‣ 3.1 Smack ‣ 3 Technique and Implementation ‣ Automated Inline-Test Generation without Relying on Method-Level Unit Tests" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">2</span></a>}}$, defined as follows:

where $T^{𝗌}$ (a set of unit tests) is the result of running a test generator $G$ on a class produced by the Extractor, i.e., $G({𝖼}^{𝗌})$.

To find values that should be used as inline-test inputs (via the $\mathrm{𝚐𝚒𝚟𝚎𝚗}$ construct) and expected output (via the $\mathrm{𝚌𝚑𝚎𝚌𝚔𝙴𝚚}$ construct), this component instruments each extracted class ${𝖼}^{𝗌}$ to collect runtime values of target-statement variables before and after as inputs and expected outputs, respectively, while running tests output by the Generator. The output of this component is a set, $E^{\text{<a href="\#S3.E3" title="In 3 Instrumentor. ‣ 3.1 Smack ‣ 3 Technique and Implementation ‣ Automated Inline-Test Generation without Relying on Method-Level Unit Tests" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">3</span></a>}}$:

where $\overline{{𝖼}^{𝗌}}$ is an instrumented version of ${𝖼}^{𝗌}$. Figure 2 shows how instrumentation inserted by the Instrumentor records values that are later used to compose inline tests; it records values of all variables in the extracted method’s argument list before executing the target statement, and values of variables being assigned in the target statement (left-hand side of assignments) after executing the target statement (lines 3-6 in Figure 2). Some special cases (not shown in Figure 2) include:

• $\mathrm{■}$

  The instrumentor does not log values of target-statement scoped variables, e.g., x in l.stream().map(x -> x + 1).collect(Collectors.toList()); $𝚡$ is invisible outside the lambda expression so extracting it would cause compilation errors.

• $\mathrm{■}$

  The Instrumentor logs return values of methods that a target statement calls, e.g., for m1(arg), the Instrumentor captures and associates the method call with a new variable m1_arg whose type is the return type of m1(). Eventually during inline-test execution, m1_arg replaces every instance of m1(arg) to avoid dependency on the original method call m1(), making the inline test self contained.

Figure 8 shows the instrumented version of the extracted code from Figure 4. Note: Instrumentor and Generator run in parallel in Smack, but we describe them in sequence to ease presentation.

This component runs unit tests produced by the Generator and collects data from instrumented code points. More formally, it outputs the set $E^{\text{<a href="\#S3.E4" title="In 4 Executor. ‣ 3.1 Smack ‣ 3 Technique and Implementation ‣ Automated Inline-Test Generation without Relying on Method-Level Unit Tests" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">4</span></a>}}$:

where $D^{𝗌}$ is data collected by running test $\tau \in T^{𝗌}$ on instrumented class $\overline{{𝖼}^{𝗌}}$. $D^{𝗌}$ maps a unit test’s name to data collected at each instrumentation point ($\tau \to data$), where $data$ contains values collected for each variable $v$ before ($D^{𝗌}[\tau ][v].\text{before}$) and after ($D^{𝗌}[\tau ][v].\text{after}$) target statement $𝗌$.

This component composes inline tests from collected data $D^{𝗌}$ from the Executor and outputs $E^{\text{<a href="\#S3.E5" title="In 5 Composer. ‣ 3.1 Smack ‣ 3 Technique and Implementation ‣ Automated Inline-Test Generation without Relying on Method-Level Unit Tests" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">5</span></a>}}$:

where $I^{𝗌}$ is a set of inline tests, one set per $\tau \in D^{𝗌}.\text{keys}$.

This component inserts generated inline tests in $E^{\text{<a href="\#S3.E5" title="In 5 Composer. ‣ 3.1 Smack ‣ 3 Technique and Implementation ‣ Automated Inline-Test Generation without Relying on Method-Level Unit Tests" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">5</span></a>}}$ to be right after the corresponding target statements in the original enclosing method of the target statement . The Transplanter also checks that inserted inline tests pass in the original CUT. An inline test that fails – e.g., due to non-determinism – is discarded. Note: the Transplanter uses ${\delta }^{𝗌}$ and ${\mathrm{𝗋𝗐}}^{-1}$ (the inverse of the Extractor’s $\mathrm{𝗋𝗐}$ procedure) to fit inline tests into the CUT. The Transplanter also uses $𝗌.\text{file}$ and $𝗌.\text{loc}$ to find information on where to modify the CUT.

This final component produces a smaller set of generated inline tests by removing redundant ones. Inserting too many inline tests into the CUT reduces readability and can lead to compilation failure if doing so exceeds the maximum allowable method size [67]. The Reducer uses a coverage-then-mutants approach [54]. The idea is to first reduce the number of inline tests by eliminating those that are redundant based on instruction coverage (of the target statement), and then adding back inline test that kills mutants that survive the initially reduced set. Doing so ideally ensures that the final reduced set has the same bug-finding ability as the original set.

Smack uses JavaParser [41] to extract each target statement into a new method and, subsequently, a class that imports all needed packages. A mapping from target statement to its line number in the CUT, and the Extractor’s modifications are also stored for later use in transplanting generated inline tests to the CUT.

Smack applies two unit-test generators – Randoop [70] and EvoSuite [21] – to the extracted methods. Other test generators can be added in the future.

Smack uses JavaParser for source-level instrumentation of extracted methods. That instrumentation logs input and output values at each target statement. Those values are subsequently used to compose inline tests. Smack logs primitive or String-typed target statement variables as is. For other non-primitive types, Smack uses XStream [98] to serialize objects into XML, persists the serialized objects to disk, and logs the file path so that inline tests can deserialize those objects at runtime.

Smack uses JUnit to run generated unit tests; other frameworks like TestNG can be supported in the future.

Smack uses JavaParser to compose inline tests, using the input and output values recorded at a target statement while executing the instrumented extracted method.

Smack again uses JavaParser to insert generated inline tests to be after the target statement in the original enclosing method. Smack first uses the mapping stored by the Extractor to locate the target statement in the CUT. Then, Smack inserts nodes for the inline tests to be right after nodes that represent the target statement in the abstract syntax tree, which is then converted to a Java file.

First, Smack uses JaCoCo [62] to collect the instruction coverage of all generated inline tests $I^{𝗌}$ for each target statement. Then, an inline test $\tau \in I^{𝗌}$ is added to an initially empty reduced set $T^{\prime }$ if $\tau$ covers instructions that those currently in $T^{\prime }$ do not cover. Then, Smack uses universalmutator [29] to generate mutants on the target statement. If an inline test ${\tau }^{\prime }\in I^{𝗌}\wedge {\tau }^{\prime }\notin T^{\prime }$ kills a mutant that survives $T^{\prime }$, then ${\tau }^{\prime }$ is added to the final reduced set $T$. But, as we discuss in § 6, the order in which inline tests are considered by the Reducer affects the minimality of $T$.

Table 2 shows the projects that we evaluate; they are all those that Liu et al. evaluated ExLi on [54, 56]. These projects (1) are single-module Maven projects; (2) are actively maintained and changed since 2022; (3) do not have failing unit tests when run alone, with JaCoCo, Randoop, or EvoSuite; (4) contain a statement in one of the four statement categories – regular expressions, string manipulation, bit manipulation, and stream operation – that Liu et al. focused on; and (5) have at least one inline test generated by ExLi. Table 2 shows an identifier (PID) for subsequent reference, project URL, the commit SHA used, number of lines of code (computed using $\mathrm{𝚌𝚕𝚘𝚌}$ [1]), total number of target statements evaluated (All), the number of those target statements that are not covered (UC) and covered (C) by developer written or automatically generated method-level unit tests. The last row shows sums across all projects.

We evaluate 957 target statements from these 31 projects (“$\mathrm{\Sigma }$” row of the “All” column in Table 2). The 312 in the “$\mathrm{\Sigma }$” row of the “UC” column are those that developer written or automatically generated method-level unit tests do not cover (§ 2.3). We subsequently call these 312 “uncovered statements” or “Uncovered312”. Among the 312, 3 use regular expressions, 284 do string manipulation, 9 do bit manipulation, and 16 call stream APIs. These are the four kinds of statements that all prior work on inline tests target for Java. Dually, the 645 target statements in the “$\mathrm{\Sigma }$” row of the “C” column are those covered by method-level unit tests. We subsequently refer to these 645 as “covered statements” or “Covered645”. Of these, 361 are covered by developer written method-level unit tests, 424 by Randoop-generated method-level unit tests, and 483 by EvoSuite-generated method-level unit tests. The kinds of these 645 statements are: 81 regular expressions, 458 string manipulation, 88 bit manipulation, and 18 stream operations.

During inline-test generation with Smack and ExLi, we run Randoop for 100 seconds for every non-test class in the project and every extracted class to generate unit tests for each project (per the Randoop user manual [96]); we set $\mathrm{𝚜𝚎𝚎𝚍}$ to 42, $\mathrm{𝚞𝚜𝚎𝚝𝚑𝚛𝚎𝚊𝚍𝚜}$ to true, and all other options as default. We run EvoSuite for 120 seconds (per a recent SBST competition [83]) per extracted class. We also set the seed to 42. We subsequently denote the Randoop-based approach as Smack-R and the EvoSuite-based approach as Smack-E.

We use universalmutator [29, 14] because it generates mutants at the source code level, allowing us to filter mutants by target statement. Also universalmutator was used in all prior work on inline-test generation [56, 54].

We run all experiments on an Intel Xeon w9-3475X @ 2.20GHz (36 cores, 72 threads) CPU, 125 GB RAM, Ubuntu 20.04, Java 8, and Maven 3.8.6.

Randoop and EvoSuite generate no unit test for extracted methods in 13 cases. Of these, method extraction failed for five because JavaParser encounters type resolution errors. Smack extracts methods for the other eight but no unit test was generated for them because, although Smack supports non-primitive types (§ 3.2), the objects in these cases are complex. For example, consider this target statement in $\mathrm{𝚛𝚊𝚕𝚜𝚌𝚑𝚊}\mathrm{\_}\mathrm{𝚎𝚡𝚝𝚌𝚕𝚊𝚜𝚜𝚐𝚎𝚗𝚎𝚛𝚊𝚝𝚘𝚛}$:

The $\mathrm{𝚖𝚎𝚝𝚑𝚘𝚍}$ variable is a $\mathrm{𝚓𝚊𝚟𝚊}.\mathrm{𝚕𝚊𝚗𝚐}.\mathrm{𝚛𝚎𝚏𝚕𝚎𝚌𝚝}.\mathrm{𝙼𝚎𝚝𝚑𝚘𝚍}$, a class in Java’s reflection API, so it is hard for Randoop and EvoSuite to generate objects of this class from scratch. Randoop and EvoSuite utilize reflection to generate other objects, but objects in reflection API (e.g., of type $\mathrm{𝙼𝚎𝚝𝚑𝚘𝚍}$, $\mathrm{𝙵𝚒𝚎𝚕𝚍}$, etc.) are not used as target objects to construct test oracles.

Inline tests fail in 23 cases due to non-deterministic (flaky) assertions. For ten of these, the inline tests’ assertions fail because serialized objects’ classes use default hash codes that depend on specific memory addresses. For instance, consider this target statement from velocity-config-tool:

The $\mathrm{𝚌𝙵𝚒𝚕𝚎}$ object is assigned the String value of $\mathrm{𝚌𝚞𝚛𝚛𝚎𝚗𝚝𝙵𝚒𝚕𝚎𝙾𝚋𝚓}$. But, since $\mathrm{𝚌𝚞𝚛𝚛𝚎𝚗𝚝𝙵𝚒𝚕𝚎𝙾𝚋𝚓}$ is of type $\mathrm{𝙾𝚋𝚓𝚎𝚌𝚝}$, $\mathrm{𝚂𝚝𝚛𝚒𝚗𝚐}.\mathrm{𝚟𝚊𝚕𝚞𝚎𝙾𝚏}$ returns the hash code of $\mathrm{𝚌𝚞𝚛𝚛𝚎𝚗𝚝𝙵𝚒𝚕𝚎𝙾𝚋𝚓}$, which is obtained from execution-specific memory addresses. Internally, non-JDK objects (such as $\mathrm{𝚌𝚞𝚛𝚛𝚎𝚗𝚝𝙵𝚒𝚕𝚎𝙾𝚋𝚓}$) are serialized into XML files (such as $\mathrm{𝟶}.\mathrm{𝚡𝚖𝚕}$), and are deserialized at runtime. One generated failing inline test is:

This inline test fails because the memory address of $\mathrm{𝚌𝙵𝚒𝚕𝚎}$ used in the $\mathrm{𝚌𝚑𝚎𝚌𝚔𝙴𝚚}$ expression changed in the next execution, causing the assertion to fail. For two other statements, the generated inline tests are flaky because they extract dynamic information from online webpages. So, they non-deterministically pass or fail as the webpages’ contents change. Consider this target statement from $\mathrm{𝚕𝚊𝚖𝚊𝚛𝚒𝚘𝚜}/\mathrm{𝚜𝚑𝚎𝚛𝚍𝚘𝚐𝚙𝚊𝚛𝚜𝚎𝚛}$:

An inline test generated for this statement uses a Google search page as $\mathrm{𝚞𝚛𝚕}$. Generated inline tests for this target statement check if $\mathrm{𝚍𝚘𝚌}$ objects in two accesses are identical, so they fail. Future work can reduce flakiness by using mocks. For other statements in this category, generated inline tests fail because XStream cannot deserialize objects due to type resolution errors.

The “# Inline tests” column in Table 3 shows passing inline tests that Smack generates before reduction. RQ4 (Section 4.5) shows that Smack’s reduction step reduces the average number of generated inline tests per target statement from 16 to 2.2.

Neither Randoop nor EvoSuite generates unit tests for corresponding extracted methods for 31 target statements because of exceptional behavior. The target statement, extracted class, and generated unit test in Figure 10 illustrates the way exceptional behavior prevents inline-test generation. There, Smack extracts the target statement on line 8 into the class on lines 15-19 and Randoop generates the test on lines 22-32 for that class. That target statement computes the substring of name starting from the last index of colon in name. However, all Randoop-generated tests use values of name with no colon. So, they all throw exceptions and Smack cannot collect all values from which to compose inline tests.

We filter out statements where Smack only generates failing inline tests: 1 for Randoop and 4 for EvoSuite have the same problem of non-determinism as those described in § 4.2.

Among Uncovered312, universalmutator generates $1815$ mutants for 258 statements. Among Covered645, universalmutator generates $2844$ mutants for 534 statements. For the other 54 statements in Uncovered312 and 111 statements in Covered645, universalmutator only generates invalid mutants or has no applicable mutator.

Table 5 shows the mutation analysis results for Uncovered312. There, “Contributions of unit-test generators” shows if Randoop (Smack-R), EvoSuite (Smack-E) or both (Union) are used. “# Stmts Run Pass” are statements for which Smack generates passing inline tests (last row of Table 3); “# Stmts w/ mutants” are statements with mutants; “# Killed” are killed mutants; “# Surviving” are surviving mutants; and “% Mutation score” is computed as $(\text{\# Killed}\times 100)÷(\text{\# Killed}+\text{\# Surviving})$.

Mutation scores on Uncovered312 are high, suggesting that Smack-generated inline tests are effective for detecting faults on statements that are not covered by method-level unit tests. Overall, we find that Smack-E is generally more effective at killing mutants than Smack-R, but both kill mutants that the other does not. Figure 11 shows the sets

of mutants killed in statements where Randoop and EvoSuite generate inline tests: 4 mutants are killed when using only Randoop (Smack-R) compared with 212 mutants when using only EvoSuite (Smack-E). Of the 212 mutants killed by only Smack-E, Smack-R has no unit tests that can help generate inline tests that kill 97. The remaining 115 mutants are killed by Smack-E when using specific characters or strings that Smack-R does not use. For example, a mutant that is only killed by Smack-E modifies the original string manipulation statement from corrected = text.replace('/', '-')... to corrected = text.replace('*', '-').... Smack-R inline tests only have $\mathrm{𝚝𝚎𝚡𝚝}$ with simple values such as only alphabetic characters, but Smack-E inline tests use a broader range of values for $\mathrm{𝚝𝚎𝚡𝚝}$, and one of them contains $\ast$, which helps kill this mutant.

Smack-R kills 4 mutants that are not killed by Smack-E because Smack-R uses a broader range of integers as start and stop indices in the substring(start, end) method. These mutants simply switch the arguments of $\mathrm{𝚜𝚞𝚋𝚜𝚝𝚛𝚒𝚗𝚐}$, and can only be killed when the $\mathrm{𝚜𝚝𝚊𝚛𝚝}$ and stop indices are not equal. In more detail, Smack-E outperforms Smack-R in killing mutants involving $\mathrm{𝚒𝚗𝚍𝚎𝚡𝙾𝚏}$ and $\mathrm{𝚜𝚝𝚊𝚛𝚝𝚜𝚆𝚒𝚝𝚑}$ – it uses the caller as an input. Conversely, Smack-R is more effective in detecting mutations involving $\mathrm{𝚜𝚞𝚋𝚜𝚝𝚛𝚒𝚗𝚐}$ because it uses diverse integers for the start and stop indices. These mutants often survive Smack-E, which frequently uses identical start and stop indices. This analysis highlights the complementary strengths of using Randoop and EvoSuite in Smack. Also, the mutation scores after the first round of reduction (not shown in our tables, 91.1%) is 5.6 percentage points lower than that of “Union” in Table 5, so Smack’s two-round reduction is beneficial.

Table 6 shows mutation testing results on Covered645. There, mutation scores are not as high as those for Uncovered312 (Table 5). Comparing the “Union” and “ExLi” rows in Table 6 shows that Smack’s mutation score (83.1%) is lower than ExLi’s (87.7%) for all 645 statements. Also, comparing the “Union” and “UnitTests” rows shows that Smack’s mutation score (83.1%) is better than that of using only method-level unit tests (75.6%). However, on all 957 evaluated target statements in this paper, Smack-generated inline tests achieve a mutation score of 88.4%, compared to 53.6% for ExLi-generated ones, and 46.1% when using only method-level unit tests.

Figure 12 compares the sets of mutants killed by Smack with Randoop only (Smack-R), Smack with EvoSuite only (Smack-E), and ExLi (which uses developer written and automatically generated method-level unit tests), on $2697$ mutants in target statements that are covered by manually-written or automatically-generated tests. We find again that (1) using Smack with Randoop and EvoSuite is better than using either alone; and (2) Smack and ExLi are complementary for mutant killing, so future work should investigate using them together more efficiently.

Figure 12 shows that Smack-R and Smack-E kill mutants that the other cannot. Smack-R kills 75 mutants that Smack-E does not (regions H, J, K, I), and Smack-E kills 622 mutants that Smack-R does not (regions D, F, G, E). We next discuss examples of mutants killed only by

Smack-R or Smack-E. Figure 13 shows a mutant [19] killed only by Smack-R. There, line 3 is the target statement and line 4 is the inline test. The mutant changes the second != (line 1) to < (line 2), so its condition, minValue != null && minValue.indexOf(".") < -1, is false. Killing the mutant requires minValue to contain "." and maxValue to be null or not contain ".". But, the inline tests generated by Smack-E use no such values.

Conversely, Figure 14 shows a mutant [75] that is killed only by Smack-E. There, line 2 is the target statement and line 5 is the inline test. The mutant swaps the arguments of replace(" ", ""). To kill this mutant, line must start with zero or more leading whitespaces followed by #. But, Smack-R-generated inline tests do not use such values. These mutation analyses show that Smack-R and Smack-E contribute complementary inline tests with respect to fault-detection ability on target statements.

Smack and ExLi are complementary as well. Figure 12 shows that Smack kills 147 mutants that ExLi does not (regions D, L, H, E, M, I), and ExLi kills 222 mutants that Smack cannot kill (regions B, C). We next discuss examples of mutants that are only killed by Smack or ExLi. Figure 15 shows a mutant [66] killed only by Smack. There, line 3 is the target statement and line 5 is the inline test. ExLi-generated inline tests for this target statement do not use a string starting with "..".

Conversely, Figure 16 shows a mutant [5] that is killed only by ExLi. There, line 4 is the target statement and line 6 is the inline test that kills the mutant because the temp value has 19 digits. But, temp values used by Smack contain at most 4 digits, making temp $\widehat{}$ (temp $>>>$ 32) and temp $\widehat{}$ (temp $>>>$ (32 - 1)) to always be equal to temp. These two examples demonstrate that Smack and ExLi generate inline tests that are complementary with respect to fault-detection ability. That is, the union of inline tests that they generate kill more mutants than when they are used separately.

Figure 17 shows a mutant [39] that is killed by method-level unit tests but not Smack or ExLi. There, line 3 is the target statement, and we elide the unit test. The mutant changes else if to $\mathrm{𝚒𝚏}$; it is not killed by inline tests because validating logic in multi-statement if blocks is out of scope for inline tests. The recently proposed block tests [30] would be more applicable in this case.

Smack’s two-round reduction aims to reduce the set of inline tests that are inserted after the target statement in the CUT, without losing the fault-detection ability of the whole set of generated inline tests. The first round starts from R0, the initial set of all generated inline tests, removes those that are redundant based on coverage, and produces the initially reduced set, R1. Then, the second round further reduces R1 by removing those that are redundant based on mutants killed, producing an intermediate set. To preserve the fault-detection ability of R0, Smack produces the final reduced set, i.e., R2, by adding to the intermediate set any inline test from R0 that kills mutants that survive the intermediate set.

The first two rows in Table 7 show reduction rates on the Uncovered312 dataset. There, we see that Smack achieves high reduction rates on Uncovered312. With Randoop (Smack-R row), coverage-based reduction in R1 yields 250 of 3,361 R0 inline tests. After mutation-based reduction and adding back some R0 inline tests to preserve the mutation score of R0, the number of inline tests in R2 becomes 337. With EvoSuite (Smack-E row), the numbers of inline tests in R0, R1, and R2 are 594, 274, and 348, respectively. Computing reduction rates as $((R0-R2)/R0)$, the reduction rates on Uncovered312 are 90.0% with Randoop and 41.4% with EvoSuite.

The third and fourth rows in Table 7 show the reduction rates for the Covered645. With Randoop (“Covered645” , “Smack-R” row), coverage-based reduction in R1 leaves 518 of 19,075 R0 inline tests. After mutation-based reduction and adding back some R0 inline tests to preserve the fault-detection ability of R0, the number of R2 inline tests becomes 625. With EvoSuite (“Covered645”, “Smack-E” row), the numbers of inline tests in R0, R1, and R2 are 3,119, 580, and 798, respectively. These reduction rates are 96.7% with Randoop and 74.4% with EvoSuite.

In addition, to reduce the set of inline tests per unit-test generator, Smack also supports unified reduction, which further reduces the set of R2 inline tests from all generators based on mutation testing. The rightmost two columns in Table 7 show the results of unified reduction. The “UR2” column shows the number of R2 inline tests after unified reduction, with 634 and 1,054 inline tests for Uncovered312 and Covered645, respectively. The “URR(%)” column shows the reduction rate of Unified R2, with 84.0% and 95.3% for Uncovered312 and Covered645, respectively. This translates to an average of 2 inline tests per target statement on Uncovered312 and 1.6 inline tests per target statement on Covered645.

We compare Smack times with the only known manual inline-test writing times from Liu et al.’s user study [55]. In that study, participants on average take 150 seconds to understand a target statement, write 1.7 inline tests per target statement, and spend 210 seconds to write an inline test. Summing these numbers, it took study participants about 507 seconds to understand and write inline tests per target statement. Table 8 shows different views of Smack’s runtime costs. The top part shows the runtime costs per inline test, per target statement, and per project. Comparing manual writing times with Smack’s average time per target statement (Table 8 row ${𝝁}_{𝒔}$(s)), Smack-R has similar costs as manually writing per inline tests, while Smack-E is much faster. Furthermore, the user-study times that we compare against do not include the time for evaluating mutation-killing effectiveness of manually written inline tests. The slower Smack-R takes, on average, 661 and 464.3 seconds to generate inline tests for each target statement in Uncovered312 and Covered645, respectively. The faster Smack-E takes 48.8 and 37 seconds to generate inline tests for each uncovered statement and covered statement, respectively. On a per inline test basis, this cost is even lower. The slower Smack-R takes 478.6 and 385.5 seconds to generate inline tests for each target statement in Uncovered312 and Covered645, respectively. The faster Smack-E takes 38.6 and 27 seconds to generate inline tests for each uncovered statement and covered statement, respectively.

Considering that inline tests are relatively new, Smack is valuable for reducing manual effort and opens new opportunities for larger-scale research on inline tests. We measure the times while running all the steps in sequence, but the tasks in each step (e.g., mutation testing) can be parallelized. We leave such optimizations as future work.

The bottom part of Table 8 shows the breakdown of end-to-end time of running Smack on Uncovered312 and Covered645 (including time to run R0 inline tests and perform mutation analysis). All times except the inline tests execution time are one-time costs. Overall, the results show that the time to run Smack on Uncovered312 is manageable, but future research should look into speeding it up some more. Specifically, it takes 44.8 hours for Smack-R and 3.7 hours for Smack-E to generate unit tests for Uncovered312 (Table 8, column Uncovered312). The breakdown of these times are as follows. Smack-R takes 161.1s to extract target statements, 41.6 hours to generate unit tests (an inherent cost of Randoop, not Smack), 2,934.5s to execute the generated unit tests, 144.8s to execute R0 inline tests, and 2.3 hours to perform mutation testing. For Smack-E, the extraction, generation, execution, R0 inline-test execution, and mutation testing times are 161.7s, 0.9 hours, 858.6s, 137.7s, and 2.5 hours, respectively. Figure 18 displays the runtime breakdown of Smack on Uncovered312 and Covered645 visually. There, it can be seen that the end-to-end times are dominated by unit-test generation and mutation testing time. Table 9 shows the runtime statistics across all evaluated projects for different sets of target statements (in seconds), along with number of target statements.

Looking at row Gen(s), we observe a huge difference in the generation times of Smack-R and Smack-E. This is due to how these unit-test generators work – Randoop generates unit tests for the entire project [96], while EvoSuite only generates unit tests for the class enclosing the target statement. Randoop takes time that is proportional to the number of classes in a project, while EvoSuite takes time that is proportional to the number of target statements. So, Randoop spends more time than EvoSuite to generate unit tests. Yet, Smack-R and Smack-E spend similar time on execute generated inline tests and mutation testing. Prior work [55] shows that one needs to inject three orders of magnitude more inline tests than what is injected into the CUT to trigger pronounced differences in inline-test execution time. But, Smack-R generates only about 6 times more inline tests than Smack-E.

In Covered645, the distribution of runtime costs for different stages are similar to those in Uncovered312 (Table 8, column Covered645). It takes 66.9 hours for Randoop and 6 hours for EvoSuite to generate unit tests for this dataset. The breakdown of these times are as follows. With Randoop, Smack takes 335.8 seconds to extract target statements, 62.5 hours to generate unit tests, 5,749.6 seconds to execute the generated unit tests, 374.8 seconds to execute R0 inline tests, and 2.7 hours to perform mutation testing. With EvoSuite, the extraction, generation, unit test execution, R0 inline tests execution, and mutation testing times are 343.4 seconds, 2.5 hours, 2,162.2 seconds, 261.9 seconds, and 2.7 hours, respectively.

Smack does not take into account useful information from the context of the target statement in the original enclosing method. For example, a conditional in that context might limit the set of values that should be used in the target statement. We use Figure 19 to illustrate this limitation. There, line 2 and line 4 restrict the inputs that can be used on the target statement on line 5 to only strings with prefix $\mathrm{𝙲𝙻𝙰𝚂𝚂𝙿𝙰𝚃𝙷}\mathrm{\_}\mathrm{𝙿𝚁𝙴𝙵𝙸𝚇}$ (“$\mathrm{𝚌𝚕𝚊𝚜𝚜𝚙𝚊𝚝𝚑}:$”), followed by $\mathrm{𝚂𝙻𝙰𝚂𝙷}\mathrm{\_}\mathrm{𝙿𝚁𝙴𝙵𝙸𝚇}$ (“$/$”). By extracting only the target statement for inline-test generation, Smack becomes unaware of this context. So, Smack uses a string that does not satisfy the contextual requirements. The resulting inline test passes, kills mutants, but such input are unlikely to be realistic. There is value in generating inline tests with and without the context of the original enclosing method. We will explore how to capture more of that context for inline-test generation in the future. For example, it may be possible to extract a backward-slice from the target statement and use that slice as part of inline-test generation. Or we could collect invariants [17] for the surrounding method and use them as constraints during inline-test generation.

We only implement Smack for Java, to enable comparison with ExLi, which was also only implemented in and evaluated on Java. Also, the availability of robust unit-test generation tools – Randoop and EvoSuite– provide an extra motivation to use Java. We plan to explore inline-test generation for other languages, such as Python (for which there is already an inline testing tool [57]).

Currently, Smack cannot generate inline tests if all the tests generated by test generators throw exceptions. However, there are no theoretical limitations that prevent inline tests from expecting exceptions. Future work can explore how to support exception checking in ITest and include tests that expect exceptions in the pipeline of automatic inline test generation.