Exploring Mutation Testing for Teaching Introductory Programming

Many introductory programming courses in undergraduate level employ some form of automatic feedback tool to help students with their learning process. The most common approach is to require students to submit code and then check it against a number of test cases prepared by the instructor [7]. While such systems can be beneficial to the learning process, they often also present some pitfalls:

1. 1.

   Students typically start with only a problem description and a blank code editor, leaving them on their own about how to proceed.

2. 2.

   After an erroneous submission, students often resort to trial-and-error, trying to fixing reported errors in an “ad-hoc” way rather than taking a step back and re-assessing the problem.

3. 3.

   The reliance on instructor-provided tests may lead students to believe that the sole purpose of the assignment is to pass the tests rather than to build up an understanding that can be applied in future situations.

4. 4.

   The availability of generative AI tools such as ChatGPT and CoPilot that can solve introductory programming problems automatically means that unsupervised students may ask the AI tool for help and obtain a solution that passes the tests, even if they do not understand it. This can give a false sense of progression that does not translate to effective learning [6].

This paper proposes an alternative type of programming exercise were students are given the code and asked to write the tests. The general form of such exercises is the following: given a correct version of a program, and one or more incorrect ones, produce a suite of tests such that the correct program is accepted and the incorrect ones are rejected. The idea is inspired by mutation testing, a well-known Software Engineering methodology for assessing the quality of test suites [4].

When teaching introductory programming, however, our primary focus is on understanding rather than engineering. Nonetheless, we believe that such style of exercises can still be beneficial for beginners:

1. 1.

   By starting from a problem description and code fragments to analyse, we avoid the initial difficulty of translating from an informal idea to a formal notation.

2. 2.

   By focusing on the differences between code fragments, we emphasize code reading skills which are often negleted in programming courses [1].

3. 3.

   Because test suites can be assessed automatically, we can still provide fast and scalable feedback to such exercises.

4. 4.

   The ability for students to came up with relevant test cases is a useful skill in itself for writing and debugging programs.

As a proof-of-concept, we have extended the Codex system to allow conducting such exercises. The remaining of this paper describes this extension. However, at the time of writing this approach has not yet been tried in the classroom.

Codex is a web-based system for setting up programming assignments with automatic assessment [10]. The system supports different programming languages (Python, Haskell, Java, C and SQL) and types of assignments (input-out tests, unit tests, property tests). It has been developed by the author of this paper and is used in several programming courses at the Faculty of Science of the University of Porto. The source code and user guide are available at https://github/pbv/codex.

Codex is organized around structured documents: both index pages and exercises are written as Markdown files.¹¹1Using many extensions (e.g. LaTeX mathematics) allowed by the Pandoc library: https://pandoc.org/ This means that exercises can be composed using a standard text editor, archived in version control systems, copied and compared for differences, etc. The system is otherwise minimal: the server renders pages, accepts submissions, stores them in a SQLite database, runs tests and renders reports. There is a minimal admin interface, but most of the setup and configuration is done via command-line tools.

Doctest²²2https://docs.python.org/3/library/doctest.html is a Python library for writing tests of functions, classes or methods embedded in the documentation. The library makes uses of Python’s reflection capabilities to discover tests automatically. Listing 1 shows an example of three unit tests included in the documentation string of a number squaring function. The tests mimic the interactive use of the Python interpreter (and indeed such examples can be copied from an interative session), making it more suitable for beginners than conventional testing frameworks because there is little extra bureaucracy for writting tests.

Since Doctest it is part of the Python standard packages, no extra dependencies have to be installed, simplifying its adoption by beginners. The library also exposes an advanced API that allows reading tests from a string or file (as opposed to documentation strings); this is used by the Codex doctest tester and also the new mutation tester presented in Section 3.

Program mutation [4] (also known as mutation testing) is a technique for assessing the quality of a test suite by generating many small variations of the code under test (called mutants) and checking that these are rejected by the test suite (said to have been killed). A mutant that survives the test suite corresponds to either: (1) a benign change that preserves the program behavior; or (2) some input combination that is not properly tested. Thus, the amount of mutants killed by a test suite can be used as a comparative indicator of test quality.

Mutated programs can be automatically derived from the original using some heuristics, e.g. changing a comparison such as less-than-or-equal (<=) to less-than (<) or vice-versa, or increasing constants by one. Several libraries exist for automatically generating and testing mutants from a codebase of specific programming languages.³³3E.g. MutMut for Python (https://github.com/boxed/mutmut) and PIT for Java (https://pitest.org/). Mutation testing is often taught in University courses on software testing or project assignments [3, 8]. It is not, however, typically covered in introductory courses.

We have extended Codex with a new mudoctest tester for mutations of Python code. The tester requires that the instructor defines both the correct program (the target) and one or more incorrect mutants, while students submit the doctests. Mutants are not automatically generated; this choice is deliberate so that the instructor can choose appropriate variations for the intended skill level and learning objectives.⁴⁴4Nonetheless, it would be still possible to generate mutants automatically as a separate offline step.

Oliveira et al. [9] present results from an experiment on using a mutation testing tool for Pascal programs with 20 beginner students. They report better scores in a post-experiment code understanding survey by the group using mutation testing against the control group.

Clegg et al. [3] used a “Code Defenders” game for teaching software testing using mutations. Players alternate between attackers and deffenders; attackers submit mutants, while defenders submit test suites. The game assigns scores to players as they interact in these roles.

In his PhD thesis [2], Clegg investigated the use mutation testing for improving the quality of test suites used by instructors when automatically grading student’s code.

Mansur et al. [8] studied the effect of teaching mutation testing to students in a Data Structures and Algorithms programming project. They found that mutation testing had a positive effect on the quality of student’s test suites.

We believe that mutation testing in teaching introductory programming in the style proposed in this paper could be beneficial to students. A combination of factors enable this, namely, the lightweight syntax of Python doctest suites and the automatic feedback provided by Codex. However, there are a number of caveats that we anticipate:

1. 1.

   Doctests scripts are interpreted textually, meaning that extra white space or the choice of delimiters (e.g. single or double quotes) may cause a test case to fail.

2. 2.

   Runtime errors during the running of test suites might be confusing to students.

3. 3.

   The terminology employed in the Codex generated reports is rather technical (e.g. “target” and “mutant”) and should perhaps be revised to better suit a beginner audience.

As further work, we would like to experiment with such exercises along more conventional ones were students are required to write code in a real classroom and get some feedback from students and instructors.

Another interesting direction is to investigate the use generative AI tools by students when solving mutation testing exercises. Some initial research exists on the use of AI tools for exercises about writing code [6], but it might not apply to exercises about writing tests. Our anecdotal experience suggests that e.g. ChatGPT can explain code differences and even suggest good test cases for simple programs. It would be worthwhile to investigate whereas students could benefit from such explanations for self study.