Integrating Questions About Learners’ Code in an Automated Assessment System

Introductory programming students are often tasked with completing assignments which require the development of code conforming to functional requirements, such as the production of specific outputs. These types of assignments foster students’ program writing skills, but do not necessarily increase students’ confidence regarding their knowledge of the course materials [6, 5]. Further, students struggle when asked to explain their own program code [8], indicating that assignment completion does not necessarily imply proper knowledge acquisition.

Programming students tend to benefit from a more efficient learning process if they seek to understand their assignments’ underlying concepts [20], which is not guaranteed by assignment completion. Further, students may struggle to understand and adequately guide their learning process towards deeper comprehension skills [11]. This obstacle is further amplified by the emergence of Large Language Models (LLMs), which are capable of producing working solutions to most introductory programming exercises [4]. An experiment carried out by Prather et al. (2024) [16] demonstrated that LLMs, when used in an unconstrained manner by introductory programming students, might be detrimental to students with pre-existing difficulties, as they create an illusion of understanding that can hinder the learning process.

Questions about Learners’ Code (QLCs) [9] were proposed as a means of assessing the program comprehension skills of novice programmers. QLCs consist of questions (and, usually, multiple-choice options for it) which are formulated against a student’s own code. In this way, students are assessed through questions which specifically target their own code, benefiting from a learning aid adapted to the context they are working on. Several contributions have been made, particularly in automating this approach for programming courses employing different languages (e.g., Java [18], JavaScript [7], Python [10]).

In this paper, we describe our experimental integration of QLCs in the Automated Assessment System (AAS) used at our institution for introductory programming courses in Java. We aim to explore what insight we can gain from integrating QLCs into students’ usual learning process. In particular, we explore the following research questions:

1. 1.

   RQ1: How does students’ success on different types of QLCs evolve over time?

2. 2.

   RQ2: Can QLCs offer insight into novice programmers’ misconceptions?

Our overall goal is to assess whether QLCs are an adequate complement to existing AASs, as we hypothesise that they can yield benefits such as: (a) providing students with personalised formative feedback, reinforcing program comprehension knowledge and offering a way to self-assess and detect misconceptions; and (b) provide instructors with a means to assess students’ progress and detect learner misconceptions, possibly in an automated manner in a large-scale course.

We describe our preliminary study, in which we implemented QLCs in an AASs and collected data on student code submissions and QLC responses. We find that analysing students’ answers to QLCs can offer insight into their learning progress and possible learner misconceptions, and how the latter evolve over time. These findings could provide an opportunity not only for students to regulate their learning process, but also for instructors to steer lectures towards addressing the most common or persistent student difficulties or misconceptions.

Introductory programming courses traditionally focus on a writing-first approach, where students learn to program by writing code that conforms to specific requirements (usually functional). Previous studies have shown that successfully writing programs does not necessarily imply deep program comprehension [5, 14], and that learners struggle to explain their own code even when they were successful in developing programs [8]. Comparatively, comprehension-first teaching approaches, where program writing is preceded by the development of program reading skills, have been shown to be effective in developing program comprehension skills [19].

A recent approach in strengthening learners’ program comprehension skills consists of Questions about Learners’ Code (QLCs) [9]. Lehtinen et al. (2021) define QLCs as questions, posed automatically by a computer to a student, which refer to concrete constructs and patterns in the program code the student has written [8]. QLCs are typically multiple or single-choice questions, which can be verified automatically, or open-ended questions, which might require manual evaluation. They may be generated by employing a mixture of static analysis (e.g., which constructs are used) and dynamic analysis (e.g., how the code behaves when executed with specific inputs or starting states). Static QLCs focus on identifying how programming constructs are employed in the code, whereas dynamic QLCs focus on program behavior (e.g., tracing variable values). The ability to trace programs is highly correlated with program comprehension and writing skills [2, 12, 13]. A recent application of QLCs consists in combining them with the concept of notional machines [1], where the type of generated questions might involve, for example, the construction of a tree for a given expression written by the student.

A previous study by Lehtinen et al. (2023) [8] investigated whether students’ success in QLCs correlated to their program writing skills (in JavaScript) and their tendency to drop out. Their findings indicate that students show more difficulty in answering dynamic QLCs than static QLCs. Yet, overall, QLC success is highly correlated with course success and program writing skills. Further, they found that students with higher difficulty in answering QLCs throughout the course are more likely to drop out.

In previous work, we developed Jask (2022) [18], a first experimental system for automatically generating QLCs for introductory Java exercises. Similarly to other studies, we found that students struggle with dynamic QLCs more than they do with static QLCs. Figure 1 presents an example of a dynamic QLC. In this paper, we continue this line of work, developing more QLC types, and focusing on how to integrate the questions in an AAS.

Figure 2 presents the concepts of a typical AAS realisation (grey-shaded area). A course is composed of several code assignments. Each assignment has several test cases, which associate arguments to expected results, and define one (or possibly more) reference solution. Students are enrolled in the course and perform submissions to each assignment. Each submission holds the student solution, which is marked with a score (most likely, automatically).

The remaining concepts of Figure 2 relate to our approach. Each assignment may be associated to one or more QLC Type (from a catalog), meaning that the AAS will generate concrete questions of those types for submissions to that assignment. Each question has two or more multiple-choice options. The text of the question and the options result from instantiating templates against the submitted code, using its structure, identifiers, and execution data in the formulation of questions and their multiple-choice options.

Since each QLC is generated from a student’s submission, there is an extra layer of variability regarding the particular identifiers, structure, and behaviour of the student’s code, which can influence both the question’s formulation and options. Further, randomly selecting inputs from an assignment’s test cases provides additional variability. These test cases are defined with meaningful values that cover usual and edge cases, and sampling from them guarantees that questions are posed with inputs which make sense for each assignment.

We implemented an experimental incorporation of QLC in the Paddle environment, an Automated Assessment System (AAS) in which students develop and submit solutions to coding assignments. Figure 3 illustrates the environment, where we can see a QLC being presented after an exercise submission.

After each submission that passes the functional tests (i.e. expected outputs are correct), students are prompted with QLCs targeting their solutions. The idea is that instructors associate QLCs that target concepts that are relevant for each specific assignment, taking the sequence of assignments into account. As this was our first experiment of using QLCs in a course, we chose to associate at most one QLC to each course assignment, and we did not allow multiple attempts in answering QLCs. All QLCs were single-choice (one correct option), with response being optional (student may leave the QLC blank). The answers were stored in a database and were used as data for our study.

We conducted a study with introductory programming students by integrating our approach in a programming course for graduate students in non-STEM fields. The course was taught using Java, where students were tasked with assignments targeting the concepts of functions, expressions, variables, control structures, arrays, references, and procedures (side-effects).

Our results show that integrating QLCs within the AAS of a programming course can offer insight into how learners’ knowledge and misconceptions evolve over time.

Regarding the overall QLC success rates shown in Table 2, we note that it is not straightforward to attribute specific causes to differing success rates, as they depend on the assignment’s context and its appearance within the course. Nevertheless, unsurprisingly, these values showcase students generally struggle more with questions related to concepts like tracing (e.g., WhichVariableValues) as opposed to static program properties (e.g., HowManyParams). This confirms the observations of previous studies [18, 7].

Our observations show that QLCs can offer insight into how students’ knowledge on elementary programming concepts evolves over time during an introductory programming course. We also identified and analysed common student misconceptions and how they persist over time. Our results show that student progress is not linear, with prior knowledge not necessarily persisting over time or transferring to assignments of higher complexity, and with misconceptions not necessarily improving over time. The sample used to derive the results was small, but we believe this introduces no significant bias or limitation to our analysis. Our findings thus reveal that, while students may appear to understand certain concepts within simple contexts, this knowledge may not necessarily transfer to assignments of higher complexity.

An implication for instructional design is that it is worth posing the same type of QLC several times over the progression modules, as concept acquisition tends to be fragile. By monitoring the QLC results, instructors may detect students’ most prominent misconceptions, in order to prioritise and reinforce aspects in lectures that students struggle the most. On the other hand, students may use QLCs as a means of self-assessment that strengthens their skills and uncovers knowledge gaps. Further, we should provide multiple attempts for answering QLC, as is not beneficial to have any sort of attrition in the platform usage.

We intend to continue augmenting our AAS with QLCs that not only aid novice programmers at each step of their learning process, but provide instructors with insight on how to structure and guide their course in order to adequately tackle student obstacles. We envision the usage of QLCs to reinforce and offer insight into students’ program comprehension skills, and to detect and mitigate misconceptions. As such, in future work we are likely to include the implementation of QLCs targeting other aspects of novice programmers’ learning process, like error message comprehension and code quality towards refactoring. We also plan to enrich QLCs with comprehensive explanations for the correct answers, as well as possible misconceptions for the incorrect options.