A Generative Artificial Intelligence Tool to Correct Programming Exercises

Generative Artificial Intelligence (GAI) has rapidly expanded across multiple domains, with education emerging as a particularly impacted sector. At the core of many GAI tools are Large Language Models (LLMs), a class of deep learning systems trained on extensive textual corpora and capable of producing coherent, context-aware natural language output. These models, such as OpenAI’s GPT series, have demonstrated impressive capabilities in a wide range of language understanding and generation tasks, making them increasingly prominent in educational technologies due to their adaptability and scalability [2].

The introduction of powerful LLM-based applications, such as OpenAI’s ChatGPT, has created new opportunities and challenges in the teaching and learning landscape. On one hand, GAI tools can support educators by generating instructional materials, summarising content, or responding to complex student queries. On the other hand, their misuse, particularly by students relying on these systems to produce unoriginal or plagiarised work, raises significant ethical, pedagogical, and assessment-related concerns. [7, 14, 1]. Recent research has explored how GAI can become a powerful ally in the classroom, especially when integrated thoughtfully into the curriculum. Its use can foster innovation in assessment, provide personalised feedback, and help teachers scale their efforts to large student groups [8]. Other approaches, such as GAMAI [9] or Socratique [15], help professors create exercises or students learn a topic and receive feedback.

In 2020, Filipe Portela proposed an innovative teaching paradigm named TechTeach, aimed at increasing student engagement through gamification and active learning strategies [11]. This approach integrates digital tools like Kahoot!, project-based learning, and collaborative exercises to create dynamic, student-centred classrooms. Over the years, the model has evolved with new techniques and adaptations being introduced annually, including innovative mechanisms for assessment using real-time quizzes [12].

Inspired by the success of this paradigm, the GAI-based correction strategy presented in this paper incorporates several principles originally introduced in TechTeach, particularly those related to engagement and formative feedback through gamified interactions.

In this context, several tools have been developed in recent years, like Gradescope, to support the automatic evaluation of programming exercises. Traditional automatic graders rely heavily on predefined unit tests and static analysis techniques, often failing to understand creative or alternative correct solutions. With the emergence of GAI, new possibilities arise for interpreting code semantically and evaluating student submissions based on reasoning and problem-solving rather than rigid test cases. These systems can complement or even surpass conventional tools in specific contexts, especially when coupled with pedagogical strategies that promote iterative learning and constructive feedback [8, 7]. The strategy proposed in this paper builds on these recent advances by combining generative AI capabilities with a structured evaluation approach. Thus, it addresses some limitations of traditional systems and leverages a more flexible and nuanced assessment process.

This study follows a practice-oriented methodology structured into four main phases: Design, Implementation, Analysis, and Interpretation. These phases guided the development and evaluation of a Generative Artificial Intelligence (GAI) algorithm that automatically assesses student solutions to programming exercises. The method was applied over two academic years in the Web Programming course at the University of Minho, involving over 200 students.

• $\mathrm{■}$

  Design Phase:

  • –

    Identify the pedagogical needs and challenges in evaluating programming exercises in large classes.

  • –

    Define the classification rubric, consisting of three levels: -1 (no understanding), 0 (partial understanding), and 1 (complete understanding).

  • –

    Design a structured statement format for programming problems to facilitate prompt comprehension by the GAI model.

  • –

    Develop and test different prompting strategies to effectively and consistently evaluate code.

• $\mathrm{■}$

  Implementation Phase:

  • –

    Develop the GAI-based correction algorithm, integrating prompt templates with programmatic evaluation workflows.

  • –

    Prepare datasets with anonymised student submissions from the Web Programming course.

  • –

    Implement automation scripts to process code submissions, interact with the GAI model (e.g., ChatGPT), and collect responses.

  • –

    Generate automatic classifications and feedback based on the GAI’s output.

• $\mathrm{■}$

  Analysis Phase:

  • –

    Compare the GAI-generated classifications with those manually assigned by experienced instructors.

  • –

    Assess the generated feedback’s consistency, accuracy, and usefulness.

  • –

    Analyse how well the rubric and prompt strategy align with the intended learning outcomes.

• $\mathrm{■}$

  Interpretation Phase:

  • –

    Reflect on the impact of the proposed strategy on assessment processes in programming education.

  • –

    Identify the strengths and limitations of the GAI approach compared to traditional manual correction.

  • –

    Discuss the implications of adopting this type of algorithm for formative and summative assessment in higher education.

All procedures adhered to ethical standards, ensuring that student data remained anonymous and protected throughout the process.

Adopting a case study methodology is appropriate for this research, as it enables the exploration of a GAI approach without compromising the course’s core learning objectives. This method is particularly well-suited for studies conducted in academic environments because it allows professors to observe student behaviours in context and relate these behaviours to their outcomes. [3].

Regarding the technological infrastructure supporting this study, several digital tools were integrated into the workflow to support each phase of the process:

• $\mathrm{■}$

  HackerRank [6] – Used as the coding environment where students developed and tested their programming solutions.

• $\mathrm{■}$

  Google Forms [4] – Employed as the primary submission platform for students to submit their final code.

• $\mathrm{■}$

  Google Sheets [5] – Served as the central hub for aggregating responses, generating dynamic prompts, and connecting to the GAI evaluation system.

• $\mathrm{■}$

  ChatGPT API [10] – Integrated with the spreadsheet to automatically analyse and grade student submissions based on customised prompts.

• $\mathrm{■}$

  ioEduc [13] – Includes the rescue system and allows students to individually and anonymously complete project assessments, both individual and group. These grades are then used by the algorithm.

The design and implementation of the GAI-based correction system were guided by a set of pedagogical and operational principles to ensure both educational value and fairness in the assessment process. These guidelines aim to align the use of generative models with authentic student evaluation practices:

• $\mathrm{■}$

  Assessment Objective: The programming exercise assesses students’ understanding of the covered topics. Students are allocated 50 minutes to respond to the proposed challenge and demonstrate their knowledge under time constraints.

• $\mathrm{■}$

  Realistic Learning Context: The assessment is framed within a broader narrative. Students are encouraged to approach the task as a practical and contextualised project, including peer-assessment and reflection components, thereby mimicking real-world development environments.

• $\mathrm{■}$

  Personalised Submission: At the beginning, students must fill out a form, including their personal data. This helps prevent plagiarism and also provides valuable context for the LLM to interpret.

• $\mathrm{■}$

  Generative Model for Evaluation: A large language model (LLM), such as ChatGPT, is used to analyse the code and the statement and to produce an assessment aligned with a simplified three-level scale instead of traditional numeric scores (e.g., 0–20). This strategy encourages qualitative feedback and conceptual mastery rather than point-based evaluation.

• $\mathrm{■}$

  Learning from Human Input: The evaluation process includes data from peer-assessment and other manual assessments (e.g., project contributions), which serve as training context and calibration reference for the LLM, helping it to interpret performance with greater nuance.

• $\mathrm{■}$

  Human Validation: To mitigate errors and ensure the quality of the automated evaluation, the system includes human-in-the-loop verification. Specifically, all extreme results (i.e., grades of -1 or 1) are manually reviewed by the professor. Additionally, a representative sample of other responses is also validated to improve the reliability of the overall process.

• $\mathrm{■}$

  Feedback Transparency: Final evaluations, including those produced by the LLM and any manual corrections, are shared with students. This ensures that learners receive meaningful feedback and can understand the rationale behind their results.

• $\mathrm{■}$

  Rescue possibility: Students should have a mechanism to contest their grades. This mechanism should include rules and may be approved automatically or manually.

The success of a Generative AI (GAI) algorithm in evaluating student programming exercises is directly influenced by the clarity, structure, and technical precision of the exercise statement. Since LLMs interpret natural language through prompt engineering, the statements must be designed to avoid ambiguity, favour structure, and anticipate edge cases. The following guidelines have been defined to ensure that exercise statements are effectively interpreted by GAI systems while also maintaining pedagogical consistency and technical correctness according to the rules defined by professor:

• $\mathrm{■}$

  Use a modular structure: Divide the statement into logical parts (e.g., Part A, Part B) when multiple technologies or concepts are involved. This approach helps both students and AI models grasp the scope and sequence of tasks. Each part can have distinct evaluation rules/prompts.

• $\mathrm{■}$

  Present requirements using ordered lists: Use bullet points (e.g., a., b., c.) or numbered lists to clearly separate tasks. Each instruction should focus on a single objective or functionality.

• $\mathrm{■}$

  Define all custom attributes and variables: When referring to identifiers like eid, SID, or contextual values (e.g., date of birth, student ID), always clarify their origin, format, and usage to prevent students’ plagiarism or copying. For example: “margin set to the last two digits of the student id (SID).”

• $\mathrm{■}$

  Maintain consistent naming conventions: Select a uniform style for IDs, variable names, and file names (e.g., camelCase, lowercase, or snake_case) and apply it consistently throughout the documentation.

• $\mathrm{■}$

  Specify expected interactions clearly: Clearly define which user actions trigger responses (e.g., clicking a button triggers an alert). If JavaScript is involved, indicate the event handlers and the expected messages or outcomes.

• $\mathrm{■}$

  Clarify data processing steps: If the task involves data transformation (e.g., calculating a value, subtracting an offset from an ID), provide an explicit formula or explanation to ensure consistent interpretation.

• $\mathrm{■}$

  Include dynamic elements and explain them: When utilising runtime-generated content (e.g., DOM manipulation, dynamic tables, loops over API data), describe the expected output structure (e.g., a table with three columns: X, Y, Z).

• $\mathrm{■}$

  Describe external resources and formats: If the task involves fetching data from external APIs or files, include the endpoint, HTTP method, expected format (e.g., JSON), and the structure of the response object. For instance: “Access the property data.URL in the JSON response.”

• $\mathrm{■}$

  Avoid vague or overly open-ended language: Replace expressions like “do something similar” or “create a generic solution” with specific technical instructions. LLMs require clear guidance to produce accurate feedback.

• $\mathrm{■}$

  In case of need, indicate what libraries or frameworks students can use: If students must use external libraries (e.g., Bootstrap, jQuery), specify the version and provide a link or state how they should be included (CDN, local file, etc.).

• $\mathrm{■}$

  Be explicit about expected outputs: Whether it is a table, form, alert, or LocalStorage entry, describe the expected structure, data, and user experience. This is essential for automated evaluation to ensure presence, correctness, and completeness.

Beyond supporting the automated correction process, the exercise statement also plays a crucial role in promoting academic integrity and preventing plagiarism. In educational settings where assignments are automatically corrected using generative models, the risk of students sharing answers or reusing solutions significantly increases if all statements are identical.

To mitigate this risk, the statement should be structured to ensure consistency while remaining adaptable for each student. A recommended strategy involves integrating personalised variables, such as the Student Identification Number (SID), name, or date of birth, into the exercise’s body. These variables can be collected at the beginning of the assessment and programmatically embedded within the task description.

For example, the student ID (SID) can define the margin of an HTML element, serve as input to a function, or determine the value passed to an API endpoint. This not only ensures that each student receives a unique version of the same pedagogical challenge, but also preserves fairness by standardising difficulty and learning outcomes across the cohort.

Such dynamic personalisation enhances the complexity of sharing solutions while ensuring consistency in assessment criteria. It also promotes deeper engagement from students, who must apply core concepts to contexts with their own data.

These guidelines ensure that statements are not only pedagogically sound but also technically compatible with automated correction systems based on generative AI. They support both human and AI comprehension, enhancing fairness, accuracy, and the quality of feedback.

The following pseudocode outlines a generic engine for generating a structured prompt to be interpreted by a large language model (LLM) like ChatGPT. This prompt aims to assess students’ programming exercises by integrating contextual data provided by students and peer assessment information.

The algorithm receives a set of textual parameters as input that define the context and content of a programming exercise:

• $\mathrm{■}$

  statement – It is the programming exercise statement;

• $\mathrm{■}$

  rules – The specific technical/structural requirements to be followed when developing the code

• $\mathrm{■}$

  variablesText – a list of variables to be used in the implementation and has students’ characteristics (e.g. BirthDay, Favourite Colour, others);

• $\mathrm{■}$

  studentCode – the code developed and submitted by the student;

• $\mathrm{■}$

  peerScore – the individual participation score attributed by peers’ assessment in the group;

• $\mathrm{■}$

  groupScore – the evaluation score attributed to the student’s performance in a group project.

The output is a fully structured prompt (promptText) that is interpreted by a Generative AI model (e.g., ChatGPT), combining all contextual elements and evaluation criteria in natural language.

To mitigate potential misclassifications and ensure fairness in the grading process, the system integrates a gamified appeal mechanism derived from the TechTeach framework, referred to as Grade Rescue. This mechanism enables students to request a grade revision if they believe their assessment was inaccurate or if they encountered exceptional circumstances that may have affected their performance.

Requests for grade revision must be formally submitted and are subject to the professor’s approval. If a request is accepted, the corresponding grade is temporarily suspended, and the student is closely monitored throughout the remainder of the subject. At the end of the term, the case is reassessed, taking into account the student’s overall engagement, consistency, and contributions to determine the final grade.

By incorporating this human-in-the-loop approach into the GAI-driven evaluation process, the system enhances its commitment to transparency, pedagogical integrity, and continuous improvement. It also empowers students to participate actively to ensure the fairness and accuracy of the assessments, with the professor as the jury.

Although the proposed approach is scalable and independent of any specific tools or platforms, it is crucial to demonstrate its application in a real-world context. The following section presents a practical example of applying the algorithm using a selected set of tools.

The correction process incorporates various digital tools into a unified pipeline that guarantees efficiency, scalability, and pedagogical relevance. The following steps outline the complete operational workflow of the algorithm:

1. 1.

   The professor designs the programming problem and publishes it on the HackerRank platform.

2. 2.

   Students are instructed to develop and test their solutions using HackerRank’s interface.

3. 3.

   Upon finalising their code, students submit it via a Google Form.

4. 4.

   Submissions are automatically collected and organised in a Google Sheet.

5. 5.

   The spreadsheet is connected to the ChatGPT API, which analyses each submission using a tailored prompt that incorporates the problem statement and a predefined evaluation rubric.

6. 6.

   The algorithm considers three primary inputs: the student’s submitted code, the problem statement, and the student’s performance in the group project (collected through the ioEduc platform). Based on these inputs, the system generates a grade in the range: -1, 0, or 1.

7. 7.

   As a quality control measure, all responses graded with -1 (indicating a lack of understanding) or 1 (indicating mastery) are manually reviewed by the professor to confirm the accuracy of the AI-generated evaluation.

This workflow enables the correction of large volumes of student submissions with speed and consistency, while still accommodating more complex or non-standard solutions.

The exercise statement includes some initial rules. It follows a controlled and semi-personalised structure to ensure fairness, consistency, and academic integrity. The key rules are:

• $\mathrm{■}$

  Duration: 50 minutes + 5-minute buffer.

• $\mathrm{■}$

  The exercise consists of a single statement divided into two distinct parts (Part A and Part B).

• $\mathrm{■}$

  Students must develop and test their code online using the HackerRank platform.

• $\mathrm{■}$

  To solve the project, students must consider their answers (e.g., birthdate, student ID, and others) and use the value for the underlined variables.

• $\mathrm{■}$

  Students can consult the class slides and previously developed project code.

• $\mathrm{■}$

  The use of Generative AI tools is strictly prohibited.

• $\mathrm{■}$

  Final code solutions must be submitted via a dedicated Google Sheets form.

The Generative AI (GAI) algorithm was designed to assess student performance by considering three complementary evaluation dimensions:

• $\mathrm{■}$

  Student’s Response to the exercise: Evaluates whether the submitted code adequately addresses the exercise statement by following the specified variables and technical requirements. The LLM directly analyses it based on the structured prompt, which already includes student variables correctly placed within the exercise statement. The result is calculated in a (0–100%) grade.

• $\mathrm{■}$

  Student’s Participation in the Project (0–100%): Based on peer assessment, each group member can indicate whether another member contributed to the project. The participation score is calculated as:

  $$\text{Participation (\%)}=100-\left(\frac{\text{notWork}}{\text{TotalAssessments}}\times 100\right)$$

  where notWork is the number of times the student was marked as a non-contributor.

• $\mathrm{■}$

  Student’s Performance in the Project (0–100%): Group members evaluate each other’s contribution on a scale from -4 to 4. The performance score is calculated as that normalises peer-assessed performance into a 0–100% scale:

  $$\text{Performance (\%)}=\frac{\sum \text{grades}+4}{8}\times 100$$

The overall grade produced by the LLM is then interpreted according to the following thresholds:

• $\mathrm{■}$

  -1: The student does not demonstrate basic knowledge (< 25%).

• $\mathrm{■}$

  0: The student demonstrates partial understanding (25%–74%).

• $\mathrm{■}$

  1: The student shows mastery of the content (>= 75%).

The spreadsheet template includes 30 columns and serves as the primary data collection and processing environment. Each row corresponds to an individual student’s response. The sheet’s structure is organised as follows:

• $\mathrm{■}$

  Column A – Student participation score based on peer evaluation;

• $\mathrm{■}$

  Column B – Student performance within the group (group-assessed contribution);

• $\mathrm{■}$

  Column I – Exercise statement and student answer - Part A ;

• $\mathrm{■}$

  Column J – Exercise statement and student answer - Part B;

• $\mathrm{■}$

  Columns K to Q – Student-specific variables collected via the pre-assessment questionnaire (e.g., birth, eid, SID, etc.);

• $\mathrm{■}$

  Column R – First prompt generated for code evaluation based on the statement (Column C) and student inputs (Column K to Q);

• $\mathrm{■}$

  Column U – Second prompt generated for code evaluation based on the statement (Column D) and student input calculated (Column L);

• $\mathrm{■}$

  Column Z – Authorship prompt generated for authorship validation and potential plagiarism detection based on Column A:Q;

• $\mathrm{■}$

  Column AB – Final result, calculated based on the outputs from both prompts (Columns Z and AA) and validated by the professor when necessary.

The missing columns are support columns used to parse the results (e.g., split values by |). This spreadsheet-driven architecture enables the automation of prompt generation, integration with the ChatGPT API, and storage of all relevant metadata needed for evaluation.

A dynamic prompt was automatically generated at Google Sheets by combining the student’s code submission with their personal data and the exercise rules. The prompt follows a two-step structure:

• $\mathrm{■}$

  Step 1 – Code Evaluation: Reads the exercise statement and the student’s code, then assesses the solution based on predefined grading rules.

• $\mathrm{■}$

  Step 2 – Authorship Verification: Cross-references the code with the student’s individual performance to identify potential copy or unauthorised use of generative AI tools.

A simplified version of the evaluation prompt used in Google Sheets is shown below:

This evaluation prompt was generated and stored in column S of the spreadsheet. In this setup, cells $K$1 to $Q$1 contain the names of student-specific variables, and the corresponding values are located in K4 to Q4. The complete exercise statement is placed in $I$2, while the student’s code submission appears in I4.

This formula dynamically integrates all necessary contextual and individualised information into a structured prompt, which the GAI model interprets to assign a preliminary grade.

A second prompt was explicitly designed to support the detection of potential academic misconduct. It cross-references the quality of the submitted code with the student’s performance within their group. By leveraging peer assessment data, this prompt helps identify inconsistencies between student effort and the code quality, factors which may suggest the use of external assistance or generative AI. The following listing illustrates the structure of this second prompt:

In this case, cells A3 and B3 store the names of the variables representing peer- and group-assessed performance, while A4 and B4 contain the student’s actual evaluation results. These values are used to construct a second, personalised prompt that guides the GAI model in making inferences about authorship and originality.

To calculate the final classification, this second prompt complements the initial code evaluation by incorporating behavioural and performance indicators, as described in Subsection 5.2 and stores in result in column AA.

Below are selected excerpts of feedback generated by the GAI algorithm, which combines code analysis with behavioural performance metrics. Each output includes a plagiarism risk assessment, an authorship judgement, a detailed justification, the corresponding final grade, and an indication of missing elements.

Plagiarism: 2  |  Human: Yes

Justification: The code presents some common errors that a student might make, such as incorrect CSS formatting and the absence of placeholders in the form. Additionally, the overall structure of the code is simple and does not perfectly meet all specifications, which is typical of work done by a student. The effort evaluation and the completed work grade indicate that the student can produce acceptable code. Therefore, the probability of plagiarism is low.

Final Grade: 70.00%  |  Missing:

• $\mathrm{■}$

  The margin of the div with id=eid should be defined by the last two digits of the SID (30px), but it was set as 30em.

• $\mathrm{■}$

  The background color of the <h3> should be based on the birth date (22/06/2003), but it was set as #22062003, which is not a valid hexadecimal value.

• $\mathrm{■}$

  The courses table is not using the proper HTML table structure (<table>, <thead>, <tbody>, <tr>, <th>, <td>).

• $\mathrm{■}$

  The form is missing placeholders for fname, course, and opinion.

• $\mathrm{■}$

  The JavaScript code contains syntax and logic errors.

• $\mathrm{■}$

  The object saved in LocalStorage should be named after the eid, but the current implementation does not guarantee this.

Plagiarism: 4  |  Human: No

Justification: The code presents various errors and inconsistencies typical of code generated by AI or copied from a source without proper understanding. Additionally, the student’s effort evaluation is 0, and the completed work grade is 0, indicating that the student did not put in any effort. Therefore, it is highly likely that the code was not done by the student.

The calculated scores ranged from 20% to 78%, with plagiarism suspicion scores varying between 2 and 4, and AI usage likelihood scores also ranging from 2 to 4.

The final grade distribution was as follows:

• $\mathrm{■}$

  -1 (Insufficient understanding): 12 students (10.17%)

• $\mathrm{■}$

  0 (Basic knowledge): 104 students (88.14%)

• $\mathrm{■}$

  1 (Advanced mastery): 2 students (1.69%)

Among the twelve students who received a grade of -1, three submitted a successful request for grade redemption through the Grade Rescue mechanism. The other students did not meet the criteria for a positive reassessment.

These results suggest that the algorithm, combined with human validation, reliably classified student performance while also offering a pathway for reconsideration in exceptional cases.

A feedback form was included in Google Forms to assess students’ perceptions regarding the test and the evaluation strategy based on Generative AI. Students were asked to rate the difficulty of the test and their level of agreement with the proposed model using a 5-point Likert scale. Table 1 summarises the results.

Analysing the results presented in Table 1, it is noteworthy that over 90% of students approved of the proposed evaluation model. Regarding the difficulty level, the majority of students (52.50%) considered the exercise to be accessible, selecting the neutral option on the scale.