Interactive Evaluation of Complex Programming Assignments Using LLM Assistant

The capabilities of LLMs in code generation and analysis have been explored extensively. Several systems like GitHub Copilot, which utilizes OpenAI’s Codex and GPT-4 models, have demonstrated impressive performance in generating functional code blocks based on natural language prompts or existing project context [2]. These systems integrate into popular development environments and assist programmers in various tasks, including code generation, refactoring, error identification, and documentation.

Research by Ziegler et al. [9] on productivity assessment of neural code completion tools found significant improvements in developer performance, particularly for routine programming tasks. The integration of these tools into programming workflows has shown potential for enhancing both efficiency and code quality.

KP Assistant was developed as a customised GPT (Generative Pre-trained Transformer) using OpenAI’s GPT Builder platform. This approach allowed for precise configuration of the model’s behaviour through specialised instructions, knowledge base integration, and capability settings. The underlying model selected was GPT-4o due to its superior performance in code understanding and analysis during preliminary testing compared to alternative models.

An important characteristic of KP Assistant is that it was fully configured to operate in the Slovak language, with all instructions, prompts, and interactions designed specifically for Slovak-speaking students. This language specification was critical for seamless integration into the Technical University of Košice’s curriculum, where Slovak is the primary language of instruction. However, some leeway was provided in terms of in-code comments or technical terms, which are by industry best practices in the English language.

Based on previous research at our department and other universities, we were able to design clear instructions for the GPT model in terms of expected answer structure and overall GPT’s persona. Numerous sources provide recommendations for teachers and instructors in undergraduate and graduate study programmes. These are usually based on the observed behaviour and work results of students. One of the largest and more recent studies with well documented results is written by García et al.[3], where they conclude that informal and more open behaviour leads to better results. Özdaş [10] came to a similar conclusion, along with a list of key characteristics of a positively viewed educator, which were easy to convert to character traits of our GPT. Configuring our GPT’s behaviour based on these and other relevant findings created a hypothetically ideal persona for our assistant.

The assistant was configured with specific instructions divided into three main categories:

• $\mathrm{■}$

  Basic Instructions: Defined the overall purpose, interaction style, and operational parameters.

• $\mathrm{■}$

  Conversation Start Instructions: Established fixed prompts for initiating different types of assessment sessions.

• $\mathrm{■}$

  Question Generation Instructions: Detailed guidelines for analysing student code, generating relevant questions, and evaluating responses.

These instructions were not provided all at once to the assistant but were served behind the scenes at predetermined phases of conversation. The whole conversation flow is visualized in the provided diagram (Fig. 1), which shows expected input and action based on it. Simply said, these instructions were served if previous input was deemed valid and sufficient for further processing, which streamlined the process of the whole conversation. Additionally, in this case we personally witnessed that hallucinations from GPT were more scarce, since it was validating itself. Naturally, this more complex design made conversation slightly slower, mainly after users uploaded their assignment. However, delay after this step was previously present as well, and the time difference was deemed insignificant (approx. 10 seconds longer in 10 testing runs).

The assistant’s knowledge base was populated with course materials, including lecture presentations and descriptions of game assignments that students could implement. This provided essential context about the course structure, learning objectives, and assignment requirements. Web search and code interpretation capabilities were enabled to allow access to the course website and facilitate source code analysis.

Three conversation starters were defined corresponding to the three assignment submission phases in the Component Programming course:

• $\mathrm{■}$

  Submission 1: Game logic and JDBC

• $\mathrm{■}$

  Submission 2: JPA + REST

• $\mathrm{■}$

  Submission 3: Web browser game with GUI

For each submission type, the assistant was configured to generate and assess five questions based on the submitted code: two easy questions (3 points each), two medium-difficulty questions (5 points each), and one difficult question (6 points). Additionally, the assistant was programmed to evaluate specific aspects of the implementation based on course criteria, such as game functionality, service component integration, code quality, and testing.

Six different game implementations were used to assess the assistant’s performance across various code structures and quality levels: Battleships, Lights Off, Color Sudoku, NumberLink, Reversi, and Connect Four. These implementations varied in complexity, code organization, and overall quality, providing a comprehensive test bed for the system.

For each game, a structured testing procedure was designed and conducted:

• $\mathrm{■}$

  Initiating a conversation using one of the submission options

• $\mathrm{■}$

  Uploading the game’s source code as a ZIP archive

• $\mathrm{■}$

  Answering the generated questions

• $\mathrm{■}$

  Analysing the final assessment and score

• $\mathrm{■}$

  Testing various edge cases and unexpected behaviours

A total of 16 students from the Component Programming course participated in testing the system, with 12 providing valid feedback. Students were instructed to follow provided steps in this order:

1. 1.

   Access KP Assistant using a provided link

2. 2.

   Select the appropriate submission – in this case, first submission option

3. 3.

   Upload their own game implementation

4. 4.

   Answer the generated questions

5. 5.

   Complete a feedback survey evaluating their experience

The survey collected both quantitative assessments (on a 5-point scale) of conversation flow, question relevance, clarity, and evaluation objectivity, as well as qualitative feedback on perceived strengths and weaknesses of the system.

Data was collected from both experimental phases, encompassing conversation logs from the system tests with the six reference games, student-assistant interaction transcripts, and comprehensive survey responses from student participants.

The analysis methodology centred on identifying recurring patterns in three critical areas: the quality of question generation, the accuracy of assessment procedures, and any usability challenges that emerged during testing. Evaluation criteria were specifically designed to measure the relevance of generated questions to the actual submitted code, verify assessment consistency against expected scoring benchmarks, evaluate the natural flow and clarity of conversations, detect instances of hallucination where questions referenced nonexistent code elements and gauge overall student perception of the system through their qualitative feedback.

Testing with the six reference game implementations revealed several key insights about KP Assistant’s performance:

• $\mathrm{■}$

  Battleships: The assistant performed well with this well-structured implementation, correctly analysing the code structure and generating relevant questions. The final assessment (33/34 points) aligned with the expected score based on code quality and response accuracy.

• $\mathrm{■}$

  Lights Off: During this test, the system was deliberately challenged by first uploading incorrect file formats and then a valid but unrelated ZIP file. The assistant correctly identified these issues and prompted for a proper submission. When the actual game code was uploaded, one hallucination occurred where the assistant asked about a nonexistent method. After being informed of this error, it generated a replacement question, but failed to maintain the correct question sequence, skipping directly to the fifth question. The final score (24/34) appropriately reflected the code quality and response accuracy.

• $\mathrm{■}$

  Color Sudoku: The assistant generated generally relevant questions, but one hallucination occurred. Interestingly, when provided with a response that pointed out the hallucination, the assistant accepted the answer as correct and awarded full points. This behaviour indicates a potential evaluation vulnerability when students identify system errors. During further testing, we were not able to replicate this behaviour in 50 attempts with the same input, but its singular occurrence proves that even when analysing a fairly common game, the LLM cannot be completely trusted.

• $\mathrm{■}$

  NumberLink: When testing different submission types, the assistant could successfully transition between them upon request. However, when challenged about point deductions, the assistant occasionally displayed flexibility in scoring that could potentially be manipulated, suggesting the need for stricter evaluation protocols.

• $\mathrm{■}$

  Reversi: This test involved a large codebase (42MB ZIP archive), which revealed limitations in the assistant’s code analysis capability. Examination of the code analysis logs showed that the assistant only processed the first ten files in the archive, leading to incomplete understanding of the codebase and subsequent hallucinations in questions.

• $\mathrm{■}$

  Connect Four: This implementation was intentionally minimal and low quality. The assistant correctly awarded low scores (3 points) for the basic aspects, reflecting the poor implementation. When asked to generate a complete working implementation of the game, the assistant appropriately refused and instead offered guidance on resources and documentation that would help improve the implementation.

The student evaluation provided valuable insights into the practical application of KP Assistant in an educational setting. The distribution of game implementations among the 12 valid student responses. We had additional games available among our submitted assignments, which allowed us to test the versatility of KP Assistant in scenarios that were not mentioned in the provided materials. Additional games included Boulder Dash, Pipes, Nonogram, Bricks Breaking, Dots, Kakuro, Checkers, and Slitherlink. Two of these games, Dots and Kakuro, were made by two individual students, which allowed us to observe grading differences among students with the same assignment but slightly varying results.

Students rated the conversation flow and clarity highly, with an average score of 4.42/5. Eight students gave the highest rating (5 – excellent), one rated it as excellent (4), and three rated it as good (3). This suggests that the assistant’s communication style and instructions were generally clear and easy to follow. The relevance of generated questions to the actual code submitted received a moderate average rating of 3.42/5. Distribution was more varied, with three excellent ratings, three very good, three good, two below average, and one poor. This metric correlates with the hallucination issues observed in the reference game testing, indicating that question generation based on code analysis remains a challenging aspect of the system. Clarity and unambiguity aspect received an average rating of 3.58/5, with two excellent ratings, six very good, two good, one below average, and one poor. The assistant’s ability to formulate clear, grammatically correct questions in Slovak was generally successful. Students perceived the final assessment as relatively objective, with an average rating of 4/5. Five students rated it as excellent, four as , one as good, and two as below average. This indicates that despite some issues with question generation, the overall evaluation framework generally aligned with student expectations of fair assessment.

Among many takeaways, most obvious was the conversational approach which allowed for a more engaging and natural assessment experience compared to traditional automated testing systems. Students appreciated the clear communication style and the opportunity to explain their implementation decisions. KP Assistant demonstrated the ability to analyze various game implementations and generate relevant questions about their specific structures and features. This adaptability represents a significant advantage over rule-based assessment systems that typically require separate configuration for each assignment type.

The system successfully integrated both question-based assessment of student understanding and automated evaluation of code quality and functionality aspects. This holistic approach aligns well with educational best practices that emphasize both product and process evaluation. Unlike traditional office hours or scheduled defenses, the assistant remains available 24/7, allowing students to practice and prepare for formal assessments at their convenience. This could be particularly valuable for distance learning scenarios or courses with large student populations. Well structured prompts allowed for relatively straightforward adaptation to different courses, programming languages, and assessment criteria. This versatility suggests potential for broader application across the computer science curriculum.

Despite its promising capabilities, several limitations emerged during the evaluation of KP Assistant. The most significant technical limitation observed was the assistant’s restricted ability to process and understand large codebases comprehensively. The processing of only the first ten files in larger projects led to incomplete code understanding and subsequent hallucinations. This suggests that improvements in code indexing and analysis are necessary for handling complex projects. The occurrence of questions about nonexistent code elements represents a critical challenge for educational applications. Such hallucinations could confuse students and potentially undermine their confidence in both their code and the assessment system. The assistant’s tendency to accept explanations about hallucinated questions as correct answers indicates a potential vulnerability that could be exploited. A more robust verification mechanism is needed to maintain assessment integrity. The most effective mitigation of this issue was the implementation of carefully crafted assignments which were designed for this use case. To utilise LLM assistant with the best results, we tailored specific assignments (games in our case) specifically with future analysis by LLM assistant in mind. We suggest creating these assignments with a maximum of 8 files in mind, since students can potentially create their own documentation or environment files.

From architectural perspective, the observed behaviour changes following an OpenAI platform update highlighted the potential instability of systems built on third-party AI services. Educational applications require consistency and reliability that may be challenging to maintain with rapidly evolving commercial AI platforms. The current implementation also requires manual ZIP file uploads rather than direct integration with development tools or submission systems, creating friction in the assessment workflow. This can be eliminated by switching to locally deployed models, which can be integrated into complex pipelines, but at the cost of convenience for both user and developer.

While students can freely access and use the custom GPT without a paid subscription, creating and maintaining such customized assistants requires a paid OpenAI subscription for developer. This introduces an ongoing cost consideration for educational institutions looking to implement and maintain similar systems long-term.

KP Assistant and similar systems should serve as supplements to, rather than replacements for, instructor evaluation. Their primary value lies in providing additional practice opportunities and preliminary feedback before formal assessment. For the Component Programming course specifically, the assistant could be utilized as a pre-defense preparation tool, allowing students to identify potential weaknesses in their implementation before the official submission. Implementation should include mechanisms for instructor oversight, such as conversation logging and the ability to review and override automated assessments when necessary. In the context of Technical University of Košice’s curriculum, such features could be implemented through the existing GitLab system, where conversation logs could be stored alongside student submissions for reference during final evaluation.

We should inform students about the system’s limitations to set appropriate expectations and encourage critical evaluation of the assistant’s feedback. The documentation for the Component Programming course could include specific guidance on how to interpret and critically evaluate KP Assistant’s responses, particularly regarding the known hallucination issues with complex codebases. Regular updates to the system’s knowledge base and instructions based on observed performance and student feedback are essential for maintaining relevance and accuracy. We should refresh lecture materials and game descriptions in the assistant’s knowledge base for each semester to ensure alignment with current course content. The experience from one academic year could inform improvements for the next.

Combining dialogue-based assessment with traditional automated testing could leverage the strengths of both approaches while mitigating their respective weaknesses. The Java-based projects in the Component Programming course already include unit tests, which could be complemented by KP Assistant’s understanding-focused questions to provide a comprehensive assessment of both code functionality and student comprehension. This implementation enhances comprehension by providing Slovak language support. Maintaining proper localization is crucial for educational applications targeted at large groups of students. Future iterations could strengthen the Slovak technical vocabulary used in programming contexts, ensuring that terminology is consistent with that used in lectures and course materials at Slovak universities.

Ideally, phased implementation beginning with lower-stakes assignments before expanding to critical assessments would allow both students and faculty to adjust to this new assessment paradigm. For example, KP Assistant could first be used only for the initial submission in the Component Programming course before potentially expanding to later submissions. Because of the adaptable design shown with KP Assistant, we could create similar tools for other programming classes, ensuring a uniform assessment experience throughout the computer science program while still meeting the unique needs of each class. This claim is supported by the fact that our assistant was able to analyze the source codes of games that were not included in the provided materials but had similar mechanisms and components inside.

We utilised our previously existing bundle of solutions to evaluate student assignments – ARENA. This system has been developed over the last 5 years, and it has become part of half a dozen subjects in our department. Further details can be viewed in the previous volume of ICPEC[5], which also refers to the previous versions of the system. One component of this system in particular, ORACLE, provides personalised feedback to the students, based on the runtime of their solutions, their coding practices and their way of designing the algorithms. Along with KP Assistant, it helps provide regular and relevant feedback to the students, which, as proven before, causes noticeable improvement in their performance. In the future, we aim to create a self-hosted variant of KP Assistant, which can be deployed and implemented into the ARENA system. This will provide us valuable data about learning patterns, common issues, misunderstandings and mistakes. Adding KP Assistant to the ARENA system will allow the LLM to access student and reference source codes on the department’s version control system, which will speed up the process of evaluation and simplify the process from the side of the students.