The Influence of GenAI on the Evaluation of Computer Programming Students in Higher Education

AI is rapidly transforming educational practices, offering capabilities ranging from intelligent tutoring systems to adaptive learning management platforms. These technologies enable personalized instruction by adapting the content and pace to each student’s profile [19, 9, 3]. Generative AI encompasses systems capable of producing content, including text, images, video, and source code, as well as solving complex logical problems. In programming education, these tools are significantly shifting student’s behavior and reshaping the development of skills such as logical reasoning, algorithmic problem-solving, and proficiency in programming languages [1]. Platforms like GitHub Copilot, ChatGPT, and OpenAI’s Codex can generate code, explain complex programming concepts, and assist in debugging, often in real time [17]. This paper analyzes the pedagogical implications of generative AI tools in computer programming teaching and learning, with a particular focus on their influence on student’s evaluation. It addresses the following research questions: (i) Which generative AI tools are most commonly used by students? (ii) What changes have these tools already brought to the evaluation process? (iii) What mitigation strategies or pedagogical approaches are educators adopting? and (iv) What challenges and concerns do teachers face in assessing students in this new context? The structure of the paper is as follows: section two describes the use of generative AI tools in computer programming learning. Section three provides a brief literature review on the assessment of programming skills in the age of AI. Section four outlines the survey methodology and questionnaire design, followed by results presentation and their analysis, and discussion. Finally, the last section presents some conclusions, key contributions, and directions for future research.

The use of AI in software development has significantly accelerated with the rise of generative AI tools. Powered by large language models (LLMs), tools such as GitHub Copilot and ChatGPT are designed to understand prompts and generate relevant source code in real time [3]. GitHub Copilot, for instance, integrates directly into development environments, offering context-aware code suggestions. These capabilities challenge the validity of traditional take-home programming assessments, as they allow students to generate entire solutions with minimal effort. While AI-generated code often meets assignment requirements, it may lack well-structured design, clear documentation, or optimization, raising critical questions about what competencies are truly being assessed. Generative AI tools are currently being used in several stages of the software development lifecycle, including [4, 15, 14]:

• $\mathrm{■}$

  Code Completion and Generation: tools such as GitHub Copilot suggest real-time code snippets, reducing repetitive coding and accelerating development;

• $\mathrm{■}$

  Bug Detection and Correction: AI models can detect syntax and logic errors and propose fixes, often integrated into modern IDEs;

• $\mathrm{■}$

  Documentation and Explanation: LLMs can generate human-readable documentation, aiding code comprehension and collaboration;

• $\mathrm{■}$

  Learning and Tutoring: these tools act as intelligent tutors, providing beginners with example-driven learning and concept explanations

Despite these advantages, such as faster development, improved code quality, and a reduced learning curve for beginners, serious pedagogical challenges have emerged, particularly in foundational programming education [13, 18]:

• $\mathrm{■}$

  Passive attitude to understanding and learning basic concepts: beginners may rely heavily on AI-generated solutions, bypassing essential learning steps such as understanding syntax, logic, and debugging. This attitude can hinder the development of core programming skills;

• $\mathrm{■}$

  Superficial Understanding of Programming Concepts: generative AI often presents solutions without explaining the underlying reasoning, leading to a “black box” perception of programming. Students may adopt these outputs without truly grasping the principles involved;

• $\mathrm{■}$

  Difficulties in debugging and handling code errors: when AI-generated code fails, students unfamiliar with debugging practices often struggle to diagnose and resolve issues. Generative tools rarely provide detailed walk-throughs of their suggestions, impeding students’ ability to troubleshoot independently;

• $\mathrm{■}$

  Problems related to ethics and plagiarism: the boundary between legitimate assistance and academic dishonesty is increasingly blurred. Instructors face difficulty distinguishing original student work from AI-assisted submissions, raising concerns about plagiarism and assessment fairness;

• $\mathrm{■}$

  Difficulties in understanding and adapting the code: AI-generated code is not always correct or efficient. Beginners may lack the critical skills needed to assess or adapt the generated output, which can reinforce misconceptions and bad practices.

While generative AI tools offer promising possibilities for enhancing personalized learning and supporting the development of more authentic, dynamic assessments, they also raise significant concerns related to academic integrity, equity, and the loss of critical thinking and creativity in student work. A primary concern is academic dishonesty. Students may use AI-generated content to complete assignments with minimal personal input, effectively bypassing the learning process. A recent EDUCAUSE survey [12] reported that more than 60% of faculty teachers observed an increase in AI-generated student submissions, intensifying concerns about plagiarism and authorship. Although tools like Turnitin’s AI detector have been introduced, their accuracy and reliability remain under debate. Research on the integration of generative AI into assessment processes is still emerging. Although many studies highlight the potential of generative AI in education, relatively few provide empirical evidence on its influence on evaluation practices [8]. According to a recent study by Huber et al. [5], both educators and students recognize the profound impact of AI on traditional assessments, particularly in fields such as essay writing and coding. Although many educators advocate adapting assessment strategies to leverage AI in fostering higher-order thinking, students express mixed feelings, often concerned about potential losses in creativity and originality. According to some authors [7], more than 50% of students enrolled in programming courses admitted to regularly using AI tools, with more than 30% employing them for graded assignments. This raises challenges for instructors, as AI-generated code is often indistinguishable from human-written code, especially when slightly modified. To address these issues, researchers such as Kizilcec and Elaine Huber [5] propose a shift toward hybrid and process-oriented assessment models. These may include:

• $\mathrm{■}$

  Live coding exams, to evaluate real-time problem-solving and syntax fluency under supervision;

• $\mathrm{■}$

  Monitoring the code development process, through screen recordings or automated trace logs;

• $\mathrm{■}$

  Code explanation sessions, requiring students to justify and analyze their submissions;

• $\mathrm{■}$

  Version control analysis, using platforms like Git to track iterative work;

• $\mathrm{■}$

  AI-inclusive tasks, such as critiquing or improving AI-generated solutions;

• $\mathrm{■}$

  Project-based learning, which emphasizes design, collaboration, and testing rather than just code correctness.

Despite the possible benefits, these approaches face implementation challenges, particularly in large classes, due to increased demands on teaching time and the need for closer monitoring. As noted by Zawacki-Richter [19], institutions must invest in faculty training, establish clear academic integrity policies, and adopt scalable and process-oriented evaluation strategies that measure both code and understanding.

To investigate the research questions about the impact of Artificial Intelligence, particularly Generative AI tools, on the evaluation process in computer programming subjects, an online survey was conducted and disseminated amongst Portuguese higher education professors, who teach programming-related subjects. The target audience for this study was reached through two main channels: public institutional websites (where contact information of the teaching staff was available) and personal academic networks. Participants were informed that the questionnaire was anonymous, ensuring the confidentiality of the responses. A total of 31 responses were collected from professors from 12 different Portuguese higher education institutions, including both universities and polytechnic institutes. The respondents teach in higher professional technical courses (TeSP), bachelor’s and/or master’s degrees, and represent 3 class formats: in-person, online, and hybrid teaching. This heterogeneity enhances the richness of the dataset, reflecting a broad spectrum of perspectives within the national context. Although it is a relatively small sample size and may not fully capture the diversity of institutional contexts, qualitative studies with limited samples can still yield valuable insights into the subject under investigation [10]. Following other recent computing education research studies [11, 6, 2, 16], the survey combined closed- and open-ended questions to gather both quantitative indicators, useful to identify general trends, and qualitative insights, providing depth and context to participants’ experiences and perceptions. The structure was as follows: Section A – Respondent characterization and curricular context: data on academic background, such as institution and teaching level, subject area, and class description; Section B – Perceptions of AI usage by students: teacher’s point-of-view on students’ awareness of AI tools, particularly code generation; Section C – Changes in Evaluation methods and Adopted Strategies regarding AI tools: investigation on whether and how assessment methods have been adapted in response to the AI usage; Section D – Future Perspectives/Challenges: forward-looking reflections about AI in computer programming assessment.

This research outlines the impact of generative AI on evaluating students of computer programming courses in higher education. Results reveal that while these tools offer significant pedagogical opportunities (such as enhanced autonomy, real-time feedback, and support for project-based learning), they also pose substantial challenges and concerns, such as the lack of critical reflection, excessive reliance on these tools, and along with difficulties in ensuring authorship. Most professors reported changing their evaluation methods, adopting strategies such as oral assessments, project-based tasks, and emphasis on reasoning. The evidence indicates a definitive shift towards a more reflective, process-oriented, and diversified assessment strategy, with the hope of preserving the integrity of computer programming learning in the age of AI. There is also a clear call for integrated actions on an institutional level, rather than isolated efforts by individual teachers, and the urgency of such support is strongly linked to the deep feeling that AI is changing the way students learn and how they are evaluated. The study contributes by offering a case study from Portuguese higher education, offering practical implications regarding emerging assessment practices (e.g., reasoning and process description, prompt analysis, AI usage detection) and contributes to research about AI-integrated pedagogy, specifically focused on assessing programming skills in higher education. Many studies focus on the use of AI as a teaching support tool, without adequately addressing its impact on the evaluation phase of the teaching-learning process. This study seeks to fill this gap by analyzing real-world practices adopted by professors of computer programming courses. The intersection of AI, teaching programming, and evaluating programmatic skills requires an integrated and critical pedagogical approach. Technology alone cannot solve the challenges of education, but it can be a valuable ally when used intentionally and consciously. Teacher training, investment in infrastructure, and a commitment to ethical and inclusive education are necessary paths to consolidate these innovations in higher education. Future work should explore students’ critical examination of generative AI tools, as well as their ethical and cognitive concerns regarding AI-assisted learning and shifting evaluation methodologies. Moreover, the evolving nature of generative AI tools warrants the need for a comprehensive focus on long-term studies on the impact these tools have on developing programming skills.