A Game of Shadows: Effective Mastery Learning in the Age of Ubiquitous AI

The rise of generative AI and Large Language Model-based tools has a significant impact on both the field of computing, and computing education. Computer scientists, educators, and practitioners are having heated discussions about the changes in the nature of programming, and its influence on learning. At the same time, there is an opportunity to use LLMs to support students better than we used to. I identify four “burning issues”:

The rapid advance of Generative AI (GenAI) in education presents both unprecedented opportunities and significant challenges for mastery learning. In my lightning talk, I focused on two critical areas: (1) Understanding students’ use of GenAI tools and (2) Fostering their adoption among educators.

Firstly, there currently is a gap in our understanding of how students engage with GenAI tools, and we should investigate the state-of-the-art with regard to their interaction patterns, prompting strategies and application. This will help inform our instructional strategies and assessment methods, ensuring we leverage these tools to enhance learning.

Secondly, we need to consider the adoption of GenAI tools among educators. So, I discussed the importance of designing and implementing support structures for educators at higher education institutions. This may include offering workshops, mentoring programs, and other opportunities of sharing knowledge, e.g., at local conferences. By training educators, we can collectively harness the full potential of GenAI.

Bloom defines one of the core tasks of teaching “Our basic task is to determine what we mean by mastery of the subject and to search for the methods and materials which will enable the largest proportion of our students to attain such mastery” (Bloom 1968, p 1). This quote suggests two fundamental questions concerning the intended learning outcomes of assignments/courses/degree programs: 1) Mastery of what? What it is that we want our students to master? This question can be discussed at diQerent levels. Micro level: e.g. concepts, fractions of larger topics that are discusses during a teaching session. Course level: what knowledge, skills and attitudes do we want our students to master after a X week course? Macro level: degree program level intended learning outcomes. What kind of competencies do we wish our graduates to have?

The second question is what we mean by mastery of something? Can we clearly define what it means to master some aspects of knowledge, skills or attitudes? Is mastery always easily observable for the teacher (or student themselves). This is a challenging issue especially when we think about macro level learning objectives. Finally, I want to raise a question to what degree we have empirical evidence that mastery learning is beneficial for all students and not only for those with good selfregulations skills.

In this lightning talk I discuss “Mastery Learning” and its challenges in modern education. I explore the question of how education can be scaled efficiently, with the solution proposed being mechanization. The idea is to break down learning into smaller, manageable pieces, which can then be addressed systematically. However, a central issue identified is that this approach often leads to isolated bits of knowledge, making it difficult for learners to grasp the overarching picture.

Connecting these fragmented pieces to form a coherent understanding is critical. The challenge, therefore, lies in bridging the gap between detailed, mechanized learning and the ability to perceive the broader context. We discuss strategies for overcoming this issue, offering solutions for how mastery learning can be improved to create a more holistic educational experience.

The increasingly powerful generative AI models have brought some benefits to software engineering. There is emerging evidence that using tools such as GitHub Copilot can increase productivity for professional software developers [1]. For novice programmers, however, there is recent evidence that while the best students can benefit from generative AI tools, struggling students do not get the same benefits [2]. In fact, struggling students can even be harmed by them. AI tools can become a crutch, leaving students with a false sense of mastery.

In my lightning talk, I proposed that we could try to combat this “widening gap” between students by explicitly teaching them how to prompt AI models to write code with Prompt Problems [3]. This could help struggling students to get the benefits of generative AI that are currently enjoyed by professional programmers and the best performing students.

Generative AI can generate solutions to simple programming problems typically used as exercises to help students learn in introductory programming courses. We can no longer rely on assessment tasks as a motivator to learn “content knowledge” because our standard tools make such tasks redundant. We need to shift student motivation from focusing on content knowledge acquisition to understanding why the content is important, and how we use the content effectively. This requires rethinking the capabilities that we expect from students and the order in which they are acquired.

For students who are motivated to learn programming, we need to focus attention on teaching students to use GenAI to support their learning. Given the high degree of flexibility in access to learning resources provided by GenAI, we should explicitly discuss metacognitive skills and encourage students to seriously reflect on the processes they are using to learn. This has significant implications for learning in general, and for Mastery Learning approaches.

Recent research on generative AI highlights its potential to enhance education through just-in-time learning materials and personalized tutoring. However, our lab has conducted multiple studies which also reveal potential harms to students.

Our first paper explores the impact of generative AI on student help-seeking and classroom social dynamics. While AI can provide immediate assistance to students, which benefits students who might experience social barriers to seeking help, our findings suggest it might inadvertently reduce student interactions. This diminished peer-to-peer engagement can lead to a less collaborative learning environment, where students no longer seek out and establish social support groups on which they have previously relied.

In our second paper, we investigate the risks generative AI poses to student metacognition. Metacognition, or the awareness and understanding of one’s own thought processes, is critical for effective learning. Our research indicates that AI can sometimes mislead students, giving them a false sense of understanding and progress.

Through these and other projects, we are working to identify the nuanced harms associated with the use of generative AI in educational settings. Our goal is to develop strategies and best practices that maximize the learning benefits of AI while mitigating its potential downsides. If successful, we hope to pave the way for the responsible use of generative AI in education, ensuring that it serves as a tool for enhancement rather than a crutch that diminishes the quality of learning.

Traditionally, in computer science courses in higher education, a teacher introduces a new concept, a student then practices that concept, and the teacher will test the student’s knowledge of that concept, likely among many others. In a Mastery Learning setting, this process gets atomized such that a student has to fully understand a concept before moving on to the next concept. However, in either case, ChatGPT, Github’s CoPilot, and other widely available AI-based large language models (LLMs) are fundamentally changing what it means to get information, code, and learn. Rather than ignore or ban the use of these tools, we can harness this moment to rethink how we teach computer science, potentially in a way to support mastery learning. If code generation is suddenly less time-consuming, but potentially wrought with errors, shifting our instruction to focus more on code reading and explaining will be crucial. Further, LLMs can help us generate the multitude of assessments that are needed as each student learns a concept to demonstrate mastery. With their powers combined, LLMs can help support more students in more mastery learning settings, completely changing the landscape of what it means to teach computer science.

Self-explanations can help learners engage with course content and to build connections between topics. However, self-explanations need to be connected to prior concepts to be effective, and many learners do not self-explain effectively. We propose using an LLM to compare expert-generated explanations with learner self-explanations and to prompt the learner to revise and elaborate on their self-explanations. In a pilot study of this approach, we found that students generally responded to LLM prompts and responded to the feedback but did not necessarily update their self-explanations accordingly. We also detected that students felt a sense of social presence with the LLM system, reacting to it having access to their self-reflections.

At the beginning of 2023 in a staffroom conversation about generative AI tools, a colleague remarked, “I don’t think my students know about these yet”. Now 18 months later, the world is a different place. A study of instructors’ perceptions of generative AI tools I conducted with a group of computing education researchers found many ideas about the benefits and challenges to teaching and assessment [1]. Probably the greatest challenge was the impact on learning. As one participant claimed, “It is a great tool … to prevent yourself from learning”. When using a generative AI tool are students handing over the responsibility for learning to the tool and in effect the tool becomes a surrogate for learning?

Mastery learning emphasizes achievement of a high level of understanding in a given topic before moving on to the next. Key aspects: clear learning outcomes, individualised pacing of learning, formative assessment, support, summative assessment. When considering a mastery learning approach the availability of generative AI tool brings particular challenges. Students using generative AI tools can take a passive learning approach, engage superficially with the tools leading to surface rather than deep learning. An over-reliance on feedback from the tools can impact negatively on development of critical thinking and problem-solving skills. With less interaction with humans students may miss important guidance and emotional support. The result is an illusion of learning where students feel they are learning effectively while not developing deep understanding.

Failure is an important part of the learning process, but most students have negative feelings about it. Students can respond differently to situations where they expect their risk of failure is high. Some students may seek human guidance from an instructor or peer, while others prefer to handle the situation privately, without anyone knowing. Fear of failure can be a barrier in mastery learning, such as if a student is not willing to take a mastery quiz. Generative AI may be able to play a role in supporting students in pushing through a situation where they are worried about failure. The availability of generative AI can also help with scaling of this kind of support, as many current course environments do not have the human resources to provide one-on-one coaching to students.

Some challenges in integrating generative AI into education in general are the diverse range of guidelines students are receiving from their instructors on what is and is not appropriate use of generative AI. This can be a result of instructors’ own lack of understanding of the use of generative AI tools, and how they can be appropriately integrated (or not) into their classrooms. Instructors’ own concerns about the ethical implications of these tools may also affect their adoption. We also need to consider that students may have their own ethical considerations about the use of generative AI, whether for reasons of privacy, climate concerns, or discomfort interacting with a machine.

Mastery learning techniques are an excellent fit for foundational programming education, especially for diverse postgraduate cohorts. These techniques emphasize the importance of allowing students to progress at their own pace, which is essential when considering the varied levels of programming knowledge within such groups. Students may come from both cognate (closely related to programming) and non-cognate (unrelated) backgrounds, which means their learning needs can differ greatly.

In light of these differences, a self-paced, practice-based approach is critical to address individual learning speeds and requirements. Tailoring learning pathways to accommodate both experienced programmers and beginners ensures that each student can work through material at a comfortable pace. This method supports a more personalized educational experience that encourages deeper understanding rather than a one-size-fits-all approach.

Despite the promise of this method, practice remains a formidable challenge in educational settings. Creating consistent, quality practice opportunities that cater to each student’s needs is often difficult, particularly when faced with the diverse range of programming skills and learning behaviors. Providing the right resources and feedback at the right time is key to making practice an ally rather than an obstacle.

Moreover, leveraging AI in educational contexts to create more effective and inclusive learning environments remains an uphill battle. While AI holds significant potential for tailoring education to individual students’ needs, the complexities involved in implementing these technologies can sometimes feel insurmountable. However, overcoming these challenges is crucial to fully realize the potential of mastery learning in programming education for diverse cohorts.

The talk explores the implementation of mastery learning in computing education and the challenges instructors and students face. Mastery learning emphasizes ensuring students achieve a high level of understanding before progressing, but the process is demanding for both parties. Instructors encounter difficulties in distilling course objectives, creating materials, and continuously assessing and providing feedback. Meanwhile, students must develop strong self-regulation skills to meet mastery criteria, presenting its own set of challenges.

Large language models (LLMs) have the potential to ease the burden on instructors as they could <could assist in refining course objectives, generating assessments, and offering adaptive feedback to students. The question, however, is how such models can be integrated effectively beyond introductory courses like CS1. Moreover, addressing these concerns calls for resources that provide clear guidance on leveraging LLM tools in a meaningful way.

Another critical aspect is the need to explore gradual adoption approaches to LLMs in mastery learning. Risks such as over-reliance, student data privacy, and loss of personalized instruction need to be mitigated. Instructors are encouraged to explore incremental adoption to balance innovation with careful oversight, ensuring that the integration aligns with educational goals.

Finally, it is important to reflect on the trade-offs associated with using LLM tools in computing education. While such technologies may streamline some instructional processes, there is a risk of losing valuable interpersonal elements of teaching.

Converting a traditional CS1 course to a Mastery Learning format is challenging, with uncertain outcomes if key needs and risks are not properly managed. Coordinating a large course under this format requires significant effort, from providing support to students in different modules to creating multiple formative and summative assessments for varying levels of mastery. Feedback must be delivered rapidly, and new lecturers must be trained, as mastery learning is not widely endorsed. Based on eight years of experience implementing a CS1 course in this format at Universidad de los Andes, Chile (see main ref), we have identified three critical factors for success: effective course management, ensuring quality in teaching and assessment, and maintaining scalability and efficiency. Generative AI technologies offer promising tools to support mastery learning by reducing barriers, costs, and risks. Key aspects of the pedagogy, such as personalized learning, assessment creation, and feedback generation, can be automated, while course management tasks like communication and staff training can also be enhanced. This raises important questions: what AI tools are most suitable for these improvements? How can they be tailored to the diverse needs of mastery learning implementations in computer science? How can we measure their impact to ensure effectiveness? Widespread adoption of mastery learning, supported by generative AI, would enable institutions to improve educational outcomes and provide students with deeper learning experiences, preparing them for future professional challenges.

This talk explores the effects of AI on mastery learning in computing education. It outlines key challenges, including increased procrastination, widening achievement gaps, and potential misinformation from AI-generated content. The discussion then shifts to innovative strategies for leveraging AI to enhance mastery learning, such as AI-powered self-regulation tools, robust feedback mechanisms, and adaptive learning paths. By addressing both the pitfalls and potential of AI in computing education, this talk aims to suggest more effective integration of AI tools in the pursuit of deep, foundational learning.