RAGent: A Self-Learning RAG Agent for Adaptive Data Science Education

Data-science courses attract students from computer science, mathematics, biology, and other disciplines [11]. This disciplinary breadth brings uneven technical preparation, posing instructional challenges. Heterogeneous backgrounds and fast-moving content generate a steady flow of conceptual, technical, and administrative questions. When instructors or TAs cannot reply promptly and precisely, learning process suffers. Static resources – discussion forums and FAQs – are often either overwhelming or too generic to capture question-specific nuance. Scalable, course-aware support is needed so staff can focus on higher-order interactions. Advances in large language models (LLMs) [12] and Retrieval-Augmented Generation (RAG) [9] offer such scalability. Although prior work validates LLMs in education [17, 13], generic models lack course grounding, and student queries are frequently ambiguous. We introduce RAGent, a specialized RAG assistant for University of Maryland undergraduate data-science courses. RAGent couples GPT-4o with a knowledge base built from lecture notes, the syllabus, and supplementary materials. Initially piloted in graduate courses, it is evaluated here with undergraduates to test cross-population adaptability. This work contributes: (1) an implementation of a university-level RAG agent built with LangChain, FAISS, GPT-4o, and Streamlit; (2) a multi-dimensional query classifier labeling questions as relevant-known, relevant-unknown, or irrelevant; (3) a self-learning loop capturing unanswered queries for knowledge base expansion; (4) a rigorous evaluation framework with categorized question sets and metrics; and (5) an analysis of feedback from 125 undergraduate students on usability and experience.

RAGent employs RAG to ground responses in verified course-specific knowledge sources [9], addressing limitations in traditional LLMs such as hallucination tendencies. The system implements a hybrid architecture combining dense retrieval with generative modeling through five stages, as illustrated in Figure 1, with detailed component interactions shown in Figure 2.

RAGent’s implementation integrates several specialized libraries: Langchain for orchestrating RAG components, FAISS for efficient similarity search, OpenAI’s text-embedding-ada-002 model, GPT-4o for response generation, and Streamlit for the user interface. A Postgres database stores relevant but unanswered questions for instructor review. This stack creates a modular architecture where components can be independently optimized as educational needs evolve or new AI capabilities emerge. Source materials are parsed and segmented using recursive character text splitting, producing coherent chunks of approximately 1000 characters with 200-character overlap to preserve context across boundaries. Each chunk maintains metadata links to its source file, page, and offset. We embed chunks with text-embedding-ada-002 (1536-dimensional vectors) and store them in a FAISS index optimized for high-dimensional nearest-neighbor search. When a student submits a question, the classification module assigns one of three relevance labels. For in-scope queries, RAGent performs cosine-similarity search and selects $K\in [3,7]$ chunks based on local embedding density. The retrieved context, classification metadata, and original query are injected into a GPT-4o prompt template that enforces citation formatting; the LLM then generates responses with inline source attributions. Queries labeled relevant but unanswered are stored in Postgres for instructor review. Verified answers are embedded and merged into the FAISS index during periodic ingestion jobs, expanding coverage without redeployment. The Streamlit interface provides real-time interaction for students, presenting answers with collapsible source excerpts. An administration dashboard allows teaching staff to monitor unanswered queries, author responses, and trigger re-ingestion when needed.

We evaluated RAGent through a dual-pronged approach combining controlled quantitative assessment with ecological user testing involving 125 undergraduate data science students at the University of Maryland. Our controlled evaluation employed a stratified test corpus with balanced representation of in-scope and out-of-scope queries across conceptual understanding, technical implementation, and administrative concerns. We incrementally tested with sample sizes of 100, 200, 300, and 400 queries to assess performance stability and scalability. For each query, we recorded classification decisions, retrieval metrics, and response characteristics, comparing outputs against the ground truth labels using standard information retrieval metrics: accuracy, precision, recall, F1-score, and error rates. The ecological evaluation engaged students in structured interaction sessions through a Streamlit-based interface. Participants posed diverse questions (administrative, conceptual/technical, and deliberately out-of-scope) and completed post-interaction feedback forms capturing both quantitative ratings and qualitative impressions. Metrics included response correctness, classification precision, system reliability, user satisfaction (10-point scale), adoption willingness (Yes/No/Maybe), and qualitative feedback. For all evaluations, RAGent operated with its complete architecture using authentic course materials, with experiments conducted on standardized configurations to ensure reproducibility. This combined approach provided both rigorous performance measurements and insights into authentic user interactions, enabling comprehensive assessment of RAGent’s educational efficacy and implementation readiness. The controlled experiments demonstrated RAGent’s ability to distinguish between relevant and irrelevant queries with high accuracy, while the ecological testing revealed strong user acceptance and identified improvement opportunities. The dual methodology allowed us to verify that performance characteristics observed in controlled settings translated effectively to real-world educational contexts, providing validation of both technical capabilities and practical utility. Through this evaluation framework, we were able to systematically assess RAGent’s performance across multiple dimensions critical to educational technology: technical accuracy, appropriate boundary recognition, response quality, user experience, and pedagogical alignment. These findings informed both our understanding of the RAGent’s current capabilities and our roadmap for future enhancements to further improve its educational value.

Our evaluation of RAGent produced both quantitative performance metrics and qualitative user insights that collectively demonstrate the framework’s effectiveness in educational settings.

While RAGent demonstrates considerable promise, our evaluation identified several opportunities for enhancement. Key work directions include knowledge boundaries for specialized queries, performance variance across query types, lack of visual capabilities, cross-topic synthesis challenges, and interface customization. Our enhancement roadmap addresses these constraints through three strategic dimensions: multimodal capabilities for visualizing statistical concepts, enhanced retrieval mechanisms using hierarchical approaches and knowledge graph representations, and adaptive interface customization based on user preferences. Additionally, we plan to implement a temporally-aware “Gradual Knowledge Expansion” that aligns with course progression, where for any given week $N$, the system will only access content from weeks $1$ through $N-1$ plus evergreen materials. This approach keeps answers temporally relevant, prevents exposure to upcoming concepts, and maintains alignment with instructional pacing. Looking beyond single-course implementations, we are developing a generalized framework with automated ingestion routines, course-agnostic prompt templates, a low-code instructor dashboard, and inter-course knowledge sharing mechanisms. This abstraction would significantly reduce deployment overhead and facilitate institution-wide scaling, transforming RAGent from a course-specific tool to a broadly applicable educational platform that can adapt to diverse disciplinary contexts while maintaining domain-specific relevance. Additional enhancement would include multimodal capabilities to support image and code visualization. Another promising direction is represented by the personalized response generation based on individual learning history and performance patterns, allowing RAGent to adapt explanation complexity and provide targeted resources in regard to each student’s knowledge level. To support broader adoption while maintaining institutional flexibility, we plan to develop comprehensive documentation and implementation guidelines to allow other institutions reproduce RAGent’s architecture using readily available components. This includes detailed technical specifications, configuration templates, and best-practice recommendations for adapting the framework to diverse curricular.

This paper has presented RAGent, a specialized RAG-based intelligent agent addressing the unique challenges of university-level data science education. By integrating LLM capabilities with course-specific knowledge retrieval, RAGent enhances academic support accessibility, consistency, and scalability. Our evaluation, combining controlled experiments with ecological testing involving 125 undergraduate students, demonstrated RAGent’s effectiveness in distinguishing between relevant and irrelevant queries while providing contextually appropriate responses. The framework showed consistent performance improvements across increasingly diverse question sets (accuracy improving from 95.0% to 96.5%), suggesting robust generalization capabilities. Student feedback was overwhelmingly positive, with 94.4% of participants expressing interest in integrating the tool into their course. RAGent’s novel contributions include: (1) a multi-dimensional query classification system that effectively categorizes student questions, (2) a self-learning mechanism that captures unanswered but relevant queries for knowledge base expansion, and (3) a contextually aware response generation approach that maintains alignment with course terminology. These features address the challenges posed by ambiguous student queries and the structured nature of course content. While our evaluation identified certain opportunities for improvement – including knowledge boundaries, performance variance, and lack of multimodal capabilities –these provide clear directions for future development through progressive knowledge expansion, multimodal support, and cross-course generalization. The broader implications of this work extend beyond a single course implementation. RAGent demonstrates how AI-augmented educational tools can bridge gaps in traditional academic support, particularly for diverse student populations with varying technical preparation. As institutions face increasing enrollment pressures, frameworks like RAGent offer a scalable approach to maintaining educational quality while accommodating individual student needs. Our future work should validate RAGent’s effectiveness across multiple institutions and diverse course contexts to establish broader generalization and identify context-specific adaptation requirements.