Are We There Yet? On Security Vulnerabilities Produced by Open Source Generative AI Models and Its Implications for Security Education

Generative Artificial Intelligence (GenAI) tools are changing the way code is written, optimized, and maintained, marking a paradigm shift in software development. Large Language Models (LLMs) are being integrated into professional development flows, providing remarkable efficiencies, automation, and accessibility [4, 7]. Simultaneously, these tools are gaining ground in education, where students can use them to work on programming assignments, dabble in new languages, or rapidly prototype software [5, 13].

However, alongside these benefits, an increasing number of studies have raised serious concerns about the security of AI-generated code [12, 11, 3]. Studies show that LLMs can generate syntactically valid but insecure code that incurs vulnerabilities such as command injection, unsafe memory usage, improper input handling, and misconfigured cryptographic routines [12, 11, 3]. This can create serious vulnerabilities for professional teams when GenAI tools are used without stringent validation or human oversight.

From other perspectives like education, GenAI offers new possibilities for learning and has the potential to revolutionize the way we learn to code. On the one hand, while LLMs can democratize knowledge and provide immediate feedback to learners, those tools could reinforce insecure coding practices if students do not understand how to identify and correct the problems embedded in their generated outputs [5]. In this duality, the need for strategies that integrate secure-by-design principles becomes apparent.

Despite increasing interest in GenAI applications, two specific research gaps remain insufficiently addressed:

1. (i)

   Lack of comparative evaluations about the security vulnerabilities in code generated by GenAI, across multiple open-source LLMs and programming languages.

2. (ii)

   Absence of strategies to handle the fact that GenAI produces code with security vulnerabilities.

Our work addresses these gaps by analyzing three open-source LLMs – Codestral, DeepSeek R1, and LLaMA 3.3 70B – across a variety of experiments related to prompt-based code generation in the programming languages Python, C, and Java. We apply multiple prompt sets to generate code and evaluate the resulting outputs using static analysis tools (Bandit, Cppcheck, Flawfinder, and SpotBugs).

The goal of this study is twofold: to expose recurring vulnerability patterns in GenAI-assisted code generation, and to propose recommendations for integrating prompt engineering that generates code with minimum vulnerabilities and secure code review into programming education, so that software developers are empowered to identify and solve vulnerable code.

Generative AI (GenAI) tools have fundamentally transformed software development, including code authoring, testing, and deployment. Using tools like GitHub Copilot, OpenAI Codex, and ChatGPT, developers gain access to automated help with tasks such as generating code, fixing bugs, and providing documentation. According to Fan et al. [4] GenAI manifests significant potential in a variety of software engineering (SE) activities, ranging from requirements engineering to code repair to Software Testing. Hou et al. [7] further consolidate this perspective through a systematic review of the literature of 395 articles, revealing emerging trends and gaps in the use of LLM in SE workflows. However, this progress comes alongside concerns about the correctness, robustness, and security of AI-generated code. Evidence from the field points to the emergent behavior of LLMs -hallucinations or unsafe default practices, for example - requiring additional scrutiny to avoid unintentionally introducing risks into development pipelines.

The security of code produced by LLMs has been the subject of recent empirical investigation. Tóth et al. [12] analyzed 2,500 GPT-4-generated PHP websites and discovered that 26% contained at least one injectable vulnerability via user-driven web requests. Similarly, Rabbi et al. [11] performed a security-centric exploration of the chatGPT-generated Python code, showing that user-adapted outputs were plagued with quality or security problems much more often. Clark et al. [3] evaluated the consistency and reliability of the ChatGPT output through the lens of Halstead complexity metrics and observed that while many outputs appear correct, errors remain and compound in prompt iterations. Jamdade and Liu [8] proposed the generation of secure code via guided prompts and partially mitigated common web vulnerabilities such as SQL Injection and XSS. A 2024 report from the Center for Security and Emerging Technology (CSET) outlines three major risk areas associated with AI code generation: the production of insecure code, model susceptibility to manipulation, and downstream security risks through AI feedback loops [6]. Their evaluation of five LLMs using consistent prompts showed that nearly half of the generated outputs contained bugs with potential for exploitation. The report also emphasizes the lack of security-aware benchmarks in current GenAI evaluation, highlighting the need for tools and frameworks that prioritize secure-by-design criteria. Despite these findings, the absence of security-oriented defaults in LLMs and the lack of automated validation pipelines remain core challenges in integrating GenAI into secure software development.

With the increasing use of GenAI tools by students, educators must now face new challenges in ensuring the code quality and secure coding habits of these students. Fernandez and Cornell [5] advocate that CS1-type introductory courses be adapted to train students not only to use GenAI tools, but also to interpret their results critically and uniquely apply verification methods. Xiao et al. The (DevGPT dataset) [14] contains more than 29,000 developer-ChatGPT conversations in which GenAI in a dev context, as well as its pedagogical highlights that GenAI is a great rely. Wang et al. [13] provided an overview of how LLMs are used in software testing education and highlighted the increasing relevance of prompt engineering and automated analysis tools in educational contexts. Despite increasing adoption, few curricula currently integrate structured assessments of GenAI-generated code or teach students how to recognize security pitfalls. This leaves a critical gap in computer science education.

In this section, we describe the experimental design, models, languages, and evaluation tools that we use to explore the security vulnerabilities in GenAI-generated code. The goal is to systematically assess how different LLMs respond to prompts likely to produce insecure code and to evaluate the severity of the resulting vulnerabilities.

We selected three large open source language models (LLMs) specialized in code generation with architectural diversity, allowing us to have efficient output and structured code in different languages. This combination allows for a meaningful comparison of GenAI behaviors across different design philosophies:

• $\mathrm{■}$

  Codestral: Based on Mistral, optimized for structured and efficient code generation [2].

• $\mathrm{■}$

  DeepSeek R1: A multilingual model aimed at code completion and automated software development tasks [1].

• $\mathrm{■}$

  LLaMA 3.3 70B: An adaptation of Meta’s LLaMA model, custom made for generating source code [9].

Each model was independently tested using consistent prompt sets designed to elicit security-relevant behavior.

To ensure a comprehensive evaluation, the code was generated in three programming languages commonly used in security-sensitive applications.

• $\mathrm{■}$

  Python: Popular for scripting, automation, and web backends, but prone to command injection and unsafe shell usage.

• $\mathrm{■}$

  C: Widely used in systems programming, known for memory management issues.

• $\mathrm{■}$

  Java: Common in enterprise applications, susceptible to improper stream handling and encoding flaws.

These languages were chosen due to their widespread adoption and the availability of mature static analysis tools.

To analyze the security of the code generated in these languages, we selected one or more static analysis tools per language. The following tools were applied:

• $\mathrm{■}$

  Bandit for Python: Bandit parses the abstract syntax tree (AST) of Python code and identifies patterns associated with security issues such as command injection, insecure hashing algorithms, and unsafe imports. It assigns each issue a severity level (Low, Medium, High) and a confidence level (Low, Medium, High) based on the rule’s reliability. In our analysis, only findings with Medium or High confidence were considered, and the original severity labels were preserved.

• $\mathrm{■}$

  Cppcheck for C: Cppcheck focuses on memory safety, null pointer dereferencing, buffer overruns, and undefined behavior. Warnings are categorized internally into severity levels using flags such as error, warning, and style. For this study, we mapped these categories to severity as follows: error → High, warning → Medium, and style or performance issues → Low.

• $\mathrm{■}$

  Flawfinder for C: Flawfinder scans source code for known risky C/C++ functions (e.g., strcpy, gets, sprintf) and assigns a numerical risk level from 0 (lowest) to 5 (highest), based on the likelihood and impact of exploitability. In this study, risk levels 0–2 were considered Low, 3 as Medium, and 4–5 as High.

• $\mathrm{■}$

  SpotBugs for Java: SpotBugs analyzes Java bytecode and reports bugs or bad practices using a built-in priority scale (1 = High, 2 = Medium, 3 = Low). These priorities reflect the estimated impact and certainty of the finding. Only confirmed or likely vulnerabilities were included in our aggregation.

Severity classifications (High, Medium, Low) were assigned directly based on each tool’s native scoring system, as described above. Manual overrides or reinterpretations were not applied. This ensures consistency, reproducibility, and tool-aligned risk interpretation across all analyzed outputs.

Having established the tools used to evaluate security vulnerabilities, we next describe the design of the prompts used to generate the code samples. The goal was to simulate realistic development scenarios where security issues are likely to emerge, thereby enabling a fair and targeted assessment of each model’s behavior under common and risky programming conditions.

Twelve prompts were carefully crafted to simulate realistic yet security-sensitive development tasks, such as executing system commands, handling user input, and managing files. These tasks were chosen because they are known to be frequent sources of security vulnerabilities in real-world applications. Each prompt was instantiated across three programming languages (C, Python, and Java) and processed by each of the three evaluated models, resulting in a total of 108 generated programs.

The design of the prompts followed a dual approach: some prompts resembled realistic development tasks with minimal instructions (e.g., “write a program that receives a command and executes it”), while others were explicitly framed to elicit unsafe behavior (e.g., “do not sanitize the input”). This second group of prompts was not intended to induce vulnerabilities for the sake of counting them, but rather to assess whether LLMs comply blindly with insecure requests without issuing any form of warning or safeguard. This strategy was used to assess whether LLMs would default to insecure coding patterns in the absence of explicit guidance – a behavior known as unsafe default generation [8, 11], which has been documented in recent empirical studies [12, 8, 11]. This setup reflects both professional and educational use cases, where ambiguous or under-specified prompts are common. Similar evaluation methodologies have been employed to test the inherent security posture and behavioral tendencies of GenAI models under both standard and adversarial prompting conditions [11].

Each generated output was statically analyzed, and the detected vulnerabilities were mapped against the Common Weakness Enumeration (CWE) standard maintained by MITRE [10]. To efficiently manage the evaluation, an automated experimental pipeline was developed, as described in the following subsection.

To support this process and ensure consistency, scalability, and reproducibility, we implemented a custom experimental pipeline that automates the entire workflow, including:

• $\mathrm{■}$

  Injecting the crafted prompts into each evaluated LLM (Codestral, DeepSeek R1, and LLaMA 3.3 70B) via Azure endpoints.

• $\mathrm{■}$

  Collecting and cleaning the generated code samples, ensuring removal of non-code artifacts and standardizing file formats.

• $\mathrm{■}$

  Compiling the code artifacts where applicable (C and Java) to verify syntactic correctness.

• $\mathrm{■}$

  Conducting static analysis using Bandit for Python, Cppcheck and Flawfinder for C, and SpotBugs for Java, with outputs mapped to Common Weakness Enumeration (CWE) standards.

• $\mathrm{■}$

  Aggregating compilation results, vulnerability findings, and severity classifications into structured datasets for comparative analysis.

Automated generation of code, compilation, analysis, and result collection reduced the need of manual intervention, helping to minimize human bias and operating errors. The full source code for the evaluation pipeline is available upon request for academic and research use.

With the pipeline in place to automate code generation and analysis, we defined a set of metrics to quantify and compare the security posture of each model. These metrics focus on vulnerability counts, severity levels, and distribution patterns across languages and models.

We assessed the generated code using the following metrics:

• $\mathrm{■}$

  Total number of vulnerabilities identified.

• $\mathrm{■}$

  Severity-weighted vulnerability index (High = 3, Medium = 2, Low = 1).

• $\mathrm{■}$

  Frequency of key vulnerability types (e.g., CWE-78, CWE-120).

• $\mathrm{■}$

  Vulnerability distribution by programming language and model.

These metrics enable comparative analysis across models and languages, offering insights into recurring weaknesses and patterns of risk.

The methodology described above ensured a systematic and reproducible evaluation of GenAI-generated code across multiple models, languages, and security dimensions. By combining prompt engineering, automated pipeline execution, multi-tool static analysis, and CWE-based vulnerability mapping, a robust foundation was established for comparative security analysis. The following section presents the results and insights derived from this evaluation.

We evaluated 108 programs generated by Codestral, DeepSeek R1, and LLaMA 3.3 70B in response to 12 prompts in Python, C, and Java. Vulnerabilities were identified using static analysis tools (Cppcheck, Flawfinder, Bandit, and SpotBugs), with severity classification directly derived from tool output.

As shown in Table 1, Codestral generated the highest number of vulnerabilities (43), followed by DeepSeek (37) and LLaMA (28).

In addition to total volume, Codestral exhibited the highest number of High severity issues (19), followed by DeepSeek R1(15) and LLaMA 3.3 70B (12), as shown in Table 2.

Figure 1 presents the severity distribution of detected vulnerabilities, categorized as High, Medium, and Low, for each model.

Table 3 and Figure 2 show that C-based outputs were consistently more vulnerable, reflecting the intrinsic risks of memory-unsafe languages. Python followed with moderate vulnerability rates, while Java exhibited the lowest overall.

Figure 2 shows the distribution of vulnerabilities across C, Python, and Java for each evaluated model.

Among the identified issues, four CWE categories emerged most frequently: command injection (CWE-78), buffer overflow (CWE-120), memory bounds violation (CWE-119), and improper input validation (CWE-20). Table 4 and Figure 3 summarize their distribution across models.

Figure 3 details the frequency of common vulnerability types (CWE categories) identified across Codestral, DeepSeek R1, and LLaMA 3.3 70B outputs.

OS Command Injection (CWE-78) and Buffer Overflow (CWE-120) emerged as the most prevalent vulnerabilities, particularly among Codestral-generated code, underscoring the critical need for secure prompt engineering practices.

• $\mathrm{■}$

  Codestral: Generated the most critical vulnerabilities, especially command injections (system() usage) and buffer overflows.

• $\mathrm{■}$

  DeepSeek R1: Produced moderate vulnerabilities; however, race conditions appeared due to unsafe concurrent file writing.

• $\mathrm{■}$

  LLaMA 3.3 70B: Demonstrated the best security posture, though still exposed to unsafe subprocess usage in Python.

In addition to quantitative data, we observed characteristic patterns of insecure code per language:

C Language:

• $\mathrm{■}$

  High prevalence of buffer overflows and command injection vulnerabilities.

• $\mathrm{■}$

  Unsafe use of scanf(), system(), and manual memory management.

Python Language:

• $\mathrm{■}$

  Command injection via unsafe use of subprocess.run(shell=True).

• $\mathrm{■}$

  Insecure file handling practices (temporary file issues).

Java Language:

• $\mathrm{■}$

  Resource leaks (missing close() on streams).

• $\mathrm{■}$

  Default encoding issues without UTF-8 specification.

The findings confirm that model choice, language, and prompt type significantly impact the security of GenAI-generated code. C remains the most vulnerable target language, while LLaMA 3.3 70B achieves relatively safer code generation across languages.

The integration of Large Language Models (LLMs) into programming education presents both opportunities and risks. While GenAI tools such as Codestral, DeepSeek R1, and LLaMA 3.3 70B can accelerate learning by providing immediate code generation and feedback, they also introduce critical security challenges.

This research emphasizes the need to integrate secure code review into computer science curricula. Teaching students how to identify vulnerabilities in GenAI-generated code, especially when working with memory-unsafe languages, will help to develop students into more security aware software engineers.

Recommended pedagogical tools include guided prompt engineering, static analysis of AI-generated code, and structured vulnerability-identification activities. By introducing those skills in immersive programming courses, instructors can get students to a point where they are able to use the GenAI tools responsibly and safely in professional software development scenario.

This study has two important limitations that must be taken into account. First, the tools used to perform static code analysis, while essential for scalable and reproducible assessment, do not detect all types of vulnerability. Dynamic analysis or manual code inspection could further improve the detection of more specific vulnerabilities; this will be taken into account in future work.

Secondly, the study is performed with 12 prompts designed to simulate common development tasks. Although these prompts were selected to cover various vulnerability patterns, they cannot cover the full spectrum of real-world scenarios, as this may depend on many variables and requirements. This limitation may affect the generation of the results. Future work should expand the set of prompts to include a wider range of functionalities and application domains.

This study presented a security-centered evaluation of code generated by three prominent open-source Large Language Models (LLMs): Codestral, DeepSeek R1 and LLaMA 3.3 70B. Using a comprehensive set of realistic prompts across C, Python, and Java, and employing multiple static analysis tools, we systematically quantified and categorized the vulnerabilities found in the generated outputs.

The results reveal that GenAI-generated code often exhibits critical security flaws, particularly when targeting memory-unsafe languages such as C. Codestral demonstrated the highest density of vulnerabilities, including a notable prevalence of OS command injection (CWE-78) and buffer overflow (CWE-120) issues. DeepSeek presented a moderate security profile, while LLaMA consistently produced relatively safer outputs, though vulnerabilities persisted across all models and languages.

From a professional development perspective, these findings underline the necessity for integrating systematic validation mechanisms and secure prompt engineering strategies into GenAI deployment pipelines. Without explicit safeguards, the risk of propagating insecure code into production environments remains significant.

Academically, this research highlights the urgent need for enhancing software engineering curricula to incorporate GenAI security awareness. Teaching students not only how to utilize LLM-based coding assistants, but also how to critically evaluate their outputs, will be crucial to building the next generation of security-conscious developers.

Future work will explore dynamic testing approaches to complement static analysis findings and extend the evaluation across additional programming languages and LLM architectures. Furthermore, investigating the impact of fine-tuning LLMs with security-enriched datasets represents a promising avenue for mitigating inherent risks in AI-assisted software development.