Can Open Large Language Models Catch Vulnerabilities?

During preliminary testing, we observed that LLMs often produced highly diverse and inconsistent CWE predictions for the same code snippet. We hypothesized that this behaviour could stem from the high complexity and overlap between many CWE categories.

To mitigate this, we constrained the classification space by selecting only those CWEs in the dataset that were better represented and more consistently labelled -second phase. Specifically, we selected all CWEs for which the dataset contained at least 100 unique vulnerable snippets. This filtering process resulted in a total of 8 distinct CWEs, each with 100 representative examples. This filtering resulted in the following 8 CWEs:

To complement the set of vulnerable code snippets, we also included 100 safe snippets, code samples without any known vulnerabilities, also drawn from the same dataset.

The final dataset used for experimentation thus consisted of 900 code snippets: 800 vulnerable examples (distributed equally across 8 CWEs), and 100 non-vulnerable examples. All code snippets were selected randomly.

The experiment was conducted using three state-of-the-art LLMs:

• $\mathrm{■}$

  Llama 3 (Meta) [12]: a general-purpose decoder-only model with 70 billion parameters, designed for language understanding and generation.

• $\mathrm{■}$

  Codestral (Mistral) [14]: a code-centric model trained on a diverse set of permissively licensed codebases.

• $\mathrm{■}$

  Deepseek R1 (Deepseek) [2]: a general-purpose model that employs a “thought process” decoding mechanism to enhance reasoning capabilities and improve performance on complex programming tasks.

With this selection, we aim to cover a representative spectrum of contemporary LLM architectures and specializations, ranging from general-purpose language models to code-specific systems. Importantly, all models included in this study are open source, which ensures transparency, reproducibility, and accessibility for the research community.

Here we started our third phase, where to constrain the classification space and better assess model accuracy, we provided each LLM with a fixed list of 8 CWE identifiers, corresponding to the vulnerabilities present in the dataset. Our prompt included this list and instructed the model to select the most appropriate CWE based on the code snippet provided. This design ensures that the task is framed as a closed-set classification problem, rather than free-text generation, enabling a measurable comparison across models.

All models were queried independently using standardised prompts in isolated sessions, with no memory of previous queries.

Each model received the same structured prompt, designed to simulate a realistic developer scenario. The prompt consists of three components:

1. 1.

   An introduction asking the model to determine whether the code is secure;

2. 2.

   A list of 8 CWE identifiers from which the model must choose the most appropriate one, if a vulnerability is present.

3. 3.

   A code snippet (in C or C++);

For non-vulnerable snippets, the model was expected to explicitly state that no vulnerability was present.

This is our fourth phase where to ensure consistency and minimize variability across models, a carefully designed prompt engineering strategy was adopted. The primary objective was to ensure that all models received the exact same task, with clear and explicit rules to avoid ambiguity.

In this context, the prompt was designed to enforce that models:

• $\mathrm{■}$

  Analyze a code snippet and determine whether it contains a vulnerability.

• $\mathrm{■}$

  If a vulnerability exists, identify the most appropriate CWE from a given list.

• $\mathrm{■}$

  Return only one CWE identifier from the list or the quote “There isn’t” if no vulnerability is found.

• $\mathrm{■}$

  Never suggest CWEs that are not in the provided list.

After multiple interactions and different results, we arrived at a prompt structured in two parts: a base instruction, common to all examples, and a second block with composed of a CWE list and the code snippet. The complete base prompt used in all tests is shown below:

⬇

You are a cybersecurity expert specialized in vulnerability detection.

You are given:

- A list of CWE identifiers (e.g., [CWE-79, CWE-89, CWE-200]).

- A code snippet to analyse.

Your task:

- If the code snippet contains any vulnerability, you must map it to the most appropriate CWE from the provided list and output only

that CWE identifier (e.g., CWE-79).

- You must always select a CWE from the provided list if there is any vulnerability, even if the match is approximate.

- If the code snippet contains no vulnerabilities at all, output exactly: There isn’t.

Rules:

- Do not output any explanation, justification, description, or additional text.

- Do not suggest vulnerabilities outside the given CWE list.

- If multiple CWEs could apply, choose the best matching one.

- Output strictly a single CWE identifier from the list or ’There isn’t’.

This block was followed by a prompt containing the specific data for each example, such as the list of CWEs and the code to analyse:

⬇

Analyze the following code snippet for vulnerabilities.

Do not explain your answer. Output only the CWE identifier or ’There isn’t’.

-CWE List:

-Code Snippet:

This approach enabled an objective and controlled evaluation of model performance, minimizing interference from external factors such as model verbosity or formatting variation.

In this last phase, we collected each answer and evaluated each model using two distinct metrics:

• $\mathrm{■}$

  Blind Detection: Whether the model was able to detect the presence of a vulnerability, of any kind, in the code snippet. In this process, we refined the response of each LLM to derive a binary outcome, indicating the detection or absence of vulnerabilities.

• $\mathrm{■}$

  Specific Detection: Whether the model was able to correctly classify the vulnerability using the appropriate CWE identifier

These metrics allow us to distinguish between the model’s general ability to detect insecurities and its ability to correctly classify known weaknesses.

We first assessed each model’s ability to detect the presence of vulnerabilities. That is, could the model identify that a given code snippet was problematic? In Figure 2, we present the performance of all three models in identifying True Positives – focusing exclusively on vulnerable code snippets.

The three models demonstrated high rates for blind detection. Llama achieved a score of 97.4%, followed by Deepseek with 84.9% and Codestral with 83.2%. These results suggest that LLMs might be highly sensitive to the presence of insecure patterns in code, and capable of flagging problematic input in most cases.

To assess the extent to which the models produce False Positives during blind detection, we evaluated them on a set of secure code snippets that had been carefully curated to contain no known vulnerabilities.

As shown in Figure 3, the models frequently misclassified safe code as vulnerable. Llama incorrectly flagged 75% of the safe snippets, followed by Deepseek (72%) and Codestral (58%). These results highlight a tendency among LLMs to overpredict vulnerabilities when operating without guidance, raising concerns about their reliability in real-world scenarios where precision is crucial.

These findings suggest that the LLMs are highly cautious – perhaps excessively. Their sensitivity to potential vulnerabilities can lead to false positives, where secure code is mistakenly flagged as problematic. This behaviour, while arguably safer in exploratory analysis, can erode developer trust in the tool, especially when integrated into real-world workflows where the majority of code may not be vulnerable.

Understanding whether language models can not only detect the presence of vulnerabilities but also correctly classify them is critical for evaluating their practical utility in secure software development. In this subsection, we present our results of the models’ ability to perform this more fine-grained task.

As shown in Figure 4, the models’ ability to perform specific detection drops significantly when compared with blind detection. All models performed similarly: Llama with 16.0%, Codestral with 16.9%, and Deepseek with 16.2%. This suggests that while LLMs may be able to recognize the presence of a vulnerability of any kind, they often struggle to accurately map it to its precise classification within the CWE taxonomy.

To further investigate the classification patterns of each model, we analysed the full distribution of CWE predictions using heatmaps. Each heatmap focuses on one ground truth CWE and shows how frequently each model predicted each CWE as a response. The Y-axis lists all possible CWE predictions, while the X-axis corresponds to the three LLMs. We included an additional row labeled “CWE-XXX”, which is intended to captures answers outside the provided list, serving to identify hallucinations or non-compliance with the prompt.

From a high-level perspective, the heatmaps reveal substantial differences in behaviour among the models, as well as important commonalities. A striking observation is that none of the three models ever predicted an invalid CWE label such as CWE-XXX. This indicates that, despite their limitations, the models did not hallucinate syntactically incorrect or non-existent CWEs. The absence of such hallucinations suggests that the models exhibit a degree of robustness in adhering to the expected CWE format when prompted explicitly.

At a finer level, the heatmaps show that the Llama model exhibits a pronounced and systematic bias towards predicting CWE-20 (Improper Input Validation) and CWE-119 (Improper Restriction of Operations within the Bounds of a Memory Buffer), regardless of the actual ground truth. This behaviour points to overfitting to dominant patterns learned during training or to an insufficient understanding of the distinguishing features required to differentiate between CWE types. While this leads to a high frequency of predictions in those two categories, it also results in extremely poor performance for the remaining CWEs, where the model seldom assigns the correct label. Consequently, beyond CWE-20 and CWE-119, Llama’s classification capability appears substantially limited.

Another relevant insight is that none of the models managed to predict CWE-787 (Out-of-bounds Write) for any example, even when it was the correct label. This consistent omission suggests that this vulnerability type poses a particular challenge to LLMs, either due to a lack of representative training data or due to its subtle distinction from related memory issues such as CWE-119 or CWE-125.

When comparing the models more closely, Codestral and Deepseek emerged as the most similar in terms of their prediction distributions. While not identical, their heatmaps display comparable misclassification patterns and similar concentration zones, implying overlapping decision heuristics or aligned training data characteristics. In contrast, Llama’s distribution stands out as significantly skewed and less nuanced.

Among individual model strengths, Codestral demonstrated a comparatively strong ability to correctly identify CWE-200 (Exposure of Sensitive Information to an Unauthorized Actor), achieving consistent matches in relevant cases. This suggests that Codestral may possess a better contextual understanding of confidentiality-related issues, or may have benefited from more robust examples in pretraining data for that class.

Overall, the heatmap analysis exposes several critical limitations in the CWE prediction capabilities of current LLMs, particularly in terms of overgeneralization and lack of coverage for less common but equally important vulnerabilities.

The present study relies on three publicly available LLMs – Llama3, Codestral, and Deepseek R1 – as released and accessible at the time of writing. As these models are still undergoing rapid development and refinement, future iterations may exhibit significantly different behaviours. Consequently, some of the observations and conclusions presented in this work may not generalize to later versions of the same models.

Another threat to validity arises from the ambiguous classification of CWEs and the pre-existing classifications in the dataset, which may contain inaccuracies. While the closed-world classification setting facilitates clearer performance comparisons across models, it oversimplifies the problem space and may not fully capture the complexity of real-world conditions, where multiple or overlapping vulnerabilities can exist.

Additionally, our dataset is composed of vulnerabilities disclosed between 2010 and 2019. Although the Big-Vul dataset remains a valuable resource, it may not accurately represent current programming practices, libraries, or emerging vulnerability patterns. As such, results obtained on this dataset may not extrapolate to newer or more diverse codebases.

An additional concern is that the models may have been trained on the same dataset used in this study, which could lead to biased responses. Such bias arises when the model encounters information that closely resembles what is present in the dataset, possibly reinforcing pre-existing patterns or associations that do not truly capture the diversity of real-world vulnerabilities.

Finally, prompt formulation and evaluation criteria, although carefully designed, may still introduce unintentional biases. Variations in phrasing, ordering of CWE options, or even formatting choices may influence model responses in subtle but impactful ways. While efforts were made to minimize these effects, their complete elimination cannot be guaranteed.