Network Attack Detection and Defense – AI-Powered Threats and Responses

Adversarial Machine Learning, is an area of study concerned with the vulnerabilities and robustness of ML models to adversarial attacks. Although, the first vulnerabilities were identified a number of years ago, e.g., [9], the contributions to this area have increased exponentially in recent years and entire conferences such as IEEE SaTML are devoted to this topic as well as regular sessions and many papers in both security and ML conferences. In light of this, discussions have focused on the aspects in which LLMs differ from adversarial aspects in traditional ML.

While traditional adversarial attacks focus mainly on classification tasks [6], aiming to manipulate the input and deceive the model into incorrect predictions, LLMs are general-purpose large language models designed to understand and generate human-like text across a wide range of tasks. Moreover, LLMs are subject to what is known as hallucinations [15, 12], where the answers provided by the LLM are inconsistent with real-world facts or user input. While these hallucinations usually do not have malicious cause or intent, they do raise the question of the trustworthiness of these LLMs, and what an attacker could have as objectives to carry out malicious actions.

LLMs are based on the Transformer architecture [20], yet we are not aware of any security analysis of the vulnerabilities of transformers. This is a topic that requires further study. Training LLMs, such as ChatGPT, is particularly expensive, both financially and in terms of the data required. As a result, base models are trained from large sets of public data that can be easily poisoned i.e. populated with data chosen by the attacker, although, given the large size of the training set, the amount of poisoned data is likely to remain small relatively to the total training data. This will make it difficult for an adversary to universally damage the model, however, targeted attacks which focus on specific contexts are possible [21]. Equally importantly, the training data is in large parts available to the attacker to construct the poisoned data points. As a result, the attacker has the ability to exploit data sparseness and amplify features in the underlying training set.

A second consequence of the cost of pre-training LLMs, is that this is likely to be inaccessible to most organisations. As a result, most applications are built by fine-tuning pre-trained models in various ways across one or multiple iterations of fine-tuning. As a result, the supply chain of the model becomes a significant concerned. Where does the pre-trained model originate from? what fine-tuning stages has it undergone? on which data? provided by whom? Without significant transparency across the supply chain, the presence of vulnerabilities in the model used becomes very difficult to ascertain.

Fine tuning is achieved in different ways and for multiple purposes. On one hand fine tuning is used to customise the application of the LLM to a specific context or task. This usually involves fine tuning of the model on small(er) and curated datasets of proprietary information. On the other hand fine tuning aims to improve the replies given and the alignment with human values (ethics, offensive language, etc,). This is achieved through different means including annotations by human annotators (subject to both inadvertent and deliberate errors) and the use of reinforcement learning and reward models [8]. Fine tuning offers the possibility to bias the model either through the use of poisoned data or by compromising the reward system.

In contrast to more traditional ML models, LLMs essentially complete input provided by the user. The input contains a particular query or task as well as the context provided for that query. Prompt engineering, i.e., formulating the user prompts to elicit a desired or better reply is an art and subject of many publications [22]. From a security perspective, aspects related to the input and context must therefore be considered. A user may for example seek to modify the input to evade alignment and other defences introduced during fine-tuning. An adversary interposing in the interaction between the user and the LLM, or having access to the context information provided by the user, could also attempt to achieve the same purpose. Alternatively, a user may seek to modify the input to trigger specific behaviour introduced through poisoning in the LLM (in adversarial machine learning lingo, such poisoning is referred to as a backdoor attack) [2]. Again, an adversary interposing may seek to achieve the same effect. Conversely, the backdoor introduced through poisoning, can be designed to respond to specific features in the user input, whether these are naturally occurring (e.g., sentiment, unusual phraseology, patterns in coding or in comments).

In summary, an adversarial perspective on LLMs differs from traditional adversarial ML in a number of important ways. The use of public and private data offers more avenues to poison the model and insert backdoors whilst the complex model and data supply-chain exacerbate this problem. Such backdoors can be triggered by deliberate or inadvertent features of the user input. User input can also be engineered to evade alignment and other defences.

Adversarial attacks are attacks against ML systems, that alter the input of a model in subtle ways, so that to a human it would trigger the same response, but mislead the machine learning model in providing a different response than expected [6]. A typical example is in computer vision [7]. When a slightly modified image is presented to a human, the human does recognise the image that is presented (animal, person, road sign, …), but the model outputs a different class than the expected one. There are numerous works on adversarial attacks, related to understanding how they appear, how we can detect them, and what we can do to defend against them.

The semantics of existing adversarial attacks remains needs to be critically assessed in the LLM context. Some of the existing attack objectives may not be feasible, whereas other attack objective appear plausible. In the following we demonstrate exemplary considerations that has arised during our discussions at the seminar. Obviously, LLMs are implemented in software, and software has bugs. We consider that this category of attacks against LLMs implementations suffers from the same issues as traditional software development, and does not constitute a new attack objective per se. Looking at specific objectives, we consider that an attack could have the following objectives:

Stealing the model

LLMs are expensive to train, because this requires a significant amount of computing power (hardware), and data (storage). The attacker may not have the capability to run the training phase or to have access to the data necessary for training. Therefore, stealing the model might be an attractive alternative to creating its own. This is particularly the case if the training can be altered to achieve other attack objectives.

Denial of service

Here, the attack objective is to ensure that the model does not respond in a timely manner. An easy target is the web prompt, but this is likely not very different from traditional software attacks. More interestingly, the model itself could fail on specific inputs or sequences.

Privacy-related attacks

Large Language Models are trained on large amounts of data, that are extremely likely to include privacy-sensitive information. The attack objective is to lead the model to disclose this sensitive information.

Systematic bias

The attack objective is to ensure that the model will respond with a systematic bias to all questions asked. This is a large attack surface.

Model degeneration

Here, the attack objective is to (slowly) lead the model to an unstable state, where the answers provided are less accurate than the ones obtained with the initial training. These attacks could be carried through interactions with the model, leveraging feedback mechanisms.

Falsified output

The attack objective is to ensure that the model will provide an attacker-desired output to a specific question. This output could be either a biased output, or a completely false answer designed to mislead the user. The degree to which this attack could be carried out is unclear. Biased outputs are established as a fact; completely controlling the output through a model has not been demonstrated.

An example of the impact of these attacks is code generation. Similarly to the malicious compiler of Ken Thompson [19], one could create a LLM used for code generation that would systematically generate backdoored or vulnerable code. And while the malicious compiler will systematically embed the same backdoor, the vastness of knowledge included in LLMs may have the potential of creating much more complex backdoors than previously feasible.

Evaluating the security of LLMs presents a multifaceted challenge for several reasons:

Data quality and origin

The training data consists of massive amounts of data largely scrapped from the Web, including human-generated content, presenting various data quality challenges such as biases, outdated information, miss-information, errors etc. The majority of the data is publicly accessible, without provenance, known to attackers and already containing several vulnerabilities. Inspecting or curating at scale is impractical.

Algorithmic & model opacity

The learning systems combine various learning tasks and complex (black-box) algorithms. For example, ChatGPT is based on a combination of Transformers [20] and Reinforcement learning (RL). Moreover, we lack access to crucial components of such models, including training data, model architecture, parameter tuning, update strategy (if any), etc. OpenAI, for example, for the most recent GPT foundation model, GPT-4, declined to publish information about the “architecture (including model size), hardware, training compute, dataset construction, training method, or similar” (citing “the competitive landscape and the safety implications of large-scale models” [1]).

Diversity in applications and user groups

LLMs find applications in a wide range of tasks such as text summarization, generation and question answering, spanning various domains such as education, customer service and healthcare. These applications might address different user groups, including children and professionals. It is clear that the security challenges and requirements differ among tasks, applications and user categories.

Rapid technological advancements

The rapid pace of advancements in LLMs poses challenges in keeping up with emerging security implications, given the variations in data, algorithms, training strategies and downstream tasks.

The role of data in AI systems is of paramount importance. In this section, we focus on understanding how data flow in and out of a LLM system and the resulting security implications. A high-level perspective of the data supply in LLMs is shown in Figure 2. Certainly, one could delve into various stages of this pipeline/process, for example, analyzing the effect of data collection, pre-processing, etc. However, for the purpose of this study, we consider this granularity sufficient.

The supply chain consists of the following components:

The LLM model

LLM models can be categorized into two types: i) pre-trained models like ChatGPT, which can be used off-the-shelf, and ii) fine-tuned models which are typically the result of further training a pre-trained model on a specific task or application dataset, e.g., on financial data.

Training data

These are large and diverse datasets from various sources used to train the pre-trained models.

Human feedback

Utilizing human feedback could be employed to enhance the performance of the model. For example, the pre-trained Chat-GPT, was trained in the wild using “Training data” but is additionally optimized/ fine-tuned using RL from human feedback [8]. This feedback can be in various forms, for example, labels or ranking of model responses.

Fine-tuning data

These are task/domain-specific data that enable the model to adapt and specialise for the desired task/domain, such as financial data.

User

A user interacts with the LLM using a language interface. Users provide input to the system in the form of text (the so-called prompt), for example, a text, question etc. and receive a text output. Users can engage in an iterative process, refining their prompts based on the mode’s responses (we refer to it as conversational model hereafter). They can also provide feedback on the responses like U, D.

User feedback data

User data, for examples, promts, conversations and feedback may be used for model update.

Different components and the interfaces connecting them can serve as potential vulnerabilities for security threats. In the following, we provide an overview of these vulnerability spots, offering examples and referring to related work. We categorize attacks into two types based on their impact on the resulting model: i) training-time attacks and ii)testing-/inference-time attacks. Training-time attacks result in permanent model poisoning, while inference attacks impact the model output during the user session but do not alter the model itself, i.e., they have an ephemeral effect.

The security of LLMs is a topic of paramount importance. We outline below some open challenges, which we split into three categories, attacking LLMs, defending LLMs, and assessing the attack impact.

Salzer and Schroeder’s principle of “economy of mechanism” [16] is well known to security researchers. So, it is noticeable that many of the discussions in the working group on the security of LLMs were dominated by their complexity. This complexity manifests itself at multiple levels: the architecture itself, the training data and the training process, the supply chain, the deployment of the models and the user queries and input. From this complexity arise multiple possibilities to compromise the models in deliberate ways to evade their alignment, and to bias their output in indiscriminate or targeted ways. Many potential vulnerabilities were discussed during the seminar. Some may be only hypothetical at this stage. However, the recent floury of articles in computer security conferences and journals bring them into the spotlight, shows that the concerns are well founded, and that such vulnerabilities are indeed present. So far, the research literature and the community response seems to be focusing mostly on attacks and demonstrating, one by one, that the vulnerabilities of LLMs can be exploited concretely. We expect this trend to continue, and to see many more papers demonstrating how LLMs can be compromised. In contrast, work on mitigating vulnerabilities is scarce at present. Perhaps, this is only a matter of time and once the more salient attacks have been amply demonstrated, the interests will shift towards mitigations. Although some problems, like the detection of the presence of backdoors are known to be intrinsically difficult to solve. Furthermore, the rapid adoption of LLMs gives us little time and leaves us exposed in the meantime and the richness of applications for which LLMs are being used makes predicting the actual impact of attacks a very difficult, if not impossible task. Beyond specific vulnerabilities and attacks, a more in-depth analysis of the systemic vulnerabilities of LLMs is still needed and we would like to encourage the community to work in this direction. Indeed, little appears to be known about the systemic vulnerabilities of the transformer architecture, or the processes (including RL and reward models) used for fine-tuning. Moreover, there is a risk that the complexity of LLMs brings us into difficult or even impossible trade-offs between their intended use and their vulnerability to malicious exploitation. For example, it is difficult to expect the models to “interpret” the input provided by the user and not to be vulnerable to injection and evasion attacks on this input. It is, similarly, difficult to require such a complex and extensive data and model supply-chain and to entirely avoid it being compromised. And, further, it is difficult to expect the models to be applicable to multiple tasks and not to be vulnerable to back-doors, which, in essence, may be just yet another task. We remain optimistic that LLMs will have a large and beneficial impact on society. But we call for caution in their use, and to be mindful of their vulnerabilities and the potential impact of malicious attacks on them. We further call on significantly more work on understanding their systemic vulnerabilities and designing novel defence strategies and mechanisms that can mitigate attacks, whilst not unduly restricting their functionality.

Weaknesses of AI models can be separated into two broad categories. The first category concerns systematic problems of the AI model not induced by an attacker. These are reflected in intrinsic problems of the model (e.g., poor performance or insufficient robustness) or extrinsic factors (e.g., poor stakeholder acceptance). The second category concerns attack induced issues with the model. Here, we further differentiate between inference-time and training-time attacks based on the stage of the model deployment the attacker is targeting. The attacker can either target the model during training (i.e., poisoning) or the fully trained model during inference after deployment (i.e., evasion attacks).

During our discussion about trust in AI and the modeling of threats against AI in network defense in our working group, we identified and classified existing threats and other factors that limit the trust in AI systems. Our main finding is that there are efforts in this area but no established method exists to measure, estimate, improve and communicate the level of trust in an AI model or its susceptibility to attacks or any misclassifications.

We define trustworthiness as the verifiable absence of unwanted properties. Explainability and transparency may contribute significantly to measure or estimate the trustworthiness. For such an assessment the full model lifecycle needs to be considered. This assessment should not only be limited to attacks, but include other unwanted properties of AI models that may exist also without the presence of attacks. Thus, for every AI model candidate, the robustness, the susceptibility to attacks, the resistance and resilience, as well as, potential attack detection and response measures need to be systematically analyzed.

We determined that widely accepted metrics, tools, and processes for the assessment are missing, both, regarding the evidence for trustworthiness, as well as, regarding the risk assessment. With the current state of the art, systematic assessments are not yet possible. Thus, it is a long-term goal to define the prerequisites for systematic and recurring risk assessments based on systematic audits of models. These are required for the secure and safe application of AI models in critical and sensitive environments.

(based on BSI report [1])

Automatic machine learning or AutoML

Approaches to automate the training pipeline setup and training itself.

Evasion attacks

An attacker plans to change the decision of an AI system during its inference (or operation) phase by subtle modifications of the model input.

White-box attack

If the attacker has perfect knowledge of the model, the features, and the data.

Grey- or black-box attack

If the output function is differentiable, which is the case for most of the currently used learning algorithms, a gradient may be computed as a prerequisite for the optimization procedure. However, this is also the case if the attacker has only limited knowledge of the target model, the feature and the data.

Substitute Model:

The model may be derived either via model stealing attacks or via newly trained models.

Black-box transfer attacks

Attacks developed for one model can in many cases be transferred to different cAI models without much effort (transferability).

Black-box query attacks

These attacks use queries to the target model combined with gradient-free optimization methods such as genetic algorithms or bayesian optimization.

Backdoor poisoning attacks and DoS poisoning attacks

These attacks corrupt parts of the training data in a targeted way.

Connectionist AI (cAI) systems

cAI systems are trained with machine learning and data.

Symbolic AI (sAI) systems

These systems may be directly constructed by a human developer.

Target attack

The attacker is able to control the decision of the AI system.

Untargeted attack

The attacker just changes the decision of a AI system in an arbitrary way.

The landscape of network defense has a long-standing history, yet the comprehensive integration of AI into this domain remains elusive, facing skepticism from both security experts and academics alike. In the face of evolving cyber threats, coupled with recent extraordinary advancements in AI, the need for re-evaluating the role AI may play in network defense becomes increasingly evident. In this working group, network security and applied AI experts joined forces to explore the intricacies of this complex problem space. Our primary focus was on systematically addressing urgent needs in network defenses, spanning across the entire defense pipeline – from pre-detection mechanisms to real-time alert analysis and post-detection response strategies. While there may be many ways to incorporate AI-based solutions, assessing whether involving AI is likely to be beneficial and justified in terms of additional complexity is not trivial. Moreover, the recent breakthroughs in the space of Large Language Models (LLMs) have renewed the ambition of building smart interfaces for human operators and analysts, hence prompting a careful re-evaluation of the potential of AI assistance in network defense.

Motivated by a fresh perspective on the application of AI in network defense, our working group took a practitioner-centric approach. Rather than assuming predefined problems, we aimed to first identify essential needs in the network defense pipeline, moving beyond the conventional alert generation step by an Intrusion Detection System (IDS). This approach challenges the common research paradigm of starting with assumed problem importance but instead promotes a pragmatic assessment of the potential of AI in network defense in alignment with real-world needs. Engaging experts from both network security and AI disciplines, we sought to reshape the discourse, emphasizing the analytical exploration of AI applications based on identified and substantiated needs within the field.

Our overarching goal was to create a roadmap that may guide future investigations with relevant and promising directions for future work. To move towards this objective, we adopted a structured approach that begins by defining a comprehensive network defense pipeline. The pipeline is scoped within the detection segment of the known NIST Cybersecurity Framework 1.1, which in turn we separate into several stages, each characterized by its own set of practical security problems. Upon identifying challenges at each stage of detection, we evaluate these challenges based on the chosen set of criteria: importance for practitioners, the potential for AI-driven solutions, and the research effort required. Our assessment is based on the consensus among the nine members of the working group, encompassing both practitioner and researcher perspectives from the network defense and applied AI domains. This report presents the current result of the analysis according to the developed assessment framework, pending a future extension through a comprehensive human study involving more practitioners.

To define the scope of this analysis, we refer to the NIST Cybersecurity Framework (CSF) 1.1 [3], developed by the National Institute of Standards and Technology (NIST) in the United States. This framework provides guidance for organizations to manage and improve their cybersecurity resilience. The core of the framework consists of five functions, each representing a key aspect of cybersecurity (Figure 1):

• $\mathrm{■}$

  Identify: Understand and manage cybersecurity risks to systems, assets and data.

• $\mathrm{■}$

  Protect: Develop safeguards to ensure the delivery of critical infrastructure services.

• $\mathrm{■}$

  Detect: Implement activities to identify the occurrence of a cybersecurity event.

• $\mathrm{■}$

  Respond: Take action regarding a detected cybersecurity event.

• $\mathrm{■}$

  Recover: Restore and maintain critical infrastructure services in the aftermath of a cybersecurity incident.

In this report, we focus our investigation on the Detect function as the one receiving primary academic attention in the field of network defense and yet lacking a comprehensive view in connection to wider real-world needs. We further break down the detection segment into concrete cybersecurity goals and position them along the general detection pipeline – its pre-detection, detection, and post-detection stages, as depicted in Figure 2. This breakdown enables us to formulate a (non-exhaustive) list of 18 network defense challenges based on what we consider to be crucial cybersecurity activities posing day-to-day challenges for practitioners. These challenges constitute the body of our analysis, where we assess the potential of AI to assist in solving them.

The central idea driving our work was the assessment of how relevant, meaningful or feasible the application of AI may be in view of the critical needs in network defense. The resulting structured framework may serve as a guide for researchers, aiding them in determining their research focus and prioritizing efforts effectively, particularly emphasizing challenges that have higher AI potential, in contrast to more elusive goals.

We utilized a three-fold evaluation to comprehensively assess each challenge – specific issue or task within the detection stage of the network defense pipeline that is under consideration. The three key rating criteria we defined are Importance, the AI Potential, and Research Effort, which are depicted in Figure 3. Below we provide a concise definition for each, along with insights into the main intuition and some underlying aspects:

1. 1.

   Importance: This criterion captures the significance of the challenge, emphasizing its role in the detection workflow. The evaluation is grounded in the practitioners’ perspective to ensure practical relevance (which is currently limited to the composition of the working group and is likely to be refined in the follow-up work).

2. 2.

   AI Potential: The assessment of the working group regarding the potential success and significance of data-driven AI methods (machine learning or deep learning) in application to a given challenge. This category encompasses three aspects, which in combination form the total AI potential:

   1. (a)

      Applicability: Examining whether the challenge can be formulated in a data-driven manner for using AI algorithms. Namely, this reflects whether the nature of the challenge lends itself well to the application of AI techniques that rely on data analysis, pattern recognition, and algorithmic decision-making. Problems suitable for a data-driven approach are those where historical information holds valuable clues about what might happen in the future: a much-desired but often limited property in cybersecurity.

   2. (b)

      Success Likelihood: Assessing the probability of solving the problem through a data-driven approach. Even if the problem can be formulated in a data-driven way, there may be serious complications in solving it. For instance, in the case of non-anticipated distributional shifts (due to the lack of representative data or dynamic changes in data at deployment), the trained AI model may lose its relevance too fast.

   3. (c)

      Impact: Gauging the potential impact and effectiveness of leveraging AI to address the identified challenge. This pertains to the expected performance of an AI solution, assuming its applicability and high success likelihood, in comparison to more traditional approaches that do not rely on intelligent automation. The estimated impact may also be influenced by, for instance, the associated costs which may or may not justify the additional complexity of using AI for the task. Interpretability, maintainability and other operational considerations behind an AI-powered solution may also greatly influence its expected impact.

3. 3.

   Research Effort: This category is orthogonal to AI Potential in the sense that it covers the research-related side of addressing the challenges, meaning the effort required to conduct a research study that reliably investigates application of AI to a given task. In our understanding, the Research Effort criterion roughly encompasses two components:

   1. (a)

      Existence of Relevant Data: This aspect considers whether there is ground-truth data within the operational context that holds significance for the challenge at hand. Having access to complete relevant data of high volume and precision is crucial for training AI algorithms. If the necessary data is readily available within the existing network defense workflows, it reduces the effort required in sourcing and preparing data for AI applications. Note that the current public availability of such corresponding data is out of scope for this analysis (although acquiring it is a significant challenge of its own for the research domain).

   2. (b)

      Algorithmic Difficulty: This dimension reflects the technical complexity associated with developing AI algorithms tailored for the specific challenge. Some challenges might demand sophisticated algorithms due to their intricate nature (for instance, those that model multi-dimensional time-series or graph-structured data), while others may be more straightforward. Challenges with higher algorithmic difficulty may necessitate more advanced AI research and development efforts.

This comprehensive framework aims to provide researchers with a holistic perspective on the challenges in network defense, enabling them to prioritize efforts based on the combined assessment of the defined properties. The deliberate design ensures, on the one hand, that researchers can identify not only what challenges are critical but also where the application of AI is most feasible and impactful. On the other hand, the framework may pragmatically illuminate the opportunities for “low-hanging fruit”, allowing researchers to address more urgent but less complex needs with a higher likelihood of success, whilst having the assurance of relevance of their study for network defense practitioners.

Here we report on the main outcome of the working group, essentially presenting the snapshot of our understanding of the potential of AI-powered network defenses in the given moment (Table 5). The current state of analysis involves 18 ranked challenges derived from in-depth discussions among academic and industry researchers composing the working group, utilizing expertise in network defense and applied AI. Moving forward, this analysis would be refined to incorporate the perspectives of more security practitioners, offering a more holistic and practically relevant understanding of the challenges at hand. Crucially, this analysis needs to be strengthened with a thorough literature review to incorporate the most recent research advancements in the area, in order to more precisely evaluate the alignment of the current research trends with the highest needs in network defense.

The aim of this discussion was to provide a framework for future investigations in AI-powered network defense, leveraging the latest advancements in AI while fostering a more comprehensive understanding of the challenges faced by practitioners. The distinguishing characteristic of the framework is the choice to move beyond the conventional realm of AI-based intrusion detection and explore areas earlier in and further along the pipeline. Our systematic approach facilitates a wide examination of the problem space from both the network security practitioner and research perspectives. It covers practical feasibility, potential impact of applying AI, and the research effort required to address the identified important challenges. By combining the expertise of network security and AI researchers, the framework and its future extensions not only offer a nuanced understanding of challenges but also foster a synergistic environment where innovative solutions can be explored while staying rooted in reality.