Defining and Fortifying Against Cognitive Vulnerabilities in Social Engineering

The talk examines why the digital environment makes us so susceptible to social engineering – because we have become so used to being manipulated and deceived by others that we don’t notice and deceive ourselves that we have choice and control. In fact, ubiquitous tracking of our online activities has created a huge information asymmetry, which Soshanna Zuboff has described as “surveillance capitalism”, and humans have adopted routines to respond to prompts to give our time, attention, and money. While we do not regard them as “attackers” in the traditional cybersecurity sense, they utilize very similar cues and exploit habits. To regain control, we need to engage in regular goal setting and planning of our activities and ration our digital engagements along the lines of Cal Newport’s “digital minimalism”. But we also urgently need reliable trust anchors to enable humans to distinguish friends from foes.

Human physiology exerts electric potentials that can be captured by computing devices. Such physiological signals allow the assessment of user states, such as cognitive workload, affect, or stress [2]. While these states can be assessed to allow users to quantify themselves, they are also a gateway for exploiting behaviors in real-time. Social engineering attacks can be tailored depending on the user states, thus increasing the likeliness of social engineering attacks. Furthermore, sensors, such as thermal cameras, are becoming more ubiquitous. Thermal cameras have recently drawn the attention of HCI researchers as a new sensory system enabling novel interactive systems. They are robust to illumination changes, making separating objects from the scene background easy. Far-infrared radiation, however, has another characteristic that distinguishes thermal cameras from their RGB or depth counterparts as it operates in the non-visual spectrum. On the other hand, the visual spectrum, i.e., human visual perception, is limited to only 1 percent of the electromagnetic spectrum. Research has shown that extending visual perception can be beneficial. To investigate the potential of the adoption of thermal imaging, we present the conducted studies to infer users’ states, e.g., cognitive load, attention type [5, 6], as well as environmental state, e.g., the presence of recording devices [7], and foot traces [8]. Our findings reflected the potential of thermal imaging to further protect the user by knowing the user’s state and nudging them when they are cognitively vulnerable. Yet, our research also explores how thermal imaging might introduce novel attacks, namely thermal attacks[9].

Objective

Due to the very different backgrounds of seminar participants (scientists, practitioners, military), this session aimed to create a common understanding of social engineering and identify state-of-the-art strategies of hackers.

Methodology

This session followed a two-step approach to develop a common understanding. Firstly, Chris Hadnagy introduced the fundamentals of social engineering. Afterward, participants were split into groups, discussing a set of questions to create a joint understanding of social engineering from different perspectives.

The talk by Chris Hadnagy aims to define the fundamentals and main vectors used by malicious social engineers in attacking their targets. We define how attackers use phishing, vishing, SMiShing, and impersonation in their attacks. By defining each of these and discussing the advancement in the technology used, we can better understand the minds of the attackers in choosing which method to use. Following this discussion, we went into depth about the stages of social engineering engagement from a practitioner standpoint. The goal was to understand the methodology used by professional social engineers.

Following the introduction and plenum discussion to obtain a common understanding, participants split into two subgroups to discuss the following questions:

Objective

This session aimed to identify grand challenges in social engineering from both a practical and academic perspective.

Methodology

To structure this working group, Chris Hadnagy first introduced the different phases leading up to a successful social engineering attack. Those steps then served as a scaffold for breakout groups in which grand challenges for each phase were identified.

Phase 1–OSINT

Open Source Intelligence (OSINT) refers to collecting and analyzing information from publicly available sources. It involves gathering data from various sources such as social media, news articles, online forums, public databases, and websites. OSINT provides social engineers with valuable information about their targets, which can be used to craft convincing narratives and exploit vulnerabilities. By utilizing OSINT, social engineers can gather personal details, interests, affiliations, and even behavioral patterns of their targets. This information enables them to tailor their approaches to appear more trustworthy and increase the chances of successful exploitation.

Phase 2–Target Selection

Social engineers target individuals who can access valuable information or sensitive systems. This could be employees of a company, individuals in positions of authority, or those with access to financial information. By targeting those with valuable data, social engineers increase the likelihood of a successful attack or fraud.

Phase 3–Attack Plan

Social engineers craft a highly personalized attack plan using the information gathered during the prior phases. They adopt different personas, using tactics such as impersonation, pretexting, or creating fake online profiles to establish credibility. They exploit emotions and trust by pretending to be someone the target knows or trusts.

Phase 4–Attack Launch

Social engineers conduct their attacks by manipulating and exploiting human psychology and trust. They use various tactics to manipulate individuals into divulging confidential information or performing actions that could compromise security. They often employ techniques like impersonation, pretexting, and phishing to trick people into believing they are someone they are not or representing a trustworthy entity. Social engineers can access sensitive information and financial data by exploiting human emotions, curiosity, and ignorance or by gaining unauthorized entry into systems. These attacks can occur through various mediums, such as phone calls, emails, social media, or even in-person interactions, to deceive individuals and bypass security systems.

Phase 5–Evaluation

Throughout the process, social engineers document their actions, record findings, and assess the impact of any successful exploits.

Phase 6–Reporting

Finally, they provide a detailed report to the organization, outlining the vulnerabilities discovered and recommending remediation measures.

Participants discussed why educating users to protect themselves from the initial phase of social engineering is complex. The participants identified the following challenges concerning Open Source Intelligence:

Creating Awareness of Own Vulnerabilities:

A key challenge is making users aware of their vulnerabilities. Users should be aware of what information is publicly shared and could be used by attackers (and which information is not publicly stored but could be accessible by attackers if the platform is breached). Moreover, many users struggle to think that they are targets in the first place.

Inference of Available Data:

A challenge is keeping an overview of available information about oneself when AI models draw conclusions based on metadata (e.g., relationships).

Social Media Exploitation:

Social media platforms often contain tons of personal information. Attackers can exploit this information to create phishing messages, impersonate individuals, or conduct other forms of social engineering.

Dynamic and Evolving Nature of Platforms:

Online platforms and available information constantly change and evolve. Keeping track of these changes and assessing the reliability of information can be challenging, for instance, how social media platforms change their privacy settings and auto-send friend requests.

Lack of Users’ Awareness of the Influence of Their Internal State:

Users’ internal states and vulnerabilities play a significant role in their susceptibility to social engineering attacks. As social engineering relies on manipulating individuals’ emotions, behaviors, and cognitive processes to deceive them into revealing sensitive information, taking specific actions, or compromising security. Yet, users are not aware of such an influence.

Based on the group discussion and the pointed-out challenges, participants discussed potential solutions to help protect the user and increase their awareness about OSINT. Attackers usually use information aggregation during the OSINT phase, where they gather information from multiple sources to create a comprehensive profile of a target. This profile can be used to craft more convincing and targeted social engineering attacks. Accordingly, participants envisioned a Cross-Platforms Search Notification System, where users would be alerted if someone searches them on different platforms, e.g., work/personal website, LinkedIn, Instagram, Facebook, and warn the user if access rates are unusually high. However, this entails technical and privacy challenges, e.g., logging search activities across platforms.

Identifying potential targets and implementing strategies to mitigate risks can be complex for both defenders and users. Participants openly discussed why this phase might be hard to mitigate. One dominant reason was that many users struggle to think they were targets in the first place. Hence, they exhibit neglectful behavior when dealing with both their own data and institutionally accessible information.

The central premise of an attack plan is coming up with a pretext for the attack. During the planning phase, the attacker uses the information gathered from OSINT to devise a strategy. This involves selecting the most appropriate attack vector—whether it be phishing, pretexting, baiting, or another method—based on the target’s vulnerabilities and the attacker’s objectives. The attacker also crafts the message or scenario they will use to deceive the target, ensuring it is convincing enough to elicit the desired response. This phase requires careful consideration of the psychological and emotional triggers that will be most effective on the target and planning for any contingencies or responses the target might have.

Attack planning also involves the creation of backstories, fake identities, or any necessary props (like counterfeit badges or websites) that will make the attacker’s approach more credible. This is where the creativity and insight of the attacker into human psychology are paramount. The success of this phase hinges on how well the attacker can anticipate the target’s reactions and prepare for them, ensuring that the attack will not only reach the target but also resonate with them, prompting the desired action or information disclosure.

One of the most effective ways to counteract the planning phase is to limit already access to information that could be gathered during OSINT. Corporations should regularly check what type of information is publicly accessible, including about their employees. Awareness campaigns about public profiles can hone employees’ sensitivity to sharing certain information about themselves or their employers.

In the attack launching phase, the attacker puts their plan into action and makes direct contact with the target. This could be through email, phone calls, social media, or in-person interactions. The attacker employs the crafted scenario to manipulate the target into performing specific actions, such as divulging sensitive information, granting access to secure systems, or even transferring funds. The success heavily relies on the attacker’s ability to adapt to the conversation flow and maintain the deception convincingly.

During this phase, the attacker must remain vigilant and adaptable, as unforeseen variables or responses from the target may require on-the-fly adjustments to the plan. The psychological manipulation skills of the attacker are crucial here, as they must build trust or authority with the target quickly. The ability to read cues from the target and adjust the approach accordingly can make or break the attack’s success.

Countermeasures can be taken individually through general awareness training or collectively through peer protection. The latter entails establishing a reporting culture in which employees inform and warn each other about new schemes they encounter. Employers should establish a single point of contact where incidents can be easily reported and that is responsible for disseminating newly identified threats. Another approach that has merit on both individual and collective levels is the introduction of friction. Artificially delaying certain actions or procedures, e.g., can create time windows in which reason can kick in, or the attack can be delayed to a point where the risk of exposure becomes too great to continue the attack. Urgency should almost always be a warning sign for an incoming attack.

Evaluating social engineering attacks, particularly those conducted as part of penetration testing, poses several challenges.

Ecologic Validity and Generalizability

While pen testers try to act realistically, their actions are still limited by legal and ethical considerations. Also, their customers might exclude certain actions (e.g., accessing sensitive and personal information about employees). Those conditions inevitably influence the ecologic validity of the findings and pose the question of to which degree the findings generalize to settings with real attackers.

Metrics

Many forms of penetration testing are still strongly limited in terms of the used metrics. For example, click rates are among the most popular metrics for phishing awareness campaigns. At the same time, these allow only very little to be learned and are questionable, as they depend on factors beyond pen testers’ control (and assessment). There is a need to rethink metrics currently in use fundamentally.

Individualization

Measures against social engineering are generally costly from a corporate perspective, as a result of which easy-to-implement solutions are favored (e.g., making users attend talks on awareness once a year). The challenge with such measures is that they might annoy users (as content might be repetitive). Also, employees might have a different level of knowledge and understanding, as a result of which some might struggle with terminology already while others might be bored. A major challenge is the individualization of measures, where users’ skills, prior knowledge, and tasks of their everyday job are considered.

Case Study vs. Large Scale

Due to the required effort and cost, campaigns and research projects often focus only on specific cases rather than large-scale approaches. While (small-scale) case studies might be well suited to identify causes and interesting aspects, more large-scale studies are required to assess effects.

Priming

For ethical reasons, users or employees might be primed; that is, they are being told upon being employed or signing up for a study that they will / might be subject to a security assessment. This inevitably changes behavior. Research is needed to understand the implications of priming and on approaches that minimize any such priming effects.

Independent Auditing

Pentesting / auditing is often conducted due to certification or due to being required by insurance companies. As a result of this, companies subsequently hire auditors. The challenge here is that pen tests and audits, in this case, are not independent. It is still an open question about how such independence can be achieved.

Completeness

While defenders must protect against any vulnerability, attackers only need to find one weakness. Pentesting usually cannot consider any aspect/attack surface.

Pentester Empathy

A particularly interesting aspect is their actions’ implications on pen testers (similar to the Milgram experiment). This is an unexplored area of research.

The group identified several challenges regarding the reporting and, in particular, the way in which recommendations are / should be made.

Turning lessons learned into positive change

While many pen tests are designed to demonstrate issues/holes that allow attackers to be successful, it is often much less clear how this knowledge can be turned into positive change.

Targeting Opportune Moments

Closely related to the abovementioned aspect, change must be carefully targeted to opportune moments, i.e., moments in which users are (more) receptive to change and appreciate it. It is well known that in situations of change (e.g., moving to a new house/office, getting a new smartphone/laptop), people are more willing to change habits, but this is hardly explored from a cybersecurity perspective.

Check-the-Box vs. Organizational Change

Penetration tests/auditing is often seen today as a necessary requirement rather than an opportunity for real (organizational) change. It remains an open challenge to move from just getting things done to a culture in which true organizational change is anticipated.

Misconceptions about being a target

Many users struggle to understand/accept that they are a target. Reports can surface convincing cases (e.g., kindergartens being targets of cyber attacks).

Response Costs / Prioritization

Cyber attacks are usually possible through many different approaches, and addressing all of them is costly. There is a need to understand better how countermeasures can be prioritized so as to maximize their impact.

Change Management / Leadership

Who should drive change is often unclear. Whereas employees often consider employers responsible for cybersecurity, employers want employees to change their habits. A challenge is how to establish a social contract.

Expressing IT Security as a Business Risk

In particular, companies and individuals struggle to accept IT security as a business risk until they become victims. Research is needed as to how the consequences of successful cyber attacks can be better conveyed.

Objective

The objectives of this session were to (1) understand how modern sensing technology can be used to assess user states affected by social engineering, to (2) think about how knowledge of those states can be used to design counter measured and (3) how attackers can exploit this knowledge.

Methodology

Thomas Kosch and Yomna Abdelrahman first introduced the capabilities of modern sensors and machine learning (see talk). Afterward, participants were divided into groups and asked to think about ways of using knowledge obtainable from sensors to defend from social engineering and also which novel attack surfaces this creates.

As previously discussed, users’ internal states significantly influence their susceptibility to social engineering attacks. Social engineering relies on manipulating individuals’ emotions, behaviors, and cognitive processes to deceive them into revealing sensitive information, taking specific actions, or compromising security. In this session, we leverage design fiction methods to ideate the role of physiological sensors in social engineering from both the attackers’ and defenders’ perspectives.

Thomas Kosch and Yomna Abdelrahman gave concrete examples of utilizing novel ubiquitous sensors like eye tracking and thermal cameras to detect and leverage cognitive vulnerabilities during social engineering. However, opportunities are not limited to these examples but rather to steer the mindset of the participants towards using modern sensors in different contexts.

Eye Tracking:

Eye-tracking data in social engineering refers to collecting and analyzing information about a person’s eye movements and gaze patterns. While traditional uses of eye-tracking are often associated with research in psychology and HCI usability studies, the application of eye-tracking data in the realm of social engineering introduces additional considerations. Here are some examples:

• $\mathrm{■}$

  Understanding Attention: Eye tracking data can provide insights into where a person directs their attention. In a social engineering context, understanding what elements or cues attract a person’s gaze can be valuable for attackers. Attackers may use this data to refine deceptive techniques.

• $\mathrm{■}$

  Phishing and Visual Deception: Attackers may leverage eye-tracking data to optimize the design of phishing emails. By understanding where users focus their attention, attackers can create more convincing and visually deceptive elements to increase the likelihood of successful social engineering attacks.

• $\mathrm{■}$

  Defensive Techniques: While attackers could utilize eye-tracking, they also hold merts for defenders. Researchers and defenders can use eye-tracking data to understand how users visually engage with social engineering attacks and security warnings. This information can inform the design of more effective alerts and communication strategies to raise awareness about potential social engineering threats.

Thermal Cameras:

Facial temperature can be influenced by emotional states, and temperature changes may reflect variations in emotions and internal states, e.g., cognitive load, stress, anger, etc. These changes could be monitored seamlessly and non-invasively using thermal cameras. While researchers utilized thermal cameras to build cognitive-aware systems in various contexts, it is unexplored in the context of social engineering, yet the potential it holds, for instance:

• $\mathrm{■}$

  Cognitive Load Detection: Recent work showed the potential of using thermal cameras to capture facial temperature to quantify cognitive load from low, medium, high, and very high. Research reflected the potential of thermal imaging to further protect the user by knowing the user’s state and nudging them not to perform any security critical tasks when they are cognitively vulnerable.

• $\mathrm{■}$

  Emotions Detection: Attackers rely on the victims’ emotions to conduct social engineering attacks. Namely, they aim to simulate emotions, e.g., fear, guilt, and stress, to make users reveal sensitive information or perform certain actions. Research revealed correlations between changes in facial temperature and these emotions. Emotional-aware systems built using thermal cameras could detect these emotions and either used by the defenders to build protective measures when such emotions are detected or by the attacker to know vulnerable moments.

Following the talk, we had an open discussion on how modern sensors could be deployed in the context of social engineering to serve both attackers and defenders. To this end, participants split into two subgroups to discuss the following questions:

The groups came up with the following ideas for novel protection mechanisms.

Mechanism 1

During face-to-face situations, use a video-based assessment of physiological data to give insights into the current level of fatigue, stress/arousal, and identify health status. Based on the signals, the system would recognize if the user is sleepy/exhausted, provide recommendations, and flag potential threats.

Mechanism 2

During reading emails, use data such as heart rate, EDA, pupil dilation, blink rate (fatigue), reading speed, eye tracking for speed of reading emails, and nonverbal body posture to determine when the user is vulnerable, and the email client color changes to indicate cognitive vulnerability.

Mechanism 3

One group proposed using physiological data not to detect victims’ vulnerabilities but the attacker’s intent. Once an attacker is detected, it is flagged.

The groups came up with the following ideas for novel possible attacks.

Attack 1

Attackers could exploit the knowledge about users’ states to tailor the attack and abuse the user during their vulnerable state.

Attack 2

Having access to heart rate, EDA, pupil dilation, blink rate (fatigue), and reading speed data, attackers could use the data to find a point when the target is most stressed and fatigued, being overly busy. During that time, a phishing email is sent from someone in authority over the target requesting immediate action.

Interestingly, the participants discussed how they can benefit from the potential of using physiological real-time data without the risk of falling into the attackers’ hands. For instance, participants proposed randomly introducing noise to the data or using differential privacy. Adding enough noise makes the attack infeasible but still allows data to be used legitimately.

So far, social engineering has been mainly discussed in association with deceptive practices aimed at exploiting human psychology for malicious ends. The goal of this session, however, was also to consider the use of its principles and mechanics for positive individual and social outcomes, i.e. for good. Social engineering (SE) can be defined as “any engineered act that influences a person to take an action that may or may not be in their best interest”. This allows us to insist on the specially designed intention (engineered) as well as to emphasize that it could be for malicious but also positive “for good” reasons. In this context, we should consider the involved ethics, understood as the thinking process about human conduct and the values on which they are based. Indeed, either for good or bad, social engineering may not respect human autonomy, transparency, or explainability, and of course, the non-maleficence principle will be strongly questioned.

During a joint walk up to the old castle ruins, seminar participants were thus presented with two leading questions and invited to discuss in present company and eventually report back. The two leading questions posed were:

1. 1.

   How would you use insights, techniques, and methods of Social Engineering to do good?

2. 2.

   What are ethical boundaries and obligations when “manipulating” people for good?

Examples discussed by participants included personal health and environmental conservation. In the realm of public health, social engineering and, specifically, nudging can play pivotal roles. For instance, designing environments that subtly encourage physical activity, such as strategically placed stairs over escalators, can significantly impact public health outcomes. Similarly, nudges in cafeterias or grocery stores, like placing fruits and vegetables at eye level, can make healthy food choices more appealing and accessible. These interventions use our natural tendencies and decision-making shortcuts for positive ends, making the healthier choice, the easier or more attractive option.

On another note, environmental sustainability can benefit from social engineering techniques aimed at encouraging eco-friendly behaviors. For example, utility companies have successfully used social norms to influence behavior by showing customers how their energy consumption compares to their neighbors, nudging them to reduce energy use. Similarly, simple prompts or reminders to recycle or making recycling bins more visible and accessible can significantly increase recycling rates. These strategies rely on our innate desire to conform to social norms and our responsiveness to environmental cues, guiding us toward more environmentally sustainable actions.

A commonly mentioned critique of nudges was the user’s agency, which might potentially be violated. Even nudges “for good” are construed by a choice architect, i.e. an individual or group of people who deem one choice better than another. Conflicting moral and value systems can, therefore, give precedence to choices that go against what the individual might have selected in a more conscious choice scenario. In the end, any technique deemed as social engineering entails some kind of manipulation. The question of which manipulation is deemed “good” or “bad” needs to be discussed in light of differing moral and ethical frameworks. People’s agency should, at best, be preserved, while transparency should always be provided about how certain choices are presented.

Objective

This session aimed to identify specific research questions that could be addressed through joint research projects and initiatives.

Methodology

Based on the grand challenges and discussions from the first day of the seminar, Mohamed Khamis synthesized different areas of research participants could vote on. Afterward, breakout groups were built where people identified specific research questions based on their interests.

We compiled a list of main research areas based on the outcomes of previous sessions. Participants then voted on which areas they would like to explore. The main areas were:

1. 1.

   Social engineering vulnerability: self-assessment and misconceptions

2. 2.

   Organizational changes to defend against social engineering

3. 3.

   Frictions and warnings: How? When? What?

4. 4.

   Evaluation of solutions: threats to validity

5. 5.

   Evaluation metrics

6. 6.

   Datasets

All areas that received four or more votes proceeded to the next stage, in which the participants chose the area they were interested in exploring further to produce research questions that can be addressed by (a) a Ph.D. thesis, (b) a research grant, or (c) a dedicated research center.

The participants voted for the following research areas:

Evaluation of Solutions

The breakout group discussed methodological challenges of evaluating solutions against social engineering attacks.

Finding and Supporting Routines Against Attacks

This breakout group discussed research questions related to the development of routines, aiming at minimizing risks from social engineering attacks.

Datasets

This breakout group aimed to identify research questions that could be answered by having access to different practitioners’ datasets.

Objective

This session aimed to identify specific topics and areas of collaboration.

Methodology

For this session, a poster was created for each research question participants identified in the previous session. Then, participants were asked to indicate their interest in each research question by writing their names on the poster. Afterwards, in several rounds, participants met at the poster to discuss the following questions: (1) Who would fund this research? What would be the scope of a project? (3) How would you pitch the topic (i.e., write an abstract)?

The following list describes the different research projects and initiatives.

Funding

NL: NCSC-NL (National Cyber Security Center), NCTV (National Coordinator For Counterterrorism and Security, Ministry of Justice and Security), Cyberveilig Nederland, HighTechCrime Police/Europol (i.e., Law Enforcement), AIVD (General Intelligence and Security Services, Ministry of Interior)

FR: ANSSI (National Cyber Security Center), Viginum (Service of vigilance and protection against foreign digital influence), Inria (National Research Institute on Informatics and applied mathematics, CNRS (National Center for Scientific Research).

DE: BSI (Bundesamt für Sicherheit in der Informationstechnik), BMI (Bundesministerium des Inneren), Cyberagentur; BKA (Bundeskriminalamt)

Pitch

Social engineering attacks have emerged as a predominant threat, exploiting human psychology to manipulate individuals into revealing confidential information, compromising their financial security, and influencing their decision-making, including voting behavior. These tactics bypass traditional cybersecurity measures by directly targeting the most vulnerable link in the security chain: the human. Recognizing the gravity and complexity of this issue, the proposed enter title here aims to serve as a pioneering institution dedicated to combating these threats across Europe.

The center will be a collaborative hub, uniting industry practitioners and academic experts to develop comprehensive strategies against social engineering attacks. Its primary objectives will include raising awareness about the nature and tactics of these attacks, enhancing the ability to detect and identify such threats promptly, and devising effective mitigation strategies to reduce their impact. By fostering a multidisciplinary approach, the center will address current challenges and anticipate emerging trends in social engineering, such as the threat of generative AI, ensuring a proactive stance against these evolving threats.

This research center will formulate goals to fortify individual privacy, financial integrity, and democratic processes against the influence of social engineering. The establishment of this research center represents a significant step forward in strengthening Europe’s resilience against these sophisticated psychological attacks, thereby protecting its citizens and institutions in an increasingly interconnected world.

Funding

DoD (Department of Defense), GCHQ (Government Communications Headquarters), EU, DFG (German National Research Foundation), Cyber Insurances

Scope

Creating threat actor ecosystem by 1) identifying trends of emerging (credible) threats and patterns of attacker interest in different attack capabilities made available in the darknet. 2) Characterize specific and emergent criminal convergence spaces from internet forums/anonymous markets to Telegram/Discord channels. 3) Quantify/qualify the effect of the appearance of an offensive capability in one of those venues for realizing an attack.

Pitch

Social engineering attacks follow a pattern and defined phases (see 4). The pre-attack phases involve OSINT, target selection, and attack planning. This project aims to model the attacker patterns to predict social engineering attacks even before happening (i.e., prevention instead of mitigation). The overarching goal is developing an investigation infrastructure that triggers based on detecting attack patterns.

As shown in Figure 1 the approach would cover different aspects. Define economic and criminological theoretical underpinnings to identify measurables within forum and telegram/discord channels to characterize communities regarding the type of support (e.g., moral hazard/adverse selection mitigation mechanisms) they provide to criminal activities. NLP topic analysis will be used to identify both discussion topics within communities and user perception/feedback related to attack technology (sentiment analysis) to characterize user interactions at scale. Language and slang challenges must be addressed, especially on less verbose channels like instant messaging platforms. Active probing with direct interaction (either as potential customers/providers) or face-to-face interviews (remote setting) with offenders/perspective offenders to investigate underlying decision mechanisms/factors (e.g., why joining community x/choosing product y to do offense x rather than product y’). Develop a model of threat selection. Relate model predictions to high-level trends in emerging attack tech with known incidents and see if there is a credible link between what these markets enable and what attackers do.

Based on this Threat Actor Ecosystem, attacker movement would be modeled by:

1. 1.

   Identifying patterns of attacker actions within network monitoring data.

2. 2.

   Coding techniques on network packets/sequencing and MITRE ATT&CK mapping to evaluate qualitatively attack processes.

3. 3.

   Building the conceptual model from that understanding.

4. 4.

   Developing a data model for detecting those patterns from network data for scalability.

5. 5.

   Evaluating the correlation of historical trends in those attack patterns with trends from our approach.

Funding

NSF (National Research Foundation), DFG (German National Research Foundation), Security companies

Scope

Social Engineer’s vishing dataset or self-data collection; project might include other modalities (keystroke dynamics, etc.)

Pitch

An increasingly popular form of social engineering attacks is vishing (voice phishing). Little effective means for protection exist today against such attacks beyond raising awareness in cyber education. At the same time, the human voice holds rich information about (1) the current user state (i.e., whether they are stressed and what their current level of awareness is) as well as (2) techniques in use to social engineer somebody (firm voice to sound authoritative, pleading tone to beg for help, etc.). This project proposes to design, implement, and evaluate in-situ interventions protecting users from falling for vishing, that is, mechanisms that are capable of detecting in real-time if a caller is trying to social engineer somebody or if the callee is being socially engineered and provide active guidance as to how the legitimacy of a call can be verified.

The project will address the following objectives:
(1) building predictive models based on real vishing data, allowing common manipulation strategies and callee reactions to be detected;
(2) designing, implementing, and evaluating interventions to assist end users during vishing calls;
(3) assessing social and ethical implications of vishing mitigation technologies and strategies.

Funding

Social Media Platforms, Security Companies, and Research Foundations.

Pitch

Social network users share personal information online that might be misused in several ways incl. social engineering. In this project, we want to investigate the usage AI-based support to inspect the information users want to share for (a) identifying content that could be misused or does not match the user’s privacy needs, (b) educating the user on sharing consequences and (c) helping the users avoiding to share such information in the future, and (d) modify the content to mitigate the probability of misuse.

Funding

Research Foundations and Cyber Security Companies.

Pitch

This funding proposal seeks support for an innovative project to leverage artificial intelligence (AI) to design personalized support systems that effectively mitigate vulnerabilities towards social engineering attacks. The proposed initiative recognizes the multifaceted nature of susceptibility, focusing on personability, age, cultural differences, and accessibility needs, among others, as critical dimensions to tailor interventions. The advent of sophisticated social engineering attacks demands a nuanced approach that adapts to individual characteristics. Our project will employ advanced AI algorithms to analyze and understand diverse user profiles to generate tailored responses and interventions resonating with users personally and build resilience to manipulation attempts.

The following ideas were identified but not further discussed:

• $\mathrm{■}$

  Utilizing AI for routine building and providing support

• $\mathrm{■}$

  Plugins and feature highlighting

• $\mathrm{■}$

  Designing social engineering interventions

• $\mathrm{■}$

  AI-based validation

• $\mathrm{■}$

  Self-assessment and self-reflection to mitigate vulnerabilities

• $\mathrm{■}$

  Design targeted support for vulnerable groups

• $\mathrm{■}$

  Understanding routines to mitigate vulnerabilities

• $\mathrm{■}$

  Characteristics of targets and social engineers