Cognitive Sensing and Interaction

My research explores how digital technologies can be designed and used to support both work and wellbeing. I am particularly interested in the bidirectional relationship between productivity and health: how the very technologies that enable us to work, such as communication, planning, and collaboration tools, can also be harnessed to promote wellbeing, and how the technologies we use to manage our health and wellbeing can, in turn, enhance our effectiveness at work. A major interest is the use of psychological theory to inform the design and evaluation of these digital systems. By grounding technology design in established theories of motivation, self-regulation, and recovery, I aim to deepen our understanding of how people interact with these systems and how technology can be shaped to foster thriving rather than overwork. In related work, I have examined how digital systems for measuring and evaluating work can shape experiences of productivity and control, particularly in remote and hybrid contexts. For example, in Remote Work, Work Measurement and the State of Work Research in Human-Centred Computing (Gould et al., 2023), we argued that the expansion of “bossware” and digital surveillance during the pandemic highlights the conceptual limits of technology-mediated understandings of work, and the need for more inclusive, representative approaches to future-of-work research. Within the context of this Dagstuhl Seminar, my interests lie in how insights from psychology and human–computer interaction can inform the responsible development of cognitive and physiological sensing technologies for everyday work. This includes questions of boundary management, trust, and ethical design, ensuring that systems which sense or augment human cognition do so in ways that respect autonomy, diversity, and wellbeing. My contribution to the seminar focused on connecting insights from psychology and HCI to the design of ethical, inclusive cognitive interaction systems.

My research investigates how we can augment human memory and research cognition. A major thread in my work focuses on cognitive alterations – from memory augmentation to attenuation and implantation – enabled by ubiquitous and immersive technologies [1, 2]. I previously explored how technology can act as a partner in remembering – helping people to reconstruct, reinterpret, or even reform their past experiences. Through studies of visual lifelogs, memory cues, and privacy-preserving prostheses, I examine how digital traces interact with our autobiographical memory and sense of self in real life or in virtual environments [3], revealing both the empowering and altering potentials of “smart” memory systems. More recently, I have turned to Generative AI as a cognitive amplifier for researchers. In collaboration with international HCI researchers, I examine how Large Whatever Models (LWMs) can transform research cycles, automate synthesis, and preserve ubiquitous research artifacts [4, 5]. I envision generative systems as active participants in scientific reasoning and collective memory, capable of extending our cognitive reach and accelerating discovery. Working at the intersection of human-computer interaction, psychology, and AI, I aim to establish hybrid systems that not only augment human memory but also reshape how we conduct, recall, and evolve knowledge. My goal is to design technologies that amplify reflection without erasing human agency – tools that help us remember, reason, and reimagine together.

My work explores how neuroscience, behavioral design, and artificial intelligence can be combined to create technologies that advance cognitive health and mental wellbeing. As Chief Product and Science Officer at Mendi Innovations (Sweden), I focus on translating neuroimaging insights into accessible neurofeedback tools, particularly through fNIRS-based brain training for attention and self-regulation. As Honorary Fellow of Neurotechnology at the University of Nottingham’s Brain Data Group, I contribute to research that spans HCI, game design, dynamic system theory, and adaptive neurotechnology, using fNIRS data analysis to study engagement and cognitive load. In parallel, I serve as an Advisor to the REACT project (Responsible AI), that aims at developing responsible, adaptive robotic systems for complex healthcare environments and co-lead an IEEE workstream developing ethical frameworks for mental health technologies. I am also involved in work investigating how at-edge neuromorphic computing and EEG-based sensing can enable low-power, real-time interpretation of neural signals in wearable systems to predict epilepsy seizure onset. Across these domains, my goal is to advance ethical, personalized, and human-centered neurotechnology that meaningfully integrates with everyday life.

JIZAI Coding is a research initiative that aims to transcend conventional programming paradigms by integrating embodied interaction, affective and physiological sensing, and co-creative AI to empower users – including those with disabilities – to solve their own challenges through code. In contrast to “vibe coding” as a metaphor for loose prompt-driven generation, JIZAI Coding envisions a resonant, adaptive coding partnership between human cognition, bodily states, and generative systems. Drawing upon the framework of Cognitive Sensing and Interaction (Dagstuhl Seminar 25422) and the notion of gut feelings (the body’s tacit cues), this project posits a future where code is not only “written” but felt, synchronized, and lived.

Human cognitive capabilities, such as learning, reasoning, and communicating, can be enhanced through various intervention approaches. While direct physiological interventions, including biofeedback and Transcranial Magnetic Stimulation (TMS) provide immediate cognitive enhancement during active stimulation, their effects might be typically transient and limited to the intervention period. My research proposes an alternative paradigm: achieving sustained cognitive augmentation by developing users’ meta-skills through information systems. Our approach leverages sensing technologies, including eye-tracking, physiological sensors, and behavioral analytics to recognize cognitive states during learning activities. By providing real-time feedback on engagement levels, comprehension, self-confidence, and mental workload, these systems enable users to develop metacognitive awareness, the ability to monitor and regulate their own cognitive processes. Systems such as intelligent e-books that adapt content based on reader interest [1], engagement gauges that visualize attention in online meetings [2], and gaze-based word-count trackers for reading with smart eyewear exemplify this approach. Unlike temporary enhancement through direct neural stimulation, cultivating meta-skills creates lasting improvements in cognitive capability, as users internalize strategies for self-regulated learning, attention management, and effective knowledge acquisition that persist beyond system use.

My research focuses on developing human-centered adaptive technologies through innovative neural wearables and large-scale biodata to sustainably enhance productivity, health, and well-being. A strong focus of my work is the development of discreet wearable systems (such as ear-EEG integrated headphones) and scalable software solutions that enable the transition from controlled experiments to everyday environments – solutions that I contribute as open-source resources to the scientific community. Working at the intersection of computer science, information systems, and neuroscience, I bridge wearable technology development with rigorous empirical research. My interdisciplinary approach addresses the fundamental challenge of creating reliable biosensing systems while ensuring these adaptive technologies genuinely serve human needs and well-being. Correspondingly the major theme in my work is understanding human cognitive and affective states through multimodal biosensing in both laboratory and real-world contexts. Through studies combining brain activity (EEG), eye tracking (EOG), heart rate variability (ECG/PPG), and behavioral data, I investigate states like cognitive load, flow, mental overload, and fatigue across settings ranging from controlled laboratory experiments to extended field studies in real-world knowledge work and home office scenarios.

At the Dagstuhl Seminar, I contributed a demonstration of the latest versions and ideas for wearable EEG headphones and explored potentials for novel use case scenarios, technical improvements and ares for collaboration like the use of wearable sensors in human-AI interaction or sensing context data to increase the reliability of everyday neural recordings.

Cognitive sensing is sometimes very difficult and not reliable. In order to obtain reliable data, it is necessary to use relatively big sensors such as fMRI. On the other hand, recent advances in computer vision (CV) technology enables us to estimate human activities by using a small number of cameras. We are currently developing methods of estimating human muscle activities by using images captured by one camera. As a future vision, it might be possible to estimate a human’s internal state, not only the muscle activities but also brain activities by monitoring the face, eye gaze, and so on using the power of CV. Also the estimation would be more reliable by integrating with other sensor information such as heart rate, blood pressure, or else, which can be obtained by a small wearable device.

My research explores how artificial intelligence can become an active collaborator in human perception and cognition. I study how computational models of behavior, physiology, and context can foster mutual understanding between humans and AI systems. By integrating signals from the body, such as interoceptive and proprioceptive cues, into interactive systems, I design and build interfaces that amplify human perception rather than replace it. A central theme in my work is amplified perception, the idea that AI can enhance our awareness of internal and environmental states, enabling more transparent, adaptive, and trustworthy interactions. This approach challenges the notion of AI as a controlling agent and instead positions it as a perceptual partner that co-regulates with the user. Through experimental and prototyping work in extended reality and wearable computing, I explore how feedback loops between humans and AI can enhance decision-making, empathy, and self-regulation. Another theme is understanding how the interaction with AI shapes the mental models and metacognition of users. For example, priming users about an AI augmentation system influences the user’s view of their own performance, even when the AI system is non-functional. I refer to this as the placebo effect of AI, which should be carefully investigated when evaluating and deploying interactive AI systems, particularly in terms of user agency.

My research explores how physiological sensing can move from passive observation to active participation in human cognition and emotion. I focus on signals from the eyes, skin, and cardiovascular system, studying how they can reveal changes in attention, fatigue, or affective state. Through wearable platforms such as smart eyewear [1], I aim to integrate these signals into feedback systems that help people better understand and regulate their mental and emotional processes.

A major theme in my work is digitalized emotion, the idea that feelings and physiological reactions can be captured, represented, and re-experienced in meaningful ways. This approach opens possibilities for sharing emotional experiences, enhancing empathy, and supporting well-being. At the same time, I explore cognitive augmentation by developing tools that extend perception and awareness through subtle feedback rather than direct control. Working at the intersection of computer science, design, and the arts, I collaborate with engineers, performers, and psychologists to study how technology can act as both mirror and partner. My goal is to establish new forms of human-machine interaction that are reflective, adaptive, and emotionally aware, helping people not only to perform better but also to understand themselves and others more deeply.

Over the past 15 years, my research has focused on how eye movement metrics can be sensed and used for interaction with wearable systems [1, 2]. Decades of work have shown strong links between ocular metrics and cognitive states such as attention, fatigue, mental workload, stress, and focus, as well as neurological conditions. My current interest lies in understanding how these eye movement dynamics can support cognitive augmentation in future wearable technologies.

With the rise of eye-worn XR devices enabling continuous eye tracking, a key challenge is translating controlled lab findings into real-world settings. This issue extends beyond eye tracking: other neuro-sensing modalities like EEG, fNIRS, and EMG face similar gaps. Despite the growing availability of consumer brain-sensing devices (e.g., Mendi, MindWave), the mapping between neural signals and higher-order cognitive states such as focus or workload remains weakly supported. Direct, universal links between these signals and complex cognitive functions are still lacking.

At the Dagstuhl Seminar, we discussed cognitive sensing and augmentation from multiple perspectives, including ethics, diversity and inclusion, and technical challenges. One of the most thought-provoking discussions centered on the fundamental gap between sensing and interpretation. Specifically, cognitive states cannot be inferred in a straightforward, one-to-one manner from individual signals (e.g., “signal X implies cognitive state Y”), as these states are influenced by multiple contextual and individual factors, particularly in real-world environments.

Looking ahead, I envision a more reliable and ecologically valid approach that integrates multiple sources of low-level physiological and neural evidence from eye trackers, brain-sensing devices, and fitness trackers, with contextual information and user feedback. By correlating these multimodal signals with higher-level cognitive states and moods in a holistic, personalized framework, we can better capture individual variability, habits, and subjective experiences. Artificial intelligence will play a critical role in linking these multimodal data streams to cognitive biomarkers and potential health indicators, ultimately enabling more robust and meaningful cognitive augmentation through wearable technologies.

The rapid development of smart devices and emerging AI capabilities holds immense potential to transform not only our daily interactions but also our pursuit of life goals. Existing user interfaces, however, often fall short of realizing this potential. Individuals with disabilities, who rely heavily on assistive technologies for daily functioning and independence, face significant challenges due to the limitations of current approaches. Consider a person with visual impairment navigating a supermarket. While traditional accessibility features, such as text-to-speech, offer some assistance with shopping, they often require a cumbersome series of interactions, hindering the user’s ability to efficiently gather information and make choices. We argue that the perceived separateness and disconnect between accessibility devices and users hinder the experience of true empowerment.

Why is that? I believe that conversations have been overly focused on user-system coexistence and compatibility (e.g., “I will use this device” and “This is a user-friendly device”) rather than integration and augmentation of ability (e.g., “This device is a part of me” and “I have evolved with technology to unlock my full potential”). A paradigm shift is needed from viewing technology as a separate tool to considering it as an integrated extension of the human body, mind, and identity. We call this new perspective of human augmentation is called assistive augmentation [1].

Creating such Assistive Augmentations pose a threefold challenge as they require 1) exploring ways to seamlessly integrate into users body or behaviour, 2) deeper understanding of the user so as to provide the right level of support while keeping the user in control 3) holistic design approach to create meaningful applications. Inspired by insights from psychology, human centred machine learning, and design thinking, we have an unprecedented opportunity to design a new generation of Assistive Augmentations.

To highlight this, since 2011, I have been exploring several example project within this theme at the “Augmented Human Lab”. This include projects such as “AiSee” [2] that allows blind users to access information simply by pointing at objects and asking questions; “Prospero” [3] a memory training assistant that is able to detect when a user can learn more efficiently; “MuSS-Bits” [4], that allows deaf users to feel the music.

In a world where technology increasingly shapes how we think, feel, and connect, my research asks how we can design interactive systems that genuinely support people’s long-term well-being. One key area of my work focuses on cognitive augmentation such as neurotechnologies. I see this as a space for reimagining how humans and technologies can evolve together in ways that extend and enrich our cognitive and emotional capacities over time.

Within this space, my work addresses three interrelated challenges. The first concerns (i) bridging the gap between cognitive sensing data and people’s lived experience, for instance, by understanding when and how feedback should be provided and how individuals can create meaningful experiences with and from their data. The second explores (ii) agency and relationship building, examining how people experience agency when augmented and how they perceive such technologies, whether as tools, partners, or even as parts of themselves. The third centres on (iii) diversity, ethics, and regulation, highlighting the need to move beyond the narrow, Western-centric, neurotypical user profiles that dominate much of the current research landscape.

Drawing on my background in HCI and psychology, I develop theories, new measures, and interactive prototypes grounded in psychological knowledge. My work aims to translate conceptual insights into actionable guidance for researchers and practitioners, advancing a vision of technology that supports long-term well-being and meaningful augmentation experiences.

Evolution has always been the main driving force behind slow but steady change for the human brain and body. However, living in the Information era, our perceptual and cognitive capacities cannot simply rely on natural evolution to keep up with the immense advancements in emerging technologies. Tools we use daily (AI agents, Chatbots, PCs, smartphones, wearables, etc.) remain largely uninformed about our perceptual levels and cognitive states, forming a so-called “cognitive gap” between the human (user) and the machine (system) [1].

My research explores how AI-powered systems can sense and enhance human cognition in real-world productivity contexts, in an attempt to bridge the “cognitive gap” by utilizing physiological sensing, machine intelligence, and human-computer interaction principles. By combining sensing modalities such as EEG [2], eye-tracking [3], and physiological signals with generative and reasoning models, my work aims to interpret and respond to human cognitive processes, such as attention, recall, problem-solving, or creativity in situ. This approach contributes to developing adaptive, cognition-aware interfaces that support decision-making, learning, and productivity without compromising autonomy or well-being.

In the “Cognitive Sensing and Interaction” seminar theme, my interest lies in connecting cognitive sensing with generative AI (GenAI) to create systems that not only infer mental states but also proactively scaffold human cognition. For instance, integrating GenAI with sensed cognitive workload can enable systems that offer context-aware explanations, personalized feedback, or cognitive offloading mechanisms in manufacturing [4], healthcare, and education. Ultimately, this research seeks to advance human-centered cognitive augmentation by integrating intelligent systems into workflows, where AI complements rather than replaces human intelligence.

Remote collaboration in mixed reality often lacks the rich, implicit signals that ground face-to-face interaction, particularly those tied to cognitive and affective states. My research bridges this gap by integrating real-time physiological sensing with egocentric perceptual data (e.g., eye gaze, hand gestures, and first-person scene understanding) to reconstruct not only the physical workspace but also the cognitive context of collaborators. By fusing signals such as electrodermal activity (cognitive load), heart-rate variability (arousal), and EEG-derived attention with behavioral cues captured from head-mounted sensors, I design systems that support mutual awareness, adaptive communication, and empathic co-experience.

This approach builds on empirical findings showing that sharing combined physiological cues significantly enhances co-presence and social awareness in MR collaboration [1]. Critically, I treat these signals not as isolated metrics, but as interdependent markers of situated cognition: for instance, aligning gaze fixations with spikes in cognitive load can reveal moments of confusion or insight, while synchronizing gesture fluency with attention metrics can indicate confidence or hesitation. Such multimodal inference enables systems that scaffold not just task performance, but shared understanding, turning remote collaboration into a co-regulated cognitive process.

Within the context of the Dagstuhl Seminar on cognition and interaction, I am particularly interested in advancing frameworks that tightly couple perception (what is sensed from the user’s egocentric viewpoint) with cognition (what those signals imply about mental state). My goal is to contribute to a vision of human–machine interaction where technology helps collaborators think and feel together, using egocentric sensing as a foundation for reflective, adaptive, and cognitively attuned remote experiences.

People are increasingly eager to know more about themselves through technology, which has so far focused on physiological data tracking. Advances in wearable technology and AI are enabling consumer neurotechnology to measure cognitive activity, presenting new research opportunities in Human-Computer Interaction (HCI). I am bringing together knowledge from cognitive psychology, human-centered design, and computer science to explore how we can design interfaces and interaction techniques for cognition-aware technologies – specifically non-invasive consumer neurotechnologies (CNTs). The goal is to help people understand and reflect on cognitive processes, such as focus, stress, and learning, to improve their mental health and well-being. I follow a human-centered approach, studying how people want to engage with cognitive data to design consumer neurotechnologies that respond to people’s needs and support healthy reflection.

The seminar topic cognitive sensing and interaction aligns with my research ambitions, especially considering how we can balance inaccuracies in sensing with the users need for explanations and accuracy, in particular when there is a mismatch between objectively sensed data and subjective experiences.

My research investigates how technology can extend and refine human cognition through adaptive, embodied interaction. I focus on how mixed and virtual reality systems, combined with physiological sensing, can measure and respond to mental states such as fatigue, attention, and cognitive load. By integrating data from the eyes, muscles, and autonomic nervous system, I aim to build immersive environments that dynamically adjust to users’ capabilities, augmenting their awareness, resilience, and performance rather than merely tracking behaviour. A central theme in my work is cognitive ergonomics: understanding how digital systems can support, rather than strain, the human mind. I explore how feedback mechanisms, visual, haptic, or neuroadaptive, can promote sustainable engagement and recovery in cognitively demanding contexts such as training, remote operations, and human-AI collaboration. These systems blur the boundary between observation and augmentation, treating physiological signals as active interfaces that mediate understanding between body and machine. Working across computer science, psychology, and design, I collaborate with neuroscientists, optometrists, and engineers to develop technologies that act not only as tools but as cognitive partners. My goal is to establish a foundation for ergonomic cognitive augmentation: technologies that help humans think, perceive, and adapt more effectively, while preserving well-being and agency in the face of increasing digital complexity.

Augmentation technologies would likely increase adoption in the coming decades, thus understanding their behavioral and societal impacts is critical. My research examines how augmentation technologies influence human behavior and societal dynamics through a prototypical approach [1, 2]: implementing systems, analyzing their effects on human behavior [3] and social contexts [4], and synthesizing insights about short- and long-term impacts. My work has revealed that high expectations of improvement using such technologies increase risk-taking behavior and impact how people physiologically integrate information [5]. Interestingly, my investigation of nocebo effects has shown that these positive responses are resistant to negative narratives, meaning a nocebo effect could not be induced in augmentation technology contexts [6]. This work establishes a foundation for understanding both individual psychological responses and broader societal implications of human augmentation.

The Dagstuhl Seminar on cognitive sensing brought together researchers to address future challenges in this emerging field. I hosted a session exploring the trajectory of cognitive sensing technologies across three timeframes: short (5 years), medium (10 years) and long-term (15+ years). Through structured discussions with seminar attendees, we identified surveillance and potential abuse by corporate or institutional stakeholders as the most critical concerns requiring immediate attention. Specific scenarios that emerged included workplace surveillance and the abuse of physiological data capture to infer people’s interests, attention, and cognitive states without consent. The session highlighted that technical advancement must be accompanied by robust alternatives against misuse to ensure responsible development of cognitive sensing technologies.

The insights from the seminar revealed gaps in understanding how unequal access to augmentation technologies shapes society. My next research phase will focus on two interconnected areas: first, analyzing asymmetric interactions between individuals with and without access to augmentation technologies to map societal impacts; and second, developing frameworks for establishing and maintaining trust in augmentation technologies. This work will provide both empirical evidence of technology-driven inequality and practical guidelines for equitable deployment of human augmentation systems.

My research examines how multimodal sensing and artificial intelligence can make people smarter. I want to combine psychophysiological and behavioral signals, eye movements, electrodermal activity, heart-rate variability, EEG, motion, voice, and context, to infer metacognitive states to build technologies that can support users in self-regulating human-AI interaction. Our recent research could show that metacognition is the bottle neck for performance in human-AI interaction [1] and we know that we can improve performance by adaptive UI by means of physiological sensing [2]. Therefore, I believe that the next step is to build metacognitively-adaptive interfaces to AI.

A second focus is measuring and shaping co-experience in digital environments [3], particularly social XR. I operationalize co-experience as the dynamic coupling of cognitive and affective states across people, quantified through physiological synchrony, and validated reports of presence and co-presence. At Dagstuhl, we have discussed, especially with Kai Kunze, which different analysis strategies and sensors could be used to quantify co-experience.

My view of consumer neurotechnology is that all of the current innovation is lab experiments and machine learning of datasets for accuracy etc are all a small step in a bigger view of our future living with consumer neurotechnology. This future is one where we have as many apps and wearables for our brain as we do for fitness programmes and long-term health tracking. Physical activity tracking has studied this long-term real-world use of wearable tech in the field of Personal Informatics. I have co-led the community of people that now look at Cognitive Personal Informatics [2, 4, 5] and the impact this will have on the academic landscape [3].

As part of this vision, we began work in 2018, performing longitudinal fieldwork with people tracking their own Mental Workload (as our initial theoretical framework). Initially this was self-reported through an app, and led to our introduction of the Mental Workload Cycle [1] – as a model for people should reach optimal mental effort levels across a whole day. This is distinct from most physical activity tracking that assumes more is always better, but closer to now more expert models of training support that models both activity and recovery.

Our in-progress work now involves longitudinal fieldwork with people that do have viable consumer neurotech. Over the last 12 months, we have engaged in a year of public engagement on consumer neurotechnology. Over 13 events, we have given 5,000 members of the public, teachers, children, parents, lawyers, policy advisors, and politicians the chance to try neurotechnology, and spoken to at least 5,000 more. My research vision is to move first-generation consumer neurotechnology towards devices that help people set goals for the brain, cognitive, and mental health, and can meaningful communicate progress towards them.

My research focuses on developing personalized wearable technologies that assist individuals with disabilities and chronic health conditions through sensing, actuation, and collaborative physical interactions. A key aspect of this work is engaging end users in the design and fabrication process using co-design tools. This participatory approach ensures that the resulting technologies are not only functional but also meaningful and empowering for the people who use them.

A key aspect that was missing in my research was cognitive considerations of co-design, actuation, and collaborative physical interactions. Critical questions include: How can cognitive sensing enhance creative co-design processes? How do we manage the complexities and adaptability of physical wearable devices wthout cognitive overload? How can cognitive sensing provide better control and agency over wearables? My main goal in attending the seminar was to broaden my research scope to address these challenges and integrate cognitive dimensions into wearable design.

In the seminar, I gained insights into the state of the art in cognitive sensing, both in commercial applications and academic research. I connected with new colleagues for potential collaborations, particularly in embedding cognitive sensing into co-design processes. These discussions opened exciting opportunities to explore how cognitive sensing can enhance accessibility and physical embodiment, how current cognitive sensing technologies can be cp-design and made with personal fabrication approaches and future ethical challenges that are worth exploring in the area.