Cognitive Augmentation

My research is exploring the use of Virtual Reality (and Mixed Reality in general) to go beyond what is possible in the real world. VR allows a user to slip into other bodies in various worlds. The self-perception of one’s body can be altered by presenting the user by varying the look of one’s body, or change how others are perceived by a simple touch gesture that utilizes social touch [1]. However, VR enable to present the user with a different viewpoint as well. In previous research, we presented the existence of a perspective continuum in VR [1]. Here, the viewpoint of the experience is altered to enable an out-of-body experience for the users, as they have control over a character via motion control. However, the users see themselves from their outside, therefore resulting in an experience where they are embodied in “two bodies”, one is the character, the other is the camera. How this experience and embodiment of the two bodies is perceived, highly depends on the users. While some feel like “they are the character” while looking at it from the outside, others have the feeling of only being an observer, or even being embodied in two bodies at the same time. This poses the question of the limits of human embodiment if one person can be present and embodied in multiple bodies at the same time? Can these bodies also take on abstract forms? Can humans be augmented to not only be present in multiple bodies, but also multiple times?

The emergence of cognitive augmentation technologies promises a range of unique benefits: they may improve accessibility and help create equitable opportunities for everybody, while also supporting people in developing, even extending, their natural abilities and skills. Yet, new social dynamics, needs, and challenges may dynamically emerge when a novel technology is “released” into-the-wild. Cognitive augmentation technology may (and likely will) create social tensions or intensify existing individual or societal vulnerabilities. Therefore, it is crucial to research social, societal and ethical implications alongside technology-driven advances of cognitive augmentation.

In my work, I ask how we might operationalize social acceptability and related social and societal phenomena as a core component in HCI research. My past research focused on social acceptability issues with body-worn cameras (see Figure 7) that can serve as assistive devices for the visually impaired or improve indoor navigation (1) but face significant social acceptability issues, e.g., due to concerns about bystander privacy (2). More recently, I expanded my research focus towards the design and fabrication of on-body interfaces, e.g., electrode sleeves (3) or hair interfaces (4), focusing on their perceptual, social and ethical aspects and their wearability in social contexts. With our current and future research, my freshly founded research group and I aim to contribute theories and models that help articulate the role technology plays in social contexts, empirical studies that test and refine these theories, and new tools and methods that help to design interfaces that meet social and societal needs.

I enjoyed my first stay in Dagstuhl to the fullest. The discussions with researchers related to cognitive augmentation were very stimulating and suggested what I should do next.

I develop assistive devices using robotics technology (mainly pneumatic artificial muscles and teleoperated robots). In our field, improving exercise and work performance are important evaluation axes. However, through our discussions at Dagstuhl, we strongly recognized that these affect cognition simultaneously. Conversely, cognitive augmentation will also affect physical activity. We need to determine whether this impact will be positive or negative.

In addition, I felt that social action is also important to ensure that these technologies reach a wider audience on an equal basis. Human augmentation technology should not be a technology for a limited number of people. Let’s bring augmentation technology to all people. I feel the important thing is to design the future. We must imagine what we want to be and what we need to do to make it happen.

As people are increasingly relying on technology to complete tasks that used to be done by hand. We are losing certain traditional skills such as calligraphy, especially handwriting in Chinese. Character amnesia is a frequent phenomenon that experienced Chinese speakers forget how to write certain characters previously well-known to them. The same goes with other delicate craftsmanship that is a tangible manifestation of the cultural heritage but is on the verge of extinction (e.g., Sichuan embroidery).

In addition to dependency on technology, younger generations find it demanding to learn to write Chinese characters or these delicate skills that often require a lengthy apprenticeship. To ensure that the knowledge and skills are passed onto future generations, cognitive augmentation technology could be used to help people learn and master these skills more efficiently and easily.

By placing sensors on the human body, technology such as brain-computer interfaces (BCIs) or electromyography (EMG) have the potential to translate electrical activities of the brain or the muscles into commands that the computer or device can understand and execute. People can be augmented to control a digital pen or brush using their thoughts or using programmed muscle memories, even if they are unable to physically perform these tasks due to injury or illness. This could help to prevent these skills from being lost due to lack of practice.

Grand challenges regarding craftsmanship augmentation include that:

1. 1.

   Can we develop augmented craftsmanship that can efficiently help us learn, master and retain a sense of control and precision that is comparable to or even much better than a skilled craftsman?

2. 2.

   How can we ensure that these skills are still appreciated when we can speed up the learning and mastering process?

My research interests lie in developing cognitive augmentation technology to help people be more attuned to themselves, others, and the environment. To explore this, we have built a series of design works such as HeatCraft, a wearable system that can generate thermal stimuli based on one’s body temperature sensed by an ingestible sensor. The stimuli’s temperature and body temperature change reversely, so when body temperature raises, the stimuli’s temperature drops down. Our study shows that this device could influence how people perceive and use their body. For example, a participant chose to drink some icy water to get the heat in winter. Another example is our recent work in progress GoChirp, an AI-powered wearable device that can continuously sense the existence of surrounding birds via recognizing bird songs, and provide haptic sensations when birds are detected. The device was designed for enhancing people’s awareness of wild lives and engaging them with nature in urban lives. In general, we hope the future workaround cognitive augmentation is not only about transhuman and augmenting one’s capability, but also considers the experiential perspective. We hope to develop more humanized cognitive augmentation technology to better support attunement.

My research focuses on human augmentation to enhance our capabilities and experiences. I study the effect and impact of human augmentation in different domains such as sports, games, health, safety & security, or engineering. Cognitive augmentation is a form of human augmentation that aims to create novel experiences using and extending human perception and cognition. In previous work, we have explored how a location-based information system in combination with augmented reality impacts work processes, awareness, and workload for hotspot policing [1]. While the location-based information system enhanced human perception and cognition, the timing of information delivery, the adaptation of content to situational characteristics and the modes of information delivery are of interest for further investigation. In more recent work, we have investigated the use of everyday objects to interact in augmented reality [2], how to provide augmented feedback for the correct execution of strength exercises, how to bridge the spatial gap between local and remote players in location-based games, how to provide collaboration awareness during complex co-located collaborative tasks, or how to support the mental imagery practice of elite athletes in preparation for competitions. While all projects are embedded in different domains, all have aimed at enhancing cognitive capabilities and experiences and provided initial insights into a deeper understanding of human and cognitive augmentation. Still, further research is necessary to identify use cases for cognitive augmentation or to determine benefits of cognitive augmentation. Can cognitive augmentation be used to facilitate presence, awareness, decision-making, knowledge recall, skill acquisition, or empathy? What kind of sensors and input do we need to create cognitive augmentation? From an ethical and societal perspective, it is further necessary to explore if we want to create new cognitive capabilities or whether we want to enhance/augment existing ones? What are possible benefits, drawbacks and ethical dilemmas for cognitive augmentation? Should, e.g., others be aware of someone using cognitive augmentation? These questions are just some of those that future work on cognitive augmentation will need to address.

While today’s devices put the world’s information at our fingertips, they do not help us with some of the cognitive skills that are arguably more important to leading a successful and fulfilling life, such as attention, grit, motivation, creativity, memory and learning, mindful decision-making, and emotion regulation. Building upon insights from psychology and neuroscience, my research group at MIT creates AI systems and interfaces for enhancing human cognition. Our designs range from assistive technologies that help people overcome disabilities to technologies that help people further develop their natural abilities and skills.

One of the areas we are particularly interested in lately is opportunities for enhancing cognition during sleep. Good sleep is crucial not just for health, but also daytime cognitive performance. There is an opportunity for HCI researchers to venture into systems and experiences that interface with and influence the sleeping mind. Commercially available devices for sleep such as Oura ring, Fitbit and Smartwatches primarily focus on monitoring and quantifying sleep. Instead, our work looks at how stimuli presented during sleep, including scent, sound and electrical stimulation issued at specific moments of the sleep cycle, can impact memory (strengthening & weakening), time to sleep onset, and depth of sleep.

I currently co-direct the En-Ability Lab at RIT. The objectives of the En-Ability Lab is to enable, empower and enhance abilities through technology and design. At the En-Ability Lab, one of my main research directions is “Immersive Accessibility” that looks at immersive technologies (AR/VR and Haptics) and accessibility. Under Immersive Accessibility, we conduct research under two main themes, 1) Making Immersive Technologies Accessible and 2) Immersive Technologies for Accessibility Applications. Under theme 1, we primarily identify accessibility limitations of existing immersive technologies and explore methods to make them more accessible. Here, one example project is SoundVizVR where we examine new ways to visualize sounds in VR using mini-maps and indicators to improve the accessibility of VR for Deaf or Hard of Hearing (DHH) users. Under the 2nd theme, we explore new ways to use immersive technologies for accessibility. For example, in the OneButtonPIN research, we examine using haptics to make the PIN code entry process more secure and accessible for Blind and Low Vision participants while the Haptic Captioning project aims to enhance captions and improve speaker indication for DHH caption viewers.

With Cognitive Augmentation, my main goal is to examine ways in which we can re-imagine existing assistive technologies to be more implicitly integrated with users.

Due to the increasing integration of cameras in everyday devices (e.g., smartphones, smartwatches, head-mounted displays) and the continuous progress of hardware and software in eye-tracking, there is a possibility of capturing users’ visual attention continuously and in real time. Sensing the human eye allows the assessment of users’ cognitive states and processes, providing crucial information for systems that aim to augment users’ cognitive abilities. We were interested in using gaze-based information in remote education to enhance meaningful learning, improve communication and collaboration [1, 2]. Here, challenges such as accurately sensing the visual attention of a crowd of distributed users in real-time [3] and the meaningful augmentation of learners’ cognitive abilities remain. In the future, we will investigate how educational systems can provide gaze-based feedback to support learners’ cognitive processes and abilities during learning and perfecting skills.

I envision a future where people can dive into game worlds. To make this happen, I previously pursued my Master’s degree in Computer Science and Information Engineering at National Taiwan University. To expand game genres, I published Game Illusionization: A workflow for applying optical illusions to video games to tell people how to make games with illusions. People can follow our workflow step by step to create their own illusion games. We offer an illusion database for people to search their desired illusions. And we also provide editors to help people build their illusion games in Unity Game Engine. I hope this work can inspire people to apply illusions to interactions more, and also make the game industry more thriving. Apart from illusions, I also research in Virtual Reality. However, VR headsets are not immersive enough. So, I think maybe dreaming is one of the solutions, where we have full sensations, and we think everything in the dreams are so realistic. Especially, I would like to research in the lucid dream. Lucid dream means we can control our dream content, just like playing video games, but in a more immersive way. Through explorations, I believe one day we could build immersive experiences for gamers.

What does it mean to be live and to experience liveness? The experience of live performance, be it theater, dance, music, comedy, or even inspirational talks, is a unique multi-sensorial phenomena that connects people in a particular moment. But the concept of liveness defies clear definition. It might be connected to co-presence in a particular space, yet equally it can transcend space and be experienced across many spaces at once, such as during live broadcast events. It might be the social connection forged in a shared experience, but can equally apply to an individual experiencing some event happening for the first time. This would suggest a link to the uniqueness of the performance, enhancing the feeling that those involved, those who get to experience it, are privy to something special. One intriguing take is that liveness is related to jeopardy, that there is always the possibility that things might go wrong – that, even in the most choreographed performance, no-one knows what is going to happen next. This, for me, is the true spirit of what it means to be live: a shared exploration by audience and performers of where they are headed and what they can discover together.

In my work, I ask how we can measure liveness, and how we can recreate it during situations that are not live – how might we augment our experiences to be more live? To do this I bring together my research in wearable computing, human activity recognition and social neuroscience, and combine this with my experience as a professional actor working in theater. I use the methodology of “theater as a laboratory” to try to uncover the social signals and behaviors that occur during live performance, and uncover what these tell us more broadly about the human condition. My work uses wearable motion sensors, wearable neural hyper-scanning and physiological sensors to study the brains and bodies of audiences and performers during live theater and dance. From a technical perspective, this work aims to solve problems related to multi-person sensing, streaming, and signal processing. From a user-experience perspective, the work aims to solve ways in which such data might be used to enhance performances – to augment our experience in some way. And from a wider perspective, the work aims to uncover what it means to be live.

I’m interested in how we use computer programming principles to systematically change our cognitive processes. “Coding your cognition” refers to the intentional and systematic process of modifying or improving one’s cognitive abilities, such as memory, attention, and creativity, through computational means. This concept is based on the idea that the mind is not fixed but plastic, which means that it can change and adapt throughout life in response to environmental and experiential factors. If we understand the intrinsic mechanisms and attributes of our mind. We can apply actuation to change them.

We are currently exploring these principles, for early stage research prototypes targeting our visual perception. In a first example, we are simulating a visual impairment experience using optical see-through glasses [1]. Users can experience both losses of central vision and loss of peripheral vision at different levels. We can raise the awareness regarding We also attempt to transfer eyegaze from one person to another [2].

These are just very early works focusing on visual perception. Compared to our cognition, we understand much more about our vision system, as it’s much easier to test and probe. Yet, in the following years, we plan to extend this approach towards higher level cognitive processes.

Imagining future computers will “weave themselves into the fabric of everyday life until they are indistinguishable from it” [1] suggests that future pervasive computers are embedded into traditional materials. Aiming at the creation of a rich and good user experience when interacting with pervasive computers, I see two opportunities. We could either embed other technology that creates a haptic sensation, or we use the concept of sensory illusion that augments our perception without the need for additional technology. Sensory illusion can extend the design space traditional materials properties are offering, e.g., can create the illusion that any surface provides a rich bandwidth of touch feedback when being pressed like a button, such as softness, wetness, and bendability.

In previous work on sensory illusion, I and colleagues applied knowledge from cognitive science to human-computer interaction (a) to better understand the sensory involvement when using certain interface modalities and (b) to explore the possibility of sensory illusion to compensate for shortcomings of interface technologies to extend the user experience. We investigated the tactile and non-visual perception of surface textures, aiming for defining texture patterns that serve as haptic cues for haptic surface texture perception [4]. In another work, we found that sonic cues bias the haptic surface perception by strengthening the haptic perception of the texture pattern if a sonic cue is given in parallel (Wolf and Bennett 2013b). We furthermore investigated how proprioception can support or replace visual feedback on pointing gesture execution [5]. Moreover, we explored the effect of visual distortion in combination with electrotactile feedback on surface illusions [3]. In this investigation, it could be shown that surface illusion can be induced through visual distortion of texture projected on a flat surface as well as through electrotactile stimuli provided when a surface is touched. Finally, I proposed concrete interaction design ideas for illusion-based pervasive interfaces considering any surface to possibly be an interface believing that windows, furniture, and even the floor we walk on will be an interface one day [2].

Cognitive augmentation is concerned with the augmentation of cognition, but due to the intertwinedness of the human mind and the body, we argue that augmentation is always bodily. We demonstrate this through a series of research design works around augmented cycling experiences, augmented social cohabitating experiences, and augmented emotion amplification experiences. The results of these works suggest interesting ways forward for augmentation research, in particular how the design of augmentation can highlight experiential aspects, facilitating playful experiences. Ultimately, with our work, we want to enhance our knowledge around the design of augmentation to help people understand who they are, who they want to become, and how to get there.
http://exertiongameslab.org