Designing Computers’ Control Over Our Bodies

The seminar began with an introductory session led by the organisers, where participants were welcomed and introduced to the seminar’s goals and structure. The day continued with a series of “PechaKucha” presentations, where each participant had five minutes to present their work and perspectives on computational control technologies. This fast-paced format helped set the stage for a deeper exploration of the grand challenges and opportunities that these emerging technologies present. The day ended with a group activity where participants identified and categorised key challenges, which were then organised into broader themes of grand challenges to be addressed throughout the seminar.

Building on the theoretical discussions from Day 1, Day 2 was dedicated to exploring the design challenges associated with computational control over the human body. Participants were invited to bring prototypes or share design concepts during hands-on sessions. These sessions facilitated a deeper understanding of how to create responsible and ethical design frameworks that accommodate the complexities of machine control over human bodies. The day also included further PechaKucha presentations, followed by group activities to synthesise insights from both the theoretical and practical design perspectives.

Day 3 bridged the gap between theory and practice, focusing on the integration of theoretical frameworks with practical design approaches. Participants worked in breakout groups to align their individual seminar goals with the continued process of identifying grand challenges. This day featured extensive discussions on how to merge theoretical insights with design principles to create holistic approaches to computational control technologies. The morning session included collaborative group work, and the afternoon focused on presenting and critiquing the developed ideas.

On Day 4, the seminar shifted towards envisioning future directions for research and development in this field. Participants engaged in speculative design sessions to imagine scenarios where computational control technologies (controlled by AI or otherwise) are fully integrated into daily life by 2050. These speculative exercises aimed to identify utopian and dystopian possibilities, encouraging participants to consider such technologies’ ethical, social, and psychological implications. The day concluded with a plenary session where groups reported on their envisioned futures and debated their desirability and feasibility.

The final day was dedicated to synthesising the seminar’s outcomes and identifying tangible next steps, particularly directed at writing an article on the grand challenges that surfaced in this area throughout this seminar. Participants worked on drafting concrete outputs, including research papers, frameworks, and guidelines that could serve as foundational documents for this emerging field. The day also included reflective sessions where participants shared their experiences, insights, and ideas for future collaboration. The seminar ended with a discussion on the formation of working groups to continue the momentum generated during the week.

This structured and intensive programme allowed participants to explore the complexities of computational control technologies from multiple perspectives, paving the way for future research and collaboration in designing humane and ethical interactions where computers assert control over the human body.

From Day 1, several key themes emerged as participants discussed the grand challenges related to computational control over the human body (Figure 4). These themes were documented in the notes taken during and after the presentation sessions:

1. 1.

   Control and agency:

   • $\mathrm{■}$

     A significant theme centred around the relationship between control and agency. Participants debated how computational systems could either enhance or diminish a user’s sense of agency. Discussions included both implicit control (where control is seamlessly integrated and less noticeable to the user) and explicit control (where the user is fully aware of and potentially directing the control dynamics).

   • $\mathrm{■}$

     Questions emerged regarding how control should be designed to balance these dynamics, ensuring transparency and maintaining a user’s feeling of autonomy.

2. 2.

   Implicit vs. explicit control:

   • $\mathrm{■}$

     The balance between implicit and explicit control was identified as a critical design consideration. Participants explored how different forms of control might affect the user’s experience, comfort, and acceptance of such technologies. For example, while explicit control allows for more transparency and user awareness, implicit control might lead to smoother, less intrusive experiences but could also raise concerns about trust and manipulation.

3. 3.

   Designing for sensory and perceptual experiences:

   • $\mathrm{■}$

     Discussions highlighted the importance of designing technologies that consider sensory and perceptual aspects, such as visual augmentations to the body or face, haptic feedback, and other multisensory experiences. Themes like the Proteus Effect (how a user’s self-perception changes through interaction with technology) and the placebo effect were discussed, pointing to the need for careful consideration of how sensory augmentations might alter user experiences and perceptions.

4. 4.

   Adaptivity and personalisation:

   • $\mathrm{■}$

     Another theme focused on the need for adaptive systems that can adjust to individual user differences, such as age, physical ability, and cognitive load. Participants raised questions about how these systems could dynamically modify their behaviours or outputs to better suit the user’s evolving needs, thereby enhancing usability and user satisfaction.

5. 5.

   Negotiation of control:

   • $\mathrm{■}$

     The notion of negotiating control between the human and the machine was a recurring topic. This included discussions on collaboration vs. automation and how users and systems can share or shift control in various contexts. The concept of “negotiation” in this space addresses how to design interactions that are flexible, adaptable, and responsive to changes in user intent or environmental conditions.

6. 6.

   Ethical considerations and dark patterns:

   • $\mathrm{■}$

     Ethical issues such as transparency, informed consent, and avoiding manipulative “dark patterns” were heavily discussed. The theme of responsible design emerged, focusing on creating technologies that respect user autonomy and ensure ethical standards are upheld. This theme underscored the importance of designing systems that can communicate their intentions clearly to users and provide meaningful choices regarding control dynamics.

7. 7.

   Social and embodied perspectives:

   • $\mathrm{■}$

     Participants explored the social implications of these technologies, examining how control over the body might affect social norms, behaviours, and interactions. Themes related to the social environment, the effect of such technologies on shared experiences, and how they could either enhance or disrupt existing social practices were identified.

8. 8.

   Awareness and performance:

   • $\mathrm{■}$

     The role of awareness, both in terms of a user’s self-awareness and their understanding of the system’s behaviour, was another key theme. Discussions also touched on performance metrics, questioning how these technologies could optimise human performance without compromising ethical standards or user comfort.

9. 9.

   Future design directions:

   • $\mathrm{■}$

     Participants speculated on future directions for design, asking how to create systems that balance autonomy, adaptability, and user comfort. They considered what frameworks could guide the ethical development of technologies that exert control over the human body and how to align these technologies with human values and needs.

These themes reflect a broad and deep engagement with the challenges and opportunities presented by emerging control technologies, highlighting the complexities involved in designing systems that can control the human body.

• $\mathrm{■}$

  Shared understanding: By the end of Day 3, there was a clearer shared understanding of the core concept of control and its complexities within human-computer interactions. This was marked by a convergence of ideas on the need for new frameworks and models that account for the diverse experiences and expectations of control.

• $\mathrm{■}$

  Framework for future research: The synthesis of challenges led to the creation of a preliminary framework outlining the key areas for future research. This included ethical guidelines, design principles, and technical requirements that would guide the responsible development of control technologies.

• $\mathrm{■}$

  Interdisciplinary collaboration: The discussions highlighted the importance of interdisciplinary collaboration, bringing together perspectives from psychology, engineering, ethics, and design to address the multifaceted nature of control technologies. Participants identified potential collaboration opportunities and established working groups to continue exploring these themes beyond the seminar.

• $\mathrm{■}$

  Engineering challenges: The group focusing on engineering challenges provided insights into the technical constraints and possibilities of implementing control technologies. They emphasised the importance of rigorous testing, user feedback loops, and iterative design processes to ensure these technologies are both effective and aligned with user needs.

These group activities set the stage for subsequent discussions and sessions, creating a collaborative environment where participants could further refine their ideas and contribute to the development of a comprehensive understanding of computational control over the human body.

The hiking tour incorporated a dedicated career advice session where participants could seek guidance and share insights about professional growth and development. Senior academics and experienced researchers walked alongside early-career participants, facilitating open and candid conversations about navigating the complexities of academic and research careers.

Participants were encouraged to switch groupings periodically, allowing them to interact with a variety of peers and mentors. This dynamic format fostered an environment where advice could be exchanged freely, and participants could gain diverse perspectives on topics such as:

• $\mathrm{■}$

  Building a research profile and securing funding.

• $\mathrm{■}$

  Balancing academic responsibilities with personal life.

• $\mathrm{■}$

  Navigating interdisciplinary collaborations.

• $\mathrm{■}$

  Strategies for publishing in high-impact journals.

• $\mathrm{■}$

  Leveraging professional networks for career advancement.

Many participants found this informal setting particularly conducive to discussing challenges and aspirations they might not typically share in more formal seminar settings.

After the hike, the participants reconvened for a group dinner, where they continued their conversations and reflected on the day’s experiences. The dinner provided an additional networking opportunity, solidifying new connections made during the hike and sharing reflections on the seminar’s themes.

The combination of physical activity, career-focused discussions, and social interaction helped deepen the sense of community among participants, enhancing the collaborative spirit developing throughout the seminar.

Overall, the hiking activity was a refreshing and productive break that enriched the seminar experience, offering personal and professional growth opportunities.

Many people enjoy “vertigo” sensations caused by intense, playful bodily activities such as riding fairground rides. Game scholar Caillois calls such experiences “vertigo play”, elucidating that these enjoyable activities result from confusion between sensory channels.

In HCI, designers are often cautious to avoid deliberately causing sensory confusion in players, but we believe there is an opportunity to transition and extend Caillois’ thinking to the digital realm, allowing designers to create novel and intriguing digital bodily experiences inspired by traditional vertigo play activities, through exploring methods of allowing computers varying degrees of control over one’s body.

The Digital Vertigo Experience framework is derived from four case studies and the development of three different digital vertigo experiences. This framework aims to bring the excitement of traditional vertigo play experiences to the digital world, allowing designers to create novel body-based games and providing players with more possibilities to enjoy exciting play experiences [3].

Body control techniques, such as Electric Muscle Stimulation (EMS), Galvanic Vestibular Stimulation (GVS), and Exoskeletons, enable computational agents to act on the user’s body. These techniques can offload control of voluntary movements to assist in physical tasks, stimulate body functions for rehabilitation, and redirect walking and reaching movements in virtual reality applications. Despite promising applications, more work is needed to outline the design space of body control techniques. Mapping out the available methods from the perspective of body entry points and targets for control signals can help researchers identify opportunities for enabling more comprehensive body control. What signal entry points could be used to control a specific body part? Which body parts can be controlled through the same entry point? Which control modalities can be combined with which sensing modalities without interference? What are the modalities most resilient to the user’s resistant attitude? Synthesizing this landscape will support design decisions in body control applications.

Tickling is a type of sensation that is associated with laughter, smiling or other similar reactions. Psychology research has shown that tickling and laughter can significantly relieve stress. Although several tickling artifacts have been suggested in prior work, limited knowledge is available if those artifacts could evoke laughter. In this article, we aim at filling this gap by designing and developing a novel foot-tickling mechanism that can evoke laughter. We first developed an actuator that can create tickling sensations along the sole of the foot utilising magnet-driven brushes. Then, we conducted two studies to identify the most ticklish locations of the foot’s sole and stimulation patterns that can evoke laughter. In a follow-up study with a new set of participants, we confirmed that the identified stimuli could evoke laughter. From the participants’ feedback, we derived several applications that such a simulation could be useful. Finally, we embedded our actuators into a flexible insole, demonstrating the potential of a wearable tickling insole [4].

Control in HCI is poorly understood. We lack a systematic approach to modelling, predicting, and benchmarking control. At the same time, however, control is arguably the central challenge of interaction design: controlling a system towards a goal. Across its diverse history, HCI has adopted two primary levels for studying control, implicit control and explicit control. Implicit control concerns itself with the low-level study of control perception. To study this, we have adopted neuroscience methods, primarily relying on intentional binding and the Libet clock, but the demands of these methods make them hard to apply for interaction design, and the results appear to have limited links with our subjective feeling of control (our feelings of explicit control). Explicit control methods are primarily “qualitative”, effectively asking “do you feel in control?” I posit that this question is broadly unanswerable and the data it yields almost arbitrary. We need better methods [5].

Bestowing computer systems with control can amplify people’s ability to input information into systems. Gesture keyboard systems enable users to write using touchscreen and other input media by sliding a finger across a keyboard. Serial single-finger typing is a closed-loop control task with performance limited by visual attention. Using a gesture keyboard, prolonged use results in motor memory consolidation of frequently used gestures for words. This allows users to transition to open-loop direct recall from motor memory, which is faster. However, for this to be possible the system has to interpret imprecise open-loop articulation using a decoder. In effect, the user gains efficiency by transitioning from precise closed-loop to imprecise open-loop and this transition is only possible because a decoder interprets the imprecise input. Effectively, control is shifted towards the computer system in exchange for efficiency. The same principle can be exploited in other applications, such as dwell-free eye typing, which removes the time-consuming and fatiguing dwell timeouts of regular eye typing by interpreting eye gaze data using a decoder. Other examples include typing in thin air and automatic activation of virtual tools in virtual reality by the user simply articulating the tool motion to trigger the tool, removing the need for the user to first select the tool. However, bestowing a computer system with this additional control may also come with a cost. It is possible to empirically measure a user’s sense of agency and we can demonstrate that, in at least one instance, at a certain point increased computer assistance results in a dramatic reduction in the user’s sense of agency.

Our bodies, serving as intermediaries between physical and virtual realities, play a crucial role in transmitting our perceptions and presenting our behaviors. Previous research primarily focused on creating one-to-one replicas of the real world as avatars in virtual environments or utilizing real-world on-body interactions to enhance immersion. This approach, however, has not fully exploited the potential of bodily interfaces. For example, if our bodies were composed of swarm particles with a fluid and liquid form in virtual reality, how would our perception adapt to such a dynamic state? Would our viewpoint fluctuate with the liquid’s movements, and how could we exert control over such a form? Future research should aim to design diverse bodily interfaces to investigate the implications for agency, ownership, perception, affordance, and interaction. This exploration is essential to comprehensively understand and enhance our presence and interaction within virtual environments [9].

In this talk, I first introduced several examples of my work related to sharing control with computers, exploring the interplay between human autonomy and technological influence. The first part focuses on ingestible play, a game based on sensors. In these games, participants swallow a digital sensor that monitors bodily information to serve as a game controller. However, control over the sensor is limited as participants cannot direct its movement within the body. In some cases, players can influence it indirectly, for example, by consuming coffee or bananas to affect digestion speed. Also, it would be hard to control the data the technology captured, as we don’t have direct control over our interior body. These kinds of play make people aware that we have limited control over our body as the invisible part of it. The second work I shared in this talk involves sharing bodily control with an intelligent exoskeleton in body games. This exoskeleton is designed to predict and actuate optimal movements during gameplay. We developed three modes for this system. In the first one, the exoskeleton autonomously moves the player’s body to compete against a computer-controlled opponent. In the second mode, one arm controlled by the exoskeleton competes against the player’s other free arm. With the last one, the exoskeleton controls one arm while the player decides whether to move the other arm to align with the AI’s predictions to play. Both projects aim to augment human capabilities – whether to enhance gaming performance or to increase self-awareness. However, these augmentations also raise important questions about how much we should allow machines to influence or determine human actions and whether we should let machines decide what is good for humans [10, 11, 17].

Our work on uncomfortable interactions explored technology interactions in which people deliberately expose themselves to discomfort. Experiences such as rollercoasters, theatre shows, and horror films have in common the requirement that people gradually surrender control to technology over the experience. Two examples: In Bronocomatic, a bucking bronco ride is controlled by the rider’s breathing. Initially, this means the rider is fully in control of the ride movement, but as the movement becomes more extreme, riders are no longer able to control their breathing, and control shifts to the machine. Teslasuit uses electrical muscle stimulation to create a range of sensations in the wearer. At the lowest stimulation, it is like a gentle touch, and users stay fully in control. As the stimulation gets more intense, the wearer becomes less in control of their body until, at full strength, the intensity is such that wearers fall to the ground and cannot control their muscles.

My focus for designing computers’ control over the body centers around bodily play and to answer the following question: How can we design for bodily playfulness? A key element in my work with bodily play is movement – and making (game) moves. I have developed two games, each of which manipulates the player to move and not move in specific ways. The first game, Space Agent, used an interactive binaural soundscape to drive the players to twist and turn. The other game, Move Maker, explored different ways of restraining the body as it had to move around playing with the game’s paraphernalia as the surrounding conditions for movement. My recommended reading is about the latter game. My work merges my background as a PhD in game and interaction design (Norwegian University of Science and Technology) and a Music and Movement Teacher (Rhythmic Music Conservatory, Copenhagen).

Since the emergence of the notion of body-machine couplings dating back to the 1950s (e.g., Clynes and Kline, 1960), interface design and engineering has increasingly explored developing systems with varying levels of human control (e.g., full, partial) and machine control (e.g., autonomous, partial) (e.g., Casado, 2013). Enactivist perspective to cognition (e.g., Thompson and Varela, 2001) can open a window into exploring the notion of control in subtle processes of situated interaction, perception and action (e.g., Rowlands, 2010). Methodologically, this approach may engender microinteractional dimensions of “computers” control over our bodies’ observable, and develop our understanding of the details about where, and how, the control is. This may offer insights about how “control” is perceived and acted upon in care contexts applying tactile robotics (e.g., Parviainen et al., 2019), or mirroring digital environments (e.g., Nygren, 2023).

Traditional digital games predominantly treat the human body as an input mechanism, overlooking its potential as an output medium. This oversight limits the depth of interaction and experiential richness possible in human-computer interaction (HCI). Body-Actuating Play (BAP) [21, 26], discussed in this seminar, is a concept where bodily control is shared between the player and the computer, enabling a novel form of play where the body is both an input and output medium. This novel concept transforms the user’s role from a mere operator to a collaborator with the computer, shifting their role to that of both influencer and responder.

As body-actuating technologies like electrical muscle stimulation and exoskeletons begin to control and augment human actions, we find ourselves at a pivotal moment in redefining the relationship between humans and technology. The potential for shared bodily control in BAP challenges our traditional notions of autonomy and control in digital play, provoking deeper reflection on the nature of the control we have over our bodies and the ethics of machine intervention in our physical selves [25].

Crucially, the exploration of BAP compels us to consider the psychological impact of sharing bodily control with such technologies. What does it mean for our sense of self when a machine can respond to our movements and direct them? How do we reconcile the benefits of enhanced physical capabilities with the potential loss of personal control over our bodies? While BAP is an exciting concept to explore, the ethical landscape of these innovations must be navigated with care, ensuring that the design of these technologies prioritises user consent, safety, and well-being.

Ultimately, BAP invites us to envision a future where humans and machines are intertwined in a seamless, symbiotic relationship, pushing the boundaries of what is possible in digital play experiences. This paradigm shift calls for a re-evaluation of traditional design frameworks within the HCI community, embracing the body as a central component of digital play. By designing future BAP experiences, we can create technologies that entertain and enrich and expand the human experience in previously unimaginable ways.

Our appearance and our interactions with the world around us are closely intertwined. How we dress, how we do our hair, and what makeup we put on signals something about who we are to others. Currently, more and more facets of our appearance are becoming adaptive and changeable through computational processes. Physically, this includes shape- and colour-changing clothing and accessories, as well as electronic tattoos, beauty products, and jewellery. Virtually, many more changes are possible, such as beautification filters, digital fashion, or blurring out of tattoos. Control over such body augmentations right now is mostly in the hands of the person themselves. However, such control might well transfer to others or to computational systems. For example, we can envision a future cocktail party setting where I put virtual name tags on people (i.e., I have control over their appearance), wear glasses that can turn heart-shaped on their own (i.e., I delegated control to a system and asked it to accentuate my flirting), or everybody relinquishes control over their looks (i.e., a central party AI takes charge to maybe ensure the party is eventful). Thus, as our appearance becomes interactively malleable, we also have to negotiate how much control is retained or delegated [29, 30].

Our bodies are intimate and personal: social, embodied, and physiological. This demands considerations for design – beyond the flesh – of computer “control” on our bodily engagement, raising 3 key questions: what do we mean by “bodies”, what do we mean by “control”, and what methods can foster important social design considerations? Bodies are not just fleshy entities; they are complex sensorial systems embedded in societal, community, and individual lived experiences, cultures and contexts, emotions, histories, etiquettes, and practices. Computers can exert control on our bodies explicitly (e.g., exoskeletons), but also implicitly, in less recognised ways. Research shows how a haptic baby sensing monitor changes parent-baby bodily interaction/communication practices: is this control? Socially oriented methodologies, such as socio-technical imaginaries/speculations, draw attention to these broader aspects of bodily engagement, to the body’s socially shaped and interpreted sensorial experiences, foregrounding dystopian/utopian discourses, changing practices, power relationships, privacy, and ethical implications.

My research focus involves developing interaction techniques for Extended Reality (XR) Applications. Building on my previous research on cross-reality interaction, we combine different input devices (e.g., tabletops, tablets, smartphones) with Extended Reality devices (e.g., Head-Mounted Displays), harnessing their complementary strengths. We refer to these as Complementary Interfaces (Zagermann et al., 2022) and utilise them for various Visual Computing scenarios (e.g., Immersive Analytics, Situated Analytics). In my talk, I introduced the vision of a “Best” Buddy, an AI- and XR-enhanced Human. “Best Buddy” consists of an ecosystem of devices and technologies that, while promising, also raise significant ethical concerns. The ecosystem comprises in-ear headphones or hearing aids, Augmented Reality (AR) contact lenses or AR glasses, and a smartphone. All devices are connected to a cloud service that uses AI technology (e.g., LLM like GPT-4o), leading to intertwined human-computer integration. In real-time, the LLM gets input from the lenses and in-ear headphones, empowering the human user with relevant information and giving hints on how to behave in specific situations. But who is in control? What are this vision’s potential Dark or Deceptive Patterns of an intertwined human-computer integration? How is “Best Buddy” positioned in the design space of intertwined systems (Butler, Angel, Influencer, Adversary)? [16]

HCI researchers have increasingly investigated bodily augmentations due to the associated benefits, such as accessibility, offering novel input space, and sensing environment. Most of these investigations have adopted a utilitarian perspective, providing impairment, mobility, and rehabilitation support. However, more recently, a subset called bodily extensions has emerged that appears to also embrace experiential aspects. These bodily extensions extend the human body beyond traditional sensing and accessibility perspectives. Initial designs were mostly rigid, focusing on short-term use, missing out on the opportunities for these bodily extensions to be integrated into everyday life. While the adoption of the same will become more prevalent owing to technological advances in the near future, there is limited knowledge on how to cater to controlling these bodily extensions with respect to the human body. In my work, I build soft pneumatic-based bodily extensions that users can incorporate into their everyday lives. I detail the design and rationale behind these bodily extensions, along with the novel scenarios they enable. Finally, I share my insights from creating these systems and the associated user experiences from our field study, hoping to better understand control over bodily extensions physically, physiologically, and psychologically [32, 31].

Haptics and physical interaction let us engage with digital systems in a variety of ways. Typically, this is structured as a system we use as a tool. However, increasing understanding of what haptic experiences are, new types of haptic devices, and capabilities for participatory design may influence our decisions about who controls what with our bodies. I will discuss how we can conceptualize haptic experiences, employ interaction modalities like implicit interaction in autonomous vehicles and intermittent interaction with fabrication devices, and then report on efforts for participatory design that give creators control over their haptic experiences and threats to bodily autonomy from touch.

Electrical muscle stimulation (EMS) has many potential applications in many fields, but the parameters necessary for EMS acceptance are poorly understood. We investigated a wide range of characteristics to gain a better understanding of EMS user acceptance. We created four scenarios on the basis of a literature review, assessed the technology acceptance of participants in an online survey with the help of the TAM model, and conducted in-depth interviews. Our results suggest that the use of EMS to enhance a VR experience is supported by a good attitude towards it and the intention to use it. We recommend that researchers and designers focus their efforts on keeping users in control of their activities rather than forcing them through EMS. We advise researchers to design systems that circumvent the involuntary nature of the system and allow the user to feel controlled. We found that the use of EMS to speed up learning was not widely accepted and that the desire to maintain control outweighed the desire to learn faster [34, 36, 35].

Galvanic Vestibular Stimulation (GVS) has been a known technique for over a century, yet its potential in Human-Computer Interaction (HCI) remains largely untapped and poorly understood. Despite its long history, there is a notable absence of systematic approaches to modeling, predicting, and calibrating GVS-based systems designed for end-user applications. This gap in methodology has limited the widespread adoption of GVS in HCI applications, despite its potential for reducing cybersickness in virtual reality (VR), enhancing balance rehabilitation, or developing novel experiences.

In parallel to GVS control of the body, I am also interested in the questions of control in situations where individuals experience a loss or lack of control over their physical self. These situations range from medical conditions like paralysis, neuromuscular diseases, or amputation, to the temporary disorientation or diminished bodily sensation experienced when transitioning from VR back to physical reality. This prompts a fundamental question: When there’s no sense of body, what does bodily control mean?

Understanding and addressing control in these contexts could help develop more accessible, inclusive, and empowering experiences. Both GVS applications and control-related challenges in HCI necessitate interdisciplinary research, combining insights from neuroscience, engineering, design, and HCI [37].