Designing the Human-Machine Symbiosis

Through the presentations above, a comprehensive list of “challenges” was collated by the time every participant had presented, providing a strong foundation for steering discussion during the remainder of the seminar toward topics that require further elaboration. It has been argued that HCI requires “grand challenges”, namely in that it provides a steering force to drive coordinated action in guiding future research, theory, design, and commercial development [3, 1, 2]. Acknowledging this need, these challenges were further consolidated into a set of concrete “Grand Challenges for Human-Machine Symbiosis”, with the intention their articulation would give guidance to researchers wishing to contribute to the development of the theory by providing specific and actionable gaps in knowledge or capability to which they can contribute through future research. Furthermore, the collation of recommended reading provided by each participant of the seminar now serves as a library curated by experts that may provide entrants into human-machine symbiosis research a foundational understanding of core concepts, theory, and knowledge that may guide their own future investigations, making the field more accessible to potential new contributors.

Following the completion of all participant presentations, a discussion took place in which the participants synthesized the concepts brought to light during the presentations, as well as the opportunities and challenges that were also highlighted. Through this discussion, it became evident that many challenges and opportunities pointed toward there being a significant need to articulate a clearer, more grounded, more comprehensive definition of what human-machine symbiosis is. This sentiment was further reinforced by many participants showing eagerness to draw knowledge from existing biological science explanations of symbiotic relationships, in conjunction with existing theories that deal with concepts like “self” (such as philosophy of mind) and describe “agents”, “agency”, and inter-agent interactions. With this considered, all 22 participants were broken into three breakout groups, with each group tasked with individually providing an articulation of what “human-machine symbiosis” is. Each group approached this in its own unique way, emerging from how its members organized to address the task of defining human-machine symbiosis, which resulted in it being articulated in terms of its dimensions, features, and underlying postulates, which each breakout group presented to the rest of the seminar in its entirety.

All participants then gathered to return to the main seminar room and a town hall discussion followed in which the participants attempted to integrate the three definitions into a unifying framework of human-machine symbiosis. A summary of the consensus suggested that central to symbiosis is a dynamic relationship between tightly coupled agents who are able to act independently on reality, yet assemble to exhibit emergent properties or capabilities that would otherwise not be possible with the constituent agents in isolation. These relationships would include at least one human and one machine agent but can include many of either. It was also agreed there could be a number of different types of these relationships that describe dynamics between the symbiotic agents. These types were drawn from biology and include: Mutualism (where both agents benefit); commensalism (where one agent benefits and one is unaffected); and parasitism (where one agent benefits at the expense of the other) however some suggested that it may be best to focus first on mutualism before exploring other types of relationships for the sake of simplicity. However, the attempt to define human-machine symbiosis also highlighted some theoretical challenges which were unable to be solved by the conclusion of the discussion, including where the line is between an algorithm and an agent (i.e. when does a program become an agent with agency?), where does symbiosis sit in relation to human-computer integration (i.e. are fusion and symbiosis still two subsets of integration?). Another ongoing question was whether symbiotic systems needed to be close to the human body, with the common concession being that on-body systems were not a requisite, but seen as more likely to form symbiotic relationships, or lend themselves to forming stronger symbiosis. Finally, the major themes and outcomes of this discussion were transcripted and translated into sections and subtitles of a future “Designing the human-machine symbiosis” paper, resulting in an initial skeleton of the manuscript, concluding the first day of the seminar.

The second day of the seminar was introduced by Ellen Yi-Luen Do, starting off with an “interactivity session” that involved live interactive demonstrations of systems and technologies relating to human-machine symbiosis. Several of the participants of the seminar had brought with them prototypes or technologies which were set up around the main seminar room. Other participants could then move about the room to try out systems and ask questions, with the intention that this hands-on demonstration of symbiosis-enabling technologies may inspire participants to ideate their own symbiosis systems.

Nathan Semertzidis demonstrated a novel electroencephalography (EEG) headset as well as transcranial electrical stimulation (tES) device, which his projects often employ in combination with a closed-loop to bilaterally sense and stimulate the brain. Participants were given the opportunity to experience a phosphine, a flash of light experienced in the visual field evoked by the stimulation of the optic nerve using the tES device. Nathan argued that these technologies employed in human-machine symbiosis systems could serve to provide a bridge between biological neural networks (the brain) and artificial neural networks (A.I.) opening up many possibilities for mental mergers with A.I. systems.

Nahoro Yamamura distributed to each participant a Lego set which, when assembled, formed a prosthetic extra finder participants could wear as a bodily extension. The finger was indented as an exercise to explore how the body might react to a prosthetic extension, and whether the brain would integrate it into its body schema. Some participants wore the finger for the entire duration of the seminar, reporting that sometimes it would be experienced as part of their body (for example describing that they got confused when they would move the rest of their fingers and the prosthetic would not move).

Zoe Xiao Fang demonstrated a lollipop that, when bitten, could deliver a tune to the brain through its bone-conductive lollipop stick. This demonstration highlighted the potential for food-related technologies to integrate with human physiology.

Andrea Bianchi demonstrated a collaborative microcontroller remote development toolkit. It was explained that the system was originally developed to facilitate the teaching of microcontroller development remotely during COVID-19-related lockdowns, however, it was hypothesized that such systems could also be used to teach and prototype symbiotic systems.

Jürgen Steimle demonstrated a software toolkit that enables the prototyping of bodily extensions and on-body interactions. Similarly, it was considered such toolkits could be used to rapidly prototype symbiotic systems.

Masahiko Inami demonstrated webcams that had been anthropomorphized as eyes, and speakers that had been anthropomorphized as mouths, which could both be fixed to any surface. This demonstration was intended to be a playful exploration of the embodiment and integration of inanimate objects.

Valdemar Danry demonstrated two AI conversational agents using large language models to duplicate existing people, allowing participants to talk with them. One was a duplication of Leonardo da Vinci, and another was a duplication of himself at a younger age. It was discussed that such AI systems could in the future form social symbiosis with their human counterparts, freeing cognitive demand by sharing tasks, appearing as one but distributing the workload.

Following on from the previous days defining of symbiosis and inspired by the interactivity session, the participants were then prompted to consider what types of symbiosis systems, areas of symbiosis theory, or application domains they would most like to further develop or contribute to. Participants then wrote their answers down on sticky notes, which were then posted on the main blackboard of the seminar town hall room. All participants then attempted to group these sticky notes in a self-organized manner until higher-level groupings emerged.

These high-level groupings were then reorganized into a set of application domains (applications human-machine symbiosis systems could be designed for). Participants were then instructed to form breakout groups with each group focusing on one application domain. Initially, these were 1) augmented body: sensory-motor 2), augmented cognition, 3) social symbiosis, 4) biological symbiosis, and 5) symbiotic play. However social symbiosis and biological symbiosis were merged into a single group due to having a low number of members in each. Each group was then tasked with ideating hypothetical examples of human-machine symbiosis systems designed for their given application domains. Each of the groups generated a great number of designs, that were then further refined into two to four designs that were explored more deeply. The groups then reconvened in the main seminar room and presented their designs to the rest of the seminar participants, concluding the day’s activities. The following presents a summary of what each group presented.

Group 1) “augmented body” ideated several systems that symbiotically augmented the body to enhance its capabilities in a variety of contexts. This included symbiosis for underwater living, which proposed engineering biological symbiotes that support human homeostasis as well as mobility in aquatic environments through technologies such as microbe-filled wearable membrane bioreactors, and “exoskins” which can sense and respond to properties of the water around it. Group 1 also ideated systems for “symbiotic eating” including microbes, smart materials, and nanorobots that can perform functions in the human mouth, stomach, skin, and on the food itself in order to modulate energy and nutrition intake, and alter the experience of eating such as taste.

Group 2) “augmented cognition” proposed several systems that work toward enhancing the capabilities of various domains of human cognition. One example was a system that augments memory processes by monitoring the brain for occurrences of error-related potentials or inability to recall information, reflexively rewinding through a log the system has kept of the day (e.g. through video or auditory recording) to find the information in its correct form. Similarly, the system could also employ predictive algorithms to allow the user to “recall” information from events yet to occur. The group also suggested that systems could capitalize on how affective states can alter memory encoding and recall to improve learning and facilitate desired forgetting.

Group 3) “biosocial symbiosis” was a combination of participants interested in investigating the social applications and consequences of symbiosis, and participants interested in symbiotic systems that have biological components. As such, the designs that came out of this group typically involved some kind of biological symbiotic system, whose use brought with it social consequences, both between the symbiont and the wearer, as well as between other members of society. These included: a second spine that gives the wearer courage through the direct injection of adrenaline which it produces but must be kept in a fish tank during non-use to avoid adrenal overdose; an implantable photosynthetic organism that gives the wearer the ability to photosynthesis but must constantly be managed else it will overgrow (and its use is also associated with a specific socioeconomic status); and a subdermal bioreactor that consumes excess nutrients from its wearer who can then sell the energy stored in the bioreactor for income.

Group 4) “symbiotic play” ideated examples that explored how symbiotic systems could be designed for games or playful experiences. This highlighted that symbiotic games and play can serve as a lens for safely identifying and exploring potential dystopic futures symbiotic systems may one day actualize. Some examples included a “prank” exoskeleton that forces its wearer to run or dance benefiting them through exercise; and swarms of interactive prosthetic “eye entities” that the user can see through, providing them with emergent abilities based on how they configure the eyes.

The third day of the seminar was initiated with a town hall discussion that reviewed the progress we had made so far in developing human-machine symbiosis theory and in the identification and articulation of its grand challenges and next steps, while also highlighting what conceptual areas required further development. Considering these points, participants then broke up into groups centered around projects, initiatives, and special interests that had accumulated on the “marketplace” whiteboard throughout the duration of the seminar. What each group did during this breakout session depended on the specific topic it was aligned with, with each group autonomously working toward its self-organized agenda. The topics covered by each group included: further developing human-machine symbiosis theory, with a specific focus on the different “levels” of symbiosis; further developing human-machine symbiosis theory, with a specific focus on the temporal, spatial, and structural properties and outcomes of symbiotic relationships; exploration of how human-machine symbiosis could facilitate deeper connections with nature, with a focus on writing a Dagstuhl seminar proposal around the topic; exploring the idea of human-machine symbiosis using biological materials, with a focus on ideating possible future research projects; and an exploration of the concept of “body hijacking” as an example of human-machine symbiosis, specifically referring to technological systems that allow one agent to take over the body of another. Each breakout group reconvened in the main seminar room and presented what they produced to the rest of the participants.

The seminar participants then embarked on a hike in Saarschleife along Mettlach, which lasted for most of the remainder of the day. The hike was intended to revive, refresh and inspire participants by breaking from the constant intensive discussion and conceptualization of the days prior, while also giving them an opportunity to exchange thoughts, ideas, and career advice, and to forge new research relationships and collaborations. Furthermore, during the hike participants saw that it was a good opportunity to keep an eye out for naturally occurring examples of symbiosis, which lead to participants drawing inspiration from examples that were encountered such as lichens, pollinator symbiosis, symbiosis between ants and plants, and the parasitism of vines growing on trees. The hike ended with drinks and dinner at a local restaurant, concluding the day.

The next day began with the initiation of a workshop led by Valdemar Danry and Pattie Maes, focusing on how recent advances in generative AI can be applied toward the development of AI-centered human-machine symbiosis relations (human-AI symbiosis). The workshop began with a presentation showcase demonstrating work the MIT Media lab’s fluid interfaces group has been doing to augment human cognition through offloading cognitive resources from the human to often wearable AI symbiotic systems, enabling the experience of enhanced memory, attention, and sensory abilities. Similarly, research conducted by others using generative AI was also showcased in order to demonstrate the widespread potential such technology may offer through symbiosis with a human partner (e.g. such as enhanced creative ability). Participants were then introduced to resources they could utilise to begin work on such systems themselves.

Following the presentation, participants then formed breakout groups around different applications that would benefit from human-AI symbiosis. The groups included: sensory augmentation and assistive tech; cognitive augmentation and assistive tech; motor augmentation and assistive tech; and play, games and sports. Each group ideated novel designs that exemplified how generative AI could be employed toward the ends of the specific application domain they were assigned with. Most groups generated a large number of designs, which were then refined to a single design that would be demonstrated to the rest of the participants after reconvening in the main seminar room. Most groups chose to present their designs and ideas in the form of a roleplay, usually with different participants acting out the role of the human user, the AI symbiont, and various components of the system.

A town hall discussion was then held focused on the writing of the “Designing the Human-Machine Symbiosis” manuscript. Participants further refined the outlining sections of the manuscript and identified other areas requiring further development the group could contribute toward. Based on these sections, breakout groups formed around the topics: defining human-machine symbiosis; methods and measures to study human-machine symbiosis; topologies of human-machine symbiosis; and the grand challenges facing human-machine symbiosis. Each of these groups contributed to the writing of the main manuscript by focusing on their aligned section. In addition, another breakout group was formed to draft a new Dagstuhl proposal for Human-Nature Interaction. Groups continued writing their sections until the completion of the working day.

The final day of the seminar began with closing remarks from the organizers. A final town hall discussion was held in which participants reiterated what we have established so far in building a theory of human-machine symbiosis and articulating a set of challenges that future research could address to make further contributions to the field. We also highlighted where there might be gaps in the concept, and what next steps could be taken to further crystalize the concept. It was also decided a slack group be formed to maintain contact between participants and facilitate ongoing discussion in the human-machine symbiosis research community. Following these closing remarks, participants spent the remainder of their time at Dagstuhl contributing to the “Designing the human-machine symbiosis” manuscript. As of writing this report, the manuscript is still currently in development.

Since the dawn of mankind, the history of the human race is reflected in the history of their tools and their usage. Many of these tools provide augmentation to our physical capabilities: power tools increase the body’s strength, bikes increase locomotion efficiency, and glasses and microscopes increase vision and the human ability to explore the world. However, more interestingly, tools also shape the way we think. It is known that “if all you have is a hammer, everything looks like a nail” (Maslow’s hammer), and to some extent, this is true for any type of tool, as they unconsciously reshape our perception of reality, our consciousness, and our understanding of how to interact with the world surrounding. In this short presentation, I show examples of digitally augmented physical tools that shape our perception of reality and give us new perspectives on how to design for supporting prototyping as an exploration activity, and virtual-physical interactions.

In biology, symbiosis sustains over long durations of time. With AI and reinforced learning, now we can see how software systems can co-evolve, somewhat similar to what Licklider envisioned in his visionary paper. However, how physical or tangible interfaces can co-evolve with humans over long periods is underexplored. I believe that computational and personal fabrication has a lot to offer in this context. Considering the slow development of hardware, we may need symbiotic software systems that model human behavior and can anticipate or predict future needs. These predictive or anticipatory symbiotic models will help us to create a seamless symbiotic adaptation of physical interfaces such as wearable devices, implantables, and other physical devices. Furthermore, the emerging domain of bio-fabrication could add a lot of value in creating co-evolving systems. In our work, we explore these directions as mechanisms to create co-evolving symbiotic physical interfaces.

We humans have had a desire to expand our own abilities through technology since ancient times. With the development of technology, autonomous machines that can achieve physical performance and information processing beyond human abilities have become possible. These autonomous machines have the potential to extend our abilities and fundamentally change our lives. However, it is not yet clear how machines can engage with the dynamic and high-intensity movements of humans, and how humans and machines can function as a unified entity. Due to differences in the characteristics and nature of motion capabilities between humans and machines, simply coupling the two is currently challenging. In this talk, I will introduce prototypes that explore how humans and machines can integrate and harmonize in dynamic motion. I will also briefly present the insights and challenges obtained from this exploration.

The miniaturisation of sensing, networking, and processing technologies has increasingly made information readily available. Taking this further, emerging Augmented Reality (AR) technologies and near-future holographic displays (such as light field and laser plasma displays) will soon allow rich visual information to be displayed anywhere, beyond the confines of small 2D screens. On one hand, these advances will allow relevant information to be more directly integrated with the activities, places or objects to which it is related. However, they will also bring significant challenges in designing useful and productive interfaces for visualising information and interacting with it. Given these coming developments, how can we leverage spatial interaction and situated information spaces to improve the way we perceive, interact with, and understand information? In this talk I will briefly introduce the motivations for my work on spatial interface design for data exploration and sensemaking.

Sensor device miniaturization and novel material breakthroughs have enabled hybrid skins as an emerging wearable form. These conformable skin interfaces are hybrid in their integration of technological function with existing cultural body art form factors and often involve social, design, and engineering challenges in their realization. These hybrid skins create a symbiosis with the human body through their direct skin contact, serving as one of the most intimate physical interactive interfaces in human-computer interaction. While engineering these hybrid skins has received increased interest in the past decade, open questions remain regarding the symbiosis relationship between these hybrid skins and the human wearer. Specifically, how do we design the agency and control of our interactions for, with, and by these hybrid skins? As hybrid skins evolve beyond the skin layer and even onto our bodily organs and cellular surfaces, what does it mean to be human in the age of symbiosis?

My interest lies in exploring material boundaries to understand conceptual relations of dualisms and how they intra-actively co-constitute each other (see Barad 1998, 2003, 2007, 2014). In my work with the skin as a feminist figuration, I am exploring different dermal characteristics such as processes of keratinisation, layeredness, and permeability, which allow me to reconfigure dualistic relations as non-oppositional. Being and knowing through the skin makes apparent how boundaries harden/keratinise or dissolve to determine the potentiality of affective mingling with other bodies (see Serres 1985/2008) – also, other “bodies of knowledge”. Their material-discursive touching might provide an ethical space for reconfiguring human-robot-relationalities. In my work I argue that the process of asymmetrical agential cutting (see Suchman 2006) is an embodied ethical practice, and that the figuration of the skin makes apparent how different ways of knowing are layered.

A way to Design the Human-Machine Symbiosis is through Fun with Creative Technology and Design. To facilitate cross-disciplinary research we need to provide the environments for creativity, a lab for making things [2]. We build physical and computational artifacts that are “objects” to think with, and this way of working helps us develop methods and tools to make better things. I argued earlier that the Design for Assistive Augmentation should take 3M’s into consideration: Mind, Might, and Magic [3], to have technology wonderfully blended in everyday life activities. Meanwhile, isn’t it time we discuss the art and science of the Human-Machine Symbiosis, and to investigate the engineering and design of such? We could be reflecting (art), understanding, and explaining (science), solving problems (engineering), and inventing and making (design). Let’s remember that the words design and program are remarkably close in their Greek and Latin roots. De (meaning out) and sign (meaning mark) and pro (meaning forward or out) and gram (meaning writing) both mean to mark out or make an explicit representation. Finally, let me quote Paul Graham here: What hackers and painters have in common is that they’re both makers. Along with composers, architects, and writers, what hackers and painters are trying to do is make good things [1].