Eat-IT: Towards Understanding Interactive Technology and Food

The seminar began with an embodied introduction round, where everyone introduced their name including a bodily gesture, repeated by everyone, in order to quickly learn everyone’s names.

Then, everyone gave their introductory talk, which participants had prepared beforehand. These talks were delivered in a PechaKucha format that was time-constrained to 5 minutes and favored visual material and aimed to present a personal account of participant’s experience with the coming together of interactive technology and food. The preparation instructions to participants asked them to present key readings on the topic of HFI that we also shared beforehand, so to have a common ground understanding of the existing literature. This collection of important writings in HFI can now serve as universal library as curated by experts in order to guide new entrants to the field (such as junior postgraduate students) that supervisions can guide them to, instead of curating such collections themselves again and again from scratch.

We had also asked participants to present their personal “Grand Challenges” they had encountered in their work so far. During the presentations, we asked everyone to use the whiteboard in the room to note down any Grand Challenges that they could identify, which we loosely grouped on the whiteboard based on the headings of “technology”, “users”, “society”, “design” and “other” inspired by the groupings previously identified in other “Grand Challenge” publications in HCI [1, 2, 3]. The “other” category, our wildcard, was considered to be either “theory”, “responsible design”, or “experience” at this stage.

The disadvantage of this format was that not everything that participants did around the topic could have been presented, however, the timing allowed to discuss all questions that arose, of which they were plenty. The associated discussions helped to identify additonal challenges that we added to the whiteboard as well as helped to refine existing ones. Discussions also helped clarify whether some of them can indeed be considered “grant” or whether they are important, but not “key” to be solved for the future of HFI.

We then formed teams in order to work on refining and grouping the challenges that were identified on the whiteboard. During a breakout group session, the teams were working individually to try to identify groupings and refining the Grand Challenges. Each team started off with one of the existing headings to see if this grouping does make sense as a starting point and applies to HFI. Results were documented and collated and then discussed together again with the entire group in order to resolve any instances where one Grand Challenge was put into different groups by the teams. Such multi-assignments were easily resolved, however, the groupings, in particular their namings, were heavily debated. Everyone agreed on the “technology” grouping, but the user grouping was relatively light in content in comparison. The society grouping was discussed in terms of whether it had the right name. Even more contentions was the “design” grouping, as it quickly turned into an “other” grouping where everything went that did not fit in anywhere else. The most debated grouping was the “other” grouping, as it was now discussed whether it should be called “sustainable design” or could be merged with the “design” grouping altogether. It was decided to resolve this the next day.

However, critical voices were also raised, questioning whether some of the Grand Challenges are not specific enough for HFI but rather apply for HCI projects more generally. Furthermore, it was questioned whether some of the proposed Grand Challenges are so big that they are outside the scope of HFI or cannot be solved by HFI, in particular those that were concerned with sustainability and associated supply chains. This was illustrated through examples where interactive technology could help address major issues around food, yet it is not a challenge for the field of HFI that hinders the field from flourishing, such as the use of interactive technology to optimize supply chains to avoid food getting off during transport due to inefficient routing: certainly an important area of concern where technology could be useful, however, not a challenge that hinders the development of the HFI field significantly.

An interactivity-style session involved participants trying out interactive systems concerning food. We had a 3D animated display that showed a person’s heart in a mobile format, which was connected to a heart rate sensor worn by the user. Such a system could display how a person’s interior body [11] responds to a particular food intake.

We also tried the biosensors used in the interactive system by Kunze et al. [4] that captured electrodermal activity (EDA) and heart rate by wrist-worn devices reflecting participants’ physiological reactions in real-time. This data is then wirelessly streamed over Wifi into a bespoke application that allows to pass it onto an OSC capable software, like touchdesigner, to visualize the data. This could be used to measure people’s responses during dining experiences even in large social groups, such as function settings like birthdays and weddings.

We also tried out sensor-equipped eyeglasses that can detect when a user touches their nose, when they blink and whether they look left or right. Most relevant, the captured data can also be used to detect chewing activity, which can be important when considering any eating behavior change-type of system.

Prior research showed that the weight distribution of a cup can result in a different flavor perception of the drink [5]. Based on this finding, Hirose and Inami [6] developed a system that allows to distribute the weight when a person drinks out of a cup: a motor shifts the weight in the drinking vessel contraption based on the angle of the cup. When the user is about to complete two-thirds of a drink, the more towards the top of the cup. User reports included comments such as that the apple juice that was used tasted more sweet after the weight distribution occurred. The system is still in early development, so several suggestions were made for further iterations, such as reducing the noise of the motor that might also influence the taste perception, and replace the linear motor actuation with a more complex algorithm that produces a more smooth weight distribution transition.

We also tried an “analog” food experiment where we ate different jelly beans while holding our nose in order to demonstrate the extent to which smell informs our overall taste experience.

Experiencing these systems first-hand not only generated further discussions around the opportunity of interactive technology to enhance or enrich the eating experience, it also inspired ideas of what other systems could be developed and how some of the Grand Challenges could be solved.

The interactivity session also brought to light the discussion of what the vision of HFI is. It was discussed whether the Grand Challenges themselves can help us identify a vision for the field, representing a bottom-up approach, or whether we should construct a vision first in order to guide the development of the Grand Challenges in a top-down fashion. The “use of interactive technology to enhance or enrich the food experience” was complemented by also considering the wider systems around it, like cooking or social interactions and deemed to be sufficient for now, although it was considered to be more of a mission rather than a vision.

In order to refine the Grand Challenges, we conducted a peer-review session in which the different teams from the previous grouping exercise peer-reviewed the work from one other group, respectively. They were also encouraged to comment, add and edit any content that was shared in an online document as they saw fit. This resulted in a more structured set of grant challenges that were more conform in the way they were articulated, bringing them more “in line” with one another. This activity helped to refine, in particular, the “technology”, “user” and “design” groupings, however, the “society” and now called “meta” groupings were not making much progress and it was decided that they needed further work.

Based on the prior activities, participants then engaged in a speculation exercise in order to refine the Grand Challenges through imagining what the field of HFI would look like in 2052 – a 30 years timeframe – if the Grand Challenges would not be solved. For this, the speculative design approach [7] was used to collectively reimagine possible, probable, plausible, and preferable futures. We encourage participants to also consider sustainable and equitable futures using a “flashlight” visualization (Figure 1).

Based on this, new team formations were asked to speculate about direct and indirect futures if a particular challenge (based on their own choosing) will not have been addressed. Additionally, participants were instructed to imagine what a newspaper article would look like that responds to this at that point in the future, which was supplemented by a speculative Tweet/news item. Together, these speculative media snippets were encouraged to develop as a way to help thinking more concretely about the Grand Challenges and their implications, cementing the idea that they are indeed “grand”, that is, key for the positive development of the field as a whole.

The resulting speculations were very useful in helping to cement the idea that the Grand Challenges we identified can indeed be considered to be key for the future of the field. In particular, the exercise helped to underline for everyone how important the work on the Grand Challenges is to help the HFI field as a whole.

In addition, the session also sparked additional discussions not previously had around what are desirable food futures and how HFI could help with that. Maybe frameworks from other, but related areas, could help with that? For example, the “SPRUCE” framework [8], developed to create a desirable future for autonomous vehicles through providing a practically-oriented structure, was considered as exemplar to what HFI could benefit from. The “SPRUCE” framework asks for “safe, predictable, reasonable, uniform, comfortable, and explainable” autonomous vehicles, raising the thought that HFI should similarly aim for “safe” (as in food safe), “healthy-in the long run” (facilitating healthy eating), individuality-appreciating (respond to the highly individual nature of food experiences, see especially the need to consider dietary choices and food allergies), local (responding to the highly local aspect of where food grows and is produced), time-sensitive (responding to the fact that food generally has a very limited shelf-life, especially when compared to other material HCI is concerned with, like plastic in personal fabrication HCI), and social (responding to the highly social nature of HFI experiences that produced its own term: commensality [9]).

Speculating about the future revealed the complexity about the topic but also brought to the fore the responsibilities we have to consider when designing future HFIs. Inspired by the SPRUCE framework, the team embraced responsible design principles, developing a dedicated SPROUT framework and vision for HFI to promote safe, personalized, responsible, original, uniform, and transformational interactive food futures.

We conducted another peer-review session in which we critically examined the refined Grand Challenges as expressed in the previous round. In particular, with this session, the goal was to arrive at a more textual representation of a Grand Challenge. For this, a template was provided based on prior structures as expressed in previous Grand Challenge publications in HCI [1, 2, 3]. This structure was provided in order to ensure that each group does not forget about particular aspects that need to be fleshed out. For example, every Grand Challenge should begin with a definition or explanation that ensures that the reader has a clear understanding of what the challenge is. The structure also aimed to remind authors to explain why the challenge is “grand” enough to find its way into the document. Furthermore, the structure asked authors to consider and express what happens if the Grand Challenge will be resolved and what happens if not. The results from this session were again documented in an online document and shared with the group that elicited feedback that helped refine the Grand Challenges.

Now that participants received feedback on their individual work both within their own group, from other groups, and the entire cohort, it was time to start dedicated writing sessions where participants worked in small teams on fleshing out the text as it will go into the article for each Grand Challenge. Participants were encouraged to look back at the structure provided above again in order to follow a common structure when expressing a Grand Challenge. In particular, participants were encouraged to work on a shared document that was provided by the organizers so that everyone could see everyone else’s work on the Grand Challenge while it was emerging, giving participants the opportunity to quickly ask questions as they emerged or avoid duplication of points being made. For this, it was useful that the Dagstuhl location had enough individual rooms and breakout spaces that allowed concentrated team work, yet these rooms and breakout spaces were very close-by so that participants could quickly stick their head into another space to ask any clarifying questions.

We also organized a hike through the local landscape in order to nurture not just the mind, but also the body, making use of the tight interlink between a healthy body and a healthy mind. We instructed participants to consider using the opportunity of walking next to each other, in contrast to sitting opposite each other, facing each other, like in a meeting scenario, especially over videoconferencing, to have different kinds of topics of conversations. In particular, we suggested to discuss each other’s careers and, speaking to the wide range of career spans represented in our participant list, provide career advice and also ask for career advice. This could make use of the opportunity to connect with people outside the topic of the seminar, yet relevant to their academic or industry career.

In order to illustrate the dangers that lurk if the Grand Challenges will not be solved, a session was conducted in which participants were encouraged to develop personas. The teams were encouraged to present the personas they developed to the other teams in an engaging and entertaining way, going beyond presentation slides. Most teams decided to role-play the scenarios around their personas, speaking to the embodied nature of food. For example, one team conducted a play around a single mother who relies too much on her cooking robot, while another team presented food made out of play-do that could nourish people at the most optimal level but requires social approval in order to offer access. These personas not only offered an entertaining perspective on the Grand Challenges, but also highlighted that solving them is not only an abstract exercise but affects real people.

Having engaged more acutely with the consequences if not solving the Grand Challenges, the participants were encouraged to further refine the Grand Challenges text document. In particular, the idea was to do thinking through writing, in which participants, in teams, collaboratively develop the text for each Grand Challenge and peer review it while it is emerging. This “thinking through writing” session resulted in several refinements of particular Grand Challenges. It also identified the need to spend more time on the categorization labels, as it was identified that they were too generic and not particular to HFI. To address this, a different visualization for the presentation of the groupings of the Grand Challenges was developed, based on prior work that also preferred an illustration [3] over a table in order to present the results [2]. With this new illustration, a new editing session was started where participants were instructed to further refine the descriptions of the Grand Challenges while also considering which ones are key for the article and which ones might need to go for brevity purposes. As part of the thinking through writing process, one group decided to develop the vision section of the article further, building on the SPRUCE framework that has by now become the SPROUT framework that aims to guide HFI practitioners new to the field but are unsure what has already been done and what one needs to consider if wanting to conduct research.

The final activity involved collating all the work that has been done so far and discussing it in a townhall-style environment. This was done to ensure nothing important had been missed while trying to attempt to give everyone an equal voice across the different Grand Challenges. Reflection across the entire group was then facilitated through the presentation of three final questions everyone should think about and then were free to articulate, which were suggested to start with “I like …”, “I wish …”, and “I wonder …”. These questions were aimed to facilitate reflection on what was achieved and could be improved upon for further seminars like this.

Human-Food Interaction is a completely novel area to me. Coming from a background in experimental psychology and computer science, I am interested in applications of Virtual Reality and user experience. In my recent transition to multi-sensory experience research, our team have been investigating how factors such as food shape and ambient lighting may affect taste experiences in Virtual Reality. As I near the end of my PhD, I am looking to explore new potential research areas such as HFI. Specifically, I am interested in how we can integrate senses other than vision and sound into VR, such as taste experiences.

Food is a social fact: commensality, is a far more complex concept than just sitting together at the dining table. It has been linked to improved health and well-being, food enjoyment, and positive affect among co-diner. Commensality is a multifaceted phenomenon that requires a multidisciplinary approach to be fully understood and thoroughly investigated. Computational Commensality is an approach that grounds on affective computing and social signal processing, bringing together models and theories from social and cognitive sciences with techniques and methods used in Artificial Intelligence and Human-Computer Interaction. The talk will start with the social side of eating to then illustrate a study investigating remote eating experiences through the lenses of Computational Commensality. The results of the study can help shed more light on how technology can foster commensality and how Computational Commensality could be leveraged to better understand commensal, and hence social, interactions.

Ferran Altarriba Bertran is an interaction designer and researcher whose work explores the design of technologies and experiences that add an element of playfulness to people’s mundane activities. In this talk, Ferran will discuss some of his recent work in the space of Playful Human-Food Interaction research. His presentation will discuss the intersection between play, food, and technology from three angles: conceptual, i.e. why are play and playfulness needed in our interactions with and around food?; methodological, i.e. how do we design play-food experiences that are contextually meaningful?; and design-oriented, i.e. what are specific actions designers can take to playfully reframe our food systems and practices? Overall, Ferran’s talk will provoke designers and researchers to embrace fun and joy as foundational values in the design of food-related technology, and will provide inspirational starting points to facilitate that move.

To date the coming together of food and technology is mostly one similar where computers and the human body were at the beginning of wearable computing: computers were initially developed to be lighter so that they could be worn. Only more recently we realized that there are more affordances besides weight that need to be considered (such as always-available, context-aware, always-on, etc.) as they allow to meaningfully integrate the human body and the computer into an assemblage. In this talk, I ask the question what the equivalent is for computer-food integration by drawing on recent experiments in our lab on the coming together of food and technology that also considers experiential, not just instrumental, perspectives.

SMARTMOBILITY is part of the interdisciplinary project SMARTACT. The main aim of SMARTACT is to develop and empirically test the efficacy of a toolbox for mobile, real-time interventions targeting normal eating and physical activity using mobile technology (smartphones, body monitoring). The SMARTACT toolbox encompasses different mobile and in-person intervention tools based on “what people do” (behavioral pattern), “why people do what they do” (psychosocial and contextual triggers of behavior), and “when people do what they do” (timing of behavior and triggers). A mobile app – the wellness diary – combines several tools to promote physical activity and normal eating: The physical activity tracking tool keeps track of users’ physical activities (e.g., step count), the food journal lets users document their food intake, the questionnaire tool gathers eating motives, and the feedback tool provides the user interactive multidimensional visualization of gathered data. The toolbox provides high quality and depth of collected data. A user-centered design approach minimizes the burden of manual data entry and maximizes ease of use. For the researcher, an export tool provides export options for data stored on the server. This data can then be used for collaborative immersive visual data analysis.

Contemporary human-food interaction design is predominantly a technology-driven endeavor, which has highlighted the functionality and novelty of computing technology. Such a technology-centric approach might outweigh the exploration of inherent affordances of food, such as the food’s material properties emphasizing its aesthetic, affective, sensual, and sociocultural qualities. Here, I introduce a new approach to food-computation integration that employs food as a primary material to realize computation. I present a “Research through Design” exploration of designing food as computational artifact through a case study called the “Logic Bonbon”, which is a liquid-centered dessert that can regulate its own flavor and visual presentation. Through the design-led exploration, I hope to unpack how computational qualities of food could be leveraged in the development of novel human-food interactions and shape the future of food innovation.

Designers and makers are increasingly interested in leveraging bio-based and bio-degradable ‘do-it-yourself’ (DIY) materials for sustainable prototyping. Self-produced bioplastics possess compelling properties such as self-adhesion but have so far not been functionalized to create soft interactive devices, due to a lack of DIY techniques for the fabrication of functional electronic circuits and sensors. I present a DIY approach for creating Interactive Bioplastics that is accessible to a wide audience, making use of easy-to-obtain bio-based raw materials and familiar tools. It enables additive and subtractive fabrication of soft circuits and sensors. Our biomaterials possess attractive properties for edible interfaces. I present an application case that realizes functional capacitive touch sensors which can be embedded within food.

One important characteristic of the human mind is that it has significant fluctuations in productivity and capacity. Our mind has ebb and flow, and is affected by various factors, some of which we do not even realize. The types of food we eat has a large impact on the state of our mind (cognitive performance, sleepiness, alertness etc.). These cognitive fluctuations manifest in patterns in human behavior and physiological signals (body temperature, eye movements, galvanic skin response etc.) related to the Autonomous Nervous System and can be captured with unobtrusive sensors embedded in glasses, granments, clothes etc.I want to develop tools that enable people to be aware on the impact of food intake on their everyday cognitive functioning and to live more healthily.

We recognise the rich potential of the human senses (vision, hearing, touch, taste and smell) when designing human-food interactions. This talk will engage the audience in a reflection on how to design multisensory food experiences driven by key questions, namely, the why (the rationale/reason), what (the impression), when (the event), how (the sensory elements), who (the someone), and whom (the receiver), associated with a given multisensory experience design, exemplified through food experiences. In other words, we need to think of the context of eating (e.g. home, restaurant, other spaces), the technology we use/design (e.g. augmented cutlery/utensils, VR), and the user and the desired experience. All that is further embedded in relevant responsible innovation framework, asking uncomfortable questions, anticipating unintended consequences and possible negative but also positive outcomes for future human-food interaction designs.

User experience design should always aim to engage users on different levels of perception, emotion, and cognition. Vision and sound are still the predominant interaction channels, mainly because they can be used to transmit large amounts of information in a short time, also digitally. Meanwhile, haptic/tactile cues have been established as the third modality, e. g. to generate more immersive entertainment scenarios. For future interactive experiences, though, and to evoke even deeper emotions, it will be crucial to include smell and taste sensations, too. Possible use cases are increased realism and sense of presence in virtual environments or shared traveling and eating. By means of multisensory integration and interdisciplinary research, we are following Sony’s purpose which is to “fill the world with emotion, through the power of creativity and technology.”

How could it be possible to “synthesize” the experience of commensal eating in a HCI setting? That is one of the questions we recently started to investigate through a new type of interface that we called Artificial Commensal Companion (ACC). ACCs could bring the benefits of commensal eating to, for example, people who voluntarily choose or are constrained to eat alone (e.g., in a hospital or rural setting, or during sanitary lockdown). In the presentation, we introduce an interactive system implementing an ACC in the form of a robot with non-verbal socio-affective capabilities. Future tests are already planned to evaluate its influence on the eating experience of human participants.

Human perception of the world occurs through several interaction channels which provide both factologic information about the individual’s surrounding and affect the behaviour and emotional state of the human in particular context. Taste and flavour sensations are deeply encoded in our brain and as such contribute to a large extent to emotions, memory and cognition, and to the user experience in the physical world. We are interested in designing possibilities to transfer and even extend experience related to taste and flavor to the virtual space. This transfer requires solving the fundamental problem of de- and re-materialization of the chemical interactions governing taste and flavor perception from real into the cyber space.

Our interaction with microbes is crucial and dictates not just a momentary gastronomic gratification but also sets the stage for future generations’ gastronomic delight and their ability to metabolise food. To secure these experiences for future generations, understanding how to nurture gut microbial diversity is important. The biggest challenge though is not just the translation of this rapidly advancing science, or its multifaceted influence on gut microbial diversity, but also its transformation into novel experiences to generate public understanding. For this, we propose the design of interactive play-based technologies to develop alternative framings that foster understanding and reflection on human-microbial interaction. Prior works emphasised the importance and the need for more-than-human perspectives for sustainable food futures. We contribute to this through the design and development of interactive playful tools. Our learnings and strategies can aid future playful design explorations towards nurturing human-microbial relationships to promote sustainable food behaviours.

After long focusing on food as an object, taking on its shape, materials, naming, packaging and rituals of use, food design has opened the door to the design of enhanced food interaction that combines the embodied sensory stimulation of food with multi-sensory or cross-modal interactions possibilities offered by technology. I’m interested in exploring the possibilities of food as a design material that can afford cultural and transformations, and experimenting with prototypes for instantiating a more general theory of interaction which can inform the design of technology enhanced experiences. Through designing for human-food interaction we necessarily need to consider all senses, how they are solicited by food, the subtleties which make the experience of eating memorable and worthy of sharing with others. Food offers us the possibility to be engaged somatically, enabling us to connect feeling, thinking, movement, and expression into one subjectivity. Learning how to acquire a somatic and experiential sensibility can help designing richer and more engaging future technologies.

The field of Human-Food Interaction (HFI) has garnered considerable currency in recent years in the field of HCI with researchers exploring how technologies can be designed to support different aspects of food practices. However, the majority of the exploration is still centered around digital or is computer/arduino-mediated, here we see an opportunity to look beyond the existing technologies and rethink food as the technology itself. In this talk, I will cover some of HAFP Research Lab’s works that address this theme.

Food is a form of media that allows us to experience creativity, pleasure, connectedness, trend-seeking, and relaxation. Eaters, as users, are motivated to consume food by many different factors: community, familiarity, convenience, culture, hedonism, functionality, health, morality, novelty, and more. Through the Design Thinking, Design Doing workshop, we aim to speculate on the consequences that may arise from the implementation of food technologies in the future. By considering the broader future of food technology, we aim to provoke discussion about potential utopian outcomes and the considerations required for the field of Human-Food Interaction in the present.

Prior human-computer interaction research shows that overlaying sounds to eating can change how people experience food, affecting how and what we eat. However, how to design such gustosonic experiences – referring to multisensory interactions between sounds and the act of eating/drinking – is not well understood. In this talk, I introduce my three gustosonic systems to arrive at the understanding of the design of playful gustosonic experiences, in particular the design of “sonic straws” which is straws that can play personalized notes. This work advances interaction design theory by contributing to the enrichment of eating/drinking through playful design, furthering the future of the food experience.