Quality of Sustainable Experience (QoSE)

Sky-rocketing energy prices have increased our awareness of resource limitations. Having worked with quality-versus-energy tradeoffs since 2010, the emerging multi-reality digiphysical experiences make me curious of their potential to reduce environmental footprints without sacrificing the essentials of the experiences. Bringing together the “Research through Design” and “Quality of Experience (QoE) by Design” [1] principles, I see a great potential to create beyond-expectation immersive experiences with sustainability in mind, for instance Extended Reality (XR) telemeetings. Thereby, creative design of experimental artefacts based on fundamental relationships between QoE and provisioning, measurements and modelling efforts will pave the way towards optimised quality-versus-energy performance, expressed for instance through measures such as “QoE per Watt” (QoEW) or “QoE per Joule” (QoEJ) [2] – or as “QoE per kWh” (QoEkWh) that relates directly to the energy bill.

Nowadays, research targeting the energy efficient use of video communication technology is an important research topic. In this respect, our team focuses on the energy consumption of two important aspects in a video communication pipeline: First, the generation, compression, and provisioning of videos, second, the consumption of videos on end-user devices. Methodologically, we usually start by measuring the energy consumption of a video system, then analyze the behavior with respect to parameters such as hardware, software, and video properties, and come up with numerical models that are further exploited to reduce the energy consumption. We noticed that next to the energy efficiency of distinct devices, the overall energy consumption of video services draws more and more attention in academia and industry. Hence, in this seminar, interesting challenges are to jointly optimize the energy consumption of distinct devices and a complete video service while keeping a high QoE for the end user.

Many processes in work environments (including the prominent publication mode in Computer Science with in-person gatherings to present research output) have relied on in-person meetings, which often require travel. For many researchers, traveling to conferences may well be a significant, or even largest, contributor to their annual carbon footprint. To be sustainable, alternative processes – such as virtual and hybrid attendance modes – need to be established. To be successful, these must meet the goals of the gathering and provide a high experience (QoE). How to make virtual and hybrid meetings enjoyable and therefore sustainable is a question directly related to QoE research. To address this, I have studied the QoE of virtual conference attendance via a survey approach that identified areas in which this mode works and also exposes its limits. As a future trend, hybrid conferences are having their moment, primarily due to the prolonged and open-ended transition period from the COVID-19 pandemic. While hybrid conference also address rising concerns relating to the carbon footprint of air travel. Further, they promote inclusiveness of members of the community, e.g., those that are not able to attend due to family obligations, budget restrictions, difficulties obtaining a visa or disability. Yet, it remains unclear of how to design hybrid conferences well to achieve a high participant QoE, which will be an upcoming challenge to the QoE community. This imposes a direct question to this seminar on how to make work processes – such as hybrid attendance – enjoyable and thus sustainable by means of QoE research.

Quality of Sustainable Experience raises several research questions which are addressing the different pillars of sustainability: human, social, economic, environmental. In particular, environmental sustainability calls for the following: What is the trade-off between QoE and CO₂ emission? How can an optimal operational point be derived in practice? Is the ratio of goodness, e.g. QoE, and badness, e.g. CO₂ emissions, e.g. energy consumption, a meaningful key value indicator (KVI) for today’s and 6G networks? This ratio goodness to badness is Kleinrock’s power metric from queueing theory. How much reduction in CO₂ emission can be achieved by a green user as compared to a high-quality user? How much reduction in CO₂ emission can be achieved by moving towards a green network? What is the relative impact on the reduction of CO₂ emissions of green user behavior as compared to green networking? Is it more relevant to focus (i) on green user behavior and empowering green user behavior or (ii) on green networking technology today and in the future in year 2030? What are the implications of solution approaches on the networking and communications technology?

I work mainly with classical video quality. Right now, I am developing a Virtual Reality Laboratory. So I expect to work more with VR/AR in the context of 5G. My main focus right now is the development of ecologically valid experiments. Ecologically valid experiments are closer to the real-life scenario. We expect that such experiments can reveal situations where quality is less important than concluded from a classical lab study. It gives an option for further optimization of network resources, limiting energy consumption. Further, quality should not be the only goal and the trade between quality and the resources used should be better understood. An important component is not only the network, but also the habits of the users, like playing music from video, not even watching. A different essential aspect of the quality system is the development of algorithms for recompression from clear energy. We have to understand quality and user behavior outside the laboratory. Only then can specific solutions be proposed.

It was good to see young academicians from so many countries, concerned about the society and the world. On the one hand, they worry that climate change could do a irreparable damage to our earth in coming years. On the other hand, most of them felt somewhat powerless, as big Governments and big industry seem to drive every aspect of life on planet. The youngsters work very hard to just have a decent life. They seem to be powerless in the current situation. Recognising this, the seminar attempted to do two things. The first was the little actions that they could carry out individually and in groups to start making some difference. The second was to dream of a future society – may be 100 years from now. What would be the norms and ways such that the people would really be empowered, free from the control of big governments and big industry. The participants knew that it was a mere beginning, but felt that even imaging a future society would be the first step to move towards such society in future.

One of my current research foci is multisensory user experience (MUX) in extended reality (XR). Given the immersive and presence experience enabled by XR, the number of XR-based applications is ever-increasing, especially for social interaction such as in games, therapy, and training. Avatars representing interactants are typically used. Non-verbal sensory signals (i.e. facial expression, gesture, gait) are essential for emotion portrayal. The MUX of social XR is determined by the extent to which intended emotions can be conveyed and recognized by the interactants. As XR technologies are highly energy-demanding, they can have a very negative impact on sustainability. The higher the avatar fidelity is, the higher the MUX quality can be, but the higher the energy consumed and costs. To address how to improve the greenability of XR, I am investigating how the avatar fidelity can be minimised but without compromising the perception and recognition of the intended emotions required for successful social interaction and quality MUX. Extensive user-based studies are designed to identify the minimum fidelity level for each type of sensory signal per emotion in a range of contexts. Advanced rendering techniques and machine learning models for adjusting avatar fidelity will be deployed. How MUX varies with different avatar fidelity levels will be evaluated. Overall, the main challenges are to scope the large problem space, considering the nature of emotion and the rapid growth of XR tech and techniques.

My work explores designing tools for participatory (re-)imagining and futuring in an age of transitions (and crises) in climate, energy, health, and social inequalities. How we experience the world (interacting with technology, but also how we encounter societal and infrastructural systems) affects how we imagine, understand, live, and what we see as possible in our collective futures, with consequences for sustainability. Design has an important role to play in engaging with imagination and futures, and the urgency of climate crisis makes this acute: enabling people to share their experiences with others, giving voice to underrepresented experiences, and turning ideas into prototypes (including interfaces) which can be experienced, used, lived with, and reflected upon. Designers can bring plural possible futures to life, in the present. I see the qualities of how we experience the systems around us as important in building more sustainable ways of thinking and acting – better connections to impacts, consequences, and each other.

While I integrate research, teaching and engagement like any good faculty member, I am a hacker of universities and find that a lot of my attention goes to less traditional work like community creation, demonstrating different ways to do things and building new institutions. I lead fellowships, research groups and programs for troublemaking practitioners and scholars, host a podcast on popular culture and social change, create welcoming and non-hierarchical environments and try to engage respectfully and generatively with the community and the world. After decades spent on the interaction of information and communications technology with organizational and institutional change, i have spent recent years more focused on sustainability, environmental justice and climate change, finding significant overlap and complementarities. A large part of my current attention is on reorienting the functions of my university around sustainability, where I focus primarily on research and engagement (and leadership, of course). I’ve learned that seeking a sustainability orientation echoes the challenges of the digital, diversity and other transformations before it, requiring not just modest changes to what we do or who we hire, but fundamental shifts for both individuals and organizations in how we see ourselves, our practices and our mission. It confronts identity, asks that we engage emotional complexities, requires us to engage our imagination, create different systems, communicate differently and address other seemingly distant considerations. I’m excited to hear what others are thinking about, experimenting with and learning as we navigate the unseen and deeper barriers that will begin to allow the sort of transformational developments that facilitate not just human survival, but thriving.

We investigate perception and experience for traditional and immersive audiovisual media, including video (e.g., high-resolution, high dynamic range), spatial audio, and technology for Augmented, Virtual and Mixed Reality (AR/VR/MR). Two types of “resources” may be considered in terms of sustainability: (1) Human mental and physical resources, for example measuring fatigue for telemeetings versus face-to-face, or the positive impact on wellbeing with mediated social presence. (2) Energy and natural resources consumed along the end-to-end chain (e.g., by one media system implementation versus another), or resources saved (e.g., meeting via videoconferencing or MR rather than travelling). In this context the question arises, how “sustainable experiences” can best be characterized, and the result be applied towards a more sustainable way of life. A holistic and collaborative approach is needed to achieve this. Here, I see the QoSE seminar as a possible crystallization point for sharpening the participants’ views and future collaborative work.

Over the years, QoE turned out to be very successful in redirecting the attention of the communication networks community towards the user. However, the current multiple crises indicate that we have to extend our perspective on both sides, leading to two key questions: (1) Do we have the technology we need, and do we need the technology we have? (2) Which world are we currently building? Recently, several new movements formed to address these issues, especially in the context of the “Vienna Manifesto on Digital Humanism” or the “Rat für Digitale ökologie Berlin”. That leads to question number (3): What can QoE research learn from, and how can QoE research contribute to this broader perspective on the Digital Change?

Rural areas are characterized by scattered settlements, lack or limited Information and Communication Technologies (ICT), hence they are disadvantaged in accessing e-learning resources. Different technologies have been proposed to address broadband connectivity for e-learning in rural and remote areas, such as the 3G UMTS operating in the 900MHz frequency band, and the Television White Spaces (TVWS), due to their wider coverage and capability to offer broadband connectivity. However, further research revealed that broadband networks configured in the best-effort approach cannot deliver video streaming with the required QoS for e-learning, which implies that users will end up unsatisfied, hence poor quality of experience. Trends in e-learning shows development of MoodleBox, which is a standalone mobile device that can provide both local broadband connectivity and e-learning resources. Potential research area here is to evaluate performance of MoodleBox in delivering multimedia e-learning contents in rural settings towards sustainable quality of experience in e-learning.

We work in the field of digital signal processing with a focus on the processing of audio and medical signals. Many applications and services use signal processing to extract information, for signal enhancement, or for transformation into other representations. While traditionally, model-based techniques played a much more prominent role, the employment of data-driven methods (machine learning, artificial intelligence) has increased significantly in recent years. This enabled significant breakthroughs, for instance, in speech recognition. However, in many cases, at the cost of increased resource consumption and corrupted privacy. While some of the current and upcoming technical possibilities are of great use to society, their employment is often discussed on an economic level only, and sustainability plays a minor role. I want to discuss how we can assess the benefit of new applications and services more holistically, including society, sustainability, and economics.

Innovation experiences result from the (often unconscious) comparison of needs and offer and lead to emotion, cognitions and actions. For a convincing innovation experience, the following areas need to be actively “managed”:

• $\mathrm{■}$

  Utility: Is the innovation addressing “real” human problems and needs which are relevant to the target customers? (exemplary tool: jobs-to-be-done approach).

• $\mathrm{■}$

  User Experience: Is the innovation providing a convincing experience in the product and usage context? Is the user able to use the innovation easily and does the user like the way the innovation looks and feels? (exemplary tool: iterative UX prototyping & testing).

• $\mathrm{■}$

  Customer Experience: Is the innovation providing a convincing experience in the broader market context? For example, does the innovation create positive moments of truth and emotional binding along the whole customer journey? (exemplary tool: customer experience blueprinting).

• $\mathrm{■}$

  Transforming innovation experiences into more sustainable ones is obviously a very challenging task in our days. From a management perspective, exemplary questions are: How can innovators create convincing sustainable innovation experiences? How should innovators deal with the current “more, better, higher” consumption mantra? Which tools and methods are helpful to create sustainable innovation experiences (e.g. sustainable business model design patterns)? etc.

Researching in the field of sustainable product development and supporting manufacturing companies, we have identified that companies struggle to integrate sustainability in their processes. In Blekinge Institute of Technology, we have developed and tested tools and methods to guide companies to adopt a strategic sustainability perspective based on the Framework for Strategic Sustainable Development, by considering a holistic view, the environmental, economic and social dimensions of sustainability, the assessment of the complete product lifecycle, stakeholders’ collaboration and a long-term perspective. Finding that the sustainable society transformation requires the participation and support of many fields, where digitalization can be used to support this transformation. Moreover, digitalization might facilitate manufacturing processes and the usability of decision-support tools to develop solutions with a higher sustainability performance. Therefore, it is essential to discuss how digitalization can support the implementation of sustainability, considering trade-offs and additional requirements e.g., knowledge, infrastructure, management, social interaction and circularity.

Remote telepresence is essential to enable connection among users at a distance, facilitating communication and collaboration, while decreasing the amount of travelling and commuting required; as such, it has become a key point in research agendas both in the European and national level to create sustainable travel habits and more liveable cities. Current telepresence solutions for telepresence have been shown to create exhaustion and fatigue, due to the unnatural way in which communication takes place, such as limited mobility and close-distance eye gaze. Extended Reality (XR) telecommunication systems promise to overcome the limitations of current real-time teleconferencing systems, enabling a better sense of immersion, enhancing the sense of presence and fostering more natural interpersonal interactions. To achieve their goals, they need to be designed keeping the user as the central perspective. How can we optimize the quality of such systems, such that they can maximise the Quality of Experience for the user, while ensuring the sustainability of their operating principles? How can we incorporate Quality of Sustainable Experience in the design, implementation and evaluation of such systems?

I work with experiences that drive ways in which people’s activities in the world can be more sustainable, both socially and environmentally. Recently, together with my team we have worked a lot with AI-driven systems such as social robots and the ways in which they could motivate/persuade people act more sustainably. (At the same time realising that hardware robots may not be very sustainable in themselves.) Methodologically, we employ human-centered design thinking, and especially co-design and co-creation approaches, both in-situ and (when needed, e.g. due to pandemic) online. We also work with industry to help them adopt Human-Centered AI (HCAI) design approaches. It is timely and relevant for sustainable experience design to move beyond human-centeredness, towards what has been labeled as “post-human”, “more-than-human” or “planetary” design by various authors. While these concepts are attractive, they are currently still quite abstract and philosophical in terms of how to apply them in practical product and service design. Furthermore, qualities of AI – proactivity, dynamism and autonomy – introduce new possibilities to the system design process. Hence, we need to define practices for integrating the needs of humans, the ecosystem and AI to advance sustainability through experience design. These practices have to take into account the needs of the planet, not just of humans.

Sustainability has been a crucial demand for all generations of wireless communications systems in terms of optimal resource allocation subject to given key performance indicators. Due to the required high data rates, low latency, signal processing complexity and other constraints, maintaining QoE and sustainable QoE is a challenging task in ultra-reliable low latency communication applications such as mobile immersive media. 6G technology shall support immersive mobile media experiences that extend over the entire continuum of digital computer-generated virtual worlds. A key emphasis in the growth of digital value platforms will be the convergence of multimodal engagement with media and the physicality of lived experience. In this context, architectures and technologies for green 6G networks shall be envisaged that offer sustainable QoE. In my current work, I conduct subjective experiments for mobile immersive media, subjective and objective quality assessment, mobile multimedia signal processing, analytical approaches on QoE-assured VR video streaming, energy harvesting in wireless networks. I am interested in discussing experimental designs for sustainable QoE, subjective and objective metrics for sustainable QoE assessment, technologies to enhance energy efficiency and low power consumption for 6G and beyond mobile telecommunication systems with application to mobile immersive media.

I am working in the broad area of networked systems and applications. Technical systems have become more and more complex, and many problems are solved adding additional resources, bringing resources closer to the users, or using programmable hardware. My work aims at designing mechanisms and algorithms to improve the operation of technical systems by enabling customization considering user-centric metrics or utility functions. While this can for instance improve system utilization, improve revenue and reduce the carbon footprint per user, it also puts additional burden on the control planes, and increases costs and computational complexity.

Hence, my work strives sustainability of ICT, but my interest also covers how new ICT systems can be used to enable applications improving sustainability, e.g., immersive haptic / XR applications further reducing traveling. For that I am trying to understand relationships between social, environmental and economic factors.

The following text represents a summary of the work by Group 2 and the subsequent review by other groups, especially Group 4, and respective further dicsussions.

For both aspects individually, experiences on the one hand, and environmental impact on the other, different measures and measurement approaches have been developed in the past. In this group 2, we have discussed a framework for characterizing “Quality of Sustainable Experiences” (QoSE), with the aim to jointly characterize the user experience and the associated sustainability of the used system or service. The overarching perspective was referred to as the “cathedral view” within the group. In the group work, the framework was primarily instantiated for the case of media technology, for the example of videostreaming, for which some quantitative measures for both the “experience” and the associated consumption have been investigated in the past. A joint footprint and QoE measurement view had been addressed before by some of the group members, too, e.g. in [1] in terms of power consumption vs. video streaming QoE, and [2] with regard to CO₂ emission and streaming QoE. In the group’s discussions, a novel component was the possible impact of the users’ awareness of the degree of sustainability of their product-related behavior, which has been integrated into the experiencing process, updating an existing model view on QoE formation [3, 4, 5, 6].

To this aim, the group has sketched a first graphical representation of the framework, see Figure 1. Here, W2I refers to the willingness to invest, that is, to consume more moderately and at lower perceptual quality, if this reduces the environmental impact. Both during system operation (left) and system / service production and operation (right), environmental resources are being consumed. It is noted that the figure acknowledges the fact that the exact consumption of resources may be difficult to attribute to individual systems and/or manufacturers / service providers (“Attribution Problem”). It is general consensus in the QoE community now, that QoE happens in the users’ minds and results from the appraisal of the experience with regard to expectations. Here, awareness and W2I were thought by the group to influence expectations, hence increasing QoE in spite of possibly lower sensory / perceptual quality. In case of the manufacturer / service provider, awareness for the sustainability impact of the “experiences they sell” may lead to a more careful handling of resources and acceptance, that users may not strive for better and better perception. On the very left hand side, examples for aspects that need to be measured or characterized are indicated. Here, at the border between system / service and “measures”, the associated cost is given as a measure, which currently still strongly impacts user expectations and decisions in terms of acceptance. Also for the providers or manufacturers, cost is a key measure, determining many decisions. Here, too, awareness may be a driver for updated, more environmentally sustainable decisions. To raise awareness, sustainability- and/or experience-related measures can be used to display and inform (red in Figure 1) about the environmental impact and the role of experience therein, positively influencing production and consumption patterns.

The ideas can be assigned to the categories of NOW, WOW and HOW as follows;

We present the emerging sets of questions and ideas related to time horizons and based on the discusssions within the group around the role of ICT and how it can be a means to drive sustainability.

The considerations tackled in this topic targeted the question whether a high QoE is really needed or not and aimed to challenge the paradigm of always more and always better. For most applications, it would be possible to define a certain minimum, basic requirement, which is sufficient to satisfy the needs of the user while avoiding that the striving for fulfillment of fundamental psychological needs becomes an act of greediness. Such “sufficiency” thresholds could be investigated for different use cases and implemented as the default configuration of applications. In this regard, the group discussed the need for a definition and better understanding of “sufficiency”: what does greedy or ungreedy behavior mean, what does it imply and what would be meaningful measures? Further, it was discussed that we should investigate which needs that drive human behavior are related to sustainability and how these could be used for pushing QoSE? A challenge here is however that the traditional test designs to evaluate quality are not suitable for this goal and would need to be redesigned. As a result, there is a need for new methodological approaches and metrics representing “sufficiency” as opposed to metrics targeting high quality. We briefly discuss the main NOW, WOW and HOW ideas, also incorporating the feedback from group 2 (De Bonos hats feedback round).