Visualizing Data on Non-Flat, Non-Rectangular Displays

I presented an overview of light field displays by first introducing light field photography. Light field displays enable virtual content to be rendered at different focal depths simultaneously and are particularly interesting for Augmented Reality. They enable blending virtual content with the physical world, as the light from virtual elements behaves similar to the light from the physical environment. It also has the potential to solve the vergence accommodation conflict. Virtual content is not limited to a single focal plane and can also be rendered much closer to the user’s eyes. Light field displays are still in their infancy, and I talked about the visual perception studies we are running to better understand their potential benefits.

Situated visualization is an area within information visualization that focuses on placing data visualizations in physical places to display contextually relevant information that can be related to people, places, or objects. Physicalizations, on the other hand, are physical artifacts that encode data through their geometry or material properties. In this talk, I provided an overview of these two concepts and highlighted their connections and overlap. By discussing a diverse set of examples of situated visualizations and physicalizations, I aimed to inspire new ideas for visualization on non-flat, non-rectangular displays.

Together, let’s consider visualization futures — imagining possible and probable future technologies, probing their implications, and using them as inspiration for visualization research. This talk highlights the potential for design futuring to (1) draw ideas from existing science fiction and speculative media, (2) craft new future visions that explore, provoke, and critique, and (3) articulate new trajectories, frameworks, and theories for data visualization as a field. Using the Vis Futures card deck, plus a range of generative and manual design tools, we’ll also ideate and elaborate new visualizations for a variety of novel display platforms.

My research focuses on democratizing the development of emerging technologies. More specifically, by establishing accessible approaches for designing and building emerging technologies such as robotics, wearables, and shape-changing interfaces. To advance the field, my research focuses not only on understanding these technologies (e.g., their design), but also how to build them (e.g., engineer them), and how to innovate with them (e.g., application). In this talk, I will go over some of the projects I have worked on around this topic across the fields of HCI, Design, and Engineering.

In this talk, I gave an overview of research examples and a taxonomy of augmented reality and robotics based on my CHI 2022 paper. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics.

This working group conducted a quick survey of the current state of the art for interacting with visualizations. Looking from an HCI perspective on what has been researched in the past, we first identified several interaction modalities, including, but not limited to, mouse and keyboard, touch and multitouch, pen, tangibles, eye gaze, mid-air gestures (hand), spatial interaction and device gestures, embodied interaction, speech, twisting, bending, deformation, proxemic interaction, and any multimodal combination thereof. Many of these modalities have already been applied in visualization research, especially in the past 15 years, with an increasing number of papers devoted to natural interaction or Post-WIMP interaction for visualization, sometimes characterized as visualization beyond the desktop. We provided several examples from the literature for each category, identified gaps, and discussed the potential. Touch input on arbitrarily shaped surfaces is still not easy, pen interaction has rarely been applied to curved surfaces, speech and multimodal interaction including speech would be a good candidate for interacting with unusual displays, and twisting, bending, deformation, etc. are still very underexplored in vis research. This situation will likely change with non-flat, non-rectangular displays being increasingly used for data visualization.

This group presented trends in visualization on “atypical” display forms, and atypical use (atypical for visualization research and design) to inform participants from outside the visualization community about relevant work related to the seminar. We noted existing visual perception work on very large displays (walls) and very small ones (smartwatches) that can be applied to study novel form factors for visualization. However, we also highlighted atypical “use”,including aspects that go beyond classic visualization tasks, designs, and evaluation methodologies, such as affective responses not only due to the visual encoding but also due to the display form and context of visualization use; aspects related to glanceability (mostly from work on smartwatches), etc. We additionally summarized situations in which visualizations have appeared on atypical “forms.” Some are extensions of classic displays, like combining mobile phones and tablets or combining them with digital tables to create new forms and configurations for data visualization, adding flexible displays as bands on smartwatches and extending the visualization surfaces, etc. Others are forms rarely used for visualization, such as drone displays that can fly, spherical displays (from big room installations to small hand-size spheres), flexible or pin-array displays used to visualize data, ambient displays embedded in the environment, and more generally situated visualizations.

We presented a summary of relevant resources on data physicalizations and shape-changing displays. Data physicalizations (DP) are physical representations of data; shape-changing displays (SCD) are dynamic physical displays that can represent varying types of data or information. We summarized research venues that approach these topics from the perspectives of visualization, human-computer interaction, and enabling technologies (systems engineering). We also highlight communities beyond academia that apply or practice data physicalization or shape-changing displays within their domain. We then discussed relevant research publications that address key research questions, including: what the design space of DP and SCD looks like; how to design and build DP and SCD; what are the “platforms” that designers use to define data physically; how we encode data physically; how people interact with physical properties; what is the role of the audience of a DP or SCD. Finally, we discussed example systems and enabling technologies that make data physicalizations and shape-changing displays possible, including digital fabrication approaches and meta-materials.

This group presented a summary of related work on mixed/augmented/extended reality (XR) visualization approaches and enabling technologies. We first summarized the visualization, immersive technology, and computer graphics venues around which this research is centered. Next we outlined a slate of recent innovations in XR hardware that have substantially reduced barriers to creating rich, complex and usable XR visualization systems, including: (1) untethered headsets, (2) table tracking in real-world settings, (3) high-resolution video passthrough, (4) fine-grained hand tracking, and (5) live environment mapping. We then highlighted a variety of recent design spaces and example systems that represent the state-of-the-art in XR visualization.

Recent years have seen an increase of visualization research that goes beyond desktop displays, this include visualization on mobile devices, physicalization, and immersive visualization using augmented reality technology. We focused on articulating the difference of the work targeted in this seminar from that prior work. Our focus targets dynamic and interactive visualizations, on physical displays that are not flat but rather have forms that integrate / fit their environment. These characteristics suggest that such displays allow embedding data in context (for example, in locations where the data is relevant) and can support more expressive, immediate, and powerful interactions. And they allow experiencing data in person rather than via a window into another place, as is the case of flat displays. Other work groups elaborated on the benefits of such displays as well as open challenges and opportunities. After defining the nature of these displays and visualizations built for them, we then considered the challenges they pose. These include how to promote displays and visualization use cases that minimize material and other resources (specialized material which may limit their life-cycle, energy use, sustainability, development cost), as well as ethical considerations (for example, ensuring equity, access, privacy, etc.).

We identified possible broad categories of displays that fit our scope of non-flat non-rectangular displays. These include real 3D displays (such as volumetric systems or 3D LED arrays), non-deformable [non-flat or non-rectangular] displays, deformable displays (including actuated, bendable, and foldable variants), modular (compound) displays made up of several components that can be rearranged, wearable or fabric displays, as well as everyday objects as displays. In addition to these form factors, we consider challenges related to materials, resolution, durability and energy consumption. The interactions appropriate for these displays are very much an open research topic. We have identified several input technologies that either exist or can be envisioned for such displays, such as on-surface input (via touch or tools) or off-surface input (mid-air gestures, speech, or proxemics). A first open research question is categorizing these possible inputs and studying good matches between input and type of display. Nevertheless, the details of how input is going to be translated to “interactions” (the words or commands that target and manipulate visualizations) depends on the type of visualizations and data presented on them.

As NFNRDs can vary greatly in form and size, identifying when data visualization on NFNRDs makes sense (and what benefits it might provide) requires taking a step back and considering the possible contexts of use. We considered several dimensions that define this context of use: application types (public / private, stationary /mobile), audience (data specialists / general public, single person / collaborative use), tasks (ambient / focus), etc. Clearly not all display types (identified in the previous working group) fit all contexts of use. We considered possible matches and identified a grand challenge focued on the interplay between application (purpose and context of the visualization), data, and displays.

Having access to displays that can be of diverse forms and sizes raises several research questions around how to display data on them. These start from visualization mapping and resolution questions – what we call “how the data follows the display”. For example, how can visualizations designed for a flat rectangular screen be rendered on other types of displays, such as a sphere or a cylinder? More generally, can we render any visualization on any display form and, if not, what are the limits we encounter (resolution, occlusion, etc)? Other questions relate to how the nature of the data can influence the display, or “how the display follows the data”. These observations highlight how designers might allow data to shape the form of actuated or shape-changing displays. We articulated the connection between data and displays and collected a set of opportunities and research challenges on this topic.

But how do we go about designing visualizations for such displays? Traditional sketching and paper prototyping methods are not enough, as they do not capture the subtleties of displaying information on forms that may distort visuals (see also subsection 6.6) or may be hard to envision on a two dimensional medium such as paper. We collected factors that make NFNRD hard to design for and considered different existing design methods, such as physical prototyping, and how these could be adapted for creating initial designs. We additionally reflected on the use of these displays, as their purpose is to be more integrated than flat displays, exploring how they could be perceived in their intended context of use is important. And thus how our design methods could ensure the context of use is taken into account even at the early stages.

Given the diverse and unpredictable form factors and sizes of NFNRD, it is important to evaluate how visualizations are read on them. The topic of evaluation is challenging as there are several factors that need to be considered – including the form of the display and the context of use. The form needs to be considered from a readability / perception perspective, in other words we need to validate if information can be correctly read and understood, as aspects like curvature, angle, etc. may affect visualization reading. Traditional user-study methods will likely be enough in these instances. But we additionally need to consider evaluating the context of use and of display placement. It is possible that data engagement will be higher with displays of unusual form factors or form factors that have forms fitting the environment. Investigating and evaluating engagement and affective responses requires different study methods, such as longitudinal observation studies or controlled studies that capture measures beyond time and error. Our group listed a set of relevant study methods, measures, and logistics