Art, Visual Illusions, and Data Visualization

Christophe Hurter (ENAC - Toulouse, FR)
Claus-Christian Carbon (Universität Bamberg, DE)
Mauro Martino (MIT-IBM Watson AI Lab - Cambridge, US)
Bernice E. Rogowitz (Visual Perspectives - New York, US)

Art, visual illusions, and data science are seemingly disparate fields, but they are deeply intertwined. Art has always been a means of visual expression and communication of complex narratives. Visual illusions have captivated audiences for centuries with their ability to “trick the eye,” and challenge our perceptions, but also to teach the beholders about cognitive and perceptual functioning in a joyful and compact way (Carbon, 2014). Data science offers powerful tools for analyzing and interpreting diverse and large data sets, using novel visualization metaphors to provide insight and new perspectives. These three disciplines offer different approaches to understanding the relationship between the visual representation and its perception. The goal of this seminar was to provide a forum for exploring synergies between these diverse disciplines. This premise is built on a strong foundation. There is a long history of artists incorporating visual illusions in their art to challenge audience perceptions Some examples include Robert Delaunay’s explorations of color contrast; Victor Vasarely’s spatial illusions; Bridget Riley’s optical illusions; Salvador Dalí’s spatial sampling illusions; M.C. Escher’s impossible constructions; Giuseppe Arcimboldo’s composite portraiture; Yayoi Kusama’s infinity rooms; Hans Holbein the Younger’s anamorphic art; and Patrick Hughes’s reverse perspective paintings. Contemporary artists now use data science to create works that explore hidden patterns and relationships in complex data sets. Examples include Laurie Frick’s use of self-tracking information gathered from personal devices to create hand-built works and installations; Jenny Odell’s combination of satellite images to create collages revealing patterns in human-made structures; Refik Anadol’s use of datasets from EEG brainwave patterns and machine learning algorithms to create immersive, data-driven installations exploring memory and cognitive processes; Nathalie Miebach’s intricate, three-dimensional sculptures visualizing climate change data and its impact on weather patterns; Giorgia Lupi and Stefanie Posavec’s hand-drawn visualizations of personal data exchanged through postcards over the course of a year; Aaron Koblin’s visualization of airplane flight paths across the United States, creating mesmerizing patterns that reveal the complexity of air traffic; Onformative’s use of face detection algorithms to search for human-like faces in satellite images from Google Earth, exploring pareidolia and pattern recognition; Ingo Günther’s illuminated globes that visualize various datasets such as population density, energy consumption, and military spending; and Ben Rubin and Mark Hansen’s collection and display of real-time text fragments from internet chat rooms and forums, revealing the ebb and flow of online conversations. Visualization researchers explore spatial, temporal, and color metaphors to represent complex data and relationships (Brooks, 1988), and are increasingly turning to visual arts and perceptual psychology for new ways to communicate complex data in more intuitive and engaging ways. This seminar brought together artists, perceptual and cognitive psychologists, and computer scientists active in the data science fields of visualization and artificial intelligence, who are not only experts in their own fields, but whose work has actively crossed discipline boundaries. Our format encouraged the organic evolution of these connections and synergies.

The combination of art, visual illusions, and data science is an interesting and captivating research area. There have been interactions between these fields, but we have not fully explored the promise and limitations of this interface. The goal of this seminar was to push the boundaries of our creativity and knowledge discovery, providing new insights and new research directions at the crossing of these multidisciplinary topics. In the standard data visualization pipeline, data are rendered onto visual geometries, like lines, fields, or three-dimensional spaces, and dimensions of the data and relationships are mapped onto these geometries. However, this 1:1 mapping is often insufficient to capture the deep complexity of the data, which can limit deep understanding. Visual illusions and art provide a new set of metaphors to the data scientist, which can bring new insights. Both the psychology of visual illusions and art are designed to draw the participant into the exploration process. Some features are revealed immediately, some emerge, and some morph and change in the exploration process. It is common in data visualization practice to focus on making salient features “pop out”. Our exploration of art and visual illusions can be a powerful vehicle for drawing research attention to the “slower,” more analytical processes (Kahneman, 2011), which are critical to understanding complex perceptual and cognitive phenomena (Carbon 2014). Moreover, advancements in artificial intelligence, particularly in generative models and machine learning algorithms, have opened new avenues for integrating AI into the creation and interpretation of visual art and illusions. AI can assist in generating complex visual patterns that challenge human perception, thereby providing deeper insights into cognitive processes and data interpretation.

The seminar consisted of a dynamic mix of plenary discussions and work in groups. Invited talks focused on core topics in art, perception, cognition, data visualization, and artificial intelligence highlighting work at the intersection of these disciplines. We allowed plenty of time to allow important questions to emerge organically, before breaking into focused groups. We began with short self-introductions (2 minutes each), followed by several invited talks to help bridge the knowledge gaps between fields. Working group topics were seeded by ideas proposed by the organizers, and emerged organically from the participants. Six groups were formed, and each provided interim and final read-out reports.

Participants were asked to present some of their work or thoughts to extend our discussions during the working groups. We received so much motivation from our participants that we had to adjust our seminar schedule. These talks explored illusions as a gateway to understanding perception, the exploration of illusions in art, the intersection of AI, art and illusions, and principles of visual perception that can reveal (or distort) features in data. Many thanks for all the highly interesting and insightful content, which triggered many additional ideas and future directions for extensive discussions. All of this reflects the general enthusiasm of the participants and presenters for this highly inspiring topic (15 minutes).

One important aspect of our seminar was the inclusion of professional artists who have worked with the concept of visual illusions, or have used AI in their creative process. To highlight their perspectives, we mounted an art show, which was supported by the Dagstuhl team. Our vernissage was on Tuesday night. Seminar participants, staff and visitors were invited to visit with the 12 artists, to discuss their work. The two exhibit rooms were open for visiting throughout the week. We also hosted a visit to an art exhibition featuring one of our artists, Bodo Korsig.

We offered our seminar members the opportunity to explore the intersections of art, science, and perception through hands-on demonstrations by experts. Participants gained unique insights into how perceptual principles shape our understanding of art and data visualization, and engaged with interactive examples that blur the boundaries between visual science and artistic expression.

Sylvia Pont:

Illusory lighting-materials-shapes demonstrations.

Marco Bertamini:

Wow we interact with mirrors.

Michael Bach:

A collection of examples, categorized by superficial perceptual dimensions, plus examples for interaction and mutual inspiration between art and vision science of perceptual illusions.

Boyu Xu:

DatAR, an Augmented Reality prototype (in HoloLens 2) designed for topic-based literature exploration in neuroscience.

Michele Banks:

“I will bring materials to do an ink painting demo which people can join. I just need water, paper towels, and something to protect the table.”

Dejan Todorović:

Effects of observer vantage point on perception of images, how the same stimulus is affected by many different contexts.

Arthur Shapiro:

A variety of color, motion, and shape demonstrations.

On Wednesday afternoon, we visited the Rheinisches Landesmuseum in Trier where we viewed historic visual illusions (ancient Roman mosaics), including large areas built from reversible-perspective Necker cubes which directly referred to our own art show organized at Dagstuhl and the expert talks by visual illusionists and scientists.

We then spent time at the beautiful outdoor Contemporanea Gallerie which displayed over 20 art works by one of our participants, Bodo Korsig. We ended the day with dinner at a winery which offered us Saar wines including a profound explanation of the wines by the head sommelier.

We conducted several brainstorming sessions, in which we explored how AI, randomness, and creativity intersect in visual illusions, perception and art , raising ethical, cultural, and societal questions. These topics guided the formation of the working groups:

Building on the rich ideas from the brainstorming phase, participants formed six final groups to delve deeper into selected topics, and form plans for future exploration, which could produce tangible outputs, such as research papers, exhibits, and workshops.

Art is a multifaceted form of expression that communicates ideas and emotions through various mediums. It often intersects with concepts like optical and visual illusions, which are perceptual phenomena that challenge our understanding of reality by misleading our visual perception. Generative AI represents a technological advancement that enables the creation of new content based on learned patterns from existing data, raising questions about the boundaries and intersections of art, design, data visualization, and artificial intelligence. The emergent features in art, visualization, and illusions do not necessarily provide a direct one-to-one mapping of data or reality, leading to both advantages and disadvantages in interpretation. The nature of “reality” itself can be debated, as it encompasses both the physical world and the constructs of our minds, a concept explored in the philosophical discipline of phenomenology, which emphasizes subjective experience. Understanding when a visual illusion serves merely as a “trick” versus when it offers meaningful insights into data visualization is crucial for effective communication. Artistic techniques like Trompe L’oeil and Mise en Abyme exemplify how visual illusions can relate to the interpretation of data visualizations, revealing complexities in our understanding.

The rise of generative AI – leveraging advanced architectures such as LLM (Large Language Model) and DM (Diffusion Model) – also prompts the emergence of new jobs and roles in both the art and technology sectors, where computer science plays a transformative role in reshaping art and democratizing access to creative expression. Tools like GPT-4, Stable Diffusion, Flux, and Midjourney enable artists and enthusiasts to create complex artworks, leading to professions like AI artists, prompt engineers, and AI ethicists. As art becomes more accessible to a broader public through these AI technologies, it raises questions about the tools used for art criticism and whether new methodologies are needed to evaluate AI-generated art. Additionally, emerging technologies such as blockchain and non-fungible tokens (NFTs) are reshaping the art world by providing new platforms for the distribution and monetization of AI-generated art, altering how we perceive and value these works. Ensuring that this democratization remains inclusive and accessible to diverse audiences and creators is essential for fostering a rich and varied artistic landscape, while also addressing challenges related to algorithmic bias, digital literacy, and equitable access to AI technologies.

The literature includes many pairwise interactions between art, visual illusions, and data visualization, but to our knowledge, this is the first attempt to bring these diverse ideas together into a coherent whole. In our discussions, activities and working groups, we explored many opportunities for publishing scientific papers and creating artistic shows around these themes.

We plan a special issue of the Journal of Perceptual Imaging, focused on illusions, art, and data visualization, edited by Kassandra Lee, Claus Christian Carbon, and Bernice Rogowitz. Other ideas were proposed, such as a book of edited papers and follow-on workshops and seminars to engage a wider community of data and visualization scientists, perceptual researchers and artists, and to inspire new research directions and new interdisciplinary curricula. Other venues for exploration for future interaction include Visual Science of Art Conference (VisAC) associated with the European Conference on Visual Perception (ECVP) VisAP, the art and visualization event aligned with the IEEE Visualization Conference (VIS), as well as relevant sessions at ICC, IUI, and NeurIPS. Additionally, we plan to create white papers for dissemination in more general forums, such as Computer Graphics and Applications and IEEE Viewpoints. An art show focusing on emergent perception across art, vision, and data visualization is also envisioned, highlighting the convergence of these fields.

Our participants bring a rich diversity of expertise, having explored the intersections between perception, art, data visualization, and artificial intelligence. Some have applied perceptual principles to create artistic works, while others have studied how we perceive and experience art. Several participants employ perception principles to enhance data visualization or use data visualization techniques as artistic tools. Many integrate artificial intelligence, either as a creative medium or as a support for artistic expression, and others utilize signal processing to analyze and characterize artistic works. Additionally, their work encompasses the study and application of visual illusions, from classical to contemporary art, and extends to scientific research on perceptual principles like Gestalt, examining their implications for both art and data visualization.

We are very grateful to Dr. Raimund Seidel and his team for their support for our art exhibit. They spent much additional time and effort to identify exhibit spaces and proposed creative options for displaying the works within operational constraints. We thank Itay Goetz for his enormous contribution to the art show and to the organization of our multi-part excursion, and extend special thanks to Dr. Klaus Reeh who welcomed us at Contemporanea Galerie für moderne Kunst in Oberbillig with original sparkling wine from the Saar region to this exquisite outdoor art exhibition where we had vivid discussions on Art, AI and the potential conditio humana that let us create art. And, above all, we give thanks to the scientists and artists who participated in this seminar, who so openly embraced different perspectives on the theme of this excellent Dagstuhl Seminar Art, Illusion, and Data Visualization.

• $\mathrm{■}$

  Kim Albrecht – Filmuniversität Babelsberg – Potsdam, DE

• $\mathrm{■}$

  Michael Bach – Universität Freiburg, DE

• $\mathrm{■}$

  Michele Banks – Washington, US

• $\mathrm{■}$

  Marco Bertamini – University of Padova, IT

• $\mathrm{■}$

  Alina Braun – BMW Group – München, DE

• $\mathrm{■}$

  Claus-Christian Carbon – Universität Bamberg, DE

• $\mathrm{■}$

  Shanthi Chandrasekar – North Potomac, US

• $\mathrm{■}$

  Oliver Deussen – Universität Konstanz, DE

• $\mathrm{■}$

  Peter Eades – The University of Sydney, AU

• $\mathrm{■}$

  Brian D. Fisher – Simon Fraser University – Surrey, CA

• $\mathrm{■}$

  Itay Goetz – Universität Bamberg, DE

• $\mathrm{■}$

  Ludwig Hanisch – Nürnberg, DE

• $\mathrm{■}$

  Christophe Hurter – ENAC – Toulouse, FR

• $\mathrm{■}$

  Sophia Huth – Berlin, DE

• $\mathrm{■}$

  Stephen Kobourov – TU München, DE

• $\mathrm{■}$

  Jürgen Kornmeier – IGPP – Freiburg, DE

• $\mathrm{■}$

  Bodo Korsig – Trier, DE

• $\mathrm{■}$

  Michael Kubovy – University of Virginia – Charlottesville, US

• $\mathrm{■}$

  Karina Kueffner – Nürnberg, DE

• $\mathrm{■}$

  Kassandra R. Lee – University of Nevada – Reno, US

• $\mathrm{■}$

  Mauro Martino – MIT-IBM Watson AI Lab – Cambridge, US

• $\mathrm{■}$

  Fiona Menzel – Technische Hochschule Würzburg Schweinfurt, DE

• $\mathrm{■}$

  Mario Alexis Emilio Michelessa – National University of Singapore, SG

• $\mathrm{■}$

  Arthur I. Miller – University College London, GB

• $\mathrm{■}$

  Claudia Muth – Universität Hof – Münchberg, DE

• $\mathrm{■}$

  Alexander Pastukhov – Universität Bamberg, DE

• $\mathrm{■}$

  Rebecca Pfiffer – freakstotable e.V. – Wirsberg, DE

• $\mathrm{■}$

  Sylvia Pont – TU Delft, NL

• $\mathrm{■}$

  Marius Raab – Technische Hochschule Nürnberg Georg Simon Ohm, DE

• $\mathrm{■}$

  Brian Rogers – University of Oxford, GB

• $\mathrm{■}$

  Bernice E. Rogowitz – Visual Perspectives – New York, US

• $\mathrm{■}$

  Arthur Shapiro – American University – Washington, US

• $\mathrm{■}$

  Ahna Skop – University of Wisconsin – Madison, US

• $\mathrm{■}$

  Dejan Todorovic – University of Belgrade, RS

• $\mathrm{■}$

  Jan Willem Tulp – TULP interactive – Rijswijk, NL

• $\mathrm{■}$

  Boyu Xu – Utrecht University, NL

• $\mathrm{■}$

  Rebecca Ruige Xu – Syracuse University, US