Navigating the Maze of Guidelines to Unify Visualization Design Recommendations

This lightning talk explores the intersection of data visualization as both science and craft, drawing from my experience in navigating the design of visualizations in practice. Emphasizing themes that balances between structure and creative expression, this talk provokes reflection on the dual nature of visualization – as a science, anchored in empirical research, perception, and statistical integrity, and as a craft, deeply expressive, intuitive, and personal. The talk challenges the notion of fixed rules in visualization, emphasizing instead the diversity of ways a message can be visually conveyed. Like language, visualization has grammar and structure – but also metaphor, rhythm, and poetry. Rule-breaking can be useful when grounded in thoughtful design. Thus, I advocate for learning from vis designers’ processes in creating visually compelling visualizations. The goal is not to prescribe, but to inspire dialogue and design grounded in human insight.

I mostly worked on data representation from geometrical encoding to visual perception of patterns. When data are transformed, encoded, and then decoded and interpreted by the end user, some information is lost.

Design guidelines can help at every step of the visualization pipeline, ensuring that as much information as possible is retained. It depends on the task we want to solve, the user, and the context in which it is used.

Generative Artificial Intelligence holds great promise in integrating a vast amount of human knowledge across all fields to provide guidance. However, they require a carefully designed loss function based on expert knowledge and a large amount of clean data to learn from them what constitutes good or bad guidelines for a prompted request.

The challenges are that the existing guidelines are limited to a few isolated cases from very different domains and abstraction levels. How do we scale up the number of cases and their variety while maintaining the data quality and meaningfulness? How can we encode such a diverse range of guidelines at various levels of abstraction? Is a language model combined with a vision model enough? Shall we use agentic approaches? Which roles for the agents? Do we have computational models that, in theory, can generate such guidelines? How do we validate such AI models?

Why are we still doing lab studies?! This critique often emerges when lab-derived guidelines break down in messy, real-world settings. But perhaps the problem isn’t the lab itself. It’s how we frame its purpose and what we expect from it.

I propose that the lab serves two critical roles. First, it helps us understand why something works (or doesn’t) through controlled studies. These studies isolate causal mechanisms, which can support generalization across formats and contexts. Success here means having explained something.

Second, the lab excels at rapid iteration. Testing and refining prototypes to make messy problems manageable. In this mode, success is measured by progress, not control. These studies are valid so long as we don’t mistake them for universal truths.

Bad lab studies, then, fail in one of two ways: they stop too early (after a single surprising result), or they ask the wrong questions (focusing on trivial manipulations or ignoring essential context). I argue that we should design lab studies with the wild in mind. Instead of pitting lab against field, we should ask: What part of the wild are we trying to bring into the lab, and why?

The relationship between visualization design guidelines and visualization research is multifaceted and can be viewed from both research and practitioner perspectives. From a research standpoint, guidelines for designing visualizations may be perceived as vague, overly general, or even conflicting. They often lack the precision and rigor expected in scientific inquiry. However, from a practitioner’s perspective, such guidelines serve as valuable tools for translating experience and domain knowledge into actionable recommendations for design practice.

Integrating these two perspectives is therefore highly relevant, though the nature of their relationship is not immediately clear. This relationship can be bidirectional: on one hand, research can lead to the formulation of new guidelines based on empirical findings; on the other hand, research can also investigate the implementation and effectiveness of existing guidelines in practical settings. Thus, the interplay between visualization research and design guidelines involves both the generation of guidance through scientific methods and the study of how such guidance is applied in real-world contexts.

Guidelines are a broad and sometimes diffuse concept while trying to span a multitude of different and diffuse use-cases and contexts. In my provocation I am arguing for the cases that this wide span of uses causes these codified guidelines to be either too rigid in their implementation, thus making them less useful in concrete application cases, or too broad to the level of being non-actionable. In particular with regard to the use of utilizing visualization to present to a diverse group of users simultaneously, we are currently limited to finding ad-hoc solutions that have to be rediscovered from first principles in every situation. So while there is a definite need for normative guidance or guidelines, the concrete description and application of such remains an open question to be solved by the community.

We like to believe that design rules (e.g., “bar charts are better than pie charts,” or “use blue for low values and red for high”) can be universally applied. However, in practice, context matters. What works well in a lab study with 100 student participants might fail in the wild with a different audience, task, or culture. So the comfort of generalization doesn’t always hold in messy real-world settings. When we rely too heavily on codified design rules, we risk ignoring specific user needs, domain knowledge, and the aesthetic or emotional impact of design choices. This can stifle creativity and result in one-size-fits-all solutions that don’t actually fit anyone well. Codified rules give designers the feeling of certainty, but that can be misleading. Following the rules doesn’t necessarily mean we’ve made a good design decision. In most cases, it just means we’ve followed a “script”.

In this lightning talk, I challenged the notion that pie charts, 3D visualizations, and rainbow colormaps are inherently “evil” by highlighting examples (based on academic papers) where, given the right context and audience, they can be effective. I also introduced one possible approach to addressing this debate: building a database that showcases various visualizations applied to the same data, aiming to bridge the gap in aligning design choices with human needs (in the spirit of tools like GraphScape). To further narrow the design space, I proposed simulating human perception using computer vision models. For instance, applying paradigms like the Just-Noticeable Difference (JND) could help evaluate whether a given visualization preserves enough perceptual signal to remain effective when substituted with an alternative.

I ended the talk with a provocation: what if the goal of visualization isn’t to “express data effectively” but to maximize user engagement, spark curiosity, and invite diverse perspectives through interaction?

Guidelines can help creators make less bad visualization (or make bad visualizations less often). However, to be effective, guidelines must have several properties. Correctness, that guidelines lead creators to the best designs, is only one property, albeit one that the academic community focuses on. However, I argue that properties relating to the actionability of guidelines – usability, credibility, and discoverability – are also important, maybe even more than correctness. This suggests that we need to develop an art/science of crafting guidelines, which has not been considered by the visualization community.

Visualization purports to be a science. It investigates perceptual and cognitive dimensions of visualizations, it develops visualization techniques with formalisms, it systematically studies the visualization design process and resulting artifacts. But visualization is a craft. Outside the walls of research, practitioners create visualizations with specific tools, develop skills, and apply these in a labor/professional context. Visualizations are created within specific contexts and settings. Visualizations are also created en masse, at frequencies and scales far larger than which they are studied. It would be insufficient to apply the current epistemologies and methodologies of visualization research to studying visualization as a craft. We need new approaches that allow us to study visualization as it actually is: collective, community-based, and culturally-situated.

What we mean by (design) guidelines in the visualization community is less than clear. Guidelines can, on the one hand, be considered loose sets of rules that are meant to make things (processes/designs/systems/tools) effective in most cases. We should perhaps name such guidelines “considerations” – something people should consider and think about but that can certainly be broken when there is a good reason to. Such considerations have the problem that they require careful application, and therefore time and thought. They also require some form of empirical and practical backing to show when and where they applied well in the past or have been successfully broken. Guidelines might be, on the other hand, more easily applied if they were only framed at the level of guardrails – rules that, if broken, will likely lead to your process/design/system/tool to fail miserably. Yet, do we have any non-trivial or obvious guardrails in the community? How do we establish new ones? Are there ever rules that cannot or should not be broken?

Visualization research often makes general claims about the usefulness of visualization as a decision aid. However, reflecting on the field’s logic of generalization, I argue that we lack adequate ways of conceptualizing decision context and codifying the role it should play in design recommendations about decision aids. My talk sketches how decision theory provides a helpful framework for reasoning about the dimensions of decision context, identifying utility as a key but under-utilized way of accounting for how values and relationships around data shape the purpose of decision aids in practice.

Traditionally, people have relied on textbooks and, more recently, the internet to access information. With the rapid rise in popularity of large language models (LLMs), driven by their impressive performance across various domains, it’s increasingly likely that users will turn to LLMs for visualization-related tasks as well. For instance, LLMs can be used to generate new visualizations or assist with evaluating visualizations during data analysis workflows.

However, this introduces a concern: users without a background in visualization may struggle to assess whether the visualizations provided by LLMs are appropriate or effective. To address this issue, we need to explore the capabilities of LLMs in visualization-related tasks and propose evaluation criteria and guidelines to assess the quality and reliability of their visualization outputs.

We identified four steps for the application of AI models in visualization design: (1) prompting, (2) style modification, (3) data-sensitive, and (4) user-sensitive modelling. While the first to types mainly serve the purpose of inspiration and prettying of visualizations, data-sensitive and user-sensitive models can explicitly incorporate guidelines for design on different levels such as appropriate mapping, aesthetics, and user-specific adaptation of the data representation. A combination of these steps can potentially solve a multitude of everyday visualization problems, without the requirement of visualization expertise to design a technique. However, the resulting visualizations are, like created by a designer, interpretations of a model and require reasoning and a way to adjust the visualization, either by small or large changes. One big challenge of the near future will be how this communication between human user and model will look like. Only prompt-based discussions might be not efficient to discuss visual content that could be easier achieved by direct interaction with the visualized result.

Looking across seminal visualization writings reveals a set of assumptions and values about what makes a visualization good. These values are: that the visualization is objective; people are universally preceptive, attentive, and predictable; and encodings are efficient. But these core values – that come from authors linked to the fields of stats and vision science – do not capture the breadth and diversity of visualizations today. They exclude emerging genres of visualizations for self-expression and rhetoric. They define a narrow range of what gets to count as a visualization, and who gets to make them. And they limit the influence of many early visualization pioneers who created visualizations from other perspectives with other goals. What other values can we align ourselves with to broaden the space of what counts as good?

Most data visualization education follows a constraints-first approach: teach guidelines and theory, then provide opportunities for application through projects. My talk argues that leading with constraints limits creative exploration before students discover their visual voice. Drawing on evidence from teaching data visualization through hands-on “feel and see the data” exercises rather than theory-first instruction, the presentation proposes an alternate way of developing visualization literacy. When students build visual intuition first–unfettered by rules–they create compelling visualizations that communicate intent effectively, even when breaking conventional guidelines. This provocation asks whether we are teaching visualization literacy or visual compliance, and when design guidelines become barriers to developing authentic visual intelligence.

In the rapidly evolving field of information visualization, rigorous evaluation is essential for validating new techniques, understanding user interactions, and demonstrating the effectiveness and usability of visualizations. These empirical studies yield practical, insightful, and innovative design guidelines for creating compelling and expressive visualizations, as well as their design choices. However, many times these guidelines are isolated, less connected, and challenging to bring together and combine implementation, leading to a situation of “Empirical Explosion – Practical Paralysis”. In this talk, I introduced the empirical explosion based on prior surveys and predicted trends in historical data to showcase how an increasing number of empirical studies might advance the visualization community, and how they are not easily applied in the real world – the provocation questions aimed to spark interesting discussions.

In this talk, I outline the origin of this Dagstuhl Seminar, namely, that the field of visualization suffers from several interrelated challenges around design guidelines. First, we generate many loosely connected artifacts – theoretical frameworks, controlled experiments, qualitative studies, design studies, and practitioner expertise, etc. Second, there are challenges with generalization and the synthesis of research with little to no common framework that connects them (i.e., there is no good “theory of visualization”). Third, the artifacts we produce are hard to access–we produce many difficult-to-read papers, not to mention issues of education and literacy, communication and misinformation, role in decision making, etc. Finally, I highlight a call to action – how do we formulate and integrate the knowledge we produce to best serve the visualization community and the world broadly?

Researchers and designers tend to focus the bulk of their effort on the accurate and efficient transmission of objective insights about data. Thus, when looking to diagnose failures in data communication, we look to intervene at steps along this transmission process: improving encoding (e.g., through better design guidelines) or decoding (e.g., by boosting data/visualization literacy). However, guidelines that are primarily concerned with encoding-decoding cannot account for the full range of behaviors we have recently witnessed – particularly around how visualizations can propagate misinformation. This provocation suggests we need to look beyond the encoding-decoding model. By drawing on sociolinguistics, this provocation suggests we need to study “social inferences”: the meanings people read into visualizations that are not about the data, but rather are about the identities and characteristics of the visualization author, and about their relationship to the reader. Initial evidence indicates that such social inferences mediate how receptive readers are to the information a visualization depicts, and how likely they are to further engage with the visualization.

People likely mistake exploratory study findings for universal design truths because they want to be told what to do to guarantee “good” design without having to think critically about design (either because they do not have the tools/knowledge/intuitions or they do not have time/motivation to put in the effort). Through this lens, critical questions arise: what are concerns about assuming study implications do equate to instructions of how to produce “good” design, and what might we do about those concerns? My primary concern is that treating laboratory findings as “rules”, in service of pithy, impactful guidelines can lead to two key problems. First, if people overgeneralize and use those rules where they do not apply, that can lead to problematic design choices. Second, if they ignore those rules because they think the rules don’t apply in their context, that can lead to designers feeling guilt, anxiety, and even shame for “breaking” rules. Yet, there are opportunities to leverage lab findings to inform design, while accounting for nuance. One approach is through tools with adjustable parameters that enable designers to weight distinct design priorities without strong constraints. Still, more knowledge is needed to produce comprehensive, actionable tools. A potential frame for moving forward is to think in terms of the goals of the designer and the tasks of the observers, and what design properties support those goals. In doing so, it is important to think in terms of abstraction, developing theories/principles that will enable generalization of laboratory findings, rather than one-off guidelines. This approach calls evaluation tools, such as linters, into question – if linters are built on incomplete knowledge about how people interpret visualizations, that can lead to flagging effective designs as “wrong”, passing problematic designs as “correct”, and becoming overconfident that a design is effective having been checked and judged as “correct”. Going forward, a key challenge is understanding how to convey nuanced lab findings in ways that are easy to understand and actionable, despite incomplete knowledge.

In visualization research, a lot of focus is placed on established design guidelines. While some of these can be effectively and illustratively communicated, they can also lead to overly authoritative generalizations – such as the belief that 3D pie charts and rainbow colormaps are always bad. In reality, the appropriateness of such choices often depends on context, and in many cases, they may not be as harmful as we assume – they are what they are: simply guidelines, not strict rules. Nevertheless, the community continues to invest considerable time in discussing and refining long-established rules and loses additional time by rejecting novel work for not strictly adhering to these easy-to-check “rules”.

In my opinion, this time could be better spent identifying the next grand challenges the field is likely to face. These challenges could provide direction for the community and support the formation of subgroups focused on driving specific areas forward. One example of a forward-looking area we’re exploring in our lab is situated visualization in augmented reality (AR). AR has the potential to become a fundamentally new medium for interacting with data and digital content. This technological shift could profoundly impact how we engage with data – much like mobile devices reshaped computing and communication over the past two decades.

To summarize, my provocation is this: we are currently spending too much time re-evaluating old guidelines that, in many cases, are already “good enough.” Instead, I argue for a more future-oriented approach – developing design guidelines, considerations, patterns, and recommendations that address the new and emerging challenges ahead.

My talk challenges the disconnect between visual analytics design guidelines developed under controlled, idealized conditions and the complex realities of how people interact with data in practice. While existing guidelines emphasize curated dashboards and data, well-formed natural language queries, and visual minimalism, the real-world use is far messier; users are frequently under time pressure, accessing data insights on mobile devices, or multitasking across contexts. Their questions are often vague, exploratory, or evolving, and the data itself may be incomplete, inconsistently labeled, or lacking the metadata needed to support traditional interface assumptions.

This gap between designed expectation and actual experience demands a fundamental rethink of our guidelines. We need to center the realities of ambiguity, intent uncertainty, and user preference if we want our tools to support meaningful sensemaking – not just in the lab, but in the wild. This talk presents three provocations to reframe how we approach the design of visual analytics tools:

Provocation 1: Dashboards are often celebrated for their polish, well-thought out layouts, rich interactivity, and text – all of which break down in dynamic, mobile, or high-pressure environments. We must move beyond guidelines merely optimized for perfectly curated dashboards, and toward design principles that accommodate real-world conditions – time-constrained and on-the-go.

Provocation 2: Natural language interfaces for data assume clean semantics and well structured queries. We need to move beyond natural language interaction guidelines that expect clarity and completeness. Real users bring ambiguity, uncertainty, and shifting goals, and our systems must be designed to meet them there.

Provocation 3: Despite design dogma around minimalism and reduced data-ink, users often prefer densely annotated charts, descriptive captions, or even pure text summaries. We must move beyond design guidelines that prioritize visual minimalism, and acknowledge user preferences and personalization for rich explanatory context – through annotations, captions, or even standalone text.

(written in response to the provocation: “Generalisation is a comfort, not a guarantee”)

I argue that we need to think about the form and the evidence basis of guidelines to bring nuance and context in a rich and informative way. Visualisation will benefit from a pluralistic approach to knowledge and ways of knowing and by valuing diverse forms of evidence to construct guidelines. We can look up to fields such as medicine and health sciences and learn from their practice in evidence synthesis but also learn from their mistakes – such as creating a hierarchy within methods and knowledge that has stifled diversity and richness of information. I would like to also argue that we also need to focus on nuance and context that matters in the world. I would like to see us developing guidelines for substantial issues, for instance, what are some guidelines to communicate climate change, social inequalities or quality of information – we need to be specific with guidelines but we need meaningful specificity.

Our research group conducts research in three areas:

1. 1.

   basic research on network, time series and geographic data perception and visualization with lab studies that derive guidelines for better data visualization based on lab experiments.

2. 2.

   development of novel visualization techniques, where the basic principles of visualization guidelines are used and extended with novel visual designs tested in lab or crowdsourced experiments.

3. 3.

   development of visual analytics systems that apply the guidelines from basic research and visualization techniques for addressing specific application needs. That leads to best practices and general guidelines for future general visual designs.

Visualization and guidelines have 4 different forms, two and two are human-applied or computer applied.

1. 1.

   Human applied:

   • a.

     practitioners create visualization based on their knowledge. This may or may not be based or use guidelines, rather their practical experience. They use a lot of context and domain information for development of visualizations.

   • b.

     visualization experts: either researchers or experts in visualization use both their knowledge of guidelines and domain. The guidelines may not be formal.

2. 2.

   computer applied

   • a.

     Computer uses a collection of formal guidelines that it mechanically applies based on the guideline application criteria (see paper reference below). No context is taken into account.

   • b.

     AI that creates visualizations. It is unclear whether and how much AI adheres / uses the existing guidelines, context and other criteria for good visualization.

For this lightning talk, I slightly modify the provocation given by the Dagstuhl Seminar organizers (“Is Visualization a Science or a Craft?”) to “Visualization: Science or Engineering?” Following Brooks‘ statement, “the scientist builds in order to study; the engineer studies in order to build” [F. Brooks, Comm. ACM 39(3), 1996], I see the visualization research community as a whole primarily as an engineering discipline. However, we draw a lot of methods, approaches, and findings from disciplines including but not limited to computer science, psychology, mathematics, social sciences, life sciences, art, and design. This breadth has an impact on how we deal with research questions centered around guidelines. One proposed consideration is to look into guidelines on the process rather than the result (i.e., visualization artifact), broadening the perspective on guidelines to make them more flexible. Another consideration is to articulate the scope, limits, and uncertainty of guidelines, thus addressing the possible issue of over-generalization. Finally, not one size fits all, i.e., there is a need for adequate guidelines, recommendations, or examples that may span, e.g., from low-level color perception all the way to a comprehensive visual analytics system. I also touch on a few other aspects that could be discussed during the Dagstuhl Seminar.

We created a zine to summarize the ideas from our working group.

We want to create “guidelines” that help designers make better visualizations. To do this, we need to understand the properties that guidelines should have in order for them to be effective at this goal. Currently, there is very little guidance on how to create and evaluate “good” design guidelines (meta guidance). We need to (1) identify and characterize the properties of good guidelines, (2) devise ways to package them, and then (3) describe how to embed them into the design process.

We followed the following steps in our working group:

Brainstorming properties: Our process began with brainstorming an initial set of properties and identifying other resources that provide guidance on guidelines. In particular, we looked to the medical domain where they have formalized evaluation criteria for guidelines development [1, 2]. This led to a large list of 30 initial properties. We grouped them into 11 themes using affinity diagramming.

Formative evaluation and iteration: Using the 11 themes, we did a cognitive walkthrough, in which we systematically compared our properties to those listed in medical references by Brouwers et al. [1] and Armstrong and Gronseth [2]. We identified similarities and differences and characterized what is applicable to visualization design guidelines. This led us to an updated list of 41 properties (organized in the 11 themes identified before) as well as insights into how to phrase and organize the properties. To support the presentation, we further organized the 11 themes into 5 groups.

Derive Packaging approach: We then worked on how the guidelines could be packaged, i.e, represented in ways to make them most useful to designers. We worked on a template that provides a number of questions that a guideline developer should answer for their audience. We then cross-checked this template against the properties. Based on that, we iterated and updated the template to improve coverage and reduce redundancy.

Characterizing next steps: With the above steps, we built the foundation for properties and packaging (template). In the next steps, we want to survey a representative sample of existing guidelines (making use of existing repositories of design guidelines [3, 4] as a starting point) to formatively and summatively evaluate the properties and the template for representing guidelines. We have only begun to discuss the third pillar – the ramifications of how to include our approach into existing design processes.

Our initial results are:

• $\mathrm{■}$

  11 property themes, summarizing the overall set of xx properties

• $\mathrm{■}$

  A packaging template with 13 prompts

• $\mathrm{■}$

  A set of insights into guideline effectiveness for visualization design

Acknowledgments: Other members of the group who have contributed to the discussions were Alexander Bock and Fabian Beck, as well as the larger group of Dagstuhl participants. Thanks a lot!