Visualization of Biomedical Data – Shaping the Future and Building Bridges

Maintaining an overview of publications in the neuroscientific field is challenging, in particular in tasks such as investigating relations between brain regions and brain diseases. To support neuroscientists in this challenge, we investigate whether using Augmented Reality can make analyses of literature more accessible and integrate them into current work practices. We explore a number of questions, such as whether interaction with a large body of literature using topics provides a useful way for neuroscientists to explore and understand specific relationships. Our assumption is that by providing overviews of the correlations among concepts, these will allow neuroscientists to better understand the gaps in the literature and more quickly identify suitable experiments to carry out. We currently provide functionality to visualize and filter direct and indirect relations and to compare the results of queries. Our visualization work is based on an analysis of the neuroscience publications in PubMed. This provides an association graph among topics involving cognitive functions, genes, proteins, brain diseases and brain regions. We describe our prototype 3D AR implementation DatAR and challenges we face.

Data quality control does not always excite visualization researchers. But why? High quality data is critical for high quality research. In this case, quantitative phase imaging is being explored for improving the creation of treatment plans for cancer patients. We developed Loon, a visualization system that uses exemplars to combine different data sources to aid in data quality control.

Many visualization tools do not directly integrate into popular computational notebook environments. This introduces an overhead to visualization, delegating purpose-buit visualizations for the end of pipelines rather than integrating within familiar workflows. We developed anywidget, an open-source toolkit to simplify extending Jupyter notebooks with custom interactive visualizations. We demonstrate the use of our toolkit to integrate a genome browser, HiGlass, into computational notebooks to enable new interactive analysis workflows.

Broad audiences are hard to define and have multidimensional interests. However, a well characterized audience is necessary to design and craft effective data visualizations. During my journey in data journalism and visualization in the newsroom of DER SPIEGEL, I got to know two simple tricks how to approach this dilemma.

1. 1.

   Show the raw data

2. 2.

   Tell at least one story

1. 1.

   Show the raw data
   In many cases it’s a good choice to not show aggregated data like means, medians and standard deviations. It is more feasible to show the original data than the more abstract and unemotional aggregates see Fig. 2.

   

2. 2.

   Tell at least one story and explain a lot as shown in Fig. 3

We should democratize teaching and knowledge about machine learning to the point where children in schools understand conceptually what is happening. I present a case of analyzing the positional (geo) data of surnames in Slovenia and how we trained over a thousand kids from primary and secondary schools in machine learning and data literacy using this data. The training included workshops for teachers, the production of video introductions for kids, and the design of quizzes of other written material (see https://orangedatamining.comandhttp://pumice.si/en).

We discussed the opportunities that recent developments in artificial intelligence (AI), including foundation models, generative AI, and an increasing ability to integrate multi-modal data, present for the visual analysis of biomedical data, and what novel needs for visual interaction might arise from an increased use of such techniques within biomedical workflows. In particular, we covered the following aspects:

• $\mathrm{■}$

  Large language models (LLMs) and other multimodal foundation models facilitate the translation of domain-specific questions and requirements into the generation, modification, and interpretation of visual representations, and might contribute to more user-adaptive and contextualized visualization systems. Initial proof-of-concepts exist to comprehensively support the visual analytics workflow through LLMs. However, they are currently limited to individual and relatively small datasets, while biomedical applications typically require the analysis of large, complex, heterogeneous, and interconnected datasets.

• $\mathrm{■}$

  In a specialized subgroup during our breakout session, we delved into the potential applications of LLMs in comprehending and replacing various aspects of data visualizations, including spatial inputs, operations, semantics, and contextual information. The versatility of LLMs has made them suitable for a wide range of natural language processing tasks, with a notable example being ChatGPT. To this end, we conducted a preliminary investigation of the usefulness of conversational interfaces provided by ChatGPT to steer users through data visualizations, extract insights, and potentially enhance or substitute visualization components. We specifically investigated the capacity of ChatGPT to alter the conceptual (e.g., data aspects), spatial (e.g., layouting or positional relationships within the visualization canvas), or semantic (e.g., meaning and interconnections between data elements) context of visualizations. A pilot study employing a simple scatterplot (2D+color) illuminated how ChatGPT could unveil or emphasize patterns and selections within the data, such as instructing to “highlight the green data points with a value of feature x higher than the average” or “order the blue data points from left to right and bottom to top” or “highlight the data points that have been selected within the rectangle.” Our preliminary findings suggest that integrating LLMs into Visual Analytics solutions holds promise for generative AI-driven Visual Analytics and guiding users through data visualizations. Looking ahead, it would be intriguing to explore whether conversational interfaces can contribute to supporting, enhancing, or altering the interpretability of visualizations. Subsequent research could delve into more nuanced interactions with complex datasets, such as investigating visualization semantics (e.g., ontologies, visual metaphors, and abstractions), exploring user feedback mechanisms for iterative improvements (e.g., semantic zooming) and the provenance of such mechanisms, or scrutinizing further language-related semantics (e.g., labeling, descriptions, and textual elements). Finally, it is imperative to consider the ethical implications associated with relying on LLMs to shape data visualization experiences.

• $\mathrm{■}$

  Machine learning approaches are already established in practice for specific subtasks, such as dimensionality reduction of -omics data, or medical image segmentation. However, the limitations of these methods, especially when trained with the limited amount of data that is available in many biomedical applications, are still not sufficiently understood. For example, most deep learning techniques that are used today do not provide an accurate indication of the uncertainty in their estimates, and cannot detect whether a given input is sufficiently similar to their training data to be reliably processed. Even after the deployment of automated systems, continuous quality control remains essential, and is best facilitated by suitable visual interfaces.

• $\mathrm{■}$

  Explaining the predictions and outputs of machine learning methods is not only helpful for the development of such methods, but is often also a requirement for integrating them into scientific workflows (which ultimately aim for mechanistic understanding, not just predictive power) or clinical decision processes that require human experts to weight evidence from various data sources. However, the computations within machine learning methods and available model level explanations do not usually map directly to human mental models. Therefore, it is non-trivial to design explanations in such a way that humans will actually understand why the model generated a specific output, as opposed to rationalizing the proposed explanation in terms of their own mental model. We believe that future explanations should go beyond the attribution maps that are commonly presented in current work, and should combine multiple modalities, such as images and text.

• $\mathrm{■}$

  How to communicate model outputs in a way that experts and, where applicable, also the general public, will make rational and informed decisions based on them, has still not been sufficiently studied.

The working group on building the basic components of a syllabus for teaching life science data visualization to diverse target audiences was formed at the beginning of the seminar and collaborated throughout the whole week. The group participants came from diverse fields (both in academia and industry), which made the discussions very productive. The main objectives and goals of the working group that were set up at the very beginning can be summarized as:

• $\mathrm{■}$

  Recapitulation of past syllabus planning activities and attempts at building the syllabus;

• $\mathrm{■}$

  Discussion about the immediate steps that can lead to a tangible output of these efforts;

• $\mathrm{■}$

  Preparation of a prototype syllabus, including learning outcomes, and drafting an educational publication that summarizes the challenges and recommendations we identified;

• $\mathrm{■}$

  Setting up a repository for the participants of the seminar to share teaching materials in the context of the syllabus.

Within the BioVis Dagstuhl Seminar series, where this seminar is already the fourth one, participants addressed the issue of missing guidelines and syllabi for teaching visualization in life sciences several times. There were two main outputs coming from these efforts: the categorization of the existing techniques and methods into a complex table (see Dagstuhl Seminar no. 21401 report), structured according to the scale (spanning from atoms through tissues to populations, see Fig.5).

The table with techniques and methods then formed the basis for the second initiative, the Spring School on Biomedical Data Visualization (https://biomedvis.github.io), where experts from selected fields prepared a talk on a given topic, derived from the table. Although this school has been running already for three years (https://biomedvis.github.io/2021/, https://biomedvis.github.io/2022/, https://biomedvis.github.io/2023/) and collects study materials that were then made freely available, it is still lacking the systematic construction of a syllabus for teaching life sciences visualization that can be shared and followed.

Thus, within this Dagstuhl Seminar, we decided to focus specifically on the challenge of creating such a syllabus. We did a first pass through the literature search to get inspiration from other disciplines or from teaching visualization in general [1]. We identified the main target audience of the teaching activities, and then used that information to determine the direction of the syllabus construction. We summarized the expected core competencies and minimal prerequisites, and then we prepared the target learning outcomes and the syllabus itself.

The learning outcomes and therefore core competencies are modeled using the terminology and concepts of Bloom’s taxonomy [2]. When designing the syllabus, the following overall learning objective was considered: General DataVis literacy with a focus on bio-medical data. The students should gain the skills to read, analyze, and understand the visual representations used in biomedical domains. By these skills, they understand the relationship between visual analysis and the application domains from biology and medicine. Furthermore, students will be able

• $\mathrm{■}$

  to use the principles of human perception and cognition in visual biological and medical data analysis;

• $\mathrm{■}$

  to understand and use visual design principles;

• $\mathrm{■}$

  to know the basics and do’s and don’ts of visualization (including best practices);

• $\mathrm{■}$

  to critically evaluate visual representations of bio-medical data and suggest improvements and refinements;

• $\mathrm{■}$

  to apply a structured design process to create effective visualizations;

• $\mathrm{■}$

  to create low-level prototypes for bio-medical data visualizations;

• $\mathrm{■}$

  to create simple interactive (web-based) visualizations;

• $\mathrm{■}$

  to communicate visualizations (orally or written).

The syllabus consists of the following elements:

1. 1.

   Introduction: What is data visualization; Why do it; History

2. 2.

   Data types in vis (as abstractions, using bio examples)

3. 3.

   Color and perception

4. 4.

   Marks and channels

5. 5.

   Visual design principles and layouts

6. 6.

   Visual scalability (as abstraction, using bio examples)

7. 7.

   (Biology) data types

   1. (a)

      Genes and Genomes

   2. (b)

      Omics (quantitative data)

   3. (c)

      Phylogenetic Trees and Hierarchies

   4. (d)

      (Biological) Networks

   5. (e)

      Molecular Structures (3D and 2D abstraction)

   6. (f)

      Images (Medical images, Light and Electron Microscopy images, Gels and Plates, photos)

8. 8.

   Interaction and faceting (e.g., brushing and linking)

9. 9.

   Low-fidelity Prototyping

10. 10.

    High-fidelity Prototyping (e.g. with Observable)

11. 11.

    Evaluation

12. 12.

    Ethics

13. 13.

    Data-Driven Storytelling

14. 14.

    BioVis software tools (Circos, Cytoscape etc.) and critique

Already within the seminar week, the participants started to sketch the first version of the educational publication and discussed potential publication venues (for example, PLOS Computational Biology Education). Therefore, finalizing and submitting this publication will be one of the main priorities after the seminar.

The repository to share teaching material among participants of the Dagstuhl Seminar was set up using the mini-MOOC software of the Biolab group of Blaz Zupan. The repository is hosted under http://books.biolab.si/books/biomedvis. The overall repository is classified as CC BY-NC-SA 4.0, however, each contributor should also make sure that the copyrights of the shared material are clearly marked.

Last but not least, we discussed a possible future direction or extension of our working group, which is the creation of an interactive, open textbook. This book could encompass the following additional materials and approaches:

• $\mathrm{■}$

  lecture notes for each component of the syllabus,

• $\mathrm{■}$

  short accompanying videos,

• $\mathrm{■}$

  interactive content, including

  • –

    quizzes with progress monitoring and authentication, where needed,

  • –

    text-based answers with AI answer verification, hints, and critique,

  • –

    gamification,

• $\mathrm{■}$

  proposals for group activities during teaching.