What You Hear is What You See? Integrating Sonification and Visualization

We defined DMR tools and libraries as complete systems that enable Rendering from Data through a systematic representation Method (e.g., mapping, model). We can differentiate between tools and libraries that facilitate this complete process, and components that support parts of this process. A full AVAID could for instance be setup as such:

• $\mathrm{■}$

  Data: a component to read/filter data (e.g., numpy for loading csv files),

• $\mathrm{■}$

  Mapping/Model: a component for mapping data (e.g., a python function to create a Csound score file),

• $\mathrm{■}$

  Rendering (Auditory Representation): CSound is a sound rendering component usable for sonifications that focuses exclusively on sound rendering (R).

• $\mathrm{■}$

  Rendering (Visual Representation): for instance the data could be plotted using scalable vector graphics (SVG) in the web browser.

We identified the following dimensions as useful for differentiating between AVAID tools and libraries:

• $\mathrm{■}$

  Platform (e.g., web, native, specialized)

• $\mathrm{■}$

  Environment (e.g., jupyter notebooks, IDEs)

• $\mathrm{■}$

  Self-contained DMR tools vs component tools vs. libraries

• $\mathrm{■}$

  Interface abstraction level (e.g., GUI, text, code)

• $\mathrm{■}$

  Popularity

• $\mathrm{■}$

  Licensing (open/closed)

• $\mathrm{■}$

  Learning curve vs. perceived value (“Return on time invested”)

• $\mathrm{■}$

  Capabilities / versatility

• $\mathrm{■}$

  Data type support (e.g., map, graph, tabular)

• $\mathrm{■}$

  Data input/output, import/export

• $\mathrm{■}$

  Output capabilities

• $\mathrm{■}$

  Static vs dynamic output (interactivity)

• $\mathrm{■}$

  Visual rendering types (pixel graphics, vector graphics)

• $\mathrm{■}$

  Rendered audio types (sampled audio vs. score / higher level representation)

• $\mathrm{■}$

  Synchronization (e.g., between auditory and visual rendering)

• $\mathrm{■}$

  Collaborative editing or production features

• $\mathrm{■}$

  Extensibility

We believe the AVAID tooling space will need to cover a spectrum of different interface abstraction levels and levels of capabilities. This is analogous to the existing tooling space in visualization. Observing the current tools and libraries state of the art, we identified the need for audio-visual tools to develop AVAIDs, and came up with a narrow research question: RQ1: “What are the needs/problems that should be solved by AVAID tools across different interface abstraction levels?”

• $\mathrm{■}$

  functionality (services) matter more than rendering systems

• $\mathrm{■}$

  suitable rendering abstractions (to avoid engine specific lock-in)

• $\mathrm{■}$

  shareability/extensibility to promote community (presets, templates, extensions)

• $\mathrm{■}$

  enable (best) integration of the visualization and sonification modalities (synchronization, Multiple Coordinated Views (MCV) between modalities,…)

• $\mathrm{■}$

  Input/Output – Import/Export – Formats

• $\mathrm{■}$

  Users lacking knowledge of visualization and sonification – good output anyway – capture intent and context

• $\mathrm{■}$

  real-time rendering for enhanced interactivity

• $\mathrm{■}$

  continuum of flexibility versatility in data handling, method, and rendering

• $\mathrm{■}$

  data types (map, graph, tabular) and data size (performance)

• $\mathrm{■}$

  production speed (prototyping)

• $\mathrm{■}$

  collaborative production

We discussed how prototyping relates to AVAIDs and libraries and identified three dimensions that form a triangle of trade-offs:

• $\mathrm{■}$

  precision (or fidelity) (e.g., mock or subsampled data, imprecise representation method, or sketchy rendering)

• $\mathrm{■}$

  aesthetics

• $\mathrm{■}$

  speed (time needed to create prototype & products)

We acknowledge the speed quality trade-off in prototyping, speed referring to inverse time needed to create the prototype/system. Along the quality direction we usually face another trade-off, the one between precision and aesthetics, where the former refers to how accurately information is conveyed to the users and the latter refers to how pleasant and aesthetically the media is judged. Altogether, a ternary “prototyping space” emerges in which different AVAIDs can be positioned. AVAIDs may serve as an enabler of this process through their capabilities.

We first discussed whether evaluation of AV idioms is always necessary. Some projects that involve AV idioms may not need to be evaluated. For example, the exploration of the design space and novel techniques may inherently be valuable and not need to be evaluated. The fields of sonification and visualization exhibit different trends in evaluating their research. While only few sonification papers include an evaluation [4], different forms of evaluations are widespread in visualization, which is (partly) due to publication venues often requiring an evaluation in order to publish. Of research on audio-visual data representations about two third include an evaluation [6]. Evaluation has the potential to reveal how an AV idiom is useful. Hence, we urge researchers to consider evaluation as part of their process, not merely as a formality.

There is currently no framework or method for the evaluation of AV idioms specifically. Instead, we identified evaluation methods from visualization, sonification, and HCI. These methods can be used and adapted to suit the evaluation of AV idioms.

First, we like to point out evaluation methods from HCI. Evaluation methods from HCI are typically user-centered, meaning that the user is central to the design process. HCI distinguishes between formative and summative evaluation methods. Formative evaluation methods on the one hand are considered from the beginning and takes an iterative approach to the design of an artifact [27]. Visualization research has adopted formative HCI methods in visualization design studies [19, 25]. Summative evaluation methods on the other hand, are conducted at the end of a development cycle and tests how well a design performs. The design is often compared against benchmarks. Research in sonification and visualization often evaluates performance measures such as reaction time and error rate [15], but learnability [2], working memory load [22] and psychophysics [8, 23] has also been investigated.

From HCI and related fields, we identified the following evaluation techniques that would be useful for researcher to explore for the evaluation of AV idioms: (1) Empirical evaluation, (2) heuristic evaluation, (3) third-wave HCI methodologies, and (4) alternative methodologies.

1. 1.

   Empirical evaluation techniques rely on experimental procedures to answer research questions. Experimental procedures can collect and analyze qualitative data such as in grounded theory [16], or quantitative data as long as the experiment is replicable. Validation of a data representation by comparing it to a second representation typically are empirical as well. Empirical evaluation techniques are widely used and accepted in visualization [17] and have been applied to sonification as well. Measures used in empirical evaluation vary widely and depend on the aspect a researcher would like to investigate. Some examples include standardized questionnaires such as the NASA-TLX on perceived workload [14], visualization specific questionnaires [10], questionnaires assessing usability of AV idioms [24], the visualization literacy questionnaire [18], among others. Visualization research has a rich tradition of mainly empirical evaluation techniques that could be adapted to evaluate AV idioms. For an introduction we could like to point the reader to An Introduction and Guide to Evaluation of Visualization Techniques Through User Studies [11] and Patterns for visualization evaluation [5].

2. 2.

   Heuristic evaluation can identify usability issues in an artifact throughout the design cycles [20]. A heuristics evaluation involves a group of experts that examine and judge an artifact, often according to a list of principles. In HCI, many heuristic principles exist [21]. However, specific visualization heuristics have been developed [9]. We think that findings from psychoaccoustics and visual perception research could inform the design of AV idioms and could act as heuristics. There is an opportunity for future research to identify and formulate heuristics for AV idioms specifically.

3. 3.

   Third-wave HCI methodologies go beyond the user-technology relationship and recognizes that interaction is a form of meaning creation. Methodologies thus focus on experiences and meaning-making in a broader context, beyond the artifact and its user [3, 13]. One such methodology is phenomenology: “a philosophical movement that seeks to analyze the relations between human beings and their world rather than as a method for describing reality” [26].

4. 4.

   Lastly, we would like to point out alternative evaluation methods. We think such methods could be valuable to evaluate AV idioms too. For example: critical self-reflection, which is used in humanities [1, 7].

Considering evaluation methods from adjacent fields is a good start. Such methods can be adapted to evaluate AV idioms. However, evaluations should be tailored to specific research goals. Hence, we identified special consideration for the evaluations of AV idioms. We came up with the following decision checklist to guide the evaluation of AV idioms:

1. 1.

   State the intent of the AV idiom and possible hypotheses and assumptions: A design is usually created with intent. This intent should be testable. Additionally, the researcher should state their assumptions and expectations. These can be formulated as testable hypotheses.

2. 2.

   Define specificities: Define who the intended user of the AV idioms is, what task(s) they will perform with it and in what context they will use it.

3. 3.

   Summative vs. Formative: A researcher should decide how they intend to evaluate their AV idiom before they start design. Do they intend to involve the user throughout development? Then formative evaluation methods will apply. Or will the researcher compare the AV idiom once it is created? Then summative methods are better suited.

4. 4.

   Define “good”: Researchers might want to evaluate if their AV idiom is better than an alternative representation. In such a case it is important to define what it means for the AV idiom to be good. What good means depends on the intent and specificities of the designed AV idiom. For example, should the designed AV idiom lead to fewer errors or should the designed AV idiom be more memorable.

5. 5.

   Choose methodology and method: The researcher should choose a suitable method based on the previous points. In both visualization and sonification, empirical methodology is often chosen. Empirical methods are especially suitable for summative approaches. This methodology include both quantitative and qualitative methods that can be used to validate a design. However, we also want to point to alternative methodologies from other fields. For example critical self-reflection which is commonly used in humanities, phenomenology which is widely accepted in design, and third-wave HCI methodologies.

6. 6.

   Choose measures: What to measure should be chosen based previous points, especially on the methods chosen and what “good” means to the researcher. For example, a researcher wants to know if their AV idiom conveys the same information to sighted, deaf and blind users. In such a case, the researcher might chose a qualitative method to measure the users’ comprehension of the AV idiom.

We recommend that researchers plan how to evaluate their AV idiom before they begin development. An AV idiom is often designed with a specific intent. This intent needs to be stated, as well assumptions and hypotheses. How to evaluate is guided by these intentions, assumptions and/or hypotheses. For example, an AV idiom is designed with the intent to help children learn about space. The evaluation should then test the learnability of the AV idiom with children.

Furthermore, design often requires specificity. AV idiom are often developed with a specific user in mind who will use the design in a specific context to perform determined tasks. These specificities need to not only be considered when designing the AV idiom but also when planning the evaluation. In the case of a summative evaluation, especially when looking to validate an AV idiom, a baseline must be chosen when evaluating an AV idiom. We concluded that AV idioms should be tested as an integrated system when exploring the design space of AV idioms. Hence, as a general rule of thumb, AV idioms should only be compared to alternative AV idioms. This is because Gibsodian perception [12], considers human perception as an active integrated system, rather than as passive and separate channels of sensation. This theory of ecological perception has enormous implications for the evaluation of multisensory interfaces! The whole is more than the sum of the parts. Evaluating channels separately will not provide a true description of the whole system. Furthermore the senses provide different kinds of information and a simply redundant interface is the least likely to provide perceptual benefits and advantages. Hence, it is not useful to compare visualization to AV idioms. However, we identified exceptions. For example, a researcher might compare an AV idiom to a visualization if they want to explore if the combination with sonification adds something useful to the visualization. Or if an AV idiom intends to be accessible for blind, sighted, and deaf individuals.

There are, of course, further factors that will influence the results of an evaluation. On the one hand, the background and personal factors of the user will play a role (e.g., musical training or auditory impairments). On the other hand, the technical setup will play a role (e.g., room/lab settings and the output device). Therefore, such factors need to be documented correctly and thoroughly considered when drawing conclusions based on the evaluation results.

However, we see the need for evaluation methods specifically developed for AV idioms. Further research should identify, adapt and extend evaluation methods from HCI, visualization and sonification useful for the evaluation of AV idioms. For example, how do we adapt evaluation methods to evaluate fully redundant AV idioms, fully complementary AV idioms, or mixed redundant AV idioms?