Visual Text Analytics

Given the underlying annotation needs, we wanted to consider how to support data exploration and improve the process of data annotation, making it more enjoyable for the annotator. Considerations included gamification techniques, or showing the impact of the time and effort through visualizing the improvement of model quality and annotation coherence during the work. Ideally, we want to create a system that could perhaps visualize the expected impact of completing a particular annotation, in order to incentivize the human annotator to participate in the interactive active learning loop. This could be translated into three steps, when it might be possible to use visualization to support the annotator in exploring and annotating the text that is to be annotated.

Step 1: How well can an existing model be applied to your data? Try different models and visualize the areas in the texts (or the data points in the data) that are surprising to the model.

Step 2: How well can heuristics rules, stemming from the expert knowledge of the annotator be applied to your data? Will there be conflicting points in your labeling function that stem from these heuristic rules? How can the labeling, and the conflicting labeling decisions be visualized? Can weak supervision help you? We discussed a few concrete ideas of how sets and weak supervision [5] could be used for visualizing when different heuristic rules, annotator decisions, and the output of machine learning models conflict (see Sect. 6.2.2).

Step 3: How can the annotator be given control of the active learning process. For example, how can the classification certainty of an active learning model be visualized in order to let the annotator use their expert knowledge to be in charge of actively selecting what data to annotate? Visualize the data in different ways in order to guide the annotator in how to choose data points to annotate.

Even though general commercial labeling tools already exist, such as LabelStudio and Prodigy, we discussed the need for a tool that allows people who have particular expertise in their data to annotate and label meaningful words, phrases, syntactic patterns, and other items in their texts. Such a tool would differ from the traditional labeling tools in that experts often have knowledge about their data and the domain which they would like to communicate to the system and model. Doing so by repetitively adding and correcting categorical labels on words misses out on the opportunity to capture this information directly. For instance, people may have heuristics about the syntactic structure of specific sentences that imply meaning, paragraph lengths that help them detect authorship or intentional misspellings of words that reveal hidden meanings in the texts. A tool that allows people to translate such domain knowledge into heuristics, rules, or otherwise learned model parameters may be particularly helpful in cases where the data is small. For example, Mehta et al. [4] showed how close reading can be supported by a tablet interface that allows experts to annotate their text while reading. The system uses these annotations as a method for generating recommendations of relationships to additional unseen parts of the text. Kochtchi et al. [3] proposed a visual system for users to annotate relationships between named entities extracted from the text. The user-defined annotations are processed and annotation rules are automatically extracted and applied to unexplored parts of the text. In general, human-based annotation enrichments can be assisted by a visual analytics tool aiming to reduce the effort of generating label functions from scratch. This also requires high-quality annotations and the models. As models may be uncertain about labels or lead to conflicts in labels. This uncertainty and the conflicts need to be communicated and resolved. Ambiguities or uncertainties in labels (multi-labels) can be seen as overlapping sets [1] that could show which labels are problematic and whether there are any patterns in the problematic labels that would lead to new annotation rules [2]. On the other side, models could also be updated and improved based on human-injected knowledge using the visually-supported label function creation (cf. Figure 20). The comparison of the human vs. machine-produced models and the correction of the misalignment between them may be another open research opportunity.

Computational linguistics has a tradition in creating rich evaluation sets (e.g., Lehmann et al. [7]). Recently, interest in tests that are carefully designed to cover specific linguistic phenomena has regained interest in the community, e.g., the introduction of checklists for evaluation [9]. Such datasets can provide valuable insights into which phenomena have been learned by a model and which not. The downside of looking at carefully designed sets is that it is not necessarily clear how representative they are for naturally occurring data. We have benchmark data that has been created by annotating naturally occurring samples for most tasks, but these are small and it is often hard to tell how representative they are when using models in the “real world”.

The best of both worlds would therefore be to combine the two and investigate various linguistic phenomena in real-world data. Additionally, it can be informative to explore the actual occurrence of various phenomena to gain insight into whether the success or failure of models to deal with them correctly matters for real world applications. If we can identify and show what phenomena actually occur, we can also gain insight into whether existing benchmarks are sufficiently similar to the data we intend to apply our models to and, thus, whether reported results are indicative of how the model will perform on our data. Visualizations that would support such exploration of data would therefore be highly beneficial for the field.

As an example, consider a set of sentences describing people’s academic achievements. Items in such a set may include:

1. 1.

   Alice obtained her PhD in Computational Linguistics from Saarbrucken University.

2. 2.

   Bob majored from MIT Business School with a Business Administration Master’s degree.

3. 3.

   Cam has a PhD from MIT.

4. 4.

   Dan has a Master’s degree in Business Administration from Saarbrucken Business School.

One point of similarity between these sentences is their topic, and, more narrowly, the event they describe (a person obtaining some degree from some academic institution). They also share various forms of linguistic similarity such as syntax, for instance, all sentences are in active form, all have a prepositional clause starting with “from”, etc.

But there are also other levels of similarity. For example, the obtained degree in items (1) and (3) is both a “PhD”, and in items (1) and (2) and (4) is both “Master’s”. Moreover, both Master’s degrees are degrees in “Business Administration”, although the syntactic structure in which this information is realized differs: “Business Administration Master’s degree” vs. “Master’s degree in Business Administration”.

Similarly, the institution slot is similar between items (1) and (4) (“Saarbrucken University”) and between items (2) and (3) (“MIT”). However, note that the institution can also be grouped differently, noting the similarity between (2) and (4), which are both “Business School”. If we had also a sentence about a person obtaining a degree from LMU, it would have been similar to items (1) and (4) if we were to consider the country in which the academic institution is based. We can also consider grouping the person who obtained their degree by, for example, their gender. Note that this introduces a notion of ambiguity or under-specification: the gender of “Cam” in sentence (3) cannot be determined based on the text alone. The topic of Cam’s degree is also not specified. Visualizations that support exploring similarity at these levels can be of high value for non-technical end users as well as NLP experts. For non-expert users it is particularly important that the interface is intuitive and supports identifying potentially useful relations they are not a priori aware of.

The use cases include both corpus exploration on the individual text level (what is the structure of the individual items) as well on aggregations on the entire corpus level, and encompass either single arguments (“which German schools are represented in the corpus”) or collections of arguments (“show me all PhDs graduates in Business from Saarbrucken”).

If we consider the traditional InfoVis Reference Model by Card et al. [4], the first two steps of the overall process/pipeline involve transforming raw data into “data tables”, which for our purposes of visual text analytics could be compared to the definition of facets/frames that should eventually be revealed by the visual representation. One issue of relying on this conceptual model in the context of dealing with text data is the richness, complexity, and ambiguity of text in comparison with the standard case of n-dimensional multivariate tabular data, which was the default case for the InfoVis Reference Model and general-purpose visualization tools.

The typical scenario of representing text data with (interactive) visual representations that we can find in the related work is driven by the particular choice of user tasks (both from the point of higher-level analytical concerns, such as revealing the main concepts, similarities, or opinions in the text data, or lower-level exploration concerns, such as providing overview or supporting navigation over a collection of text documents) and the available data. Here, we could consider an example such as PEARL by Zhao et al. [13] which focuses on the exploration of emotions expressed in a single user’s messages on Twitter – the visual representations used by this tool rely on the particular set of computational analyses which were selected according to the task, but also the particular data formats and the respective constraints and peculiarities. PEARL could thus be viewed as a specialized solution, but cannot be considered a general-purpose text visualization approach.

Jigsaw [10] is another meaningful example to partly illustrate some of the data framing concepts (and issues) for text data representation. It was created to support intelligence analysts in investigative work to uncover hidden patterns. Its underlying data structures contain relationships between entities (people, organizations, location) and documents, which can be visualized in different ways, via a multitude of views. Each Jigsaw view was purposefully designed to support a specific facet of the data, and multiple views are needed to provide complementary perspectives on the data. Compared to the expectations of NLP experts, the design of Jigsaw lacks the support for representing the relationships between entities across facets and at multiple levels of hierarchies and scales present in the data (e.g., the semantic hierarchies, the entity relations, the structural progression of content, etc.)—such a general visual text representation technique would be very valuable for experts and lay users alike for a variety of use cases.

Considering the generalizability spectrum, the solutions such as Voyant Tools ²⁵²⁵25https://voyant-tools.org/ lie closer to the more general side, while providing support for several common linguistic analyses and rather lightweight visual representations, including word clouds [11], which seem to be appreciated by the general public and even users from particular knowledge domains, however, this representation is rather controversial with respect to the perceptual considerations and usability concerns, as studied in the related work [1, 5, 17], and it also does not address a number of possible higher- and lower-level tasks.

The considerations mentioned above led us to another iteration of our discussion, with the deficiencies of the existing visual representations of text for complex, yet generic scenarios motivating the need to reconsider the underlying data structures, analyses, and tasks besides the visual encoding or interaction technique on their own.

One of models/frameworks that we found to be relevant to such scenarios is the multilayer network model, which recognizes that the complexity of relationships between entities in real-world applications is better embraced as several interdependent layers/levels rather than a simple graph approach usually used in network visualization. A recent book publication provides an overview of the current state of the art in this emerging field [8].

For example, each word or phrase in a sentence can have a type (location, person, organization, …). Each type can be seen as a layer within a multilayer network. Moreover, the types (e.g., locations) can be part of a hierarchy (e.g., city $\to$ area $\to$ state $\to$ continent) that are also layers of the hierarchical structure. The similarity between the sentences can thus be seen as a multilayer network, where each aspect of the similarity is a type of element in the sentence. Each word or phrase in the sentence would be a node of the network and similarities would be connections between elements. The order of words/phrases forms a sequence which needs to be considered in the network visualization. Thus, the exploration of similarities and their semantics can be seen as an analogous exploration process in multilayer networks. The view on words, phrases, sentences, paragraphs, and documents relates to multi-scale exploration of networks. This opens interesting ideas and challenges from a visualization perspective.

One example of an existing visual analytic approach relevant to this discussion is FacetAtlas [3]. FacetAtlas embodies some of the principles of representing rich and complex information while allowing for dynamic exploration via a compact representation of multilayered concept networks extracted from text. Each layer represents a different facet, and relationships are computed via topic similarity; layers are superimposed and facet-level relationships are encoded as set overlaps.

Taking into account all such expectations from the experts and lay users, data models and computational approaches, and visual representation and interaction concerns, we envision one possible way forward as a flexible pipeline of data transformation and representation for text data (see Figure 21), which starts with capturing the facets and relationships on the data and domain-specific analyses side (which might include ordinal, n-ary, and heterogeneous relationships – which can also be related to the multi-layer network framework) and providing (parameterized) visual encodings that reveal the contents of these facets, but also patterns of interactions across these facets/frames.

In large-scale NLP models (e.g., BERT, GPT-2), one mechanism to probe for biases is to utilize prompts to try and understand if the model was associating particular roles more frequently with certain protected classes (e.g., gender, ethnicity). However, this probing is often highly sequential and can be ad hoc. It does not necessarily have the capability to reveal hidden biases; instead, it may lead to only the discovery of expected biases (as opposed to unexpected biases). Current approaches for understanding if biases exist in such models include comparing completions for two prompts. Observing the visual comparison of the completions can illuminate potential biases when references are made in the text that indicate forms of social biases. However, these visual comparisons often lead to follow-up questions about specific words or phrases. Current tools do not support this iterative refinement and exploration. We discussed how visualization tools could be designed to foster this iterative exploration, including:

1. 1.

   Comparing multiple models trained on different datasets for a single prompt.

2. 2.

   Comparing multiple prompt completions for a single model.

3. 3.

   Visually constructing complex prompt completion templates through the use of SentenTree or WordTree visualization techniques.

4. 4.

   Visualizing the embedding space of prompts for a given model.

It is at this intersection where visualization could serve as an effective mechanism to support the exploration of biases.

1. 1.

   Can text visualization help us to identify problems in the training data (e.g., various types of biases throughout the complete pipeline)?

2. 2.

   Can text visualization facilitate prompt engineering?

3. 3.

   Can interactivity support identifying bias in large datasets?

4. 4.

   Can interactivity support framing perspectives and stances on bias?

5. 5.

   How can we detect, warn and communicate about potential bias shifts over time?

To that end, it would be imperative to explore which visualizations are suitable for:

• $\mathrm{■}$

  Communicating bias detected by automatic data processing.

• $\mathrm{■}$

  Identifying bias, its size, and aspects in the original dataset.

• $\mathrm{■}$

  Discovering hidden and unknown bias in large corpora.

• $\mathrm{■}$

  Comparing bias between models.

• $\mathrm{■}$

  Steering models towards less bias.

• $\mathrm{■}$

  Letting the user remove or mitigate bias (e.g., by selecting and checking datasets without bias).

• $\mathrm{■}$

  Exploring the “path” of bias from a dataset, via model to visualization – source, size, amplification, and transformation of bias through the pipeline.

Throughout the design process, considerations of the user’s background (e.g., NLP experts or application domain professionals, e.g., social scientists) should be considered. Furthermore, the visual design must be careful not to suppress or amplify detected biases from the data and models. A key step to this is identifying which biases are caused by the data representations, visual representations, and cognitive biases of domain experts using the visualization [5, 1]. Another is understanding how biases are introduced, changed, removed, and even elicited in the design process and resulting pipelines (Figure 27). Finally, such designs would need to be systematically evaluated to explore the potential for unintended biases (pointer to evaluation group).

Language models, such as GPT-3, can be used as a reflection upon and summary of a large corpus of text (and also the behavior of different models). We discussed using such models to compare various collections of texts, for example, training multiple instances of the model across a variety of related corpora, in order to compare the completions the variously trained models would make. Previous work [2] has explored the comparison of corpora of tweets mentioning “China virus” to those using the term “SARS Cov-2” by looking at the variety of completions generated by the models when given carefully selected prompts. One of the challenges of this type of research is to design the prompts that are used to generate the completions. This requires some domain knowledge and expertise in prompt design, as some prompts will limit the completion space significantly. For example “Dr. Fauci is a _” limits the space significantly more than “Dr. Fauci is _”. How do we support the exploration of the prompt space in an interactive manner? We discussed a design in which a prompt could be used to generate a set of N (e.g., 1000) completions across multiple models. These completion sets could be turned into vectors in which each unique completion is given a cell and the cell is filled with the probability of the completion for the given model ³⁰³⁰30https://pair.withgoogle.com/explorables/fill-in-the-blank/. The distance between these completion vectors could represent how different (or biased) the models are on the given prompt. Given a set of possible prompts, they could be evaluated through such a pipeline to help guide an investigator toward which prompts are most likely to show a signal across models. One avenue for generating such a list of prompts could be to create a sentence with an open “slot” and a rule for its completion. For example “[NAME] is a _”. The NAME slot here could be filled by mining the input training data for the top N names in the collection. This would generate N prompts which could be fed through each language model, generating the completion vectors in order to compare. In an interactive loop, one could choose a prompt, view the completions across models, then use those completions as starting points to generate new prompt rules as shown in Figure 28.

The NLP community makes extensive use of vector embeddings which relate to words, sentences, paragraphs, documents and other text elements. The vectors tend to be high dimensional (300 dimensions or more) and are often projected down to two dimensions for visualization purposes. Such visualizations then support tasks, for instance, identifying clusters in the data, inspecting the spatial structure of the space (which words/sentences relate to each other, and how are the relations organized globally), and identifying similar items or neighborhoods of similar items. Another use case is the debugging of learned representations [3].

In most related research, visualization systems use projections like PCA, t-SNE, or UMAP [5, 4]. However, projecting the high dimensional vectors to two dimensions is inherently a lossy process, and the resulting visualizations are often misleading in various ways, which in turn may lead researchers to incorrect conclusions. Interactive tools such as t-viSNE [2] can help to alleviate some of these issues, but interactivity also requires the user to know specifically what kind of distortion they are looking for. A static visualization that effectively shows additional information, which is usually lost in standard projection visualization, can go a long way in supporting more accurate interpretations of the results. In particular, a static visualization can be used to highlight the points in the visualization where an interactive exploration is required. What we aim for in our static design is to support this requirement.

There are two general types of errors that are caused by projection of embeddings to lower dimensions [1]. First is the geometric error that distorts the distances but keeps the neighborhood of points intact. Second is the topological error that disrupts both the neighborhood and the distances. Examples of geometric errors are stretching or compressing, which increase or decrease the distances between embeddings compared to the original space (see Figure 29, (a) and (b)). Examples of the topological errors are gluing or tearing, which merge neighborhoods or disrupt a neighborhood into multiple neighborhoods, respectively (Figure 29, (c) and (d)).

Based on our discussions, we find it valuable to study the nature of noise in the data. In order to be able to communicate how much the visualization of various clusters of the data can be trusted once projected into the two-dimensional space, we first need to be able to measure to what extent the clusters are well-formed since the distortions visible in the two-dimensional visualization can be both caused by the projection or by the inherent noise in the data.

Once we have a way to quantify the noise relative to the general geometry and topology of the data, we aim to proceed with the visualization aspect. Our solution is to use cartographic techniques to depict both geometric and topological distortions in 2.5 dimensions. We have composed a list of visual encodings to help communicate the distortions in the visualization of data (cf. Figure 30).

Compression error: In order to communicate the geometric distortions in the visualization caused by compression, we propose to use contour lines, which are often used to communicate height differences in cartography. The areas that are compressed can contain more contour lines, indicating that in the original space there is larger distances between points. The number of contour lines can communicate the extent to which an area is compressed.

Stretching error: The distortion caused by stretching can be communicated with lines connecting the areas that are stretched out in the visualization. The extent of stretch can be communicated by means of arrow heads placed on the line. More arrow heads indicate smaller original distance between two areas.

Gluing error: The gluing error is caused by two or more neighborhoods being mapped to the same position. We distinguish between the neighborhoods by visualizing them in overlaying layers. One can think of this as a multistory parking garage where every story is representing a different neighborhood in the original space.

Tearing error: The tearing of the same neighborhood into multiple neighborhoods is visualized by means of “ghost points” where we duplicate the separated points with a more transparent visualization (ghost-like) and put them close to the points of the original neighborhood. Another way to represent this error in the visualization is to make use of color-coded “wormhole” shapes that indicate which neighborhoods are originally connected in the initial space.

Text data is rich and ambiguous, and there is no single canonical data representation or universally accepted interpretation. Ambiguity can be found in the text at several levels, from the grammatical level (e.g., run/run) to the semantic and conceptual level (relations between concepts and what the concepts mean, e.g., let’s go to the bank). Ambiguity in language may lead to different interpretations of the text by the annotators, which may affect their decisions. But even in cases where annotators have a similar or even identical understanding/interpretation of the text, their annotation decisions may still differ in the end due to, for instance, their different backgrounds and world perception.

In the cases where the divergence between the annotators’ decisions is high, the task can be more closely specified (and/or simplified) and the annotation guidelines can be further refined, until a desired level of inter-annotator agreement (IAA) [2] is reached. Traditionally, high IAA implies higher data quality (in terms of reproducibility of data labeling). But due to the nature of language itself and the various meanings that exist, assuming high agreement (and by consequence, a single ground truth) is at best an idealization—this neglects genuine disagreement that can provide valuable insights about language use and interpretation. For some text interpretation tasks, too restrictive guidelines might even hide the fact there is no real ground truth for this particular task.

There are many examples of visualizations that create an overview of the results of text annotations, in particular for visualizing the co-occurrence of different annotation categories. Approaches range from using standard visualization techniques (e.g. bar charts, scatter plots, Sankey and chord diagrams, and treemaps [11] to develop new visualization techniques specific to the text annotation task [12]. To the best of our knowledge, however, user interfaces for showing the difference in annotation categories between different annotators mainly focus on (i) individual annotation instances (e.g., Yimam et al. [14]), or (ii) IAA-statistics for the annotators, such as matrices showing the inter-annotator agreement for pairs of users (e.g. Grosman et al. [8]). That is, the aim of these visualizations is not to provide the user with an overview of the differences in annotation choices. This is natural, as these interfaces typically are created for curators who use them for annotation adjudication [4]. That is, curators who aim at merging each conflicting annotation into the one correct annotation and not at exploring the annotated dataset in order to gain insights into the reasons for disagreement, for instance, insights into what extent there is a single ground truth for the annotation task.

We believe that visual text analytics provides opportunities for mutual benefits of research within NLP and text visualization to provide insights into the nature of the disagreement. In addition to visualizing disagreement between annotators in terms of labeling divergences, there are additional factors that contribute to disagreement and could be visualized. For instance, the annotators could be given the opportunity to not only annotate, but also specify the level of uncertainty for the annotation, or to opt-out of annotating an instance due to it being too difficult. Also the time it takes for the annotators to make an annotation decision could be gathered as an information type showing the difficulty of the task.

An NLP system and its visualizations should be evaluated in light of these four dimensions: target audience (who?), list of tasks to support (what for?), scales to consider in the tasks (linguistic objects of interest), and models supporting the tasks at the given scales (how?).

Audience: We start with the end-user who defines what they want to perform in the system (goals). We can define different User groups (from larger to smaller in terms of the system knowledge and expected functionality): Debugger or Developer of the system, Linguist or NLP Expert in charge of the modeling, Domain Specialist using the system with no particular knowledge of NLP or CS, and General User, where each member of a group has the ability to perform the tasks of their own group and the smaller groups contained within. Considering the use of visual text analytic approaches for applications with real-world data and real-world implications [3], the importance of carefully considering the intended audience and their goals for the design and evaluation of visual text analytics cannot be overstated.

Tasks: We then define the tasks that will achieve each of the users’ goals. A task is the combination of a User+Verb and attributes; it can lead to a side effect (change in the visualization) or to a result (list of items). For example, overview (depends on the visual representation, no parameter), query, search (return a list of items), cluster (visualize groups and return a list of groups), etc. In addition to such general prototypical tasks, specialized tasks can be domain-specific, but also specific to the linguistic model (aimed at the NLP specialist) or to the programmer, allowing the respective user to understand the internal data structure of the NLP and visualizations. These tasks are not useful for the domain expert or to a lay user, for instance. Further potential tasks can be identified in the previous works on the more general task taxonomies in information visualization and visual analytics (e.g., [1] and [5]) as well as the sources focusing on text visualization and visual text analytics, such as the work by Tofiloski et al. (2015), for instance.

Scale: Each task can be applied or affect the text at a different level of granularity: word, phrase, entities, tree structure (syntax), proposition (semantics), sentence, section, paragraph, document, and corpus.

Models: Layers of linguistic information are encoded and structured differently: parse tree, part-of-speech tags, XML-encoded text (metadata), stand-off annotation (e.g., relevant spans in text), topics, etc. The model influences the tasks in the sense that it allows some tasks to be done more or less accurately, and with diverse levels of uncertainty, quality, errors, and artifacts. The models also constrain the scale of possible tasks.

Given the complex, multifaceted, and potentially ambiguous nature of text, there is a wide error range in NLP outcomes, as we cannot expect NLP systems to work perfectly in every circumstance, e.g., for each task and for each dataset. This is something users must take into account when leveraging text visual analytics systems for a particular problem, and is reflected in the extent users trust automated outcomes and take them into account. User trust is an evolving process [9], and it reflects perceived system performance over use and the risk associated with decision-making. For example, spelling and grammar correction in documents is a relatively mature problem with low impact for errors, and something users are generally comfortable delegating to automation. On the other hand, automatic summarization of patient records for clinical decision-making is an open research problem with significant consequences for wrong medical choices.

Apart from design considerations to foster adequate trust, from an evaluation perspective, it is also important to consider how trust can be measured, and whether it matches NLP capabilities. There are many challenging components to measuring adequate trust. It entails (a) an understanding of the limitation areas of NLP algorithms within a particular application, (b) assessing user perception of the limits of automated outcomes, (c) assessing how well user perception matches real capabilities of automation, and (d) how trust levels evolve over time. While there is research in understanding and designing for trust [13], including visual (text) analytic systems [7, 6], strategies for measuring trust are still not widely adopted in the evaluation of visual text analytics systems. Moving forward, we posit that the community should consider developing and fostering the use of standardized and easily applicable metrics and instruments (e.g. questionnaires) for measuring trust, which should be applicable at different stages of use (i.e., measurable over a period of time).

One of the challenges of evaluating visual text analytic approaches is related to the breadth and variety of concerns regarding computational and interactive, human-centered aspects of such approaches and tools. The efforts focusing on systematic analysis, categorization, and formulation of guidelines for evaluation (cf. the work by Isenberg et al. [10]) would thus be beneficial for researchers, practitioners, and end-users of such visual text analytic tools. Here, the topics and dimensions would include an interdisciplinary analysis/view and include both NLP/ML-focused aspects (e.g., the methods and measurements of intra- and inter-annotator agreement, or the measurements of NLP model performance for a given task beyond standard ML metrics such as accuracy or F1-score) as well as visualization and interaction-related aspects (e.g., the methods of usability measurement and long-term adoption observation). The concerns and dimensions of data annotation, computational models, text visualization approaches, and users’ trust discussed above are all highly relevant to such systematic analyses. While the complexity and variety of visual text analytic problems would most likely not allow us to recommend a single prescribed method for validation, the outcomes of the work on systematic analysis would result in a collection of guidelines that would benefit the research efforts and applications.