Dimensions of Examples: Toward a Framework for Qualifying Examples in Programming

When we consider programming to consist of several interleaving and repeating activities, examples play different roles in different activities and have different lifetimes. From our example-focused perspective, we consider three types of activities:

Exploration.

Activities to better understand or recall how to use data, tools, and abstractions to approach a programming task. In exploratory phases, programmers benefit from reading examples in documentation, on Q&A platforms, from advice given directly by peers, and increasingly from generative AI chats. During the activity, they might increasingly come up with their own ad-hoc examples to try something out or explore a problem in a debugger using a run-time example. Hence, exploration narrows down the lifetime of examples, as initial knowledge might be derived from long-lived media, while the eventual verification of whether newly written code performs as expected might happen with a non-persistent ad-hoc input.

Evaluation.

Activities where programmers put a (step towards a) solution to test. In contrast to exploratory activities, where examples tend to increase programmers’ understanding, the role of examples now shifts toward providing feedback and verifying existing behavior. Typical examples are those found in unit tests or benchmark data. Programmers might come up with their own inputs they like to test or resort to those specified in the requirements. Edge cases can become more prominent.

Explanation.

Activities that accompany the dissemination of programs and knowledge about them. Programmers might write or edit examples in comments, tests, or documentation that should serve a wider audience and serve as input for exploration activities.

We understand these activities to be recursive, as exploratory activities often entail a rapid succession of smaller exploration and evaluation steps, possibly discarding experiments before moving on.

We argue that knowing which activity an example should support is crucial to understanding not only its scope but also the tooling involved.

Although primarily geared toward verifying program behavior, unit tests are a code-level mechanism to specify examples, name them, and make them executable through a test runner. When writing unit tests, programmers can benefit from existing features of their programming environment, such as syntax and error highlighting, code completion, or debuggers. To better modularize test logic, many testing frameworks offer constructs to implement fixtures, which are reusable and reproducible example instances of certain system components. The language PyRet¹¹1https://pyret.org/ (last accessed 2025-05-05) even has syntactic support to specify exemplary tests using a where-clause after a function definition. Examples specified in tests are required to be executable, which sets a lower boundary on their complexity (e.g., requiring authors to substitute parameters through mock objects when they cannot be left out entirely) and frequently cover edge cases that are not necessarily representative but can still serve an important illustrative function.

The use of examples in programming motivates the design of environments that specifically support examples beyond a program’s regular code. Several designs have been proposed that treat examples as first-class entities in the programming environment. These examples can then be used as entry points to provide fine-grained dynamic data that helps explain and evolve programs by example. A few examples of such systems are described below:

Exemplars.

In the Newspeak programming language, methods can be accompanied by example parameters and class instances (Exemplars [3]) that enable programmers to always have a run-time instance of that method available.

Example-centric Programming.

By tracing the execution of code examples through the code they illustrate, Example-centric Programming displays the evolution of concrete data next to its relevant code [5].

Babylonian Programming.

Babylonian Programming [18] introduces editor elements to specify and select example instances and parameters for a section of source code (e.g., a method) and provides probes – an in-situ editor feature – that display dynamic data yielded by executing the active examples at the code location where they occur. This mechanism works across module boundaries. It can circumvent side effects and simulate user inputs using replacements that insert an exemplary return value for a selected expression. Implementations have been presented for Squeak/Smalltalk [19], JavaScript in Lively4 [18], the polyglot GraalVM runtime in VSCode [16], and the Godot game engine [11].

Kanon.

Kanon [17] is a domain-specific example-based live programming environment for data structure manipulation. The consequences of each code statement are explained graphically.

SnapPy.

The SnapPy [6] tool for Python takes example-based programming one step further and not only presents the current state of an example at run-time but also allows programmers to demonstrate by example what should happen with the current state, prompting the system to synthesize new code. This is an instance of programming by example [8].

With the introduction of example-aware UI components, programmers no longer need to care about where and how to implement them at the code level and gain the capability to view dynamic data within their code editor. Editor components that are not restricted to text can use rich visualizations, e.g., plotting a value over time rather than printing it as a number, viewing the content of a drawing buffer, or presenting structured data as tables or collapsible trees.

With closer proximity to code and fine-grained entry points near the code of interest, we gain the capability to provide more immediate feedback on every code change, as we can scope execution to just the relevant part rather than the program’s main entry point. Correspondingly, tools can support more experimentation.

We define complexity as the amount of data or actions that the example contains. Simple ways to increase complexity are including more items in a collection, constructing a deeper graph or tree structure, or using longer strings. Often, the least complex example for functionality is an edge case and not necessarily representative (e.g., the empty list), while adding complexity has diminishing returns (e.g., sorting a list of 15 items is rarely more insightful than 10 items), which implies the existence of a sweet spot that can change per use case.

Exceptionality corresponds to how far outside the expected or typical distribution an example is situated. Examples that rank highly in this dimension will often be called edge cases. They can be useful to understand how the system handles error conditions, e.g., by providing invalid or incomplete inputs or constructing a rarely anticipated precondition like missing file permissions. The more unlikely or unimportant the example for the considered domain, the higher its exceptionality.

What constitutes exceptionality depends on context: the empty list is more exceptional when computing the arithmetic mean, and less exceptional when used as the default for an optional parameter in an API. In the security domain, what looks like an exceptional example might as well illustrate a typical attack vector that the implementation must mitigate.

This dimension is connected to complexity, as examples that are too simple or too complex tend to be exceptions.

An example can explain the same logic using meaningless inputs or data that is more representative of the domain. Examples include using familiar names of people, places, products, etc., to identify domain objects compared to using “foo” and “bar” or “Lorem Ipsum”. In situations where some logic becomes part of the larger system, e.g., a sorting algorithm being used in a calendar app, sorting a list of integers might still explain the sorting routine in isolation, but sorting a list of events by closest date would illustrate its use in the context of the domain.

Whether closeness to the domain is better depends on the context in which the example is used: for code that parses a user’s name, using “First Last” as the example string can help during debugging to see if first and last names have indeed been parsed correctly into their respective variables. In this case, the example would be closer to the implementation than the domain. On the other hand, not using realistic examples can cause bias, e.g., programmers not expecting non-Latin characters in a name or neglecting middle names.

Scope complements the complexity dimension by describing to what degree the example connects and integrates multiple aspects of the illustrated concept. For example, a plotting library could illustrate in a single example the use of different colors, line styles, markers, logarithmic axes, and labels³³3For concrete examples, see the Matplotlib documentation at https://matplotlib.org/stable/gallery/index.html (last accessed 2025-03-21). While this often increases complexity, complexity itself can be increased without affecting scope by just adding more data without varying plot parameters.

Demonstrating the interplay between libraries, adding context that might be hard to deduce for novices, and explicitly specifying optional or default parameters can all add to scope. In this way, scope determines which related aspects beyond the example’s core are included.

The opposite would be reductionist examples that explore a single aspect in isolation. To understand multiple aspects, multiple examples would be needed. Although this often results in simpler examples, dependencies and interactions between parameters would still remain abstract.

Making examples executable often requires them to be as concrete as possible, down to instantiating primitive data types.

When executability (see Subsubsection 3.2.2) is not a concern, examples can gradually abstract away irrelevant details using symbolic placeholders (that the user needs to fill in before execution). For example, for parsing a unified diff we might explain the header of a change as @@ -2,3 +2,4 @@, but also as @@ -a,b +c,d @@ or @@ -range_removed +range_added @@, increasing the abstractness of our example string. Depending on the parsing logic, we might even succeed in executing the more abstract examples up to a point.

Concrete does not necessarily mean primitive data as in strings or numbers: One could also explain monads using the example of the list type – in this case, concrete “data” would actually mean functions (the function to wrap an element in a single-item list and the flatmap function constitute concrete versions of the unit and bind operators).

In testing, more abstract examples serve as inputs for property-based testing or concolic execution [20]. In such settings, symbolic parts of an example can still be considered during execution.

This dimension is the degree to which the example relies on non-code explanation. Depending on whether the example is viewed as part of external documentation, code, or within an example-based programming environment this might take the shape of (rich) text surrounding the example, code comments, or the use of extra features introduced by tooling (e.g., a descriptive name as in some Babylonian programming environments).

Executability corresponds to how little manual effort stands between seeing the example and observing its effects. In test suites, notebook-style programming, and example-based programming environments, examples are usually executable by construction. Examples more removed from the programming environment might be one copy-and-paste action away from running or might need adaptation or preparation (e.g., when dependencies or preconditions are not made explicit in the example or are only stated textually).

We observe that tools can greatly increase the executability of examples by hiding the effort of setting up the context in which the example can run: While writing a unit test often requires a separate module or file, importing the unit under test, defining a test case class or method, writing setup code, and invoking the test runner, an editor supporting Exemplars or Babylonian programming can provide a user interface where any parameter can be directly set to an example value right where the method is defined and execution results are automatically obtained by running the code on every change.

In the cognitive dimensions of notations framework [2], this dimension describes the capability of a notation to vary and convey information without changing the semantics of a program, such as positioning of elements, highlighting, or formatting. Although currently underused in example-based programming environments, such features promise to make examples more comprehensible. For example, letting users color a part of an example could allow them to convey importance, but also allow dynamic tooling to reuse this color to highlight outputs related to that part of the example.

This dimension maps the degree to which the example can be “experienced” through on-the-fly changes. Explorable Explanations [22] are a proposal that encourages active reading by experimentation with examples included in a document. Kanon [17] supports domain-specific visualizations of data structures that can be edited. While these interactive representations are often domain-specific, recent developments aim to support general-purpose live objects. For example, Exploriants [1] supports interactive UIs as examples, including a playable game.