PShapeTrace: Linking Drawing Instructions with Visual Outcomes in Processing

Processing is initially created to serve as a software sketchbook [10]. Although it is based on Java and shares its fundamental features and syntax, Processing allows for a more concise syntax by omitting class definitions and other structural elements. For example, the following program creates a 500×500 white canvas and draws a line from the center to the mouse cursor’s position:

  void setup() {
    size(500, 500);
  }
  void draw() {
    line(width/2, height/2, mouseX, mouseY);
  }

The setup function is invoked once at the beginning of the program, while the draw function runs continuously on each frame to update the canvas. Built-in variables such as mouseX and mouseY capture the current mouse position so that graphical interactions can be implemented easily. Compared to Java, Processing provides a more accessible way to produce graphical output with minimal code, making it suitable for beginners with no prior programming experience.

Processing programs are edited and executed using the Processing Development Environment (PDE). The PDE treats a directory as a program. Source files with the .pde extension in the directory are automatically recognized as part of a single program.

Processing programs are compiled into Java bytecode and executed. The generated code is a subclass of PApplet, which provides built-in functions like line and text as Java methods. User-defined methods, such as setup and draw, are automatically invoked by the generated code.

Several tools have been proposed to support the development of graphical applications. Santos [12, 13] introduced GUITA, a tracing tool that links UI components to their corresponding source code. This system allows users to click on UI widgets created during program execution and navigate directly to the relevant lines of source code. Additionally, users can inspect all code segments that influence a given widget. However, this approach assumes that UI components are created as objects (specifically using the SWT library). As a result, it cannot be directly applied to Processing programs, where frames are rendered dynamically through drawing functions rather than object-based components.

Ko et al. [6] proposed Whyline, a tool that links the visual output of a program to its runtime behavior. To enable users to focus on programming without needing to learn syntax, the tool maintains a list of instructions and graphical entities that users can control, similar to those in Scratch and Snap. Therefore, the tool can utilize graphical properties such as the sizes and positions of these entities to generate questions. However, the approach is not applicable to Processing, as we aim to allow programmers to use arbitrary drawing instructions.

Chan et al. [3] proposed DRT, a tool that monitors the state of GUI programs and tracks the sequence of function calls executed when a GUI state changes. Dang et al. [4] extended this approach to dynamic applications such as games. They developed a tool that identifies the instruction sequences responsible for screen changes. Our tool shares similarities with these systems but offers a more granular mapping between individual graphical elements (e.g., rectangles and lines) and their corresponding drawing functions. This distinction arises because prior studies focus on Feature Location [14], which helps developers locate the implementation of complex GUI functionalities. Our study aims to analyze the execution of individual function calls.

Huang et al. [5] introduced a debugging environment leveraging Live Programming techniques to assist in diagnosing GUI-related bugs in web applications. Their system tracks changes in the Document Object Model (DOM) and CSS, enabling users to trace the execution sequence and analyze view transitions in real time. They reported that this approach significantly reduces debugging time. Although their target applications differ, our study shares the same goal of enabling real-time program analysis during execution.

The tool is implemented as a source file that integrates into a target program for analysis. It overrides built-in drawing functions of the Processing language, such as text and rect, which are defined in the PApplet class. These overriding methods record function calls and execute the original behavior of the built-in functions.

The installation process consists of the following steps:

1. 1.

   Place the tool file (tool.pde) in the same directory as the program’s source files. The Processing Development Environment (PDE) automatically recognizes the tool file as part of the target program.

2. 2.

   Add the method call extraSettings(); to the setup method. This initializes the tool by detecting the window size set by the size method and preparing a dedicated area for the tool interface.

3. 3.

   Rename the draw method in the target program to drawMain. This allows the tool to override the draw method.

The installation process is designed for first-year computer science students who may be unfamiliar with configuring operating systems and applications. Since the tool requires only basic file operations and minor code modifications, it avoids complex setup steps such as plugin installations. Additionally, the presence of the tool in the source code makes it clear whether it is active or not.

Figure 2 shows a screenshot of the breakout game running with the tool. The original program’s execution area appears in the upper left of the window. The tool records drawing function calls and their actual parameters, and then displays these recorded function calls on the right side. For example, the first line on the right indicates that the drawMain function calls the background function with three integer arguments. Displaying function calls as a simple list was based on the assumption that first-time users could comprehend it without prior training and map it directly to the code they had written. The tool recognizes a frame by detecting calls to the background method, which clears the window content. The function call list represents how the current frame is rendered.

The tool also provides an interface for handling execution history at the bottom of the window. This section includes the current frame number, a timeline, and a pause button. The timeline visualizes recorded frames, where a blue bar marks the current frame, and beige indicates past frames. To manage memory usage efficiently, the tool retains the function calls and rendered images of the most recent 1,024 frames, discarding older ones.

In addition to the function call list, the tool offers three interactive features:

• $\mathrm{■}$

  Pause Button: Clicking the pause button suspends the program’s drawing process by skipping the drawMain function. Clicking it again resumes execution. This feature allows users to inspect the current frame in detail. Figure 3 shows a screenshot with execution paused. The red pause button indicates that drawing is suspended.

• $\mathrm{■}$

  Replaying a Past Frame: Clicking on the timeline replays a past frame by displaying the recorded image and function calls, enabling users to inspect previous states. In Figure 3, the blue bar in the timeline represents the currently selected past frame, while the red bar highlights the latest frame. Users can also navigate frames using the left and right arrow keys.

• $\mathrm{■}$

  Highlighting a Function Call: Clicking a shape in the program area highlights its corresponding function call in red. In Figure 3, clicking on the ball rectangle located on the left side highlighted a rect function that rendered the shape. The function call indicates that it was executed within the drawMain function with four actual parameters. If multiple shapes overlap, repeated clicks on the same location cycle through the shapes sequentially.

These features help users quickly analyze how their code generates graphical output, enhancing their understanding of programming concepts.

The tool is integrated into a target program, which runs as a standalone application. This approach prevents interaction between the program and the Processing Development Environment (PDE). For example, the tool cannot trace a function call in the function call list back to its corresponding line in the PDE editor. Instead, it only has access to line numbers from a temporary Java source file generated during compilation. Exploring better interaction methods while maintaining ease of installation is part of our future work.

The tool also has several limitations due to its implementation:

• $\mathrm{■}$

  Incompatibility with dynamic window resizing: The tool assumes a fixed window size and does not support programs that dynamically change window dimensions.

• $\mathrm{■}$

  Potential timing disruptions: The pause button may interfere with timing calculations in programs that rely on timers.

• $\mathrm{■}$

  Interference with mouse and keyboard inputs: Mouse and keyboard interactions with the tool may be captured by the program’s mousePressed and keyTyped methods, potentially leading to unexpected behavior.

• $\mathrm{■}$

  Restriction on overriding built-in functions: Users cannot override Processing’s built-in functions because they are already overridden by the tool.

• $\mathrm{■}$

  Error messages referencing the tool file: A syntax error in the target program may produce an error message pointing to a line in the tool file rather than the original source code.

We believe these limitations are not significant concerns for introductory courses.

The proposed tool introduces overhead in terms of processing speed and memory consumption, as it observes the execution of a target program and adds functionality to visualize drawing functions. We measure the tool’s impact on frame rate (FPS) and maximum heap memory usage.

Since Experiment 1 confirmed that the tool’s performance impact is minimal, we proceeded to evaluate its usability in Processing programming tasks.