Foundations for a New Perspective of Understanding Programming

Research on program comprehension has a fundamental limitation: program comprehension is a cognitive process that cannot be directly observed, which leaves considerable room for misinterpretation, uncertainty, and confounders. In Brains On Code, we are developing a neuroscientific foundation of program comprehension. Instead of merely observing whether there is a difference regarding program comprehension (e.g., between two programming methods), we aim at precisely and reliably determining the key factors that cause the difference. This is especially challenging as humans are the subjects of study, and inter-personal variance and other confounding factors obfuscate the results.

The key idea of Brains On Code is to leverage established methods from cognitive neuroscience to obtain insights into the underlying processes and influential factors of program comprehension. Brains On Code pursues a multimodal approach that integrates different neuro-physiological measures as well as a cognitive computational modeling approach to establish the theoretical foundation. This way, Brains On Code lays the foundations of measuring and modeling program comprehension and offers substantial feedback for programming methodology, language design, and education. With Brains On Code, addressing longstanding foundational questions such as “How can we reliably measure program comprehension?”, “What makes a program hard to understand?”, and “What skills should programmers have?” comes into reach. Brains On Code does not only help answer these questions, but also provides an outline for applying the methodology beyond program code (models, specifications, requirements, etc.).

Observing communication is a revealing way to indicate comprehension about code at many different abstraction levels. Using an analytic lens from linguistics, we can precisely describe this communication and thus enable us to make inferences about a person’s comprehension of a program. Not only are the speaker’s actions important, but the agency of the listener is vital to establishing a desired states of shared attention (i.e., both parties are thinking about the same thing) and shared intentionality (i.e., the recursive knowledge that they both know the other is thinking about the same thing they are). When both speaker and listener communicate together, they can begin to comprehend code and take actions on it as a single distributed cognitive unit. The pair’s joint knowledge can be used to execute changes to the code that may have been difficult or impossible for each of them apart.

The effectiveness of this kind of communication is not robust however, when one or both members of the pair identify with physical or cognitive disabilities, e.g., a programmer is blind or low vision, or another has ADHD or dyslexia. In our research, we employ AI techniques in computer vision, speech recognition, NLP, and physiological sensors to interpret, translate, and convey information between speaker and listener. This increases the likelihood that people of mixed abilities can successfully communicate about code, achieving a desired state of shared intentionality that illustrates their joint distributed comprehension and enables them to efficiently make changes together.

One of the main research questions in our research group is “How can we reliably measure program comprehension?”. To answer this question, fMRI and eye tracking studies were conducted. When I joined the team, in 2021, we enlarged the research field from our research group from fMRI to EEG. Since then, we conducted two EEG studies. The first study included the challenge of the new methodology which comes along with using EEG instead of fMRI. In the second study, we took a closer look at the influence of the baseline. We thereby adapted our study design to incorporate four different baselines. Therefore, we can afterwards investigate the effect of the baseline within one study, eliminating a lot of other influencing factors.

When starting to work on understanding program comprehension using fMRI ten years ago in collaboration with Janet Siegmund, there was no blueprint how to approach the topic experimentally. First, I had to learn that empirical research was not very abundant at that time even though programming is such an important economic and societal topic. In the previous years, however, we have seen much progress in empirical research on programming, including brain imaging. Now it is about time for in depth discussions on how to pursue the topic further by teaming up with interested researchers from different disciplines and to start developing a theory of programming. In my talk (see attached slides) I contribute my neuroscientific perspective on program comprehension and discuss my view of sequential processing and cognitive sequencing as key component of programming and how to deal with the dynamics and individuality of program comprehension based on my experience from studying the dynamics of rule-based category learning.

Studying eye movements during reading provides valuable insights into natural-language text comprehension and also lends itself to application in program comprehension. Using English and Java as exemplary languages, differences can be found between natural-language text and code reading, as well as in how novice and expert programmers read code. For instance, when reading natural-language text, a larger part of the text is looked at directly than when reading code. Moreover, during code reading, expert programmers attend to the main-method much sooner than novices. However, this line of research also brings about methodological challenges like event detection and correction of spatial errors. Addressing these is an ongoing effort.

The way that humans interact and absorb information delivered through technology is of interest to researchers in many fields. The accurate assessment of cognitive states such as arousal, fatigue, stress, task difficulty is essential to identifying and defining cognitive processes and testing models of how cognitive processes operate and interact. The individual and the situation can affect the measurement of cognitive states from a single type of sensor. Measurements from multiple sensors, when combined, produce a more robust measurement of cognitive states (CS) such as mental overload. Adaptive filtering or other techniques of CS can then be used to potentially find misconceptions and potentially improve task performance. For the last several years, we have designed and executed experiments about individual differences of the way people perceive, search, and understand information presented in various multimedia environments. We have used a suite of passive physiological sensors (eye fixations, skin conductivity, body temperature, heart wave form, electroencephalography, and pressures on mouse) to better understand the processes that facilitate seeking, filtering, and shaping relevant information during tasks such as understanding computer programs. To understand and solve problems, programmers must have a level of program comprehension established. There are many variations and levels of program comprehension, dependent on individual programmers as well the specific code itself. Our research focuses on deriving changes in cognitive state information from the patterns of data acquired from the user from physiological sensors. If companies incorporate technology already available in Augmented Reality glasses and cell phones, CS information could potentially be used to give feedback in classroom or industry settings. For example, if many programmers show confusion or are misled by the code being reviewed, it may indicate there is a need to improve various aspects of it such as documentation or style.

The concept of flow has been studied for decades across a wide variety of contexts from work to hobbies, and is a critical aspect of engineering productivity, however non-disruptively measuring flow has remained difficult. In this work, we take a mixed methods approach to understanding and measure how software engineers experience flow. We introduce a logs based metric called “flocus” that leverages machine learning and a comprehensive collection of logs data to identify periods of related actions (indicating focused behavior), and validate this metric against self-reported time in flow or focus using diary data and quarterly survey data. Our results indicate that we can determine when software engineers at a large technology company experience flow or focus using flocus. Extending this approach to incorporate other signals such as physiological data or eye tracking has the potential to get us closer to measuring a flow state.

The keynote, while focused on eye movements in programming, covers a wide variety topics, including: (1) basics of eye movements, (2) basic metrics, (3) advanced metrics, (4), cognitive load, (5) eye movements in programming, and (6) future challenges.

Understanding how novices reason about coding at a neurological level has implications for training the next generation of software engineers. I first briefly talk about our work using neuroimaging (fNIRS) to measure the neural activity associated with introductory programming. In this work, we relate brain activity when coding to that of reading natural language or spatial visualization. In contrast to some studies with more expert programmers, we find that while programming, reading, and spatial visualization are all neurologically distinct for novices, there are more significant differences between prose and coding than between spatial visualization and coding. We also find a neural activation pattern predictive of programming performance 11 weeks later. I conclude with a discussion of future education-related directions I hope neuroimaging research explores going forward, including understanding how external factors such as native natural language, or learning disabilities (e.g., Dyslexia) impact program comprehension and learning.

This working group discussed the current state of the art of eye tracking use for research in computing and gaps in the literature and eye tracking challenges.

Participants were generally in consensus that the field of program comprehension currently lacks an explicit taxonomy codifying the research space. As a result, participants worked together to brainstorm categories and dimensions that should be included in such a taxonomy. The goal of this taxonomy was to give researchers a practical guide of dimensions to consider when designing a program comprehension study. The discussion was moderated by Dr. Andrew Begel. In the rest of this section, we present the preliminary taxonomy categories determined by the seminar participants.

• $\mathrm{■}$

  Level of Comprehension: Letters, Lexemes, Words, Program structure, Program semantics, Program intent, Program rationale

• $\mathrm{■}$

  Reason for Comprehension: Design, Knowledge retention, Immediate use, Understanding

• $\mathrm{■}$

  Comprehension Success Criteria: How well is the intent or idea of some code transferred to its audience? (e.g., more formally, code comprehension has succeeded when Comprehension(Coding(original_idea)) = original_idea)

• $\mathrm{■}$

  Code Precision: Pseudocode, Natural Language, Primitive Language, Symbolic Programming Language (like PERL or APL), Math

• $\mathrm{■}$

  Amount, Type, or Organization of Code: Code, Program, Snippet, Architecture, Software, Docs, Modules, Functions, Libraries, Comments, Error messages, Logs

• $\mathrm{■}$

  Domain of the Code: Production Software, Experimental or Prototype, Educational, Scientific Computing, Data Processing, Games, Art, Business, Systems, Security, Etc.

• $\mathrm{■}$

  Code Language Aspects: Templates or Generics, Classes, Strong or Dynamic Types

• $\mathrm{■}$

  Modality: Physical Code Representation (e.g., visual, audio, tactile, features to support presentation such as syntax highlighting), Mental Model Representation

• $\mathrm{■}$

  Presentation (can do any of these as any modality): Text, Graph, Data Structure Graph, Blocks, Flow Charts

• $\mathrm{■}$

  Experimental Task: Reading, Writing, Explaining (consider the target audience – see code as a boundary object below), Communicating, Teaching, Tracing, Debugging, Reasoning about variable values, Dependencies, Judging design quality, Refactoring, Fixing Bugs or Security Vulnerabilities, Adding Features, Code Review, Revising, Code Summarization, Writing Documentation

• $\mathrm{■}$

  Experimental Interventions (Real time): Syntax highlighting, Font, Heatmap, Define/Use/ Navigate, Structural Elision Collapse / hide parts of code, Provided code summary, code layout, error messages

• $\mathrm{■}$

  Experimental Time Scale: Session, Sprint, Milestone, Day, Week, Semester

• $\mathrm{■}$

  Code as a Boundary object: Between coder and computer, Between coder and coder, Between coder and tester, Between coder and user, Between coder and boss, Between student and teacher, Between computer and computer, Between coder and generic outside audience, Size of the audience (either code readers or code users), etc.

• $\mathrm{■}$

  Number of Code Readers: Solo, Pair, Mob, Team, Cohort, Replacement

• $\mathrm{■}$

  Simultaneity: Interactivity, Synchronous Communication, Asynchronous Communication, No access to code author, No access to friends

• $\mathrm{■}$

  Person or Programmer In Your Study: Demographics (e.g., gender, ethnicity, cultural background), Skills or Expertise, Job or Industry, Psychometric tests, Native natural languages, Cognitive difference (e.g., neurodiversity, spatial reasoning ability), Physical Disability, Pregnancy Brain, Mental Health, Identity

• $\mathrm{■}$

  Experimental Metrics: Time on task, Accuracy, Speed, Reading Distribution – Spatial/temporal (e.g., fault localization for debugging), Subjective ratings / self reporting, Perceived difficulty or understanding, Continuous or discrete, Cost, Affect, Motivation, Self-efficacy, Biometric and cognitive metrics (e.g., focus, cognitive load, gaze path, brain activity),

• $\mathrm{■}$

  Code Metrics: Code complexity Size Debuggability Maintainability Readability

• $\mathrm{■}$

  Experimental Hypothesis: from a given hypothesis, you can choose different elements from various categories in the taxonomy.