2026-10-05T21:32:40Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:27981 2026-10-05T06:44:02Z ddc:004 open_access

Characterizing Feedback Statements in Machine Learning Jupyter Notebooks Shome, Arumoy Cruz, Luís Spinellis, Diomidis van Deursen, Arie Empirical software engineering machine learning Jupyter notebooks software testing assertions mining software repositories Background. Machine learning development in Jupyter notebooks is iterative and feedback-driven. Practitioners author statements that reveal information about program execution and use this information to decide what to do next. We call these feedback statements and identify two forms: exploratory statements that display values for visual inspection and validation statements that enforce conditions programmatically through assertions. Aims. Many failures in ML systems do not surface as exceptions, and consequently escape the crash-based analyses that dominate prior empirical work on ML notebooks. This study examines what practitioners check to catch the failures that would otherwise pass silently, by characterizing feedback statements that encode the practitioner’s mental model of what the code should do and what could go wrong. Method. We mine 297,851 publicly available Python Jupyter notebooks from GitHub and Kaggle, and extract 1,092,780 feedback statements. We sample 816 statements through proportional stratified sampling from semantic clusters obtained from CodeBERT embeddings, and apply grounded theory and open coding to manually label and analyze each statement. Results. We contribute a taxonomy of feedback statements in ML Jupyter notebooks, organized along the functional intent of the statement and the ML pipeline stage in which it appears. The taxonomy reveals that feedback in ML notebooks is overwhelmingly exploratory, and that the two platforms host qualitatively different modes of ML work. We further map our taxonomy to an existing crash taxonomy and find that our taxonomy captures defensive practices against silent failures that crash analysis cannot observe. Conclusions. Our findings indicate that notebook source should be treated as a confounder in studies of ML developer practice, surface opportunities for notebook tooling, and motivate empirical study of silent ML failures. We release the corpus of 1,092,780 feedback statements and the codebook, to support replication and tooling research. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Arumoy Shome and Luís Cruz and Diomidis Spinellis and Arie van Deursen 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.13 urn:nbn:de:0030-drops-279814 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.13 eng https://creativecommons.org/licenses/by/4.0/legalcode