Document Open Access Logo

Characterizing Feedback Statements in Machine Learning Jupyter Notebooks

Authors Arumoy Shome , Luís Cruz , Diomidis Spinellis , Arie van Deursen

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.13.pdf
  • Filesize: 0.84 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Empirical software validation
Keywords
  • Empirical software engineering
  • machine learning
  • Jupyter notebooks
  • software testing
  • assertions
  • mining software repositories

Metrics

Abstract

Background. Machine learning development in Jupyter notebooks is iterative and feedback-driven. Practitioners author statements that reveal information about program execution and use this information to decide what to do next. We call these feedback statements and identify two forms: exploratory statements that display values for visual inspection and validation statements that enforce conditions programmatically through assertions.

Aims. Many failures in ML systems do not surface as exceptions, and consequently escape the crash-based analyses that dominate prior empirical work on ML notebooks. This study examines what practitioners check to catch the failures that would otherwise pass silently, by characterizing feedback statements that encode the practitioner’s mental model of what the code should do and what could go wrong.

Method. We mine 297,851 publicly available Python Jupyter notebooks from GitHub and Kaggle, and extract 1,092,780 feedback statements. We sample 816 statements through proportional stratified sampling from semantic clusters obtained from CodeBERT embeddings, and apply grounded theory and open coding to manually label and analyze each statement.

Results. We contribute a taxonomy of feedback statements in ML Jupyter notebooks, organized along the functional intent of the statement and the ML pipeline stage in which it appears. The taxonomy reveals that feedback in ML notebooks is overwhelmingly exploratory, and that the two platforms host qualitatively different modes of ML work. We further map our taxonomy to an existing crash taxonomy and find that our taxonomy captures defensive practices against silent failures that crash analysis cannot observe.

Conclusions. Our findings indicate that notebook source should be treated as a confounder in studies of ML developer practice, surface opportunities for notebook tooling, and motivate empirical study of silent ML failures. We release the corpus of 1,092,780 feedback statements and the codebook, to support replication and tooling research.

Cite As Get BibTex

Arumoy Shome, Luís Cruz, Diomidis Spinellis, and Arie van Deursen. Characterizing Feedback Statements in Machine Learning Jupyter Notebooks. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 13:1-13:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.13

Author Details

Arumoy Shome
  • Delft University of Technology, The Netherlands
Luís Cruz
  • Delft University of Technology, The Netherlands
Diomidis Spinellis
  • Delft University of Technology, The Netherlands
Arie van Deursen
  • Delft University of Technology, The Netherlands

References

  1. Martín Abadi et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015. Software available from tensorflow.org. URL: https://www.tensorflow.org/.
  2. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. Software Engineering for Machine Learning: A Case Study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, May 2019. URL: https://doi.org/10.1109/icse-seip.2019.00042.
  3. Charles Betz. Managing Digital: Concepts and Practices, 2018. [Online; accessed 2024-05-05]. URL: https://pubs.opengroup.org/opengrouppress/managing-digital/chapter-app-deliv.html.
  4. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D Sculley. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. In 2017 IEEE International Conference on Big Data (Big Data), pages 1123-1132. IEEE, 2017. URL: https://doi.org/10.1109/BIGDATA.2017.8258038.
  5. Eric Breck, Neoklis Polyzotis, Sudip Roy, Steven Whang, and Martin Zinkevich. Data Validation for Machine Learning. In MLSys, 2019. Google Scholar
  6. Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. Goal Question Metric Paradigm. Encyclopedia of Software Engineering, 2:528-532, 1994. Google Scholar
  7. François Chollet et al. Keras. https://keras.io, 2015.
  8. Taijara Loiola De Santana, Paulo Anselmo Da Mota Silveira Neto, Eduardo Santana De Almeida, and Iftekhar Ahmed. Bug Analysis in Jupyter Notebook Projects: An Empirical Study. ACM Trans. Softw. Eng. Methodol., 33(4):101:1-101:34, April 2024. URL: https://doi.org/10.1145/3641539.
  9. Konstantin Grotov, Sergey Titov, Vladimir Sotnikov, Yaroslav Golubev, and Timofey Bryksin. A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts. In Proceedings of the 19th International Conference on Mining Software Repositories, MSR ’22. ACM, May 2022. URL: https://doi.org/10.1145/3524842.3528447.
  10. Mark Haakman, Luís Cruz, Hennie Huijgens, and Arie van Deursen. AI Lifecycle Models Need to be Revised: An Exploratory Study in Fintech. Empirical Software Engineering, 26(5), July 2021. URL: https://doi.org/10.1007/s10664-021-09993-1.
  11. Andrew Head, Fred Hohman, Titus Barik, Steven M Drucker, and Robert DeLine. Managing Messes in Computational Notebooks. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-12, 2019. Google Scholar
  12. Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. Taxonomy of Real Faults in Deep Learning Systems. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20. ACM, June 2020. URL: https://doi.org/10.1145/3377811.3380395.
  13. Wenyuan Jiang, Diany Pressato, Harsh Darji, and Thibaud Lutellier. Exploring the Jupyter Ecosystem: An Empirical Study of Bugs and Vulnerabilities. In 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 92-103, October 2025. URL: https://doi.org/10.1109/ESEM64174.2025.00052.
  14. Yuan Jiang, Christian Kastner, and Shurui Zhou. Elevating Jupyter Notebook Maintenance Tooling by Identifying and Extracting Notebook Structures. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 399-403. IEEE, October 2022. URL: https://doi.org/10.1109/icsme55016.2022.00047.
  15. Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. An In-Depth Study of the Promises and Perils of Mining GitHub. Empirical Software Engineering, 21(5):2035-2071, September 2015. URL: https://doi.org/10.1007/s10664-015-9393-5.
  16. Mary Beth Kery, Marissa Radensky, Mahima Arya, Bonnie E John, and Brad A Myers. The Story in the Notebook: Exploratory Data Science Using a Literate Programming Tool. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1-11, 2018. Google Scholar
  17. Pavneet Singh Kochhar and David Lo. Revisiting Assert Use in GitHub Projects. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE’17. ACM, June 2017. URL: https://doi.org/10.1145/3084226.3084259.
  18. Fernando Martinez-Plumed, Lidia Contreras-Ochando, Cesar Ferri, Jose Hernandez-Orallo, Meelis Kull, Nicolas Lachiche, Maria Jose Ramirez-Quintana, and Peter Flach. CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science Trajectories. IEEE Transactions on Knowledge and Data Engineering, 33(8):3048-3061, August 2021. URL: https://doi.org/10.1109/tkde.2019.2962680.
  19. Leland McInnes, John Healy, and Steve Astels. HDBSCAN: Hierarchical Density Based Clustering. Journal of Open Source Software, 2(11):205, 2017. URL: https://doi.org/10.21105/joss.00205.
  20. Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software, 3(29):861, September 2018. URL: https://doi.org/10.21105/joss.00861.
  21. Mohammad Mehdi Morovati, Amin Nikanjam, Foutse Khomh, and Zhen Ming Jiang. Bugs in Machine Learning-Based Systems: A Faultload Benchmark. Empirical Software Engineering, 28(3), April 2023. URL: https://doi.org/10.1007/s10664-023-10291-1.
  22. Mojtaba Mostafavi Ghahfarokhi, Arash Asgari, Mohammad Abolnejadian, and Abbas Heydarnoori. DistilKaggle: A Distilled Dataset of Kaggle Jupyter Notebooks. In Proceedings of the 21st International Conference on Mining Software Repositories, MSR '24, pages 647-651, New York, NY, USA, July 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3643991.3644882.
  23. Nadia Nahar, Haoran Zhang, Grace Lewis, Shurui Zhou, and Christian Kästner. A Meta-Summary of Challenges in Building Products with ML Components – Collecting Experiences from 4758+ Practitioners. In 2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN), pages 171-183, Melbourne, Australia, May 2023. IEEE. URL: https://doi.org/10.1109/CAIN58948.2023.00034.
  24. Moses Openja, Foutse Khomh, Armstrong Foundjem, Zhen Ming (Jack) Jiang, Mouna Abidi, and Ahmed E. Hassan. An Empirical Study of Testing Machine Learning in the Wild . ACM Trans. Softw. Eng. Methodol., 34(1):7:1-7:63, December 2024. URL: https://doi.org/10.1145/3680463.
  25. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic Differentiation in PyTorch, 2017. Google Scholar
  26. Jibesh Patra and Michael Pradel. Nalin: Learning from Runtime Behavior to Find Name-Value Inconsistencies in Jupyter Notebooks. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, pages 1469-1481, New York, NY, USA, July 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3510003.3510144.
  27. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python, 2011. URL: https://doi.org/10.5555/1953048.2078195.
  28. Jeffrey M. Perkel. Why Jupyter is Data Scientists’ Computational Notebook of Choice. Nature, 563(7729):145-146, October 2018. URL: https://doi.org/10.1038/d41586-018-07196-1.
  29. João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks. In 2019 IEEE/ACM 16th international conference on mining software repositories (MSR), pages 507-517. IEEE, 2019. URL: https://doi.org/10.1109/MSR.2019.00077.
  30. Fotis Psallidas, Yiwen Zhu, Bojan Karlas, Matteo Interlandi, Avrilia Floratou, Konstantinos Karanasos, Wentao Wu, Ce Zhang, Subru Krishnan, Carlo Curino, et al. Data Science Through the Looking Glass and What We Found There. arXiv preprint, 2019. URL: https://arxiv.org/abs/1912.09536.
  31. Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. KGTorrent: A Dataset of Python Jupyter Notebooks from Kaggle. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 550-554. IEEE, 2021. URL: https://doi.org/10.1109/MSR52588.2021.00072.
  32. Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. KGTorrent: A Dataset of Python Jupyter Notebooks from Kaggle, 2021. URL: https://doi.org/10.5281/ZENODO.4468523.
  33. Adam Rule, Aurélien Tabard, and James D Hollan. Exploration and Explanation in Computational Notebooks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-12, 2018. Google Scholar
  34. Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-15, 2021. URL: https://doi.org/10.1145/3411764.3445518.
  35. David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. Hidden Technical Debt in Machine Learning Systems. Advances in neural information processing systems, 28:2503-2511, 2015. Google Scholar
  36. C.B. Seaman. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering, 25(4):557-572, July 1999. URL: https://doi.org/10.1109/32.799955.
  37. Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein, and Aditya G. Parameswaran. "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1):1-34, April 2024. URL: https://doi.org/10.1145/3653697.
  38. Arumoy Shome. Dataset and Replication Package. URL: https://figshare.com/s/cc75732cf3ca0c5f9247.
  39. Melina Vidoni. Evaluating Unit Testing Practices in R Packages. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, May 2021. URL: https://doi.org/10.1109/icse43902.2021.00136.
  40. Jiawei Wang, Tzu-yang Kuo, Li Li, and Andreas Zeller. Assessing and Restoring Reproducibility of Jupyter Notebooks. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pages 138-149, 2020. URL: https://doi.org/10.1145/3324884.3416585.
  41. Yiran Wang, Willem Meijer, José Antonio Hernández López, Ulf Nilsson, and Dániel Varró. Why Do Machine Learning Notebooks Crash? An Empirical Study on Public Python Jupyter Notebooks. IEEE Transactions on Software Engineering, 51(7):2181-2196, July 2025. URL: https://doi.org/10.1109/TSE.2025.3574500.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact