Document Open Access Logo

A Multivocal Review of AutoML Practices, Challenges, Opportunities and Open Issues

Authors Keerthiga Rajenthiram , Faezeh Amou Najafabadi , Ilias Gerostathopoulos , Patricia Lago

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.27.pdf
  • Filesize: 0.98 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Machine learning
Keywords
  • Automated Machine Learning (AutoML)
  • Multivocal Literature Review
  • YouTube Videos
  • Gray Literature
  • Empirical Study
  • Research–Practice Gap

Metrics

Abstract

Background. Automated Machine Learning (AutoML) is increasingly embedded in software and data engineering workflows, automating the design and optimization of machine learning (ML) pipelines. AutoML has been extensively synthesized through secondary studies that map tools, algorithms, and evaluation practices. However, these reviews predominantly reflect a research-centric perspective and provide limited evidence on how AutoML is actually used, adapted, and experienced in practice. Traditional surveys and interviews capture only a narrow slice of the practitioner community, leaving open how well academic claims about AutoML align with day-to-day engineering realities. 

Aims. This study aims to (i) assess to what extent AutoML practices and challenges reported in existing secondary studies reflect practitioners' experiences, and (ii) identify mismatches between research and practice by empirically comparing academic claims with practitioner experiences shared on YouTube. 

Method. Following established guidelines for multivocal literature reviews in software engineering, we conduct a multivocal evidence synthesis combining 16 peer-reviewed secondary studies (systematic and multivocal literature reviews) with 30 practitioner-created YouTube videos in which engineers, data scientists, and ML practitioners describe their AutoML workflows, workarounds, and pain points. 

Results. Our analysis reveals a marked imbalance: academic reviews emphasize algorithm selection, search strategies, and benchmarking, whereas practitioners foreground data quality and cleaning, resource and cost constraints, tool robustness, scalability, and integration into production pipelines. We also observe areas of alignment, for example around hyperparameter optimization and the need for reproducible workflows. 

Conclusions. Treating YouTube videos as curated gray literature under established multivocal review guidelines, our study shows that they provide complementary, practice-grounded evidence about AutoML adoption. The identified gaps and alignments suggest concrete directions for empirical AutoML research that more directly address operational and organizational constraints in real-world software engineering settings.

Cite As Get BibTex

Keerthiga Rajenthiram, Faezeh Amou Najafabadi, Ilias Gerostathopoulos, and Patricia Lago. A Multivocal Review of AutoML Practices, Challenges, Opportunities and Open Issues. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 27:1-27:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.27

Author Details

Keerthiga Rajenthiram
  • Vrije Universiteit Amsterdam, The Netherlands
Faezeh Amou Najafabadi
  • Vrije Universiteit Amsterdam, The Netherlands
Ilias Gerostathopoulos
  • Vrije Universiteit Amsterdam, The Netherlands
Patricia Lago
  • Vrije Universiteit Amsterdam, The Netherlands

Supplementary Materials

References

  1. Online Appendix for this review. https://doi.org/10.5281/zenodo.20282909, 2026.
  2. Replication package for this review. https://doi.org/10.5281/zenodo.20283067, 2026.
  3. Muhammad Junaid Ali, Mokhtar Essaid, Laurent Moalic, and Lhassane Idoumghar. A review of AutoML optimization techniques for medical image applications. Computerized Medical Imaging and Graphics, 118:1-35, 2024. URL: https://doi.org/10.1016/J.COMPMEDIMAG.2024.102441.
  4. Florian Bachinger, Jan Zenisek, and Michael Affenzeller. Automated machine learning for industrial applications – challenges and opportunities. Procedia Computer Science, 232:1701-1710, 2024-01-01. URL: https://doi.org/10.1016/j.procs.2024.01.168.
  5. Maroua Bahri, Flavia Salutari, Andrian Putina, and Mauro Sozio. AutoML: state of the art with a focus on anomaly detection, challenges, and research directions. Int. J. Data Sci. Anal., 14(2):113-126, 2022-08-01. URL: https://doi.org/10.1007/s41060-022-00309-0.
  6. Rafael Barbudo, Sebastián Ventura, and José Raúl Romero. Eight years of AutoML: categorisation, review and trends. Knowl. Inf. Syst., 65(12):5097-5149, 2023. URL: https://doi.org/10.1007/s10115-023-01935-1.
  7. Bo Chen, Xiangyu Zhao, Yejing Wang, Wenqi Fan, Huifeng Guo, and Ruiming Tang. A Comprehensive Survey on Automated Machine Learning for Recommendations. ACM Trans. Recomm. Syst., 2(2):13:1-13:38, April 2024. URL: https://doi.org/10.1145/3630104.
  8. Yi-Wei Chen, Qingquan Song, and Xia Hu. Techniques for automated machine learning. SIGKDD Explor., 22(2):35-50, 2021-01-17. URL: https://doi.org/10.1145/3447556.3447567.
  9. Ronald Chenail. YouTube as a Qualitative Research Asset: Reviewing User Generated Videos as Learning Resources. The Qualitative Report, 16(1):229-235, January 2011. URL: https://doi.org/10.46743/2160-3715/2011.1048.
  10. Liam Connor, Rebecca Kelly, Saoirse Murphy, Eoghan McCarthy, and Edward Murray. Analysis of AutoML tools in the world of automated deep learning. Fusion of Multidisciplinary Research, An International Journal, 5(1):541-555, 2024-01-14. Number: 1. URL: https://fusionproceedings.com/fmr/1/article/view/61.
  11. Aline Marques Del Valle, Rafael Gomes Mantovani, and Ricardo Cerri. A systematic literature review on AutoML for multi-target learning tasks. Artificial Intelligence Review, 56(2):2013-2052, 2023-11-01. URL: https://doi.org/10.1007/s10462-023-10569-2.
  12. Radwa El Shawi, Mohamed Maher, and Sherif Sakr. Automated machine learning: State-of-the-art and open challenges. CoRR, 2019. URL: https://doi.org/10.48550/arXiv.1906.02287.
  13. Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106:101-121, February 2019. URL: https://doi.org/10.1016/j.infsof.2018.09.006.
  14. Dhruvi Gosai and Minal Patel. Advancing the machine learning workflow: A comprehensive analysis of auto machine learning (auto-ML) techniques and future investigations. In Ashish Kumar Luhach, Dharm Singh Jat, Uttam Ghosh, Xiao-Zhi Gao, and Pawan Lingras, editors, Advanced Informatics for Computing Research, pages 308-323. Springer Nature Switzerland, 2025. URL: https://doi.org/10.1007/978-3-031-84062-3_25.
  15. Xin He, Kaiyong Zhao, and Xiaowen Chu. AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212:106622, 2021-01. URL: https://doi.org/10.1016/j.knosys.2020.106622.
  16. Shubhra Kanti Karmaker (“Santu”), Md. Mahadi Hassan, Micah J. Smith, Lei Xu, Chengxiang Zhai, and Kalyan Veeramachaneni. AutoML to date and beyond: Challenges and opportunities. ACM Comput. Surv., 54(8):175:1-175:36, 2021-10-04. URL: https://doi.org/10.1145/3470918.
  17. Brad McKenna, Michael D. Myers, and Michael Newman. Social media in qualitative research: Challenges and recommendations. Information and Organization, 27(2):87-99, June 2017. URL: https://doi.org/10.1016/j.infoandorg.2017.03.001.
  18. Alejandro Morales-Hernández, Inneke Van Nieuwenhuyse, and Sebastian Rojas Gonzalez. A survey on multi-objective hyperparameter optimization algorithms for machine learning. Artificial Intelligence Review, 56(8):8043-8093, August 2023. URL: https://doi.org/10.1007/s10462-022-10359-2.
  19. Mohamed M. Mostafa, Ali Feizollah, and Nor Badrul Anuar. Fifteen years of YouTube scholarly research: knowledge structure, collaborative networks, and trending topics. Multimedia Tools and Applications, 82(8):12423-12443, 2023. URL: https://doi.org/10.1007/s11042-022-13908-7.
  20. Michael Quinn Patton. Qualitative Research & Evaluation Methods: Integrating Theory and Practice. Sage, Thousand Oaks, CA, 3 edition, 2002. Google Scholar
  21. Yannis Poulakis, Christos Doulkeridis, and Dimosthenis Kyriazis. A survey on AutoML methods and systems for clustering. ACM Transactions on Knowledge Discovery from Data, 18(5):1-30, 2024-06-30. URL: https://doi.org/10.1145/3643564.
  22. Fernando F. Prado and Luciano A. Digiampietri. A systematic review of automated feature engineering solutions in machine learning problems. In Proceedings of the XVI Brazilian Symposium on Information Systems, SBSI '20, New York, NY, USA, 2020. Association for Computing Machinery. URL: https://doi.org/10.1145/3411564.3411610.
  23. Luigi Quaranta, Kelly Azevedo, Fabio Calefato, and Marcos Kalinowski. A multivocal literature review on the benefits and limitations of industry-leading AutoML tools. Information and Software Technology, 178:107608, 2025. URL: https://doi.org/10.1016/j.infsof.2024.107608.
  24. Mohaimenul Azam Khan Raiaan, Sadman Sakib, Nur Mohammad Fahad, Abdullah Al Mamun, Md. Anisur Rahman, Swakkhar Shatabda, and Md. Saddam Hossain Mukta. A systematic review of hyperparameter optimization techniques in convolutional neural networks. Decision Analytics Journal, 11:100470, 2024. URL: https://doi.org/10.1016/j.dajour.2024.100470.
  25. Keerthiga Rajenthiram, Pauline Delporte, and Patricia Lago. AutoML: A tertiary study of phases, methods, tools, and frameworks. In Domenico Bianculli, Hassan Sartaj, Vasilios Andrikopoulos, Cesare Pautasso, Tommi Mikkonen, Jennifer Perez, Tomáš Bureš, Martina De Sanctis, Henry Muccini, Elena Navarro, Mohamed Soliman, and Uwe Zdun, editors, Software Architecture. ECSA 2025 Tracks and Workshops, pages 313-328. Springer Nature Switzerland, 2026. URL: https://doi.org/10.1007/978-3-032-04403-7_26.
  26. S Ravishankar and Battineni Gopi. A survey on recent advancements in auto-machine learning with a focus on feature engineering | journal of computational and cognitive engineering. URL: https://ojs.bonviewpress.com/index.php/JCCE/article/view/720.
  27. Imrus Salehin, Md Islam, Pritom Saha, S M Noman, Azra Tuni, Md.Mehedi Hasan, and Md Baten. AutoML: A systematic review on automated machine learning with neural architecture search. Journal of Information and Intelligence, 2, 2023-10-01. Google Scholar
  28. Imrus Salehin, Md Shamiul Islam, Pritom Saha, S. M. Noman, Azra Tuni, Md Mehedi Hasan, and Md Abu Baten. AutoML: A systematic review on automated machine learning with neural architecture search. Journal of Information and Intelligence, 2(1):52-81, 2024. URL: https://www.sciencedirect.com/science/article/pii/S2949715923000604.
  29. Sasan Salmani Pour Avval, Nathan D. Eskue, Roger M. Groves, and Vahid Yaghoubi. Systematic review on neural architecture search. Artificial Intelligence Review, 58(3):73, January 2025. URL: https://doi.org/10.1007/s10462-024-11058-w.
  30. Margaret Sampson, Jordi Cumber, Claudia Li, Catherine M. Pound, Ann Fuller, and Denise Harrison. A systematic review of methods for studying consumer health YouTube videos, with implications for systematic reviews. PeerJ, 1:e147, September 2013. URL: https://doi.org/10.7717/peerj.147.
  31. Alex Serban, Koen Van Der Blom, Holger Hoos, and Joost Visser. Adoption and effects of software engineering best practices in machine learning - supplementary material. Publisher: Zenodo Version Number: 0.0.9. URL: https://doi.org/10.5281/ZENODO.3946453.
  32. Wuyou Sui, Anna Sui, and Ryan E Rhodes. What to watch: Practical considerations and strategies for using YouTube for research. Digital Health, 8:20552076221123707, September 2022. URL: https://doi.org/10.1177/20552076221123707.
  33. Junchi Tian and Chang Che. Automated machine learning: A survey of tools and techniques. Journal of Industrial Engineering and Applied Science, 2(6):71-76, 2024-12-01. URL: https://doi.org/10.70393/6a69656173.323336.
  34. Jonathan Waring, Charlotta Lindvall, and Renato Umeton. Automated machine learning: Review of the state-of-the-art and opportunities for healthcare. Artificial Intelligence in Medicine, 104:1-12, 2020. URL: https://doi.org/10.1016/J.ARTMED.2020.101822.
  35. Doris Xin, Eva Yiwei Wu, Doris Jung-Lin Lee, Niloufar Salehi, and Aditya Parameswaran. Whither AutoML? understanding the role of automation in machine learning workflows. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, pages 1-16. Association for Computing Machinery, 2021. URL: https://doi.org/10.1145/3411764.3445306.
  36. Han Yuan, Kunyu Yu, Feng Xie, Mingxuan Liu, and Shenghuan Sun. Automated machine learning with interpretation: A systematic review of methodologies and applications in healthcare. Medicine Advances, 2(3):205-237, 2024. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/med4.75. URL: https://doi.org/10.1002/med4.75.
  37. Ruiqi Zheng, Liang Qu, Bin Cui, Yuhui Shi, and Hongzhi Yin. AutoML for Deep Recommender Systems: A Survey. ACM Trans. Inf. Syst., 41(4):101:1-101:38, March 2023. URL: https://doi.org/10.1145/3579355.
  38. Marc-André Zöller and Marco F. Huber. Benchmark and survey of automated machine learning frameworks. Journal of Artificial Intelligence Research, 70:409-472, 2021-01-27. URL: https://doi.org/10.1613/jair.1.11854.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact