Document Open Access Logo

SLURM-Managed HyperParameter Optimization

Authors Anusha Chattopadhyay , Hendrik Borras , Bernhard Klein , Holger Fröning

PDF

File

Thumbnail PDF
PDF
OASIcs.PARMA-DITAM.2026.2.pdf
  • Filesize: 0.83 MB
  • 13 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Distributed algorithms
  • Software and its engineering → Distributed systems organizing principles
Keywords
  • Hyperparameter optimization
  • Asynchronous Successive Halving Algorithm (ASHA)
  • Experiment management
  • SLURM
  • SEML
  • Open Source

Metrics

Abstract

Hyperparameter optimization (HPO) is essential for achieving state-of-the-art performance in machine learning, yet it is computationally demanding, particularly on shared or resource-constrained clusters. We present a system that integrates the Asynchronous Successive Halving Algorithm (ASHA) with SEML, the SLURM Experiment Management Library - an experiment orchestration layer that provides declarative configuration, provenance, metric tracking, and robust SLURM job management. The resulting open-source tool enables scalable, fault-tolerant HPO on SLURM-managed infrastructure: SEML handles experiment specification, versioning, and scheduling, while ASHA performs asynchronous early stopping and resource reallocation to concentrate computation on promising configurations. Overall, the system streamlines experiment lifecycle management, enables distributed evaluations with minimal manual effort, and reduces the time required to reach high-quality configurations compared to conventional Grid and Random Search methods under similar compute budgets.

Cite As Get BibTex

Anusha Chattopadhyay, Hendrik Borras, Bernhard Klein, and Holger Fröning. SLURM-Managed HyperParameter Optimization. In 17th Workshop on Parallel Programming and Run-Time Management Techniques for Many-Core Architectures and 15th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2026). Open Access Series in Informatics (OASIcs), Volume 141, pp. 2:1-2:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/OASIcs.PARMA-DITAM.2026.2

Author Details

Anusha Chattopadhyay
  • Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Germany
Hendrik Borras
  • Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Germany
Bernhard Klein
  • Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Germany
Holger Fröning
  • Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Germany

Acknowledgements

The authors acknowledge support by the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG.

Supplementary Materials

References

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623-2631, 2019. URL: https://doi.org/10.1145/3292500.3330701.
  2. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-305, 2012. URL: https://doi.org/10.5555/2503308.2188395.
  3. F. Brückerhoff-Plückelmann, H. Borras, S. U. Hulyal, L. Meyer, X. Ji, J. Hu, J. Sun, B. Klein, F. Ebert, J. Dijkstra, L. McRae, P. Schmidt, T. J. Kippenberg, H. Fröning, and W. Pernice. Uncertainty reasoning with photonic bayesian machines, 2025. URL: https://doi.org/10.48550/arXiv.2512.02217.
  4. Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag., 29(6):141-142, 2012. URL: https://doi.org/10.1109/MSP.2012.2211477.
  5. Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023. Google Scholar
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. URL: https://doi.org/10.1109/CVPR.2016.90.
  7. Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 240-248, Cadiz, Spain, 09-11 May 2016. PMLR. URL: https://proceedings.mlr.press/v51/jamieson16.html.
  8. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, volume abs/1412.6980, 2015. URL: https://doi.org/10.48550/arXiv.1412.6980.
  9. Klaus Greff, Aaron Klein, Martin Chovanec, Frank Hutter, and Jürgen Schmidhuber. The Sacred Infrastructure for Computational Research. In Katy Huff, David Lippa, Dillon Niederhut, and M Pacer, editors, Proceedings of the 16th Python in Science Conference, pages 49-56, 2017. URL: https://doi.org/10.25080/shinma-7f4c6e7-008.
  10. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  11. Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze, editors, Proceedings of the Third Conference on Machine Learning and Systems, MLSys 2020, Austin, TX, USA, March 2-4, 2020, volume 2, pages 230-246. mlsys.org, 2020. URL: https://proceedings.mlsys.org/paper_files/paper/2020/hash/a06f20b349c6cf09a6b171c71b88bbfc-Abstract.html.
  12. Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: a novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(1):185:1-185:52, 2017. URL: https://jmlr.org/papers/v18/16-558.html.
  13. Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. Tune: A Research Platform for Distributed Model Selection and Training, July 2018. URL: https://doi.org/10.48550/arXiv.1807.05118.
  14. TorchVision maintainers and contributors. Torchvision: Pytorch’s computer vision library. https://github.com/pytorch/vision, 2016.
  15. MongoDB, Inc. MongoDB Manual, 2025. Accessed: 2025-07-10. URL: https://www.mongodb.com/docs/.
  16. Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging AI applications. In Andrea C. Arpaci-Dusseau and Geoff Voelker, editors, 13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, Carlsbad, CA, USA, October 8-10, 2018, OSDI'18, pages 561-577, USA, 2018. USENIX Association. URL: https://www.usenix.org/conference/osdi18/presentation/nishihara.
  17. Andy B. Yoo, Morris A. Jette, and Mark Grondona. Slurm: Simple linux utility for resource management. In Dror Feitelson, Larry Rudolph, and Uwe Schwiegelshohn, editors, Job Scheduling Strategies for Parallel Processing, pages 44-60, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. URL: https://doi.org/10.1007/10968987_3.
  18. Daniel Zügner, Johannes Gasteiger, Nicholas Gao, and Dominik Fuchsgruber. SEML: Slurm Experiment Management Library, 2023. URL: https://github.com/TUM-DAML/seml.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact