Document Open Access Logo

FaaSLoad: Fine-Grained Performance and Resource Measurement for Function-As-a-Service

Author Mathieu Bacou

PDF
Thumbnail PDF

File

LIPIcs.OPODIS.2024.22.pdf
  • Filesize: 1.32 MB
  • 21 pages

Document Identifiers

Author Details

Mathieu Bacou
  • Samovar, Télécom SudParis, Institut Polytechnique de Paris, France
  • Inria Saclay, Palaiseau, France

Acknowledgements

While the paper’s author is the main developer of FaaSLoad, many other people contributed to it. Their contributions are listed in a dedicated file in FaaSLoad’s repository. Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). The author thanks Gaël Thomas for his edits, and Mylène Chameron-Moindrot for her help on figures. Lastly, many thanks to the anonymous reviewers of the conference for their comments.

Cite As Get BibTex

Mathieu Bacou. FaaSLoad: Fine-Grained Performance and Resource Measurement for Function-As-a-Service. In 28th International Conference on Principles of Distributed Systems (OPODIS 2024). Leibniz International Proceedings in Informatics (LIPIcs), Volume 324, pp. 22:1-22:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024) https://doi.org/10.4230/LIPIcs.OPODIS.2024.22

Abstract

Cloud computing relies on a deep stack of system layers: virtual machine, operating system, distributed middleware and language runtime. However, those numerous, distributed, virtual layers prevent any low-level understanding of the properties of FaaS applications, considered as programs running on real hardware. As a result, most research analyses only consider coarse-grained properties such as global performance of an application, and existing datasets include only sparse data.
FaaSLoad is a tool to gather fine-grained data about performance and resource usage of the programs that run on Function-as-a-Service cloud platforms. It considers individual instances of functions to collect hardware and operating-system performance information, by monitoring them while injecting a workload. FaaSLoad helps building a dataset of function executions to train machine learning models, studying at fine grain the behavior of function runtimes, and replaying real workload traces for in situ observations.
This research software project aims at being useful to cloud system researchers with features such as guaranteeing reproducibility and correctness, and keeping up with realistic FaaS workloads. Our evaluations show that FaaSLoad helps us understanding the properties of FaaS applications, and studying the latter under real conditions.

Subject Classification

ACM Subject Classification
  • Computer systems organization → Cloud computing
  • General and reference → Measurement
Keywords
  • cloud
  • serverless
  • Function-as-a-Service
  • measurement
  • performance
  • resource utilization
  • dataset generation
  • workload injection

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Supplementary Materials

References

  1. Apache OpenWhisk utility for deploying and managing OpenWhisk projects and packages. URL: https://github.com/apache/openwhisk-wskdeploy/tree/master.
  2. Interface JvmCpuMonitorMXBean. URL: https://www.ibm.com/docs/api/v1/content/SSYKE2_8.0.0/openj9/api/jdk8/jre/management/extension/com/ibm/lang/management/JvmCpuMonitorMXBean.html.
  3. multiprocessing - process-based parallelism. URL: https://docs.python.org/3/library/multiprocessing.html.
  4. PEP 684 – A Per-Interpreter GIL. URL: https://peps.python.org/pep-0684/.
  5. perf: Linux profiling with performance counters. URL: https://perf.wiki.kernel.org/index.php/Main_Page.
  6. The Linux kernel documentation: Memory Resource Controller: Misc. interfaces: stat file. URL: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html.
  7. threading - Thread-based parallelism: Timer Objects. URL: https://docs.python.org/3/library/threading.html#threading.Timer.
  8. What AWS Lambda’s Performance Stats Reveal: Memory Usage, April 2019. URL: https://thenewstack.io/what-aws-lambdas-performance-stats-reveal/.
  9. Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana- Maria Popa. Firecracker: Lightweight Virtualization for Serverless Applications. In Ranjita Bhagwan and George Porter, editors, 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020, Santa Clara, CA, USA, February 25-27, 2020, pages 419-434. USENIX Association, 2020. URL: https://www.usenix.org/conference/nsdi20/presentation/agache.
  10. Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein, Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt. SAND: Towards High-Performance Serverless Computing. In Haryadi S. Gunawi and Benjamin C. Reed, editors, Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, MA, USA, July 11-13, 2018, pages 923-935. USENIX Association, 2018. URL: https://www.usenix.org/conference/atc18/presentation/akkus.
  11. Lixiang Ao, Liz Izhikevich, Geoffrey M. Voelker, and George Porter. Sprocket: A Serverless Video Processing Framework. In Proceedings of the ACM Symposium on Cloud Computing, SoCC 2018, Carlsbad, CA, USA, October 11-13, 2018, pages 263-274. ACM, 2018. URL: https://doi.org/10.1145/3267809.3267815.
  12. Daniel Balouek, Alexandra Carpen-Amarie, Ghislain Charrier, Frédéric Desprez, Emmanuel Jeannot, Emmanuel Jeanvoine, Adrien Lèbre, David Margery, Nicolas Niclausse, Lucas Nussbaum, Olivier Richard, Christian Pérez, Flavien Quesnel, Cyril Rohr, and Luc Sarzyniec. Adding Virtualization Capabilities to the Grid'5000 Testbed. In Ivan I. Ivanov, Marten van Sinderen, Frank Leymann, and Tony Shan, editors, Cloud Computing and Services Science - Second International Conference, CLOSER 2012, Porto, Portugal, April 18-21, 2012. Revised Selected Papers, volume 367 of Communications in Computer and Information Science, pages 3-20. Springer, 2012. URL: https://doi.org/10.1007/978-3-319-04519-1_1.
  13. André Bauer, Haochen Pan, Ryan Chard, Yadu N. Babuji, Josh Bryan, Devesh Tiwari, Ian T. Foster, and Kyle Chard. The globus compute dataset: An open function-as-a-service dataset from the edge to the cloud. Future Gener. Comput. Syst., 153:558-574, 2024. URL: https://doi.org/10.1016/J.FUTURE.2023.12.007.
  14. João Carreira, Sumer Kohli, Rodrigo Bruno, and Pedro Fonseca. From warm to hot starts: leveraging runtimes for the serverless era. In Sebastian Angel, Baris Kasikci, and Eddie Kohler, editors, HotOS '21: Workshop on Hot Topics in Operating Systems, Ann Arbor , Michigan, USA, June, 1-3, 2021, pages 58-64. ACM, 2021. URL: https://doi.org/10.1145/3458336.3465305.
  15. Marcin Copik, Grzegorz Kwasniewski, Maciej Besta, Michal Podstawski, and Torsten Hoefler. SeBS: a serverless benchmark suite for function-as-a-service computing. In Kaiwen Zhang, Abdelouahed Gherbi, Nalini Venkatasubramanian, and Luís Veiga, editors, Middleware '21: 22nd International Middleware Conference, Québec City, Canada, December 6 - 10, 2021, pages 64-78. ACM, 2021. URL: https://doi.org/10.1145/3464298.3476133.
  16. Robert Cordingly, Hanfei Yu, Varik Hoang, Zohreh Sadeghi, David Foster, David Perez, Rashad Hatchett, and Wes Lloyd. The Serverless Application Analytics Framework: Enabling Design Trade-off Evaluation for Serverless Software. In WoSC@Middleware 2020: Proceedings of the 2020 Sixth International Workshop on Serverless Computing, Virtual Event / Delft, The Netherlands, December 7-11, 2020, pages 67-72. ACM, 2020. URL: https://doi.org/10.1145/3429880.3430103.
  17. Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guanglu Yan, Chenggang Qin, Qixuan Wu, and Haibo Chen. Catalyzer: Sub-millisecond Startup for Serverless Computing with Initialization-less Booting. In James R. Larus, Luis Ceze, and Karin Strauss, editors, ASPLOS '20: Architectural Support for Programming Languages and Operating Systems, Lausanne, Switzerland, March 16-20, 2020, pages 467-481. ACM, 2020. URL: https://doi.org/10.1145/3373376.3378512.
  18. Sadjad Fouladi, Riad S. Wahby, Brennan Shacklett, Karthikeyan Balasubramaniam, William Zeng, Rahul Bhalerao, Anirudh Sivaraman, George Porter, and Keith Winstein. Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads. In Aditya Akella and Jon Howell, editors, 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017, Boston, MA, USA, March 27-29, 2017, pages 363-376. USENIX Association, 2017. URL: https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/fouladi.
  19. Martin Grambow, Tobias Pfandzelter, Luk Burchard, Carsten Schubert, Max Xiaohang Zhao, and David Bermbach. BeFaaS: An Application-Centric Benchmarking Framework for FaaS Platforms. In IEEE International Conference on Cloud Engineering, IC2E 2021 , San Francisco, CA, USA, October 4-8, 2021, pages 1-8. IEEE, 2021. URL: https://doi.org/10.1109/IC2E52221.2021.00014.
  20. Ryan Hancock, Sreeharsha Udayashankar, Ali José Mashtizadeh, and Samer Al-Kiswany. OrcBench: A Representative Serverless Benchmark. In Claudio Agostino Ardagna, Nimanthi L. Atukorala, Rajkumar Buyya, Carl K. Chang, Rong N. Chang, Ernesto Damiani, Gargi Banerjee Dasgupta, Fabrizio Gagliardi, Christoph Hagleitner, Dejan S. Milojicic, Tuan M. Hoang Trong, Robert Ward, Fatos Xhafa, and Jia Zhang, editors, IEEE 15th International Conference on Cloud Computing, CLOUD 2022, Barcelona, Spain, July 10-16, 2022, pages 103-108. IEEE, 2022. URL: https://doi.org/10.1109/CLOUD55607.2022.00028.
  21. Artjom Joosen, Ahmed Hassan, Martin Asenov, Rajkarn Singh, Luke Darlow, Jianfeng Wang, Qiwen Deng, and Adam Barker. Serverless Cold Starts and Where to Find Them, 2024. URL: https://arxiv.org/abs/2410.06145.
  22. Artjom Joosen, Ahmed Hassan, Martin Asenov, Rajkarn Singh, Luke Nicholas Darlow, Jianfeng Wang, and Adam Barker. How Does It Function?: Characterizing Long-term Trends in Production Serverless Workloads. In Proceedings of the 2023 ACM Symposium on Cloud Computing, SoCC 2023, Santa Cruz, CA, USA, 30 October 2023 - 1 November 2023, pages 443-458. ACM, 2023. URL: https://doi.org/10.1145/3620678.3624783.
  23. Vasileios Katevas, Georgios Fatouros, Dimosthenis Kyriazis, and George Kousiouris. Embedding automated function performance benchmarking, profiling and resource usage categorization in function as a service DevOps pipelines. Future Gener. Comput. Syst., 160:223-237, 2024. URL: https://doi.org/10.1016/J.FUTURE.2024.05.051.
  24. Jeongchul Kim and Kyungyong Lee. FunctionBench: A Suite of Workloads for Serverless Cloud Function Service. In Elisa Bertino, Carl K. Chang, Peter Chen, Ernesto Damiani, Michael Goul, and Katsunori Oyama, editors, 12th IEEE International Conference on Cloud Computing, CLOUD 2019, Milan, Italy, July 8-13, 2019, pages 502-504. IEEE, 2019. URL: https://doi.org/10.1109/CLOUD.2019.00091.
  25. Pascal Maissen, Pascal Felber, Peter G. Kropf, and Valerio Schiavoni. FaaSdom: a benchmark suite for serverless computing. In Julien Gascon-Samson, Kaiwen Zhang, Khuzaima Daudjee, and Bettina Kemme, editors, 14th ACM International Conference on Distributed and Event-based Systems, DEBS 2020, Montreal, Quebec, Canada, July 13-17, 2020, pages 73-84. ACM, 2020. URL: https://doi.org/10.1145/3401025.3401738.
  26. Djob Mvondo, Mathieu Bacou, Kevin Nguetchouang, Lucien Ngale, Stéphane Pouget, Josiane Kouam, Renaud Lachaize, Jinho Hwang, Tim Wood, Daniel Hagimont, Noël De Palma, Bernabé Batchakui, and Alain Tchana. OFC: an opportunistic caching system for FaaS platforms. In Antonio Barbalace, Pramod Bhatotia, Lorenzo Alvisi, and Cristian Cadar, editors, EuroSys '21: Sixteenth European Conference on Computer Systems, Online Event, United Kingdom, April 26-28, 2021, pages 228-244. ACM, 2021. URL: https://doi.org/10.1145/3447786.3456239.
  27. David Schall, Artemiy Margaritov, Dmitrii Ustiugov, Andreas Sandberg, and Boris Grot. Lukewarm serverless functions: characterization and optimization. In Valentina Salapura, Mohamed Zahran, Fred Chong, and Lingjia Tang, editors, ISCA '22: The 49th Annual International Symposium on Computer Architecture, New York, New York, USA, June 18 - 22, 2022, pages 757-770. ACM, 2022. URL: https://doi.org/10.1145/3470496.3527390.
  28. Mohammad Shahrad, Jonathan Balkind, and David Wentzlaff. Architectural Implications of Function-as-a-Service Computing. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2019, Columbus, OH, USA, October 12-16, 2019, pages 1063-1075. ACM, 2019. URL: https://doi.org/10.1145/3352460.3358296.
  29. Mohammad Shahrad, Rodrigo Fonseca, Iñigo Goiri, Gohar Irfan Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. In Ada Gavrilovska and Erez Zadok, editors, Proceedings of the 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020, pages 205-218. USENIX Association, 2020. URL: https://www.usenix.org/conference/atc20/presentation/shahrad.
  30. Roberto Starc, Tom Kuchler, Michael Giardino, and Ana Klimovic. Serverless? RISC more! In Proceedings of the 2nd Workshop on SErverless Systems, Applications and MEthodologies, SESAME 2024, Athens, Greece, 22 April 2024, pages 15-24. ACM, 2024. URL: https://doi.org/10.1145/3642977.3652095.
  31. Dmitrii Ustiugov, Theodor Amariucai, and Boris Grot. Analyzing Tail Latency in Serverless Clouds with STeLLAR. In IEEE International Symposium on Workload Characterization, IISWC 2021, Storrs, CT, USA, November 7-9, 2021, pages 51-62. IEEE, 2021. URL: https://doi.org/10.1109/IISWC53511.2021.00016.
  32. Dmitrii Ustiugov, Plamen Petrov, Marios Kogias, Edouard Bugnion, and Boris Grot. Benchmarking, analysis, and optimization of serverless function snapshots. In Tim Sherwood, Emery D. Berger, and Christos Kozyrakis, editors, ASPLOS '21: 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Virtual Event, USA, April 19-23, 2021, pages 559-572. ACM, 2021. URL: https://doi.org/10.1145/3445814.3446714.
  33. Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart, and Michael M. Swift. Peeking behind the curtains of serverless platforms. In Haryadi S. Gunawi and Benjamin C. Reed, editors, Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, MA, USA, July 11-13, 2018, pages 133-146. USENIX Association, 2018. URL: https://www.usenix.org/conference/atc18/presentation/wang-liang.
  34. Ian H. Witten, Eibe Frank, and Mark A. Hall. Data mining: practical machine learning tools and techniques, 3rd Edition. Morgan Kaufmann, Elsevier, 2011. URL: https://www.worldcat.org/oclc/262433473.
  35. Tianyi Yu, Qingyuan Liu, Dong Du, Yubin Xia, Binyu Zang, Ziqian Lu, Pingchao Yang, Chenggang Qin, and Haibo Chen. Characterizing serverless platforms with ServerlessBench. In Rodrigo Fonseca, Christina Delimitrou, and Beng Chin Ooi, editors, SoCC '20: ACM Symposium on Cloud Computing, Virtual Event, USA, October 19-21, 2020, pages 30-44. ACM, 2020. URL: https://doi.org/10.1145/3419111.3421280.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact