Document Open Access Logo

Evaluating CI/CD Security Best Practices in the GitHub Actions Documentation

Authors Lukas Boschanski , Marco Vieira

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.15.pdf
  • Filesize: 1.1 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Documentation
Keywords
  • CI/CD security
  • Software documentation
  • Security best practices

Metrics

Abstract

Background. GitHub Actions, the most widely used Continuous Integration and Continuous Deployment (CI/CD) platform, is frequently involved in large-scale software supply chain attacks. While prior work has focused on detecting vulnerable CI/CD pipelines, research has rarely considered preventive perspectives such as analyzing Security Best Practices (SBPs) in CI/CD documentation. 

Aims. We investigate the completeness of the official GitHub Actions documentation from a security perspective and identify potential improvements to documenting secure CI/CD practices.

Method. We employ an Large Language Model (LLM)-based documentation mining pipeline to extract security advice items verbatim. We derive actionable CI/CD SBPs from the OWASP Top 10 CI/CD Security Risks framework and conduct a qualitative analysis by mapping the extracted advice items against these best practices to assess which best practices are explicitly covered. 

Results. Across 639 documentation pages, we identify 459 security-related advice items, of which 288 are actionable, but only 50% of them map to actionable SBPs. These items primarily address insecure configurations and insufficient credential hygiene. Furthermore, only half of all security advice visual alerts have an adequate warning type. 

Conclusions. The GitHub Actions documentation lacks a consistent methodology for incorporating SBPs. Moreover, coverage of established CI/CD security best practices is uneven across OWASP risk categories, with several categories receiving little to no actionable guidance.

Cite As Get BibTex

Lukas Boschanski and Marco Vieira. Evaluating CI/CD Security Best Practices in the GitHub Actions Documentation. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 15:1-15:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.15

Author Details

Lukas Boschanski
  • University of North Carolina at Charlotte, NC, USA
Marco Vieira
  • University of North Carolina at Charlotte, NC, USA

Supplementary Materials

References

  1. Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L. Mazurek, and Christian Stransky. You Get Where You're Looking for: The Impact of Information Sources on Code Security. In 2016 IEEE Symposium on Security and Privacy (SP), pages 289-305, May 2016. ISSN: 2375-1207. URL: https://doi.org/10/ghd865.
  2. Yasemin Acar, Christian Stransky, Dominik Wermke, Charles Weir, Michelle L. Mazurek, and Sascha Fahl. Developers Need Support, Too: A Survey of Security Advice for Software Developers. In 2017 IEEE Cybersecurity Development (SecDev), pages 22-26, September 2017. URL: https://doi.org/10/hb4q6d.
  3. Jessy Ayala, Yu-Jye Tung, and Joshua Garcia. A Mixed-Methods study of Open-Source software maintainers on vulnerability management and platform security features. In 34th USENIX Security Symposium (USENIX Security 25), pages 2105-2124, Seattle, WA, August 2025. USENIX Association. URL: https://www.usenix.org/conference/usenixsecurity25/presentation/ayala.
  4. David Barrera, Christopher Bellman, and Paul Van Oorschot. Security Best Practices: A Critical Analysis Using IoT as a Case Study. ACM Transactions on Privacy and Security, 26(2):13:1-13:30, March 2023. URL: https://doi.org/10/hb4q6f.
  5. David Barrera, Christopher Bellman, and Paul C van Oorschot. A close look at a systematic method for analyzing sets of security advice. Journal of Cybersecurity, 9(1), January 2023. URL: https://doi.org/10/hb4q6g.
  6. Olga Bedrina. The State of CI/CD in 2025: Key Insights from the Latest JetBrains Survey | The TeamCity Blog, October 2025. URL: https://blog.jetbrains.com/teamcity/2025/10/the-state-of-cicd/.
  7. Christopher Bellman and Paul C. van Oorschot. Systematic analysis and comparison of security advice as datasets. Computers & Security, 124:102989, January 2023. URL: https://doi.org/10/hb4q6h.
  8. Giacomo Benedetti, Luca Verderame, and Alessio Merlo. Automatic Security Assessment of GitHub Actions Workflows. In Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses, pages 37-45, November 2022. URL: https://doi.org/10/hb4q6j.
  9. Lukas Boschanski and Marco Vieira. CI/CD Security Best Practices and GitHub Actions Documentation. Software (visited on 2026-08-24). URL: https://github.com/bluuuk/gh-cicd-sbp-docs
    full metadata available at: https://doi.org/10.4230/artifacts.27689
  10. Philipp Burckhardt. Trivy under attack again: Widespread github actions tag compromise exposes ci/cd secrets, March 2026. Accessed: 2026-04. URL: https://socket.dev/blog/trivy-under-attack-again-github-actions-compromise.
  11. Ramaswamy Chandramouli, Frederick Kautz, and Santiago Torres-Arias. Strategies for the integration of software supply chain security in devsecops ci/cd pipelines. Technical Report 800-204D, National Institute of Standards and Technology, Gaithersburg, MD, February 2024. Approved 2024-01-31. URL: https://doi.org/10.6028/NIST.SP.800-204D.
  12. Continuous Delivery Foundation. CI/CD Cybersecurity Guide, 2025. URL: https://cybersecurity.cd.foundation/_print/docs/.
  13. Omer Gil, Aviad Hahami, Asi Greenholts, and Yaron Avital. GitHub Actions Supply Chain Attack: A Targeted Attack on Coinbase Expanded to the Widespread tj-actions/changed-files Incident: Threat Assessment (Updated 4/2), March 2025. URL: https://unit42.paloaltonetworks.com/github-actions-supply-chain-attack/.
  14. GitHub. Documentation done right: A developer’s guide, 2024. URL: https://github.blog/developer-skills/documentation-done-right-a-developers-guide/.
  15. Google. Langextract: A python library for extracting structured information from unstructured text using llms, 2026. Accessed: 2026-05-18. URL: https://github.com/google/langextract.
  16. Peter L Gorski, Emanuel von Zezschwitz, Luigi Lo Iacono, and Matthew Smith. On providing systematized access to consolidated principles, guidelines and patterns for usable security research and development. Journal of Cybersecurity, 5(1), January 2019. URL: https://doi.org/10/hb4q6q.
  17. Peter Leo Gorski, Sebastian Möller, Stephan Wiefling, and Luigi Lo Iacono. "I just looked for the solution!"On Integrating Security-Relevant Information in Non-Security API Documentation to Support Secure Coding Practices. IEEE Transactions on Software Engineering, 48(9):3467-3484, September 2022. URL: https://doi.org/10/hb4q6r.
  18. Yacong Gu, Lingyun Ying, Huajun Chai, Chu Qiao, Haixin Duan, and Xing Gao. Continuous Intrusion: Characterizing the Security of Continuous Integration Services. In 2023 IEEE Symposium on Security and Privacy (SP), pages 1561-1577, May 2023. URL: https://doi.org/10/hb4q6s.
  19. Jiangnan Huang and Bin Lin. Revisiting Security Practices for Github Actions Workflows. In 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC), pages 73-77, 2025. URL: https://doi.org/10/hb4q6t.
  20. Igibek Koishybayev, Aleksandr Nahapetyan, Raima Zachariah, Siddharth Muralee, Bradley Reaves, Alexandros Kapravelos, and Aravind Machiry. Characterizing the Security of Github CI Workflows. In 31st USENIX Security Symposium (USENIX Security 22), pages 2747-2763, Boston, MA, August 2022. USENIX. URL: https://www.usenix.org/conference/usenixsecurity22/presentation/koishybayev.
  21. Daniel Krivelevich, Omer Gil, and Ory Segal. OWASP Top 10 CI/CD Security Risks | OWASP Foundation. URL: https://owasp.org/www-project-top-10-ci-cd-security-risks/.
  22. Yusuke Kubo, Fumihiro Kanei, Mitsuaki Akiyama, Takuro Wakai, and Tatsuya Mori. Action Required: A Mixed-Methods Study of Security Practices in GitHub Actions. In NDSS. The Internet Society, February 2026. URL: https://doi.org/10/q64h.
  23. Xiaofan Li, Yacong Gu, Chu Qiao, Zhenkai Zhang, Daiping Liu, Lingyun Ying, Haixin Duan, and Xing Gao. Toward Understanding the Security of Plugins in Continuous Integration Services. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 482-496. ACM, 2024. URL: https://doi.org/10/hb4q6v.
  24. Tanner Linsley. Postmortem: Tanstack npm supply-chain compromise, May 2026. URL: https://tanstack.com/blog/npm-supply-chain-compromise-postmortem.
  25. Xiaoyue Ma, Lannan Luo, and Qiang Zeng. From One Thousand Pages of Specification to Unveiling Hidden Bugs: Large Language Model Assisted Fuzzing of Matter IoT Devices. In 33rd USENIX Security Symposium (USENIX Security 24), pages 4783-4800. USENIX, August 2024. Google Scholar
  26. Rami McCarthy Merav Bar. s1ngularity: supply chain attack leaks secrets on github: everything you need to know, August 2025. URL: https://www.wiz.io/blog/s1ngularity-supply-chain-attack.
  27. Siddharth Muralee, Igibek Koishybayev, Aleksandr Nahapetyan, Greg Tystahl, Brad Reaves, Antonio Bianchi, William Enck, Alexandros Kapravelos, and Aravind Machiry. ARGUS: A Framework for Staged Static Taint Analysis of GitHub Workflows and Actions. In 32nd USENIX Security Symposium (USENIX Security 23), pages 6983-7000. USENIX, August 2023. Google Scholar
  28. National Security Agency and Cybersecurity and Infrastructure Security Agency. Defending Continuous Integration/Continuous Delivery (CI/CD) Environments. Technical report, National Security Agency and Cybersecurity and Infrastructure Security Agency, June 2023. URL: https://media.defense.gov/2023/Jun/28/2003249466/-1/-1/0/CSI_DEFENDING_CI_CD_ENVIRONMENTS.PDF.
  29. Lorenzo Neil, Harshini Sri Ramulu, Yasemin Acar, and Bradley Reaves. Who Comes Up with this Stuff? Interviewing Authors to Understand How They Produce Security Advice. In Proceedings of the Nineteenth USENIX Conference on Usable Privacy and Security, SOUPS '23, pages 283-299. USENIX, 2023. URL: https://www.usenix.org/conference/soups2023/presentation/neil.
  30. Greg Ose. What’s coming to our github actions 2026 security roadmap, March 2026. Accessed: 2026-04. URL: https://github.blog/news-insights/product-news/whats-coming-to-our-github-actions-2026-security-roadmap/.
  31. Ziyue Pan, Wenbo Shen, Xingkai Wang, Yutian Yang, Rui Chang, Yao Liu, Chengwei Liu, Yang Liu, and Kui Ren. Ambush From All Sides: Understanding Security Threats in Open-Source Software CI/CD Pipelines. IEEE Transactions on Dependable and Secure Computing, 21(1):403-418, 2024. URL: https://doi.org/10/hb4q6x.
  32. Mark Quinlan, Aaron Ceross, and Andrew Simpson. The Efficacy Potential of Cyber Security Advice as Presented in News Articles. Interacting with Computers, 37(1):30-48, January 2025. URL: https://doi.org/10.1093/IWC/IWAE048.
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250, 2016. URL: https://arxiv.org/abs/1606.05250.
  34. Elissa M. Redmiles, Amelia R. Malone, and Michelle L. Mazurek. I Think They're Trying to Tell Me Something: Advice Sources and Selection for Digital Security. In 2016 IEEE Symposium on Security and Privacy (SP), pages 272-288, May 2016. URL: https://doi.org/10/gnrpc8.
  35. Elissa M. Redmiles, Noel Warford, Amritha Jayanti, Aravind Koneru, Sean Kross, Miraida Morales, Rock Stevens, and Michelle L. Mazurek. A Comprehensive Quality Evaluation of Security and Privacy Advice on the Web. In 29th USENIX Security Symposium (USENIX Security 20), pages 89-108, 2020. URL: https://www.usenix.org/conference/usenixsecurity20/presentation/redmiles.
  36. Robert W. Reeder, Iulia Ion, and Sunny Consolvo. 152 Simple Steps to Stay Safe Online: Security Advice for Non-Tech-Savvy Users. IEEE Security & Privacy, 15(5):55-64, 2017. URL: https://doi.org/10/hb4q63.
  37. Andrew Stewart and Christopher Hobbs. Systematic analysis of security advice on the topic of insider threats. Computers & Security, 154:104411, July 2025. URL: https://doi.org/10/hb4q67.
  38. Mindy Tran, Yasemin Acar, Michel Cucker, William Enck, Alexandros Kapravelos, Christian Kastner, and Laurie Williams. S3c2 summit 2202-09: Industry secure suppy chain summit, 2023. URL: https://doi.org/10.48550/arXiv.2307.15642.
  39. Christoph Treude, Justin Middleton, and Thushari Atapattu. Beyond accuracy: Assessing software documentation quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pages 1509-1512. ACM, 2020. URL: https://doi.org/10/hb4q68.
  40. Greg Tystahl, Jonah Ghebremichael, Siddharth Muralee, Sourag Cherupattamoolayil, Antonio Bianchi, Aravind Machiry, Alexandros Kapravelos, and William Enck. Cosseter: Github actions permission reduction using demand-driven static analysis. In 2026 IEEE Symposium on Security and Privacy (SP), pages 2628-2645, 2026. URL: https://doi.org/10.1109/SP63933.2026.00067.
  41. Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C. Gall, and Massimiliano Di Penta. Configuration smells in continuous delivery pipelines: A linter and a six-month study on GitLab. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pages 327-337. ACM, 2020. URL: https://doi.org/10/hb4q69.
  42. Laurie Williams, Giacomo Benedetti, Sivana Hamer, Ranindya Paramitha, Imranur Rahman, Mahzabin Tamanna, Greg Tystahl, Nusrat Zahan, Patrick Morrison, Yasemin Acar, Michel Cukier, Christian Kästner, Alexandros Kapravelos, Dominik Wermke, and William Enck. Research Directions in Software Supply Chain Security. ACM Transactions on Software Engineering and Methodology, 34(5):146:1-146:38, 2025. URL: https://doi.org/10/g9qww8.
  43. Xinyu Zhang, Siddharth Muralee, Sourag Cherupattamoolayil, and Aravind Machiry. On the Effectiveness of Large Language Models for GitHub Workflows. In Proceedings of the 19th International Conference on Availability, Reliability and Security, ARES '24, pages 1-14. ACM, 2024. URL: https://doi.org/10/hb4q7g.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact