Document Open Access Logo

CORA: Config Risk Analyzer - Predicting Risk of Configuration Changes at Scale

Authors Weiyan Sun , Audris Mockus , Brian Ellis , Jun Ge , Madigan Kim , Sahil Kumar , Gursharan Singh , Matt Steiner , Siri Uppalapati , Nachiappan Nagappan

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.86.pdf
  • Filesize: 0.83 MB
  • 22 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software configuration management and version control systems
  • Software and its engineering → Risk management
Keywords
  • Configuration changes
  • risk prediction
  • defect prediction
  • code freeze
  • software supply chain
  • gradient boosting
  • service reliability

Metrics

Abstract

Configuration changes - modifications to feature flags and service parameters - are important to operating services, but they can cause severe outages (SEVs) with substantial service disruption. While code diff risk prediction is well studied, configuration diff risk has received little attention. The code-oriented Diff Risk Score (DRS) model provides limited discriminatory power for config diffs because config risk stems not from code complexity but from the importance, blast radius, and operational readiness of the affected services. We describe the development of CORA (COnfig Risk Analyzer), a dedicated risk model for config diffs. A key insight is that the config-path-to-service mapping bridges from what changed to which services are affected, enabling risk assessment in terms of service criticality rather than code properties.
CORA evolved through three iterations. A logistic regression model with config-specific features achieved an 11% improvement in recall at 5% gating over the DRS model on config diffs. Used for a freeze period, it reduced config gating while outage impact decreased and config diff landing volume increased 91.4%. A unified LightGBM model trained on Meta-wide data achieved a 25.95% improvement in recall at 10% gating, with organization-level improvements ranging from 15% to 66%. An enhanced model with enriched service features and SEV-severity-aware training (having also explored Bayesian hyperparameter optimization) achieved a 22% improvement in recall at 15% gating, intercepting meaningful additional high-severity outage impact in backtesting. To our knowledge, CORA is the first system to apply predictive risk modeling specifically to configuration changes.

Cite As Get BibTex

Weiyan Sun, Audris Mockus, Brian Ellis, Jun Ge, Madigan Kim, Sahil Kumar, Gursharan Singh, Matt Steiner, Siri Uppalapati, and Nachiappan Nagappan. CORA: Config Risk Analyzer - Predicting Risk of Configuration Changes at Scale. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 86:1-86:22, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.86

Author Details

Weiyan Sun
  • Meta Platforms, Inc., Bellevue, WA, USA
Audris Mockus
  • Meta Platforms, Inc., New York, NY, USA
Brian Ellis
  • Meta Platforms, Inc., Bellevue, WA, USA
Jun Ge
  • Meta Platforms, Inc., Menlo Park, CA, USA
Madigan Kim
  • Meta Platforms, Inc., New York, NY, USA
Sahil Kumar
  • Meta Platforms, Inc., Bellevue, WA, USA
Gursharan Singh
  • Meta Platforms, Inc., Menlo Park, CA, USA
Matt Steiner
  • Meta Platforms, Inc., Menlo Park, USA
Siri Uppalapati
  • Meta Platforms, Inc., Sunnyvale, CA, USA
Nachiappan Nagappan
  • Meta Platforms, Inc., Bellevue, WA, USA

Supplementary Materials

References

  1. Rui Abreu, Vijayaraghavan Murali, Peter C. Rigby, Chandra Maddila, Weiyan Sun, Jun Ge, Kaavya Chinniah, Audris Mockus, Megh Mehta, and Nachiappan Nagappan. Moving faster and reducing risk: Using LLMs in release deployment. In Proceedings of the 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2025. URL: https://doi.org/10.1109/icse-seip66354.2025.00045.
  2. Bram Adams and Shane McIntosh. Modern release engineering in a nutshell - why researchers should care. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), volume 5, pages 78-90, 2016. URL: https://doi.org/10.1109/SANER.2016.108.
  3. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2623-2631, 2019. URL: https://doi.org/10.1145/3292500.3330701.
  4. Fran Baum, Colin MacDougall, and Danielle Smith. Participatory action research. Journal of Epidemiology and Community Health, 60(10):854-857, 2006. URL: https://doi.org/10.1136/jech.2004.028662.
  5. Kaiwen Chow, Yanbing Wang, Qingsen Wang, and Andromeda Hailu. ServiceLab: Preventing tiny performance regressions at hyperscale through pre-production testing. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 185-202. USENIX Association, 2024. URL: https://www.usenix.org/conference/osdi24/presentation/chow.
  6. Daniel Alencar Da Costa, Shane McIntosh, Weiyi Shang, Uirá Kulesza, Roberta Coelho, and Ahmed E. Hassan. A framework for evaluating the results of the SZZ approach for identifying bug-introducing changes. IEEE Transactions on Software Engineering, 43(7):641-657, 2017. URL: https://doi.org/10.1109/TSE.2016.2616306.
  7. Carsten F. Dormann, Jane Elith, Sven Bacher, Carsten Buchmann, Gudrun Carl, Gabriel Carré, Jaime R. García Marquéz, Bernd Gruber, Bruno Lafourcade, Pedro J. Leitão, et al. Collinearity: A review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36(1):27-46, 2013. URL: https://doi.org/10.1111/j.1600-0587.2012.07348.x.
  8. Todd L. Graves, Alan F. Karr, J. S. Marron, and Harvey Siy. Predicting fault incidence using software change history. IEEE Transactions on Software Engineering, 26(7):653-661, 2000. URL: https://doi.org/10.1109/32.859533.
  9. Mislav Grubić, Josip Balić, Ilias Leontiadis, Soteris Demetriou, Dat Duong, and Kaushik Veeraraghavan. Conveyor: One-tool-fits-all continuous software deployment at Meta. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 539-555. USENIX Association, 2023. URL: https://www.usenix.org/conference/osdi23/presentation/grubic.
  10. Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E. Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi. A large-scale empirical study of just-in-time quality assurance. IEEE Transactions on Software Engineering, 39(6):757-773, 2013. URL: https://doi.org/10.1109/TSE.2012.70.
  11. Leo Katz. A new status index derived from sociometric analysis. Psychometrika, 18(1):39-43, 1953. URL: https://doi.org/10.1007/BF02289026.
  12. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30 (NeurIPS), pages 3146-3154, 2017. URL: https://proceedings.neurips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html.
  13. Hossein Keshavarz and Meiyappan Nagappan. ApacheJIT: A large dataset for just-in-time defect prediction. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR), pages 191-195, 2022. URL: https://doi.org/10.1145/3524842.3527996.
  14. Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30 (NeurIPS), pages 4765-4774, 2017. URL: https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html.
  15. Audris Mockus. Insights from open source supply chains. In Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), page 3, 2019. Industry Keynote. URL: https://doi.org/10.1145/3338906.3342813.
  16. Audris Mockus. Securing LLM-based software supply chains. In 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), Workshop on LLMs in Software Engineering, 2023. Invited talk. Google Scholar
  17. Audris Mockus, Rui Abreu, Peter C. Rigby, David Amsallem, Parveen Bansal, Kaavya Chinniah, Brian Ellis, Peng Fan, Jun Ge, Bingjie He, Kelly Hirano, Sahil Kumar, Ajay Lingapuram, Andrew Loe, Megh Mehta, Venus Montes, Maher Saba, Gursharan Singh, Matt Steiner, Weiyan Sun, Siri Uppalapati, and Nachiappan Nagappan. Leveraging risk models to improve productivity for effective code un-freeze at scale. ACM Transactions on Software Engineering and Methodology, 2025. URL: https://doi.org/10.1145/3722216.
  18. Audris Mockus, Randy L. Hackbarth, and John Palframan. Risky files: An approach to focus quality improvement effort. In Proceedings of the 9th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 691-694, 2013. URL: https://doi.org/10.1145/2491411.2494572.
  19. Audris Mockus and David M. Weiss. Predicting risk of software changes. Bell Labs Technical Journal, 5(2):169-180, 2000. URL: https://doi.org/10.1002/bltj.2229.
  20. Nachiappan Nagappan and Thomas Ball. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th International Conference on Software Engineering (ICSE), pages 284-292, 2005. URL: https://doi.org/10.1109/ICSE.2005.1553571.
  21. Changhai Nie and Hareton Leung. A survey of combinatorial testing. ACM Computing Surveys, 43(2):11:1-11:29, 2011. URL: https://doi.org/10.1145/1883612.1883618.
  22. Jalaj Pachouly, Swati Ahirrao, Ketan Kotecha, Ganeshsree Selvachandran, and Ajith Abraham. A systematic literature review on software defect prediction using artificial intelligence: Datasets, data validation methods, approaches, and tools. Engineering Applications of Artificial Intelligence, 111:104773, 2022. URL: https://doi.org/10.1016/j.engappai.2022.104773.
  23. Phacility. Phabricator. https://www.phacility.com/phabricator/. Open-source code review platform, originally developed at Facebook.
  24. Md Tajmilur Rahman and Peter C. Rigby. Release stabilization on Linux and Chrome. IEEE Software, 32(2):81-88, 2015. URL: https://doi.org/10.1109/MS.2015.31.
  25. Ernest T. Stringer and Alfredo Ortiz Aragón. Action Research. SAGE Publications, 5th edition, 2020. Google Scholar
  26. Chunqiang Tang. Meta’s hyperscale infrastructure: Overview and insights. Communications of the ACM, 68(2):52-63, 2025. URL: https://doi.org/10.1145/3701296.
  27. Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl. Holistic configuration management at Facebook. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP), pages 328-343. ACM, 2015. URL: https://doi.org/10.1145/2815400.2815401.
  28. Linus Torvalds. Linux code freeze. Linux Journal, 1994(1es):4-es, 1994. Google Scholar
  29. Chadd Williams and Jaime Spacco. SZZ revisited: Verifying when changes induce fixes. In Proceedings of the 2008 Workshop on Defects in Large Software Systems (DLSD), pages 32-36, 2008. URL: https://doi.org/10.1145/1390817.1390826.
  30. Yunhua Zhao, Kostadin Damevski, and Hui Chen. A systematic survey of just-in-time software defect prediction. ACM Computing Surveys, 55(10):1-35, 2023. URL: https://doi.org/10.1145/3567550.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact