Search Results

Configuration changes - modifications to feature flags and service parameters - are important to operating services, but they can cause severe outages (SEVs) with substantial service disruption. While code diff risk prediction is well studied, configuration diff risk has received little attention. The code-oriented Diff Risk Score (DRS) model provides limited discriminatory power for config diffs because config risk stems not from code complexity but from the importance, blast radius, and operational readiness of the affected services. We describe the development of CORA (COnfig Risk Analyzer), a dedicated risk model for config diffs. A key insight is that the config-path-to-service mapping bridges from what changed to which services are affected, enabling risk assessment in terms of service criticality rather than code properties. CORA evolved through three iterations. A logistic regression model with config-specific features achieved an 11% improvement in recall at 5% gating over the DRS model on config diffs. Used for a freeze period, it reduced config gating while outage impact decreased and config diff landing volume increased 91.4%. A unified LightGBM model trained on Meta-wide data achieved a 25.95% improvement in recall at 10% gating, with organization-level improvements ranging from 15% to 66%. An enhanced model with enriched service features and SEV-severity-aware training (having also explored Bayesian hyperparameter optimization) achieved a 22% improvement in recall at 15% gating, intercepting meaningful additional high-severity outage impact in backtesting. To our knowledge, CORA is the first system to apply predictive risk modeling specifically to configuration changes.