Resilient Software Configuration and Infrastructure Code Analysis

Configurable software makes up most of the software in use today. Configurability, i.e., the ability of software to be customized without additional programming, is pervasive, and due to the criticality of problems caused by misconfiguration, it has been an active topic researched by investigators in multiple, diverse areas. This broad reach of configurability means that much of the literature and latest results are dispersed, and researchers may not be collaborating or be aware of similar problems and solutions in other domains. We argue that this lack of a common ground leads to a missed opportunity for synergy between research domains and the synthesis of efforts to tackle configurability problems. To provide a foundation for addressing these concerns, we suggest how to bring the communities together and propose a common model of configurability and a platform, ACCORD, to facilitate collaboration among researchers and practitioners.

From a distance, Infrastructure as Code should really be called Infrastructure as Configuration. DevOps, SRE and other engineers dealing with infrastructure are mostly managing pure configuration data by writing, editing and auditing JSON, YAML or similar serialization formats.

Purely data-oriented languages like JSON might be fine for managing small and simple infrastructure, but when the size and complexity of a configuration grows, data languages don’t seem to be the right tool anymore.

There is no way to reuse data and have a single source of truth, with all the pain and inconsistencies that duplication inevitably brings with time. We can’t express data dependencies either. For example, the open ports of a firewall may depend on which services are enabled on a server. In JSON, everything must be hardcoded.

Infrastructure tools have thus incorporated programming features in their languages (Terraform, Ansible, Puppet, etc.). But those are often unplanned and unprincipled, resulting in a complex system.

In the end, Configuration Management looks like the poor sibling of Software Engineering. What about tests, types, LSP integration for real-time feedback, completion, and documentation? What about modularity, code reuse, and abstraction?

In this talk, I will present Nickel, a configuration programming language I am currently working on at Tweag, to help finally enter the era of Configuration as Code. I’ll discuss more specifically the approach of Nickel to modularity, which is how to write small and reusable configuration snippets that can be combined into a complex configuration based on a merging operation. I’ll talk about type-checking and built-in schema validation as well.

Configuration is in more places than people imagine. Every part of your tech stack – databases, apps, schemas, services, workflows, policy, models, and networking must be configured. Not only that, there are dependencies between these configurations. With cloud, multi-cloud, IoT, and edge computing, the number of things to configure within growing systems has exploded. The consequences of getting the configuration wrong have only worsened over time.

We are in the middle of a major shift where configuration is becoming a first-class citizen across your stack. New engineering roles dedicated to solving configuration problems have emerged: Platform, Site Reliability, Resilience, Observability, Data, “DevOps” and “YAML” engineers all deal with easing configuration toil. There are many tools and approaches that aim to help developers deal with growing configuration complexity: Configuration Management, Infrastructure as Code, GitOps, Policy as Code, and finally, Infrastructure as Data. The industry needs something to unify all these roles, approaches, and challenges and break down configuration silos. CUE (cuelang.org) does exactly that.

In this presentation, you will hear:

• $\mathrm{■}$

  hard-won insights and experiences of configuration at scale, culminating in the design of CUE,

• $\mathrm{■}$

  how configuration can go wrong,

• $\mathrm{■}$

  the need for testing and validation,

• $\mathrm{■}$

  how CUE is paving the way for a holistic approach to configuration via a language, tooling, and APIs that support a vibrant configuration ecosystem.

Configuration is inherently cross-cutting. Combing configuration, therefore, requires commutative and associative composition. In this tutorial, we show how to use CUE to model all configuration aspects of a distributed system into a single space. Configuration aspects can be data, schema, policy, validation, and templates, or any combination thereof.

This talk provided an overview of the qualitative analysis we conducted to summarize the industrial gray literature (e.g., blog posts, tutorials, white papers) on IaC.

In particular, it provided a general definition of IaC and a broad catalog outlined in taxonomy consisting of ten and four primary categories for best practices and bad practices, respectively, both language-agnostic and language-specific ones, for three IaC languages of best and bad practices for widely used IaC languages (i.e., Ansible, Puppet, and Chef).

Our findings highlighted that the IaC development and maintenance field is in its infancy and deserves further attention.

We report on industrial configuration problems outside the field of infrastructure as code. We focus on our work on automotive configuration which goes back to the 1990s and continues until today by providing software to the industry in the context of our Steinbeis Transfer Centre STZ OIT.

Automotive configuration is managed on two levels using (a flavor of) Boolean Algebra. Every equipment or sales option is represented by a Boolean variable x, where x=true represents the presence of the option in a car, and x=false represents the absence, such that a valid car order is given by a complete valuation of all variables. The upper level, model description, comprises the configuration rules for an entire model line of cars (e.g. Mercedes C-Class). The lower level, the Bill-of-Materials, contains the list of all materials needed for the model line. Each material also has a selection formula, and the material is needed to produce a car order if the selection formula evaluates to true under the order.

Problems on the upper level include the computation of forced options, which must be contained in any car, and impossible options, which cannot be contained in any car. Problems on the lower level include the detection of car orders which will select alternative materials or will fail to select a necessary material. These problems are efficiently solved by modern CDCL SAT-Solvers, although the typical variance of the model description is practically infinite (more than a trillion).

Beyond the detection of configuration errors, we proceed to configuration optimization. As an example, we describe the problem of computing the configuration of optimal (minimal) sets of test vehicles on which a given set of testing demands can be carried out.

In addition, we report on some other industrial configuration verification problems whose solutions we published in the past. This includes the detection of misconfigurations of Storage Area Networks (SAN) and of Apache Webservers, the analysis of LINUX Kernel configurations, and the analysis of the completeness of the online help system of a line of computer tomographs.

The shell is a critical part of modern software operations. I have three tools for helping you work with the shell:

• $\mathrm{■}$

  libdash, bindings to the dash parser in OCaml and Python

• $\mathrm{■}$

  smoosh, an executable small-step operational semantics tested against the POSIX spec

• $\mathrm{■}$

  pash-annotations, specifications of common commands

The continuous development of new services and the time pressure imposed by the development of new features commonly result in the introduction of non-optimal and temporary solutions. As a result, the software architecture is commonly not compliant with the originally designed one, and nobody has a complete view of the whole system. Static and dynamic analysis methods can help to reconstruct the architecture of the system.

In this talk, we showed some possible techniques for reconstructing the architecture from different perspectives: the infrastructural, the structure of the system analyzed statically or dynamically, and the structure of the organization.

Last we presented a roadmap of possible shared research work towards the development or the extension of tools for IaC architectural reconstruction.

DevOps unifies software development and operations in cross-functional teams to improve software delivery and operations (SDO) performance. Ideally, cross-functional DevOps teams independently deploy their services, but the correct operation of a service often demands other services, requiring coordination to ensure the correct deployment order. This issue is currently solved either with a central deployment or manual out-of-band communication across teams, e.g., via phone, chat, or email. Unfortunately, both contradict the independence of teams, hindering SDO performance – the reason why DevOps is adopted in the first place.

We conducted a study on 134 IT professionals, showing that, in practice, they resort to manual coordination for correct deployments even if they expect better SDO performance with fully automated approaches. We find that Infrastructure as Code (IaC) automates deployments for single teams, falling short of decentralized deployments across groups. To enable testing and automation advances for decentralized organizations, we need mature IaC solutions that embrace and consolidate software engineering principles.

To address this issue, we proposed $\mu$s ([mju:z] “muse”), a novel IaC system automating deployment coordination in a fully decentralized fashion, still retaining compatibility with DevOps practice – in contrast to today’s solutions. We implement µs, demonstrate that it effectively enables automated coordination, introduces negligible definition overhead, has no performance overhead, and is broadly applicable, as shown by the migration of 64 third-party IaC projects.

Answer set programming (ASP) is a powerful tool that folks in PL/FM/SE do not know about. I give two case studies applying ASP to synthesis (once for Datalog, once for RBAC policies); I also explain how ASP relates to SAT (its parent) and SMT (its sibling).

This talk presents our journey and experiences in transposing static analyses from application to infrastructure code.

We realized the need to analyze infrastructure code while designing the Chaokka automated tester. Chaokka injects cloud-specific faults during the execution of a test suite to assess the resilience of a cloud-native application against these faults. The approach uses delta debugging to speed up the exploration of the corresponding fault space, which we expect to benefit further from architectural insights extracted from the supporting infrastructure code (e.g., how many replicas the application maintains of each service, …).

The second part of the talk focuses on analyses for infrastructure code in general and Ansible in particular. To this end, we present two static representations of Ansible code. First, we describe a structural model akin to an AST and its applications in empirical research on Semantic Versioning in the Ansible Galaxy ecosystem. Second, we present a program dependence graph representation that captures an Ansible script’s control and data flow. We describe the challenges faced when building such graphs caused by unconventional and undocumented Ansible semantics. Finally, we describe two applications of Ansible program dependence graphs, namely the detection of variable-related smells and a graph pattern mining approach to defect detection.

With Infrastructure as Code (Iac), the process of configuring complex infrastructure has been simplified and made more accessible to developers. However, with the new expressive power of IaC and the consequentially more complex cloud infrastructure deployments, understanding this complexity has emerged as an unsolved issue. In particular, estimating the cost of an Infrastructure as Code deployment requires understanding the pricing models of every cloud resource being used, as well as an understanding of the interactions between the resources. Existing work either relies on historical usage metrics to predict cost (which has limited utility for new deployments), or on coarse-grain static cost analysis that ignores interactions between resources. We pose as a challenge to the community the need for fine-grained static cost analysis techniques for IaC that incorporate reasoning about the interactions between resources. We walk through a motivating example of this problem that demonstrates the need for such analysis as well as the complexity of the problem.

The anatomy of infrastructure code, as well as all elements and abstractions necessary in writing and maintaining that blueprint, are addressed, among other IaC formats, in the industrial standard for IaC, namely, the OASIS “Topology and Orchestration Specification for Cloud Applications” (TOSCA) industrial standard adopted by as many as 60+ big industrial players worldwide.

Paraphrasing from the standard specification itself, “TOSCA […] uses […] service templates to describe cloud apps as a topology template, […]. TOSCA provides a type system of node types to describe the possible building blocks for constructing a service template and relationship type to describe possible relations”. TOSCA-ready orchestrators such as Cloudify or Apache Brooklyn normally come with a considerable number of reusable TOSCA nodes (e.g., a MySQL DB or a WordPress host), while more are proliferating in both research and practice.

Although several alternatives to TOSCA exist (e.g., HashiCorp Terraform, Ubuntu Juju), for the sake of simplicity, we focus on outlining the anatomy of infrastructure code using the technology-agnostic TOSCA standard notation and its intended levels of abstraction.

The behavior of large systems is guided by their configurations: users set parameters in the configuration file to dictate which corresponding part of the system code is executed. However, it is often the case that, although some parameters are set in the configuration file, they do not influence the system runtime behavior, thus failing to meet the user’s intent. Moreover, such misconfigurations rarely lead to an error message or raising an exception. We introduce the notion of silent misconfiguration, which are prohibitively hard to identify due to (1) lack of feedback and (2) complex interactions between configurations and code.

In this talk we present ConfigX, the first tool for the detection of silent misconfigurations. The main challenge is understanding the complex interactions between configurations and the code they affected. Our goal is to derive a specification describing non-trivial interactions between the configuration parameters that lead to silent misconfigurations. To this end, ConfigX uses static analysis to determine which parts of the system code are associated with configuration parameters. ConfigX then infers the connections between configuration parameters by analyzing their associated code blocks. We design customized control- and data-flow analysis to derive a specification of configurations. Additionally, we conduct reachability analysis to eliminate spurious rules to reduce false positives. Upon evaluation on five real-world datasets across three widely-used systems, Apache, vsftpd, and PostgreSQL, ConfigX detected more than 2200 silent misconfigurations. We additionally conducted a user study where we ran ConfigX on misconfigurations reported on user forums by real-world users. ConfigX easily detected issues and suggested repairs for those misconfigurations. Our solutions were accepted and confirmed in the interaction with the users who originally posted the problems.

Setting up complex, large-scale infrastructure is an iterative process that includes installing dependencies and adjusting parameters within a vast space of configuration options and eventual infrastructure testing in a trial-and-error fashion. The result of this process serves as the underlying base for the deployment and execution of computation defined in the program code. This ad-hoc style of infrastructure setup cannot scale to operations with scalable compute instances (i.e., cloud computing). Infrastructure as code allows to express all infrastructure concerns in the form of code, enabling automation to achieve scalability, transparency, and reproducibility. However, creating code for infrastructure inherently differs from writing program code. The fast feedback cycles required for the iterative setup process outlined above are not feasible when infrastructure code is written in the same way as program code. Thus, infrastructure code is commonly written retrospectively in a cumbersome back-and-forth process querying configuration parameters that are prone to human error. My proposed research aims to intercept lower-level system interactions by experts setting up systems interactively and infer higher-level actions. We generate action sequences guided by the existing infrastructure serving as an oracle goal state.