Empirical Evaluation of Secure Development Processes

In practice, there is a significant difference in what different reviewers look for in code reviews and their performance. The job of security experts is to find the one critical bug. In contrast to this, the general developer is likely to accept reviews that consist of more than 70% true positive findings.

While it is essential to find as many security bugs as possible, a significant practical barrier is false positive findings that hinder addressing security effectively. False positive findings have to be resolved by humans that come with their individual preferences and might be more effective in handling specific kinds of false positive findings. To optimize the outcomes of code reviews, it is important to tailor these closer to the target audience of the review. To this end, it could be beneficial to learn what are the usual most favorite false positive findings of different stakeholders.

A general approach for optimizing tool-assisted reviews could be to learn from current false positives to mitigate them in the future. Such learning must be prepared by intensive data mining. However, to optimize the reviews, we must identify the false positive findings to avoid in the future, we must rank all the identified false positives according to some criteria that have yet to be identified. One possibility could be a rating of how critical a specific kind of finding is to fix. The intuition is that developers will start to work on the most critical finding kinds and benefit there the most from fewer false positives. Another possibility could be to consider manual downranks of developers.

For all of these metrics, one practical barrier is to get hands on the necessary data, which is usually not publicly available. One would have to massively collect practical feedback from developers that label the findings of review tools. Since it is not feasible to request a single developer to label all findings of each analysis tool, one could divide the work among multiple parties.

This huge effort in labeling, which is a huge barrier to academic studies to improve the analysis tools, is also a huge barrier in practice when a new tool should be deployed. After the deployment, the outlined task has to be performed to identify the findings of the new analyzer that must be fixed. Here, the application of the new analyzer on only newly written code can help to reduce the number of findings. Developers can focus on the new code they are working on and improve it while reducing the risk of the outcomes of the new analysis tool being ignored entirely. Still, there will be less precise results than expected, since many details have to be trained, including optimizing the configuration of the new tool and also the developers themselves.

Despite these outlined challenges in getting hands with the number of findings and false positives, incremental scans and synchronizing scans across branches are huge barriers. In particular, a false positive identified and labeled in one branch should be also labeled on the other branches on which it is present as a false positive finding.

While these individual observations and ideas sound reasonable, it remains to gather data in an empirical analysis to precisely identify how large the effort of revies for developers is. Thereby, we have to be careful about what exactly to measure. Among others, there are findings in published works that show that developers struggle with configuring checks, which increases their effort in the end. Also, the findings themselves will not change just by tweaking their distribution. Another question is whether we can transfer our own experiences from applying security analysis tools on open-source projects to the integration of the tools in a large toolchain. In the second scenario, one has to consider multiple stakeholders participating in the review. The security team can help in configuring the analysis tools but others have to run them.

When it comes to the effectiveness of a security review, experiences from the industry have shown that success metrics are essential. Since having a non-exploitable system is usually the major goal, one could consider rating findings according to their exploitability. While fixing a buffer overflow might address a true positive finding, this does not necessarily improve the system’s security. However, not having an exploit does not prove the effectiveness of static analysis since we might just not be aware of an exploit, yet. Also, it has been shown that there is a correlation between the pure number of issues in a file and the likelihood of a vulnerability, not taking any additional metrics such as exploitability or criticality into account. Still, even such simple metrics as the plain number of findings might not be applicable due to changes. Therefore, a purely empirical approach by counting the number of findings is probably not the right one.

Even assuming a success metric, the issue of insufficient resources for fixing all issues remains and it is questionable whether it makes sense to deploy tools that can find more or other kinds of vulnerabilities when even we are practically not able to fix all the existing critical vulnerabilities. When analysis tools are applied in classes, students report many findings and question the usefulness of the tool for identifying real issues.

Since the pure deployment of additional analysis tools is not the solution, we have to be more selective in which analysis tools we deploy for what purpose. We need means to properly compare analysis tools to decide on which ones to deploy. To this end, we have to focus more on an analytical comparison of what the tools do for a qualitative comparison of different analysis tools.

Such a comparison of security analysis tools could be based on databases of labeled findings and the rules of the tools themselves. One issue in this regard is the availability of such information which is usually only partly publicly accessible and if so only for a single tool. Comparing rulesets among tools and estimating the overlap between tools is a yet unsolved challenge. Comparable to mutation testing, an assessment of static analysis tools could be realized by artificially introducing bugs. Whether vulnerable samples generated by ChatGPT are suitable as a baseline is currently unclear. In the end, we still rely on people building a limited number of ground truths. Recent work mainly focuses on a quantitative comparison of how many bugs in a known data set can be detected by which tool.

The detection of possible vulnerabilities and their rating is only one of the steps in effectively securing a system. After deciding on what are the concrete vulnerabilities that threaten a system, these have to be fixed. Here, plenty of research has been done in the direction of automated fix generation. While these fixe generators are effective in generating fixes, they are yet not used in practice. In the end, the generated fixes can be seen as a blueprint for fixing a detected issue but manual checking of the generated fixes is non-neglectable. In practice, simple automated dependency updates are often rejected by developers which raises the question of how good such tooling has to get.

As in the reviews themselves human peculiarities seem to play an essential also in fixing identified vulnerabilities. Developers might not be satisfied by just accepting proposed fixes while their fix was compatible with the proposed one. Further, even when a vulnerability is fixed by a developer in one team, most likely it will not be fixed in other teams working on the same code in another branch. Often organizational overhead but also lack of communication prevent the effective propagation of security fixes.

Alike to this manual problem in fixing multiple versions of a system, also static analysis works only on a single version but the fix is needed on all versions. While developers have this often in their minds at least for the variants for which they are responsible, tools currently entirely lack such features.

Besides identical duplicates of one bug, we also have to consider additional instances of bugs in similar locations. In the end, humans tend to do the same mistake more than once. Currently, we rely on them to remember these additional locations after one instance has been detected.

One of the challenges in finding such additional locations of bugs is the significant impact of context information that makes a bug probably only so some variants or branches. Therefore, it is essential to check bug-fixing code before it is pushed to other branches.

While static code analyzers have concrete instances on which they can work, design reviews are more challenging. While effective formal methods exist, they are often only feasible for some systems due to their huge overhead. One fundamental problem is usually the lack of design specifications such as models on which a design review can be performed.

While the recovery of models to use in a design review is in principle possible, the main question is how does such a review process look like. In the end, static analysis is well-integrated into today’s development processes, while this is not the case for model-based design reviews. While it is favorable to also have such integration for design reviews, there is the danger of the process becoming more important than the product itself.

While static code checks immediately work on the concrete artifacts, for design reviews we have to identify a suitable degree of abstraction. Models can range here from a single very detailed model that is close to the source code to an abstract component diagram with data flows comparable to data flow diagrams in threat modeling.

Depending on the degree of abstraction, different security issues might be identified but most likely not detailed security vulnerabilities such as those identified by static code analyzers. Still, given suitable traceability between the models and code event the low-level static code analysis can benefit from the integration of design models into the process. Among others, design models contain information about elements that are not part of the code but with which it interacts. This information can be leveraged to tailor static code analysis such as taint analysis based on the planned interaction with external entities.

Altogether, design models allow us to systematically structure systems, divide them according to security concerns by creating insulation capsules around security-critical parts, and plan concrete security protections according to the division. Due to possibly non-trivial constraints such structurings and analysis have to be supported by tools. We have to provide architects with guidelines on how to design a secure system. In this regard, anti-pattern catalogs and associations between design patterns and suitable security patterns could help. However, the extraction of such patterns is still open work. Here, pattern mining from repositories could be suitable.

To get more benefit out of such design reviews than an initial plan, their integration into the development process is necessary. Whenever there is any change, we have to find means to reflect it as well in the code as in the design models. However, this integration would allow us also to provide developers with easy-to-comprehend information about changes such as explicitly showing how dependencies among components change when a specific call is added to the implementation. On the downside, an attacker might use exactly this benefit of easily accessible information about security measures to plan an attack.

To be practicable and applicable, we need easy and cheap processes that allow us to build such models incrementally. Here, in particular, scalability is a major concern. While we have had such security-by-design approaches already for a relatively long time, it is unclear what exact improvements we need for bringing them into practice. Nevertheless, examples such as formal methods demonstrate that this is possible.

In security reviews one has to take the perspective of an attacker which is a special case compared to other domains such as safety. Attackers are actively looking for opportunities instead of things happening accidentally. To this end, as a reviewer, you have to always keep all possibilities in mind, while usually lacking the necessary context knowledge. The main task of tooling for design reviews and static code analysis is to help in identifying and presenting unknown dangers.

When considering entirely manual code reviews, reviewers usually tend to report the more obvious findings and the completeness is usually questionable. Still, practices such as pair programming and reviewing the commits of others have been proven to be effective. Here, tool support can help in facilitating these practices, e.g., by suggesting reviewers based on the touched artifacts.

In contrast to manual reviews, tools are often more complete but usually at the cost of precision. Targeting this issue, the question is which low-level findings should be shown to developers. Among others, tools should not only provide huge lists of low-level findings but automatically derive suggestions of suitable security patterns based on the identified dangers. For example, the static analysis identifies what a component is doing and suggests patterns corresponding to that. But even just highlighting the use of critical APIs could be beneficial.

The ultimate goal of the deep integration of tools in the planning and review process is to allow the development of security mechanisms that would be infeasible to plan only manually. For example, we could target more fine-grained rights management. Starting on a high abstraction at the system level, this should be systematically pushed into the applications. This would allow us to make sure in the application code that some parts only have certain permissions and would allow for more control over third-party libraries. The main problem could be getting developers to use the more complex structures that probably result from this. Here, tooling can make such rights management or other security measures easier to include.

We conclude with a summary of open research problems that we identified in the discussion above.

Despite static analysis tools already being widely deployed, their configuration is still an open problem that hinders their effective use. We have to identify simpler ways of configuring static analysis tools. Therefore, it is essential to judge the quality of the results of a tool with a specific configuration. We need to identify suitable measures of security to support such a comparison. But we not only have to be able to compare configurations of an individual tool, but we need means to systematically compare different analysis tools to allow effective tool selection.

We need a deeper investigation of the effectiveness of security measures. To suggest suitable security measures, we have to know if specific measures that have been realized prevented the vulnerabilities for which they have been planned. Related to this it should also be checked what are the cost of specific measures and what is relation to their benefits. Additional measures should be mined from repositories and relations to other principles such as design patterns should be extracted.

Since we aim at integrating design models and design reviews into the development process, we have to identify suitable levels of abstraction to do so. The question is what impact do different views on security have on planning and checking secure designs? This integration and effectiveness could be increased further by creating relations with other security artifacts such as CVEs.

• $\mathrm{■}$

  What: A discussion breakout session with academics from software engineering, security, and related fields, as well as people from industry about the (research-related) tensions and opportunities between their objectives.

• $\mathrm{■}$

  Why: Being research fields generally close to industry, both Software Engineering and Security often rely on direct interaction with and feedback from industry to push and adapt research ideas further.

• $\mathrm{■}$

  Outcomes: A number of entities and approaches were identified as impactful opportunities for “bridging the gap” between industry and academia, namely institutions like Fraunhofer, conferences like SecDev, and events like developer summits and industry workshops.

The discussion was initiated with a question from the academic side: What problems does industry face, why don’t they share (them with researchers)? To which industry responded that they do share their problems, but academia considers most of them to be not interesting problems. Academia brought up that they are restricted in problem / topic selection by a number of factors, namely that they need to be publishable (and fit specific venue requirements) and enable a career both for junior researchers and involved graduate students. Based on this response, the discussion moved on to if industry problems without solutions are still interesting to academia. As an example, the initially low adoption of test generation in industry because of flaky tests was brought up, a problem which was then picked up by academics. There was agreement that beautiful / interesting problems from industry can be picked up by academia and that both industry and academia consider their respective problems to be interesting.

It was then discussed whether the discussion framing is part of the problem, namely that industry asks themselves what academia can do for them vs. academia asks how they can solve industry problems. Industry pointed out that the academia framing in reality appears to them to be more along the lines of “show me your problems so I can solve the interesting ones”. It was proposed that academics might not accept research (problems from industry) because they might not fit their mental model. Based on that, the general question on whether the academic side is open to expanding what they work on was posed. Boundaries imposed by existing publishing venues were identified as a potentially limiting factor, highlighting opportunities to develop better-suited venues targeting industry problems.

Following the statement “Always go after the practical problems”, the discussion turned to how to actually turn research results into applicable results in industry and what or who is required to bridge that gap. It was pointed out that there are organizations with the specific goal of bridging that gap (e.g., Frauenhofer in Germany). It was also discussed that bridging the gap likely involves efforts from all involved parties: Industry needs to understand the current state of research and be willing to apply the published and applicable findings. Researchers need to have a mind about what is practical and be willing to receive industry feedback on the applicability of their findings.

The next discussion point was based on a slide from Alex Stamos’ 2019 USENIX talk about “Tackling the Trust and Safety Crisis” (a pyramid putting what is actually talked about at USENIX at the top vs. other, more impactful areas of InfoSec for industry below it). Based on that slide, the point was brought up that a lot of the innovation in the InfoSec area actually comes from industry and that academia runs after industry. This situation posed the question on whether academia might have been trapped in the valley of industry impact, resulting in pushing researchers away from more fundamental research approaches. As an example, more foundational theories in other areas were brought up, like in the fundamental theories of databases, compilers, and operation systems.

Developer Summits were brought up for a way to bring industry developers together to discuss hard problems. It was postulated that these summits work (for both industry and present researchers) because industry people love to talk to industry people about all the cool stuff they have been doing, and researchers are present soaking it up. A question was posed on how to publish findings from these summits, with the discussion leaning more towards opinion pieces or initial exploratoritive approaches to identify industry’s problems. As a research idea for validating perception, “100 questions from/for industry” was introduced. As challenges for hosting developer summits the discussion included: who to contact and how to bring smaller companies together (that can’t send someone to summits. It was mentioned that including developers from smaller companies also presents an opportunity, because if smaller companies can send someone, they probably have greater oversight and decision power over the tech in the company.

Another discussed challenge was that people love to talk about things they plan on implementing in the future, not the experiences they actually made, with a suggested solution for research being to carefully listen and have the technical background to discern these answers. After the break, the discussion continued about Workshop / Discussion events ($\sim$ 2 h, Chatham house rules), with patterns for success of these events being identified as networking, critical mass, and no shortcuts. A challenge was brought up in that it is not always possible to directly trace the impact of conversations you had at these events and not every idea actually being able to be traced back.

The next discussion point was around the question of why industry research groups / departments / teams fail. The discussion centered around if there actually has to be a good probability of failure in research, that even the failed cases have to be spun as successes to publish in academia, and the potential cultural split for research departments vs. other teams because they focus on long term problems.

Based on this potential cultural split, the next discussion point focused on the differences in objectives for industry and academia, namely that “research” in academia and industry (might) not mean the same thing. Points included the need to differentiate between short term solutions and long term solutions, with industry required to provide value immediately and academia being more oriented towards long term. Another point was how to define a good goal or bad goal, with research trying to address real problems that are relevant for the industry, having at least a minimum impact.

The final discussion point focused on recapping both breakout sessions, with tension points between industry and academia including: Academia only grabbing the problems they think are interesting and then leaving, with industry internships having the potential to better bridge this gap. Industry problems are uninteresting for academia because they often are just constraints for business reasons with obvious solutions and research can not help with that. Messy/complex industry systems in general, with the problem that if the better / correct solution doesn’t lead to better outcomes, is it really better? The breakout closed with a recap of the potential of places for academia and industry collaboration such as (industry) conferences, Fraunhofer, and developer workshops.