Reframing Technical Debt

In recent years, we have witnessed a significant increase in research focusing on how Machine Learning (ML) techniques can be used for software quality assessment and Technical Debt Identification. However, the derived methodologies and tools lack transparency due to the black-box nature of the employed machine learning models, leading to decreased trust in their results. In addition, refactoring is the most prominent way to repay Technical Debt in the software development industry. In the literature, numerous tools provide refactoring suggestions based on well-established approaches. However, two of the main shortcomings of these approaches are: (a) that these tools usually provide many refactoring suggestions that the developer cannot parse, and (b) in several cases, the software engineers disagree with some suggestions. These problems decrease the trust of developers in the refactoring tools/approach, eventually leading engineers to drop out of using refactoring support. The lack of human trust in AI-based decision support systems has (among others) led to the rise of the ethical AI movement, calling for transparency and trustworthiness of AI systems. From our perspective, these challenges open up a new research direction in the field of Technical Debt Management that can introduce approaches relying on Explainable AI and Two-Way Learning methodologies for making the cooperation of software engineers and AI systems more efficient and acceptable for humans, putting them more actively in-the-loop and at the center of the software quality assurance process.

Software Improvement Group provides tools and consulting to software development organizations for -among others- measuring quality (including maintainability and architecture sustainability) and Technical Debt in their software systems. These services are applied incidentally as assessment and advice services, periodical reporting, and continuous monitoring.

We propose the following principles to make such services trustworthy and practical for practitioners:

• $\mathrm{■}$

  Objective and fact-based: based on (static) analysis of code and other artifacts

• $\mathrm{■}$

  Adoption of international standards and use of well-defined quality models provide transparency

• $\mathrm{■}$

  Automated and repeatable measurements to support continuous, “real-time” monitoring without manual effort

• $\mathrm{■}$

  Following a risk-based approach that aggregates fine-grained findings based on the risk categories of those findings to ensure that the conclusions are dominated by the presence or absence of higher-risk issues

• $\mathrm{■}$

  Benchmarking ensures that results provide relevant ratings and the ability to set feasible objectives, as well as fair comparison across teams, systems, and organizations

Here are some examples of models and tools that are in use by our customers for each of the Technical Debt topics identified for the Dagstuhl:

• $\mathrm{■}$

  Value-creation: a quality model for code maintainability, an architecture quality model for architecture sustainability and evolvability, and a qualitative model for calculating the Technical Debt principal based on measurements on the code base

• $\mathrm{■}$

  Tooling: tooling that supports continuous monitoring, integration in CI/CD pipelines, dashboards of Technical Debt and its trends, and visualization of architectures

• $\mathrm{■}$

  Architecture: an architecture quality model that quantifies the sustainability and evolvability of an architecture and a model for reverse engineering the realized architecture

• $\mathrm{■}$

  Data collection: Technical Debt data is derived from source code snapshots, (deltas of) quality ratings, SCM-based info such as commit information

• $\mathrm{■}$

  Socio-technical aspects: for example, the latter information is used to identify and report socio-technical aspects about team organization (vs. the architecture) and team interactions

Applying these models in practice has shown their ability to provide useful insights, which can help development organizations manage their Technical Debt better.

Technical Debt Management is essential but often sidelined over time, even in organizations initially committed to it. At CodeScene, we use the CodeHealth metric to identify and prioritize refactoring targets. CodeHealth is an aggregated file-level metric based on established code smells known to increase the cognitive load on developers. Empirical studies on proprietary projects have shown strong links between CodeHealth, defect density, and development velocity. Yet, maintaining momentum in CodeHealth improvement remains challenging for many organizations.

To address this, we are developing value models from anonymized customer data to tie Technical Debt remediation to business value. Additionally, we analyze code smell removal during pull requests to estimate the value of prevented CodeHealth declines. Our aim is to equip organizations with “what if” insights to support return on investment analyses for ongoing maintainability improvements, ultimately helping them sustain focus on code-level Technical Debt Management.

The lack of a universal agreement on what constitutes Technical Debt and how it manifests in software artifacts – especially in source code – poses challenges for Research, Practice, and Education in Software Engineering: The absence of a commonly agreed-upon “ground truth” for Technical Debt instances threatens the validity of new approaches and empirical studies. Tools for Technical Debt Management used in practice also suffer from limited credibility, as different approaches produce widely varying Technical Debt estimates and recommend divergent mitigation actions. Furthermore, without a widely accepted benchmark, effectively training students and young professionals becomes challenging. However, aligning findings from multiple Technical Debt identification approaches/tools and validating them with expert feedback could establish trustworthy reference datasets. Such Technical Debt benchmarks would create valuable opportunities for Technical Debt Management, enabling Machine Learning models to be trained and adapted to domain-specific Technical Debt issues.

Academic research offers a well-defined concept of Technical Debt, distinguishing it from other software quality issues. In contrast, industry practitioners use the term more broadly to encompass any quality defect. While research focuses on Technical Debt within source code, practitioners find that architectural shortcomings or outdated technology stacks (e.g., obsolete libraries) often pose costs comparable to code-related Technical Debt. Research highlights that accepting Technical Debt involves trade-offs, such as accelerating time-to-market. Yet, in practice, these decisions are frequently made unconsciously, with their adverse effects manifesting much later.

The distinction between academia’s precise view and industry’s broader interpretation of Technical Debt is less critical than one might think. The tools and processes required to tackle these software quality challenges are mainly identical. A key strategy for managing Technical Debt or general software quality is implementing a feedback loop. This loop incorporates a comprehensive software system analysis, examining not just the source code but also architecture, technology stacks, and other artifacts like build scripts, requirements specifications, and test cases. Insights from the development process, such as defect data or ticket resolution times, also feed into this loop.

To cope with today’s large and complex systems, this analysis should be as automated as possible while allowing for necessary manual interventions for aspects that cannot (yet) be automated. The results of the analysis are translated into actionable tasks for software engineers. Trade-offs inherent in software development are best managed when the feedback process is moderated by a role empowered to make such decisions, like a product owner.

Over the past decade, CQSE has assisted various organizations in establishing these data-driven feedback loops, effectively managing Technical Debt and improving the quality of hundreds of software systems encompassing millions of lines of code. Our experience demonstrates that this approach is cost-effective and significantly enhances software quality without sacrificing the ability to make strategic trade-offs, such as temporarily accepting certain issues to expedite time-to-market.

The adoption of the Object Management Group (OMG) Automated Source Code Quality Measures (ASCQM) standard as ISO/IEC 5055:2021 prompted the Consortium for IT Software Quality (CISQ) to revisit its Automated Technical Debt Measure (ATDM) to align with this new standard and integrate insights from recent experiences, particularly regarding the use of structural insights to prioritize Technical Debt better.

Despite these efforts, significant challenges persist with ATDM adoption. Technical Debt is still broadly perceived negatively, with improved identification of ATDM items often proving counterproductive by visibly increasing the “Technical Debt principal.” The additional focus on architecture-level ATDM items introduced in ISO/IEC 5055:2021 has only amplified this effect.

The popularity of the Technical Debt metaphor, while helpful in raising awareness, often creates a false sense of comprehension regarding its effective management. The metaphor oversimplifies the reality of Technical Debt in software engineering, where unique approaches can be applied to reduce debt, especially when concentrated debt becomes unsustainable. However, current models are inadequate in capturing critical decisions, such as the choice to retire and replace debt-heavy components, instead of addressing each Technical Debt item individually. The lack of mature models for these strategies means they are largely absent from aggregated measurement results.

Although the Contextual Technical Debt Measure (CTDM) was introduced in the ATDM standard to account for project-specific contexts and encourage adoption by development teams, uptake remains slow. Companies often prioritize Technical Debt measures for inter-project and team comparison rather than using them to empower teams to manage their debt within their own contexts.

Even with tools that can automatically detect architectural nonconformities and generate “as-is” architectural blueprints, architecture-level Technical Debt is frequently overlooked.

The recent gathering in Dagstuhl, Germany, of representatives from research, industry, and software vendors provides a valuable opportunity to leverage collective experience, address these ongoing challenges, and chart a productive course for the future.

Domain knowledge does not equal software expertise. Domain knowledge is about business rules, stakeholder needs, and essential complexity. Technical Debt means nothing in the abstract. Everyone has it, but no fixes are common. Rules and metrics are useless out of the box, useless at a portfolio level, and maybe useless at a project level; they need context. This is especially challenging when the domain knowledge is highly technical (e.g., radio astronomy or climate modeling). To resolve this, we can automatically extract context using, for example, machine learning assessment of key problems or longitudinal data on previous priorities. We can also use qualitative insights (e.g., Quality Attribute Workshops) or developer surveys. Longer term, we might try large language model recovery of relevant domain context, although how helpful this is, is unclear, such as labeling Technical Debt instances or assessing impacts. Incorporating domain knowledge will make Technical Debt Management more relevant.

Technical Debt remains a critical challenge in software development, appearing in all aspects of the development and operations workflow, including code, documentation, tests, deployment, logging, and monitoring. Symptoms of accumulating Technical Debt can be found in such artifacts, which are produced by either human actions or automated tools.

To effectively manage Technical Debt, integration into the existing team’s workflow is essential, ensuring synergy rather than imposing an additional burden. This integration must capture the perspectives of different stakeholders, including developers, team leads, and product managers, to account for technical, operational, and business needs. By bridging practical challenges and research insights and grounding benchmarking in quantitative metrics and qualitative developer experiences, Technical Debt Management can evolve into a holistic process that considers multiple dimensions of software quality.

A critical approach involves combining evidence from various workflow artifacts – akin to a “sensorial fusion” – to understand the larger picture of Technical Debt. By correlating symptoms from tools and developers’ perspectives, it is possible to improve identification, prioritization, and targeted intervention strategies.

Effective Technical Debt Management is not about eliminating debt entirely but about making informed trade-offs that respect the evolving business and development context. Shared ownership, human perception, and psychological safety are key to openly documenting Technical Debt, empowering teams to discuss and make context-aware decisions. A workflow-driven Technical Debt Management approach should align technical challenges and business objectives to pave the way for more sustainable and efficient software development processes.

The Google Engineering Productivity Research Team has been examining Technical Debt as a hindrance to developer productivity for several years. Our work on Technical Debt employs a variety of quantitative and qualitative methods but has focused heavily on survey research with quantitative and qualitative analysis of structured survey items and open-ended questions. Initially, we sought to better define what engineers mean when they talk about Technical Debt, iteratively refining how we ask about this topic in survey questions and how we measure and report it. Presently, we measure and report on ten types of Technical Debt that vary in their sources, implications, and severity. Our team has also collaborated with expert practitioners at Google on frameworks and processes for the effective management of Technical Debt. Our latest research has examined engineers’ comments regarding Technical Debt in the context of the recent increased focus on achieving higher engineering velocity (and perhaps, accordingly, less caution about incurring Technical Debt). Our aim is to achieve a better understanding of how processes and structure can help engineering teams have a healthy, sustainable, and practical approach to incurring, managing, and paying back Technical Debt.

Our research team at Google takes a mixed-methods approach to understanding and measuring Technical Debt. Our surveys give us developer perceptions of the impact of 10 forms of Technical Debt and are regularly used for prioritization. However, both engineers and managers report the need to see a “heatmap” of where Technical Debt is in the code; subjective data is helpful for knowing there is a problem, but it doesn’t provide the extent of impact. The objective data would be helpful for relative prioritization, discussions with stakeholders, and evaluating the impact of company-wide improvements to processes and tools that are intended to better fix or manage debt.

To evaluate potential objective metrics of Technical Debt, we attempted to correlate them with developer perceptions from the survey. We did this for three forms of Technical Debt but found no correlations with 117 metrics. It may be that these metrics are insufficient at their core, but it also may be that our research method was flawed as we had to make assumptions about whether a developer may have encountered each Technical Debt issue in their work. It also might be that Technical Debt, at its essence, is the difference between what the system is and what the developer believes the system “ought to be,” and therefore, this problem may be inherently intractable. More research needs to be done, and we call on others in the community to validate Technical Debt metrics against developer perceptions of the impact of Technical Debt on productivity.

Technical Debt is pervasive within all software; it exists across all development phases and grows unchecked when not deliberately managed. Accumulated Technical Debt is debilitating and an existential threat to long-life software systems. Proactive Technical Debt Management streamlines construction and integration of new software features and provides decision trade-off insight (potential Technical Debt prevention) across the software ecosystem.

Technical Debt measurement tools merged into the software development environment empower the software engineering experience while yielding higher quality, easier to change software products. When applied to reusable software, full-lifecycle Technical Debt Management leads to broader acceptance and adoption.

A practical Technical Debt Management implementation includes 3 fundamental components: 1) Technical Debt Primer, 2) Technical Debt Inventory Process and 3) Technical Debt Management System. The Technical Debt Primer captures organizationally consistent terminology, common understanding and complete descriptions for the Technical Debt Item Inventory and the Technical Debt Management System. The Technical Debt Inventory Process is an integrated software development activity employing a tool-agnostic analysis and decision framework with predefined and contextual attributes. Software developers identify/define, prioritize, and repay (closeout) Technical Debt Items within the backlog, concurrent with feature development and defect correction, as part of Agile workflows. The Technical Debt Management System utilizes automated (and limited manual) metrics capture, hierarchically integrated within dashboards, to inform software managers and leadership on Technical Debt concerns using data analytics.

Known Technical Debt Management Gaps remain:

1. 1.

   There is no universal set of Technical Debt Management metrics

2. 2.

   Technical Debt benchmarks are insufficient; standardization is needed to consistently measure and track Technical Debt

3. 3.

   Code Technical Debt Management tools are in their infancy; tools to measure other forms of Technical Debt, e.g., architecture, are embryonic

4. 4.

   A universal set of Technical Debt root causes should be defined to provide consistent reference for risk management and impact analysis.

The 2024 Dagstuhl experience has brought together practitioners, researchers and the tooling community. Together, we’re poised to take the next step – to further refine, automate and validate the Technical Debt Management process.

ABB, a major company in the industrial automation domain and develops software-intensive systems that include distributed systems, servers, real-time controllers, edge gateways, intelligent devices, cloud backends, and mobile applications. The value of ABB’s products and systems is increasingly driven by software, but the presence of Technical Debt in this software can be problematic. This is because industrial automation applications often need to be maintained for extended periods. Despite their lower change velocities compared to consumer software, Technical Debt in these applications still incurs interest payments, requiring costly migrations and updates.

Past efforts to tackle Technical Debt have involved applying source code analysis tools and surveying developers. While these approaches have some benefits, such as making Technical Debt visible and facilitating communication, they are often incomplete and lack structured processes. A culture that recognizes the importance of managing Technical Debt is crucial, but it must also balance the risk of over-engineering with the need for reasonable, cost-effective efforts in industrial domains.

Future approaches to managing Technical Debt should include purpose-built tools that address architectural debt. These tools need to be capable of handling informal architecture sketches, incorporating external information about component suppliers, supporting technology decisions, and aligning with business goals. They should also suggest architecture best practices and provide proven solutions. Although current program analysis tools are helpful in identifying and visualizing code smells, there is a gap in dedicated, agile, low-overhead solutions for managing Technical Debt.

Customizable LLM-based agents offer a promising solution to support developers by identifying Technical Debt, framing existing issues, and brainstorming approaches to address them, such as re-designing parts of the code. These agents could help developers generate appropriate source code and foster a culture of continuous Technical Debt Management, integrating seamlessly into regular development practices.

Technical debt is an inevitable aspect of software development that impacts innovation, maintenance, and efficiency. For large companies like SAP, managing Technical Debt systematically can save costs and boost productivity, but this requires a cultural shift.

A shared understanding of the value and risks associated with Technical Debt among both business and technical stakeholders is essential. This can be fostered through straightforward, user-friendly processes for managing Technical Debt, supported by tools that minimize developer overhead while accommodating necessary complexity under the hood.

At SAP, we have taken/are planning several measures to improve the management of Technical Debt:

• $\mathrm{■}$

  Playbook: Creating and distributing a straightforward playbook for systematic Technical Debt Management, along with providing training sessions on how to prevent and handle Technical Debt

• $\mathrm{■}$

  Feedback: Regularly gathering developer feedback on Technical Debt based on our developer experience survey

• $\mathrm{■}$

  Technical Debt Budget: Elevating Technical Debt items to first-class citizens in portfolio and sprint planning, securing the necessary capacity and budget for effective management and resolution

• $\mathrm{■}$

  Proof of Benefits: Demonstrating the return on investment (ROI) from reducing Technical Debt, such as increased developer satisfaction, improved Technical Debt feedback, cost savings, and increased development velocity, by leveraging central DORA and FLOW metrics as well as user research

• $\mathrm{■}$

  Technical Debt Mitigation Techniques: Providing a “toolbox” for mitigating common types of Technical Debt more efficiently. This includes large-scale refactoring techniques to address migration debt and leveraging GenAI for validating and enhancing service and API documentation

Despite our advancements, challenges remain that require further investigation:

• $\mathrm{■}$

  Developing leading indicators/early warning signals for identifying critical Technical Debt and supporting decision-making (cost vs. value of Technical Debt)

• $\mathrm{■}$

  Providing Technical Debt Management tools that integrate seamlessly with existing development processes and tools and leverage all available data.

Addressing these key issues will require a joint effort from researchers, tool vendors, and practitioners.

This talk focused on the Report to the Congressional Defense Committees on National Defense Authorization Act (NDAA) for Fiscal Year 2022 Section 835 Independent Study on Technical Debt in Software-Intensive Systems. It emphasized unique United States Department of Defense (DoD) challenges but highlighted the socio-technical aspects of the findings and recommendations. DoD, having many legacy systems with significant Technical Debt and strict funding rules (colors of money), needs to share best practices and provide additional guidance to individuals and teams empower programs to incorporate Technical Debt Management into software development lifecycle activities as a core software engineering practice. The talk also noted that DoD should invest in/utilize Technical Debt research areas as well as improve tool support and data collection. Technical Debt Management must become a valued component of sustainable growth and product quality for DoD throughout the software development lifecycle.

My name is Carolyn Seaman and I am a professor and researcher in Baltimore, USA. I have been involved in funding, conducting, supervising, and publishing Technical Debt research since 2009. The area I am most interested in pursuing at this point is Technical Debt decision-making. Research and development on Technical Debt identification over the last 15 years has yielded a host of ways for software practitioners to gather data on the Technical Debt that exists in their products, but many fewer ways to make effective decisions about what Technical Debt to pay off and when. However, it’s hard to evaluate decision-making approaches. Evaluation often relies on historical decisions, but these cannot be assumed to be optimal. Technical Debt researchers need collaborators from disciplines that have studied decision-making, such as finance, economic decision-making, and psychology.

Technical Debt emerges from decisions or situations throughout the software development cycle, jeopardizing evolution (most development workflows depend on it) and software maintenance due to its significant impact on internal software quality. Therefore, its identification is cardinal to managing the software project’s well-being and ensuring the product’s long life. However, identifying Technical Debt, including architecture Technical Debt, is not straightforward. Different context variables and factors (peopleware, processes, products, business, etc.) influence Technical Debt Management (see is Technical Debt Management platform at https://tdmframework.labculturadigital.com.br/). Technical Debt goes beyond source code. As far as it has been discussed in the sessions, a lack of grounded indicators can make it challenging to indicate Technical Debt precisely. Its multidimensionality reduces the possibility of collecting just a simple measure to level Technical Debt. Considering the multiple perspectives Technical Debt can be perceived, and assuming Technical Debt items influence their interpretation, it is necessary to investigate further how Technical Debt can be perceived and measured through combining objective, subjective and qualitative data. Many context variables and features can jeopardize this investigation, such as the lack of contextualization, traceability, explainability, and transparency. It justifies a synergic collaboration between software stakeholders, tools vendors, and researchers aiming to share data and knowledge on Technical Debt. Such a collaboration will support us in a multi-dimensional measuring and observational perspective regarding Technical Debt.

Technical Debt is a term for software solutions that are beneficial in the short term but impede future change. Previous research on Technical Debt indicates various management-related causes. We analyzed the perspective of IT managers on Technical Debt since they usually have a major influence on deadlines, the project’s budget, and setting up a Technical Debt Management process. We obtained and analyzed data from 16 semi-structured interviews and a three-person focus group discussion. We found that all IT managers understood the Technical Debt concept. They considered Technical Debt Management to be an essential topic, though almost none of them had set up a Technical Debt Management process so far. We identified three major IT managers’ concerns regarding Technical Debt Management: (1) communicating about Technical Debt with business managers as well as developers, (2) establishing a Technical Debt Management process as IT managers seem to be missing guidance on how to introduce a sustainable Technical Debt Management process, and (3) dealing with vintage systems, i.e., old potentially debt-ridden but still valuable systems. In this work, we additionally developed a model specifying causes and consequences visible to business stakeholders, causal chains, and vicious cycles. The V4CTD model of Visibility, Cycles & Chains of Causes & Consequences of Technical Debt extends the Technical Debt conceptual model and facilitates communication on Technical Debt with business stakeholders.