Is Technical Debt Limiting Your Company’s Competitiveness?
For decades, discussions within information technology departments about technical debt have occurred with insufficient engagement from executive management and the board. It’s time for boards to increase their awareness and understanding of this issue so they can add value to the conversation.
Technical debt has become a formidable encumbrance to sustaining competitiveness in the digital age. Today, agility and resiliency have emerged as strategic imperatives for long-established incumbents exposed to “born digital” players with architecture built optimally from the ground up.
As the half-life of business models continues to compress, responding to these market entrants becomes a challenge for organizations that have multiple layers of architecture reaching back decades. As wave after wave of cutting-edge technology solutions hit the streets to address pressing market needs, this struggle is like trying to turn a battleship as if it were a speedboat.
The need for resiliency, flexibility, scalability and security has expanded the technical debt topic from an IT discussion of a tactical, operational issue to a broader strategic imperative. Directors and executives need to understand the issue of technical debt better to overcome the constraints it imposes on the business — especially on the enterprise’s ability to compete effectively in the digital age.
Exercise, healthy diet and relaxation are all keys to good health. If these keys are neglected in favor of focusing on what may appear in the moment to be more pressing matters, a person may experience health challenges with age. Similarly, technical debt accrues as developers take shortcuts, accept assumptions and make other day-to-day trade-off decisions that result in the need to rework or restructure computer code (called “refactoring”) of existing technology solutions without altering the solutions’ external behavior.
“Technical debt” is a concept that refers to the cost and magnitude of additional rework caused by choosing technology solutions that are easier to implement over the short term instead of the best overall solution for the long run. As an organization makes these decisions, and as technology continues to evolve, the cumulative effect of layers upon layers of code and architectural approaches can result in an amount of technical debt that stifles the company’s ability to innovate and compete.
Not well understood by many business executives or boards, technical debt becomes a significant issue as the agile developers of today focus relentlessly on delivering software quickly to enrich customer experiences, digitize products and services, inform decision-making, and improve operational performance. Agile solutions necessitate management support for refactoring to avoid the further accumulation of technical debt; otherwise, developers over-architect or over-code software. The point is that trying to become an agile organization without a commitment to refactoring is a recipe for disaster. It bogs down systems, delays product launches, creates instability and wastes money — not the ideal way to compete in an environment that places a premium on speed and efficiency.
The question arises: Why should directors care about technical debt? In an environment dominated by emerging technologies, disruption of business models and a critical need for resiliency, technical debt can slow an organization’s ability to respond to emerging market opportunities and demands. It is often likened to monetary debt in the sense that if it is not addressed, it can accumulate “interest” over time in the form of the extra effort required in future development activities resulting from prior design choices. Most importantly, as technology evolves, the cost of “repaying” the debt becomes harder and more expensive. Therefore, in a rapidly advancing technological environment, kicking the proverbial can down the road accrues additional cost.
Not all technical debt is bad. There are times when it is incurred with the purpose of bringing a product to market or responding to an emerging opportunity or risk more quickly. For example, a company may make a conscious decision to take on debt for a specific business outcome, such as debugging known problems now with an action plan to “pay it back later” after more thoughtful consideration is given to a design that accommodates future requirements. Another example would be a short-term decision to patch up a piece of code quickly to release a product and then fix it immediately after the release.
However, it can become a serious issue if technology isn’t refreshed, or shortcuts taken are not subsequently addressed. All too often, the work to reduce technical debt is delayed due to competing priorities. That’s where the problem becomes worse, as these decisions can cause further accumulation of technical debt over time. Left unchecked, the debt can grow slowly and insidiously until it becomes the proverbial ball and chain that can prevent an organization from keeping pace with its nimbler rivals.
The archaeology of legacy technology. In many organizations, particularly longstanding enterprises, mapping the technology that supports mission-critical operations can be like an archaeological dig — one that exposes the kind of technical debt that can put an organization at serious risk. On the surface, the shiniest, newest technology supporting websites, mobile solutions and advanced analytics may exist. But dig below the surface, and one is likely to find layer upon layer of highly interdependent, complex systems — some dating back four decades or more.
To illustrate, 92 of the top 100 banks, along with 71 percent of Fortune 500 companies, still rely on mainframe technology for much of their core processing. Reuters estimates that US$3 trillion in daily commerce flows through core banking systems running on those mainframes. Most of these systems are made up of about 220 billion lines of code written in COBOL, a programming language developed over 60 years ago; in fact, COBOL code makes 95 percent of ATM transactions possible.
In most cases, these platforms reliably perform the functions they were designed to support, but continued deployment of this aging technology presents several risks. Among them is reliance on an aging and shrinking workforce with the knowledge and skills to support this technology. Much of this workforce is nearing retirement, and almost all training and education available today is focused on more modern technologies.
The tip of the iceberg. Unfortunately, it gets worse. The complex, monolithic design of these systems and the processes that support them are not well-suited to the fast-paced, agile nature of today’s digital world. As a result, organizations dependent on these aging platforms often find it difficult to respond to market opportunities or risks or to adopt emerging technological capabilities promptly. In some cases, the platforms and their complex integrations also create security risks and challenges in responding to regulatory compliance demands.
Mainframes and COBOL are certainly not the only forms of technology debt. It can exist in the form of outdated versions of databases and operating systems that can result in systems that are no longer supported by vendors and may be exposed to security vulnerabilities. Applications that have not received vendor upgrades may suffer from a lack of critical functionality or reduced stability. Hardware that has not been refreshed may be vulnerable to an increased likelihood of failure. The very architecture that defines the environment can also represent technical debt in the form of siloed, duplicative or fragile solutions. Each of these things represents a potential contribution to an organization’s technical debt.
On top of all these risks is a significant economic impact. Gartner reports that typical organizations spend over 70 percent of their IT budget on simply operating their technology platforms — and in some industries that figure is as high as 77 percent. That leaves precious little budget for enhancements and innovation. And the problem seems to be growing, as the percentage of operating costs has grown from 67 percent in 2013 to 71 percent in 2017, while IT spend dedicated to transforming the organization has shrunk from 13 percent to 10 percent during that same period.
While this problem may not be fully attributed to technical debt, a correlation is not hard to draw. Not surprisingly, the more technical debt that exists in an organization and the more acute these problems become, the heavier the ball and chain.
Paying the piper. Why not just upgrade, replace or even abandon aging systems and wipe out the technical debt? If only it were that easy. The decision to invest in reducing technical debt must survive the very prioritization gauntlet that likely resulted in the creation of the debt in the first place. Depending on the size and nature of the debt, the effort to address it is likely to delay other business imperatives, making executive management and board support for such efforts essential.
Once a decision is made to deal with technical debt, there remain critical issues to consider as well as risks to mitigate. Technology debt often exists in the mission-critical platforms that support daily operations of an enterprise. So, upgrading or replacing this infrastructure requires careful planning, effective testing and active risk management.
Fortunately, there are several approaches available to mitigate technical debt. The good news is that more modern technologies and architectural patterns offer the promise of reducing the risk associated with paying down technical debt. The options presented below to address the debt are not mutually exclusive. In fact, these options may be combined to create an organization’s overall road map that is tailored to its specific situation and risk appetite.
- Greenfield — For some organizations, there may be an opportunity to build a new infrastructure based on modern technologies. This new infrastructure may support new products or markets, but alone it is not likely to address the technical debt of the existing enterprise. This approach has the benefit of starting from a digital-first clean slate, and lessons learned from deploying new infrastructure can be applied to the legacy environment in the future. Goldman Sachs chose such an approach when it established its Marcus brand in 2016.
- Quarantine — In some cases, technical debt can simply be ring-fenced or quarantined to isolate it from the rest of the environment. This approach can be effectively deployed in the infrastructure and supporting business processes designated for retirement.
- Preserve and protect — For some stable infrastructure, the right approach may be to build a services layer around the system. That defers the inevitable need to upgrade or replace the systems in question and can effectively extend the life of critical systems assets, as well as provide the time to plan their replacement or retirement effectively.
- Simplify and rationalize — For organizations that have grown through mergers and acquisitions and not completed a rationalization process along that journey, simplifying and rationalizing their infrastructure may allow them to address some forms of technical debt.
- “Big Bang” — Some organizations may choose to do a full replacement and upgrade to more modern infrastructure. That is a high-risk, high-reward proposition. Although it may hold the promise of “ripping the bandage off” to reduce technical debt aggressively, any organization contemplating this option should carefully consider and manage the associated risks, as evidenced by the problems encountered by a bank in the United Kingdom earlier this year.
- Phased upgrade and replacement — One of the most likely approaches to dealing with technical debt is a phased upgrade or migration to newer technology platforms. This approach represents a migration path that is risk-sensitive and reduces technical debt over time. In some instances, it requires building a parallel infrastructure with temporary “scaffolding” to support the transition to the desired future-state environment.
These options can be combined with several more modern technologies and architectural patterns as organizations seek to deal with existing technical debt and avoid further debt in the future. For example, cloud solutions offer several benefits related to avoiding future technical debt while shifting some of the maintenance burden to the cloud services provider. However, cloud computing options do not let the user organization abdicate its responsibility for monitoring and managing technical debt.
Service-oriented architectures, including application programming interfaces (APIs) and microservices, are another approach that can be applied in many ways. For example, they can be used to “wrap” and “decompose” legacy systems in the “preserve and protect” option described above. They can be an important tool in the implementation of the “phased upgrade and replacement” option as well. More importantly, the decomposition of large, complex, monolithic systems into smaller components offers support strategies that help manage technical debt, create agility and enable innovation.
To face the future with confidence, directors need to understand the extent and level of the enterprise’s technical debt and inquire of management as to where they are in creating and executing a plan to address it. That discussion is important to the board because technical debt impairs a company’s effectiveness to respond rapidly and continually to emerging market opportunities, competitive threats and customer demands. Organizations that built their legacy applications for operational optimization now face formidable challenges as new business realities demand ever-higher levels of resilience in adapting business processes and systems to the digital economy; thus, organizations need to select the best approach to modernize.
Questions for Boards
Following are some suggested questions that boards of directors may consider, based on the risks inherent in the entity’s operations:
- Do the board and executive management have visibility into the extent and nature of the entity’s technical debt?
- Is there an effective means to measure technical debt?
- Does the board understand its impact on the company’s business and innovation strategy? For example, is the amount spent to maintain solutions disproportionate to the amount that the organization is investing to innovate and enhance its capabilities?
- Is there an actionable road map to mitigate the risk and cost of technical debt?
- Is there a clear target-state architecture and a road map to achieve it?
- Is the preoccupation with budgets and deadlines, or perhaps other behaviors, preventing the organization from proactively addressing the risks related to technical debt?
- Is there active governance in place to ensure effective trade-off decisions so that technical debt is managed actively on an ongoing basis?
- Is there a strategy and governance process that considers the management of technical debt in support of established business goals, including innovating a clear target-state architecture and a road map to achieve it?
- Is the organization agile and adaptive enough to recognize the impact of emerging technologies and changing business models and capitalize on, endure or overcome them with timely adjustments to its strategy and infrastructure? Are these assessments shared with the board?
- If answers to any of the above questions are “no,” are steps being taken to eliminate these and any other barriers?
How Protiviti Can Help
Protiviti has worked with 60 percent of the Fortune 1000® and 35 percent of the Fortune Global 500®, as well as smaller companies — including fast-growing technology organizations, both pre-and post-IPO. We assist boards and executive management and have a proven track record of bringing innovative solutions that help companies solve some of their most difficult business problems. Our technology strategy offerings help our clients deal with technical debt by:
- Providing visibility into the amount and nature of technical debt through:
- A technical debt evaluation: Assessing existing infrastructure to determine the extent to which technical debt is preventing the attainment of business and innovation goals.
- Developing and executing an actionable road map to mitigate the risk of technical debt through:
- A technical debt mitigation strategy: Developing blueprints, approaches and road maps to mitigate existing technical debt.
- Program and risk management: Providing program and risk management support for the execution of plans to mitigate the impact of technical debt.
- Establishing an effective strategy and governance process for ongoing management of technical debt through:
- Technology strategy and governance: Developing technology and governance designed to align business and IT strategies, minimize technical debt accumulation, and support a nimble enterprise architecture.
- Technology investment portfolio management: Rebalancing the portfolio to create the needed capacity, both human and financial, to address technical debt.
“Technical Debt: The Silent Company Killer,” by Falon Fatemi, Forbes, May 30, 2016.
“Good Technical Debt vs. Bad Technical Debt,” by Fadi Stephan, Excella blog, March 21, 2016.
“Wanted at Banks: Young Tech Pros with Old-Tech Smarts,” by Penny Crosman, American Banker, July 15, 2014
“Banks scramble to fix old systems as IT ‘cowboys’ ride into sunset,” by Anna Irrera, Reuters, April 9, 2017.
“COBOL Is Everywhere. Who Will Maintain It?” by David Cassel, The New Stack, May 6, 2017.
“Gartner, IT Key Metrics Data 2018: Executive Summary,” by Linda Hall, Eric Stegman, Shreya Futela and Disha Badlani, December 11, 2017, pages 42-45, available for subscribers at ID G00341718.
“Architectural pattern” is a technical term for a general, reusable solution that is commonly available and addresses a commonly occurring problem. Examples in analytics and business intelligence include transactional reporting, operational analytics, business analytics, predictive analytics and prescriptive analytics. In artificial intelligence, examples are speech recognition, natural language processing, machine learning, robotic process automation, and image and video analysis.
“Why Goldman Sachs Is Lending to the Middle Class,” by Zeke Faux and Shahien Nasiripour, Bloomberg Businessweek, June 29, 2018.
“TSB IT Crisis: Bank Chief Paul Pester Steps Down with £1.7M Payout,” The Week, September 4, 2018.
(Board Perspectives: Risk Oversight - Issue 109)