The pressure to innovate, collaborate and accelerate implementation in software development has never been greater. The demand for ever-increasing speed and efficiency in bringing new products and services to market has led to the confluence of software development and technology operations, and a related shift in culture within those previously segregated functions. This new approach to software delivery, known as DevOps, has become standard operating procedure in software development for many organizations.
As business methodologies have evolved in the direction of flexibility and speed, compliance and control requirements have become even more strict and demanding. These requirements, such as segregation of duties, for example, impose restrictions that are not always compatible with the fast and flexible collaboration between developers and IT operations. It has become increasingly apparent that, just as Agile methodologies, IT service management and “lean” practices enabled the creation of DevOps, DevOps must integrate security, risk, compliance and regulatory controls within itself to create a faster, more flexible and secure approach to software delivery — “DevSecOps,” in industry jargon.
To get there, companies that deploy a DevOps methodology must approach their technology control environments from the perspective of the key control objectives and risks, rather than attempt to fit “standard,” auditor-defined control activities into their process. There is more than one way to meet a control objective for ISO, COBIT, SOX and other standards, but many audit and compliance professionals are not sufficiently familiar with the DevOps process to imagine a DevOps-friendly approach to controls. In fact, DevOps-friendly controls offer a number of improvements over traditional control activities, and in many cases, can more efficiently and consistently satisfy the control objectives within a well-designed and implemented DevOps process.
What Is DevOps?
DevOps — a concatenation of Development and Operations — is a fast and flexible approach to developing and delivering software to the business and marketplace. DevOps evolved from Agile and related methodologies, which accelerated the system development life cycle (SDLC) by breaking down barriers between development, quality assurance, product management, operations and the business. DevOps is a natural progression of that, automating many of the manual steps involved in developing, testing and distributing developed code to end users.
DevOps is based on an infinitely repeating cycle of Continuous Development, Continuous Testing, Continuous Integration, Continuous Deployment, and Continuous Monitoring of the system development life cycle. The stages in blue highlight key phases in Development, and the stages in orange indicate key activities traditionally falling under Operations.
Challenges of Traditional Controls in a DevOps Environment
While software development has been evolving, most IT control frameworks have remained rooted in the waterfall delivery mindset of traditional software delivery. This has created a few challenges, specifically:
- DevOps-based processes often eschew traditional control activities leading to control/compliance “failures.” This has led some auditors to suggest that DevOps cannot be adequately controlled using traditional control methodologies.
- Traditional controls impose non-value-adding activities onto DevOps-based processes, creating inefficiency and leading to workarounds.
- DevOps-based processes tend to differ across teams and organizations — more so than traditional development — which means that controls need to be defined based on the specific development and release process. Traditional controls have always been a good practice; however, alternative controls are now more relevant than ever.
Because of these challenges, IT and DevOps practitioners often say that outdated controls have become a drag on the SDLC and pose a competitive risk. Some have gone so far as to suggest removing controls altogether — but that is not likely, given the increasing focus of regulatory authorities and customers on the risk and compliance practices of companies they regulate and do business with.
Compliance requirements and industry standards require organizations to have very specific IT controls in place, such as formal change management processes for logging, reviewing and approving changes to production. And while IT may be changing the way it develops and distributes software, the compliance requirements themselves are not changing.
In addition to compliance requirements, customers are becoming increasingly interested in how organizations approach risk and compliance due to heightened awareness of the risk for potential security breaches and sensitive data exposure. Many customers now want and need to understand the risk and compliance positions of companies they do business with before they adopt their products and services.
What Have Some Companies Tried?
In an effort to reconcile the incompatibilities of traditional controls and DevOps practices, some organizations have adopted a two-speed, or bimodal, IT approach, applying traditional controls to systems identified as “in scope” for compliance and audit purposes (e.g., finance, payroll, general ledger), while leaving systems deemed as not having a financial reporting risk to their own embedded controls. This model is considered to be an interim solution, however, given it limits the benefits that can be achieved by DevOps and creates additional overhead.
Many DevOps teams are aware of the built-in controls that come with DevOps tools, including continuous integration suites, static analysis checkers, automated scripting and testing, and automated packaging and deployment. More often than not, however, DevOps teams are not provided with clear guidance for implementing and using the tools to support the underlying control objectives that need to be achieved by the organization for compliance purposes. As a result, they under-invest in the implementation of those tools, which limits their effectiveness in supporting an organization’s control framework and fails to take advantage of these automated control capabilities.
A More Organic and Collaborative Approach
A more holistic approach to solving the problem incorporates an adaptive risk strategy into an organization’s SDLC program to organically embed controls, risk management and regulatory compliance into the operating environment without impeding the rapid and continual improvement of the services and operating models. The scope and focus of such a program should be flexible and address a variety of functional disciplines, methodologies and leading practices, including Agile and DevOps. DevOps control activities may be different than traditional control activities, but they can satisfy the same control objectives as required by regulatory and risk management frameworks.
Our suggested approach to DevOps control activities is segmented into five overlapping work streams that help establish an ongoing program designed to achieve continual improvement. These five work streams are as follows:
- Review: Review and analyze the existing operating model, including customer personas, mission, key principles, processes and culture, skills assessment, training programs, feedback channels, organizational model, and technology map. Activities include documentation reviews, subject-matter expert interviews, and management-level working sessions.
- Assessment: Once the elements of the service and operating model have been identified, opportunities for improvement are identified through customer and employee surveys as well as group-based management-level assessment sessions. The surveys and assessments sessions can be adapted to match the desired type of improvement — in this case, more agile, fluid, yet effective controls.
- Recommendation: Assessment results will reveal areas for potential control improvement. In some cases, the control activities may already exist as part of the DevOps environment, without being previously recognized as such. For example, when it comes to change management controls, the peer review process for identifying and fixing issues during development can be far more efficient as a control activity than traditional testing and/or bug fixes post-release to production. Another common example is leveraging hash algorithms and artifact control procedures and modification logs in lieu of access limitations and segregation of duties placed on DevOps teams during the delivery life cycle. Controls of this nature act as a substitute for traditional control activities, yet still mitigate the risk for which they were intended.
“DevOps as a methodology offers multiple benefits and is transforming the speed and quality of software development across all industries. Its full adoption requires a fundamental shift in how we think about design and implementation of the technology process and control environment, including how to validate its effectiveness. DevOps is not just an IT transformation but a top-to-bottom change in how the enterprise interacts with and operates its technologies.”
— Jason Brucker, Managing Director, Protiviti
- Implementation: As recommendations are defined, they are prioritized based on their effectiveness in mitigating risk, regulatory and compliance needs, as well as the costs and benefits to the organization. Controls may need to be added, removed or redesigned to ensure they mitigate the underlying risks without affecting the speed and agility DevOps provides to the delivery teams. Delivery teams should clearly understand the benefits of the controls that are being designed and implemented and view them as enablers as opposed to compliance hindrances in order to promote further innovation and evolution of the control framework. If this win-win dynamic is achieved, delivery teams will proactively seek out and communicate ways to further enhance the control framework and leverage automated DevOps capabilities working hand in hand with compliance teams.
- Continual Improvement: Once the first four work streams have been established, there needs to be a process in place to review and reassess the operating model on a recurring basis and repeat the entire process, continually identifying and initiating improvements. DevOps continuous improvement tools and technology can be adapted to this purpose, to align risk strategy with DevOps and establish a foundation for control development, implementation and monitoring. These DevOps tools are faster and more flexible than traditional development methodologies, and they also are highly accountable, with version control and artifact repositories, virtualization and containerization tools, continuous integration, as well as on-demand provisioning that addresses many of the access control and segregation concerns of regulators. While terms like “continuous integration” and “continuous delivery” may sound intimidating to an auditor, that continuous process is capable of including robust quality assurance that actually improves security, prevents fraud, reduces post-release failures and accelerates security patches and fixes as new vulnerabilities arise in a changing cybersecurity landscape. Continual improvement activities are intended to help bridge the knowledge gap for auditors.
The value of the approach described above is that it is highly collaborative, builds on existing DevOps tools and practices and teaches organizations how to achieve their own continuous improvement process, rather than imposing one from the outside.
A global software developer and a leading provider of DevOps tools needed to improve the maturity of its controls in advance of an initial stock offering. The challenge was to design change management controls that would meet Sarbanes-Oxley Act (SOX), SOC2, ISO27k and other global compliance requirements compromising the company’s well-known brand of agility and speed to market. Protiviti helped design and evaluate technology controls across all of the company’s products and financial systems.
Because the software company owned the development and release tools, they were able to re-engineer the tools to include a number of critical changes. Those changes included limiting the number of source code repositories with production deployment capabilities, and a requirement that all code goes through both peer review and a “green build” screening to ensure that it passes all unit tests before it is released. The deployment server cryptographically signs code artifacts to ensure that only signed code makes it to release (preventing circumvention of the peer review process). Finally, at the end of the development cycle, code must be pulled (requested) for deployment by an administrative account in operations that developers don’t have access to.
While the primary purpose of these changes was to improve controls, it also improved the efficiency of the process as it automated a number of checks (such as peer review) and helped improve code quality. Crucially, there was no adverse impact on the speed of release.
“Controls are often viewed as barriers that slow down the release cadence and make the engineers’ jobs harder. However, Protiviti’s experience demonstrates that in many cases controls improve the efficiency of the development/release process by automating a number of (otherwise manual) checks, helping to improve code quality and reduce the time spent on defect resolution. This ultimately enhances customer experience and trust.”
— Ewen Ferguson, Managing Director, Protiviti
Integrating risk, compliance and regulatory controls into DevOps is the next natural step in the “shift left” movement in the evolution of systems development and distribution. Older waterfall methodologies are, clearly, no longer fit for purpose in many cases. By applying DevOps tools and principles to replace outdated IT general controls with integrated, automated controls, auditors and risk managers can not only continue to comply with existing regulations, but they can do so more reliably, more efficiently and at a speed compatible with agile and flexible DevOps processes. Further, the integration of continuous monitoring and controls can add value by building additional checks and fail-safes into the DevOps process, improving the quality of code.
How Protiviti Can Help
Whether organizations are considering DevOps methodologies or have already made the shift, Protiviti has the experience and resources to help ensure that their controls are suited to their needs and aligned to keep them both agile and compliant. Below are some ways we can assist organizations in that process.
- Gap analysis. Protiviti works with organizations to assess current and desired future-state DevOps processes and controls. We outline the key gaps that need to be addressed for achieving successful transformation and provide prioritized recommendations to address opportunities for improvement.
- Capability maturity model. We capture and compare current-state DevOps maturity with best practices and industry standards. DevOps investment will be compared to derived business value to gauge trending of return on investment.
- Future-state DevOps processes. We design and modify processes to efficiently meet business requirements, along with a comprehensive control framework to ensure adequate compliance and audit coverage.
- Prioritized initiatives. We deliver a list of risk, compliance and regulatory control priorities, approved by management, to ensure organizations maintain the right focus for future implementation phases, along with an implementation road map that clearly outlines the tactical implementation path.
- Practice adoption and technology implementation. With the approved prioritization of recommended initiatives, Protiviti works with the management team to convert those recommendations into actionable improvements.