How Does DevOps Work in Large IT Departments?

by Alan S. Koch, PMP

It’s tempting to talk about DevOps like it’s a magical cookbook of utopian solutions. Of course, that’s not true — and when we’re dealing with the large enterprise world most of us live in, it’s especially not true. However, we are seeing exciting real-world results with DevOps-style practices and workflows, even in some of the most unlikely organizations: the U.S Marine Corps, Target, Hilton and Bank of America are just a few of the organizations we’ve worked with that are beginning to really see meaningful transformations as a result of DevOps initiatives. The Q&A extracts below show the wide range of experiences and possibilities.

Q: What general themes have you noticed as you go through these DevOps engagements in large organizations?

A: These large organizations feel the pain! Siloes, turf to protect, MBO, heavyweight processes … all of this makes implementation slow slow slow! The “Large Data Center Migration” project Patrick DuBois was working in 2007 when he first formed the ideas that evolved into the beginnings of the DevOps movement took place in this world. (It was at a government ministry in Belgium.) Small shops often don’t have as many of these pains, probably because people have to wear many hats, and silos of activity are either weakly defined or absent altogether.

Q: Is it practical to adopt DevOps in big huge orgs?

A: Yes. DevOps is the only practical way to address the pains these organizations have. But it is challenging for precisely the same reasons that it is needed.

Q: When DevOps comes up, we hear about the Silicon Valley unicorns. But aren’t there are some big differences between them and groups like government agencies, big banks, big retailers?

A: The biggest difference is history. Large organizations (especially military and financial services companies) have histories of formal processes that are built around hierarchical organizational structures. DevOps is a much bigger change in that context than in more entrepreneurial companies like the web-based unicorns.

Their legacy systems are another big difference. As the web-based unicorns grew, their systems were re-engineered to use Service Oriented Architecture (SOA) and microservices, which provide a system architectural environment that enables easy implementation of many DevOps practices. SOA is rare in the more traditional large enterprises, adding yet another challenge to the implementation.

Q: What kind of issues come up in these large organizations that lead them to pursuing DevOps practices?

A: One company I lead through a DevOps Executive Workshop came to us because they were trying to get DevOps going, but had trouble getting traction. They had the common complaint of not being able to get software changes implemented in anywhere near the time the business needed.

The executives in the workshop each represented a silo of the IT organization, and there were more than a dozen of them seated around the table. Their core issue (which they couldn’t see) was territory protection. Each executive was concerned that their team looked good, was efficient, and protected its boundaries. There was no “we” in their language, and that was what was getting in their way.

Unfortunately, the CIO (who was also at the table) was unwilling to force the collaboration issue, so in the context of protective behavior, we couldn’t make much progress in one day. They need some intensive executive coaching to make the changes at the top!

Q: Talking about DevOps means talking about automation. Many folks think of DevOps configuration management tools like Chef or Puppet that automate ops tasks and infrastructure, but do DevOps practices encourage automation in any other areas of IT — even outside of ops/infrastructure/sysadmin type functions?

A: Testing! And test automation applies to Ops as much as to Dev. If you only automate one thing, testing will provide the biggest benefit. You cannot achieve any kind of reasonable velocity unless you can quickly, easily, and consistently verify that whatever you are changing doesn’t break something else.

Automated Regression Testing is the key to all DevOps practices. It enables all three of Gene Kim’s Three Ways.

• The First Way — Flow — you can’t make significant gains in velocity unless testing can be done quickly.
• The Second Way — Feedback — is the whole reason for testing, and automating testing shortens the feedback loops.
• The Third Way — Continuous Learning — can only happen when you have a continuous flow of insights. The mistakes and failures that continuous testing uncovers are important sources of those insights.

Q: DevOps is said by DevOps thought leaders to be the twenty-first-century key to more secure systems and more secure organizations. Why is that, and do you agree?

A: I most certainly agree. DevOps is not just Dev and Ops; it must embrace all of IT, including InfoSec. In most large organizations, InfoSec is their own silo, and it is implemented as a gate very late in the development process, which impedes flow, and provides feedback too late. As a result the Dev team doesn’t learn about security.

Bringing InfoSec into the conversation and making them part of the collaboration allows us to push security to the left in our processes, just as it allows us to push quality to the left. Integrating security standards and checks early in the process improves flow (by eliminating the gate at the end), and provides the non-security professionals with the feedback they need to continually learn about security. As everyone becomes more security-conscious, and as security is built into the process, our systems become more secure.

Q: What are some of the most common problems and challenges you’ve run across in the early stages of DevOps adoption?

A: Executive management often doesn’t realize that they need to change, too! They think DevOps is a technical change that the technical people need to implement (which it is). But they don’t grasp the magnitude of organizational change that is required in their companies. Hierarchical structures, MBO, and even their whole philosophy about how to structure and manage a large organization need to change.

Q: DevOps is closely associated with Agile principles. We think of “Agile for Ops” and using Kanban to manage operational WiP in IT departments, and applying lean principles for IT management. All of these concepts are closely associated with Agile. We also know that there is a major movement around “scaled Agile” or “big agile.” But a lot of the tools used to scale up an Agile practice seem to lie completely outside of IT —tools like product roles, program management, finance, and such. Yet most people think of DevOps strictly as a “hard IT” phenomenon. Is there a connection between DevOps and Agile at scale?

A: They are very connected. You can’t have a smooth-functioning IT department if the Ops side of the equation is not connected to the Dev side. The Agile-at-Scale practices that an organization builds have to connect to Ops.

Yes, Agile-at-Scale goes all the way up to portfolio management and product management. But it also follows all the way through to the details of building an architectural runway, and the release train delivering value to the company. Both of those are where Dev meets Ops. Any organization that is pursuing both Agile-at-Scale and DevOps cannot be successful unless these approaches connect in meaningful ways. Call it “Agile-at-Scale-Ops.”

Q: There is a lot of talk about DevOps being incompatible with ITIL—especially ITIL Change and Release Management. Can the differences be reconciled? Can large IT shops practice both ITIL and DevOps?

A: ITIL appears to be incompatible with DevOps because it predates DevOps. ITIL V3 was written in 2007 (before the term DevOps had even been coined), and the most recent update was in 2011 (before DevOps was on the radar for large enterprise IT shops). ITIL describes traditional IT practices that DevOps leads us to replace—not because those practices are the only way to implement ITIL, but because that was what the authors knew at the time.

ITIL and DevOps share precisely the same goals and objectives:

• Provide IT services to IT’s customers in a way that is responsive to their changing needs
• Ensure the quality, stability, and security of those IT services
• Transform IT from a cost center into a strategic asset of the organization

Every ITIL process is relevant to a shop that is implementing DevOps. The key is to start with the Goal and Objectives statements for each process and ask, “What does DevOps teach us about this?” You will end up achieving ITIL’s goals using practices that the authors did not anticipate five or ten years ago — practices that I am sure they would applaud today.