“Fitting Elephants with Ballet Shoes” – DevOps Implementation in Regulated Enterprises

If we were the sort of company where my job title was ‘storyteller and growth hacker’, rather than plain old ‘marketing executive’, and our developers were known as ‘souschefs’ or ‘code ninjas’, then our homepage, rather than telling you about our product’s enterprise application release automation functionality, might instead tell you what we do in more imaginative terms – “helping elephants to do ballet”, “turbocharging your tank” or, as I heard a number of times at the DevOps Enterprise Summit 2016 – helping carthorses to run like unicorns.

One key issue which can hinder DevOps adoption in large enterprises is not just the question of whether elephant can be taught to dance – it’s the one that comes before this – of whether enterprises are practically able to integrate new technologies and processes into their highly regulated organisations. Or, to put it another way, can you fit an elephant with ballet shoes?

In the past, it has been suggested that:

  • Agile processes are not compliant
  • Dev/Ops teams must always be separate to properly maintain compliance
  • DevOps only works in small companies

 

Thankfully, these complaints are no longer so common, given the number of enterprises that have achieved substantial DevOps adoption in the last few years. Organisations in different verticals will be facing a variety of diverse internal compliance policies that might affect DevOps implementation. The one that most commonly causes issues is the Sarbanes-Oxley Act of 2002 (SOX)

  • SOX – requires segregation of duties.
    • No one employee can deploy a code change into production on their own – this can complicate the self service aspect of DevOps
    • Employees must only able to access systems they work with – this means access gates must be strictly enforced
    • Each employee must have a specific defined duty at each step in the software lifecycle – this makes role sharing more difficult

 

At the 2016 DevOps Enterprise Conference, Robert Scherrer (head of application engineering at SIX) mentioned that getting DevOps implemented in a highly regulated organisation can often seem like a battle between IT and auditors. This is understandable – in a SOX compliant organisation, development, testing and operations supposedly have to be separated – while DevOps works in a contrary way – to break down silos between each of these disciplines. On top of this, the nature of legal procedure means that these standards are updated more slowly than technology and process improvements are occurring inside enterprise organisations. It is also often the case that internally implemented directives that aim to simplify the achievement of compliance requirements will generally overfulfill them – meaning that they introduce unnecessary constraints on IT staff activities. For all these reasons, constantly working to fulfill these standards can often feel regressive from an IT efficiency point of view.

 

The reality of the situation doesn’t necessarily have to be one of two counteractive forces working against each other, however:

  • When an organisation implements the correct deployment automation or configuration management tool as part of DevOps, it is possible to define precise organisation-specific access controls so that nobody has root access that doesn’t need it. It is also possible to automate the process of creating approval gates for SOX compliance – one person can write the code and another user can be required to check it before it goes into production. Equally, a tester can request a update to a managed environment and a release manager can approve it – and the tool will enforce the necessary system constraints.
  • Similarly, automation software enables the continuous implementation of very strict process workflows, which can be necessitated by enterprise compliance policies – e.g. production releases must be deployed to QA and Test prior to being able to be deployed into production.
  • Another benefit of Continuous Delivery solutions used as part of DevOps initiatives is that a number of important quality gates can be automated – machines can now test performance, scan logs for errors and perform functional test that integrate human approvals – all of which are recorded in detailed logs that are ready for audit.
  • Good automation tools will also create fully audible logs that shows who made which change, where and when, along with associated tickets or defects raised, and any updates to the bill of materials created. This will mean that audit actually becomes easier post-DevOps-implementation, rather than more difficult – manual logs are liable to contain errors, which are much less likely when the documentation activity is baked into the tool.
  • Configuration management tools should also be able to do one of two things. Either it will be able to automatically enforce desired state for servers, meaning that deviations from compliant configurations are far less likely, or they will be able to track configuration drift and enable users to compare how the machine or configuration should look from how it does look – enabling the organisation to quickly determine if any of the changes made are outside of what is compliant or recommended.

 

So actually, implementing DevOps properly can mean that an organisation finds itself in a great state for audit, with automatically generated logs of every change that has ever occurred. It will also make it far easier to control process flows in the company, as they are enforced by software.

 

If you’re looking to implement DevOps in a highly compliant organisation, here are some suggestions that may ease the process:

  • One size doesn’t fit all: Don’t think that you can apply a so-called ‘standard’ DevOps implementation to your organisation – every organisation has different regulatory requirements, a different structure and different goals. You won’t be able to accurately assess every potential issue right at the beginning (no, Waterfall doesn’t work!), but it’s important to gather some clear ideas about what makes your organisation unique – and in turn, how your DevOps transformation will have to be unique in line with this. [example here?]
  • Establish ownership: Choose a team to lead your DevOps implementation. This can be an existing one, or a totally new one, but make sure they are in a position to innovate and collaborate across the business – they must have visibility into the activities of all other involved teams, and direct channels for communication with them.
  • Involve senior staff early in the process: Getting real buy-in from high-level people is going to greatly ease your implementation. If everyone is sharing a common goal it will reduce friction at all stages of the implementation process, and a top down approach to this will speed it up.
  • Support implementation with training: Equally, buy in across the whole company is even better. It’s important that people at all other levels of the hierarchy are given a full understanding of what the organisation is trying to achieve, and how they can help that process along, and this can’t just be achieved over chats at lunch.
  • Involve auditors early in the process: It’s important to fully discuss all plans with auditors before you begin the implementation process – it will be much easier to build compliance into the system if you understand all regulatory requirements at the start, rather than trying to fulfill them retroactively.
  • Find the source of all internal directives: internal directives that work to simplify compliance are generally more stringent than the actual laws and guidelines they aim to enforce. Making sure that your DevOps implementation only fulfills regulatory requirements rather than internal directives will help to eliminate unnecessary box ticking and enable you to streamline your adoption process.
  • Don’t silo your adoption: make sure that scaling agile processes across your whole business, not just the IT department, is part of your plan – DevOps aims to break down silos and only applying these processes to one part of the organisation risks creating larger scale silos between DevOps/non DevOps business functions.
  • Be confident: begin your transformation with a small-scale proof of concept, and expect to be pleasantly surprised by the metrics. If you aren’t, don’t jump to the conclusion – it may well be that you’ve just not implemented DevOps processes or tools in the correct way for your organisation. Failures will often be a more important part of your DevOps transformation than the successes.
  • Be patient: This is not something you’ll be able to achieve overnight. Yes, large enterprises such as Barclays have recently implemented substantial DevOps implementation in record time, but you may find that there are more natural barriers to adoption in your organisation that you’ll have to overcome – and this may well be a very gradual process.
  • Communicate: The key to large scale DevOps implementation is effective communication. You want to give as many people as possible as much visibility as possible over the entire process. Bring people together, eliminate office politics and build an office culture where people feel comfortable asking for help, and happy to give it.

 


 

This is something we’ve spent a lot of time thinking about at MidVision – our Application Release Automation solution RapidDeploy was first implemented in Lloyds Banking Group back in 2008, and many of our customers work in highly regulated verticals like financial services and healthcare. We know what enterprise DevOps looks like from the trenches, and we’ve gained a good practical understanding of what works and what doesn’t. If you’d like some help with implementing DevOps in your organisation, get in touch today