TLDR
- Stop blaming other people for failures.
- Talk to each other.
- Fix your systemic issues.
- Go read the Phoenix Project https://www.goodreads.com/en/book/show/17255186
As two wise men once said;
DevOps has nothing to do with technology
If you have worked in tech for more than ten minutes, you’ve heard of DevOps. You can’t scratch your ass without someone preaching its benefits or talking about a tool that will deliver DevOps nirvana.
You know it’s a real thing because anybody who’s anybody has a DevOps solution to sell you.
There’s an excellent chance that DevOps has been implemented in your organisation, and you are actively fucking it up right now. If you’re very unlucky, someone went on a bender with the consultants and you’ve implemented DevSecOps.
It’s easy to see the signs of DevOps bullshit:
- There is a DevOps team in the business.
- The Developers and Operations staff don’t attend the same meetings or standups.
- Management has gotten the devs creating their own infra and ops are writing code.
Anyone has the word DevOps in their title.
The original principle was excellent, effective, and made everyone’s life easier, while improving business outcomes:
- Cross-function collaboration.
- Feedback Loops.
- Experimentation and learning.
The problem is, it’s difficult to sell. A C level isn’t going to sign off on a million-dollar spend to hear “The secret to better product development is everyone collaborating and not blaming other teams for mistakes”. Doesn’t matter how good the slide deck is.
If you want to make a buck off DevOps, you need something tangible and measurable. Something management can point at and say, “See that? I purchased and implemented that. Look at my chart of the things I’ve measured!”. That’s the sort of concrete result they can put in their KPI’s.
So the ass hats in marketing kept the acronym, tossed out the core messages and “productized” a bunch of shit to sell budget holders.
They came up with diagrams like these, for DevOps tool touchpoints. The goal here is to make sure there is something for you to buy at each stage of the “journey”. It’s got nothing to do with continuous improvement, and everything to do with continuous purchase orders.
If you really want to establish DevOps, stop focusing on the tooling and turning devs into ops people or vice versa. Rolling out GitHub Actions and making the ops team learn Cloud automation with Python isn’t DevOps.
That’s just the same old corporate tribalism, but now you’ve got new tools to learn and people doing a ahit job because they don’t know how to do anything well.
So what the hell is it then?
To illustrate the rest of this post, let’s use a simple example that can illustrate the various aspects of DevOps.
Marketing would like to change the font on the company website because it’s become “stale”.
Culture of collaboration and shared responsibility
First and foremost, DevOps is a cultural change within the organization from the top down. The goal is to break down silos, increase cross-function communication and a sense of shared ownership. It has to be:
Driven by the leadership team, bottom-up change never works.
Everyone has to want it; you can’t force people to be collaborative and accountable.
Everyone in the food change needs to stop saying “them” and say “we” instead. The goal is that:
People with development, server, and operations skill sets collectively design and implement a solution that meets business requirements. They each contribute their expertise and engage with peers and the broader organization to ensure the best possible outcome.
In our simplified example:
- Devs work to replace the 10,000 baked-in style= and <font> entries.
- Infra enables the deployment to the servers and CDN.
- Ops makes sure the site still performs as expected and is monitored.
- If, at some point, that solution falls over:
The same people contribute to issue resolution and hold each other accountable for service delivery. If marketing insists that issues are resolved at one in the morning, then you ALL resolve it at one in the morning.
You built it together; you own it together.
At no point do we blame the ops team for shitty servers, the developers for bugs, or the security team for being paranoid control freaks.
If you find yourself constantly discussing how others have failed you, you’re either part of the problem or you’re in the wrong company.
Tips for staff
- If you prefer to stick in your lane, not collaborate with others, and don’t want to be accountable for the outcomes, then DevOps is not right for you. Fortunately, there are heaps of jobs (most, actually) outside this culture, and you’ll probably thrive in a bank or government department. Do everyone a favour and fuck off. Honestly, you’re going to hate it, and nobody in the team wants to put up with your shit. Just go away.
- If the company is faking it, and just going through the motions, then you have two choices. If you don’t care, then continue on as usual. If you do believe in DevOps, and it’s an important way in which you wish to work, find a new job.
Tips for management
- If you aren’t a true believer in the concept of DevOps, I would avoid any org doing/implementing it. Cultural change is extremely hard and will take an enormous effort on your part and those around you. If this change is being forced on you from above, you’re going to hate it and make everyone miserable.
- If the business is just going through the motions and has no real intention of practising DevOps, then you definitely want a new job. That’s a no win situation, as you’ll never be able to report success upwards and all your reports will hold you in contempt for passing down bullshit.
- Not everyone is going to get on board. Some people just don’t want it, or don’t have the emotional maturity and mindset to do it. Find these people and purge them from your organisation. You’re not a therapist, and it’s not your job to make people shift their attitudes. Just one resistant person in your team is enough to undo everyone’s effort.
- CIO/CTO’s: If you’re considering DevOps for any reason apart from real cultural change, just don’t. Do everyone a favour and just throw the strategy paper you paid for in the trash. You’ll be happier, your reports will be happier, and you’ll save a lot of money.
- HR: For fucks sake, don’t get involved in rolling out a cultural change and then insist that everyone has to “be taken along the journey”. Some people don’t want the journey, can’t make the journey, or just find joy in fucking up other people’s road trips. Enable managers to purge these people from your company.
Feedback Loops
Feedback is a very scary concept for nearly everyone. It puts us at risk of criticism, having our actions second-guessed, and in the worst case, getting fired. This is particularly true of technology functions because we are constantly making people’s lives difficult. When nearly everything you do annoys the shit out of somebody, you learn to avoid any opportunity for feedback.
If you’re not engaged in getting and providing feedback, you’re not doing DevOps. When was the last time you asked anyone in your organisation what they thought of your output? When was the last time you had an honest conversation with your peers about their output?
DevOps is built around feedback and the principle that having a short, accurate feedback loop will lead to improved outcomes. Every action you take is an opportunity to gain some form of feedback. I would go so far as to say that an organisation is doing anything, and there isn’t feedback, then it shouldn’t have done it.
Feedback isn’t just what your end-users say about your product/service; that’s just the obvious stuff. It comes in various forms and at multiple steps along any given process. For example:
- Direct customer feedback.
- Indirect customer feedback.
- Unit and integration testing.
- Pull Requests.
- Continuous Deployment success/failure.
- Systems performance metrics
are all critical forms of feedback on work. Here’s the most important thing:
Don’t let feedback happen passively. You have to expect it and if it isn’t there, go find it. Moving a ticket through to done isn’t good enough. Get some sort of external confirmation that what you did, and how you did it, was right.
You’ll often see diagrams like this one, that show an endless loop of DevOps.
Unfortunately, in most organisations, this loop is far too long to have value. You’ll need to identify ways of getting feedback inside each phase of the loop.
For the font change example, our DevOps teams can get feedback from:
- Validate Business Assumption:
- Do you really understand what Marketing wants?
- Do they understand? What if that font is used in images and now the sites will have mixed type in it?
- Make some changes and get Marketing’s feedback.
- Measure your change rate and tell the PM how long it really takes.
- Show examples of the change on the real site, not the Figma or other mock-ups. Do they still like it?
- Testing
- Ensure that the changes pass unit and integration testing.
- Does the new font deploy and load, but trash the site layout?
- Ops
- How are you going to get Ops to deploy these changes?
- Does Ops have concerns the font is in a third-party CDN outside your control?
- Is Security happy that a third-party file is being loaded with the site?
- Review site performance to ensure that it is comparable with the previous releases.
- CICD
- Ensure the refactored assets deploy from Development > Test > Staging without breaking.
- Is the CICD process a brittle pile of crap? Don’t just leave it there and move your ticket to done.
- Are there unit tests at all? It’s easy to pass CICD testing if you don’t have any.
Finally, reach out to Marketing and find out the results of the change. Did this change drive traffic up or down? Was there any impact at all? As a developer, you may not have much control over what gets in the backlog, but it’s still good to know what the outcome of your efforts is.
If you’re in a management position, this is your chance to hold internal stakeholders accountable. If Marketing is forcing through things like UI changes and can’t demonstrate the results, call them on it. They are burning through your budget and priorities; they should be able to justify why that happened.
If nobody can show the impact, good or bad, then why was it done at all?
Tips for staff
- Be willing to challenge your assumptions, especially if you’ve been in a role for a long time. Any incumbent system will be filled with “because that’s how we do it” operating models, and they almost certainly need to be changed.
- Feedback can be brutal, even when well intended. It can make us feel like we are at risk, and heaven help you if you’ve got imposter syndrome. You’ll have to push through that emotional resistance and find a way to be open to feedback.
- Conversely, if you are strongly confrontation avoidant, then you may find it hard to give other feedback. Especially if those people are more senior or long-term employees. Try to focus on the system/thing you are trying to change, rather than the person following the system. It sounds cliché, but feedback is not personal. Managers provide performance feedback, you’re providing systems feedback.
Tips for Management
- You need to make your teams feel safe when challenging the status quo and providing feedback. Without that buffer from you, you’ll discover that your team has very little to say about anything.
- Feedback has to be cross-functional, so if there are discrete dev and ops teams, you and other managers are going to be responsible for building the bridges between them.
- If you are being forced to implement DevOps but know it’s not real, try to avoid setting staff up to fail. Make sure everyone knows that they shouldn’t try to implement feedback. Just because your manager is being a dick, doesn’t mean you have to.
Experimentation and learning through failure
Last but not least, we come to the topic of experimentation and learning through failure. Something that companies hate and avoid paying for, wherever possible.
Everybody outside tech, especially the CFO, sees this kind of activity as dead money and a source of unnecessary risk. It’s difficult to measure its material return to the business, and it cuts into the budget allocated to business projects. Moreover, people are conditioned to avoid failure. They certainly don’t want to be involved in enabling it.
You won’t find big tech products in this space; there’s no money in it. Vendors push products such as testing tools and staff training in the hopes that marketing can convince people Continuous Improvement means fixing bugs faster or training staff.
Experimentation
Experimentation has nothing to do with technical skills or tooling, and it needs to include a wider audience than just DevOps. You need the product team on board; that’s where experimentation starts. If product demands are rigid, it’s very difficult to be flexible in its creation while doing development.
More importantly, experimentation in product is far more valuable than experimentation in low-level technologies.
Experimentation is not about trying out new tech frameworks, software, or hardware because it’s the latest fad. It’s certainly not about swapping to a tech you prefer or learning something new to pad your CV. Go do that on your time or in an organization that can support dead wood.
The goal is to change systems by taking small, calculated risks, that can be tracked to measure the level of success or failure they have.
Meaningful examples would be:
Changing the checkout UI/UX to increase sales.
Change the CICD pipelines to reduce build time.
Change the configuration of the server build to reduce overhead of systems tools.
Update the version of an existing application to identify if new features have a meaningful benefit.
Remove manual processes by automating them.
There is one crucial form of experimentation that all tech teams must do, and that’s preemptively upgrading core systems. This is often ignored because it’s a pain in the ass, isn’t very glamor, and the business often doesn’t see the value. After all, if it ain’t broke, why fuck with it?
That’s a much longer conversation, but upgrade strategies (or lack of them) are one of the main generators of risk, delay, and lost opportunity in any business. If you are not preemptively upgrading any given system, you are filling your tech debt ledger with red. You’re going to pay that bill, one way or the other. The longer you defer it, the higher the compound interest.
Learning through Failure
The concept of taking small risks and learning from them is not revolutionary. We’ve been doing this since the first human tried cracking nuts with a rock. For some reason, when we do anything in technology, we think it’s some incredible revolution or makes us a special, unique snowflake. Meanwhile, the rest of the world just calls this process “science”.
The logic is simple:
- You can’t find a solution without taking action.
- Actions carry risk.
- So identify and minimize the risks.
- Take action in measured, repeatable increments.
- Measure and validate the results.
The trick is to avoid acting like Robert Cocking.
Robert thought he could invent a better parachute, and considering the parachutes of the day sucked, that was a great idea. The problem was, Robert thought like an executive and went all in with a big bang launch of his product. Which resulted in him being spread from one end of a field to another.
So unless you want your feet shoved up your ass at 9.8 m/s, don’t be like Robert. Learn through small, controlled failures that can be recovered from.
And don’t believe anyone who says failure can be avoided. Those people are either morons or work in sales and want you to buy something.
While some organisations allow for experimentation and learning through failure, they often do a poor job of the learning part. They forget the crucial part of learning, called education. There is value in one person learning how to deploy a web server ins a docker container. There is a lot more value in that person then teaching everyone else how to do it.
Learning is an expensive process that carries inherent risk. Once knowledge is gained, it is very cheap to share. If you don’t have a formal process/ritual of sharing knowledge with peers in your organization, then you’re not doing DevOps. That’s just a costly self-education program.
Tips for Staff
- Try to find small, discrete and safe things to do at the start. It’s better to get a few wins on the board before you start trying to rip apart large systems. This gives your manager a positive message they can pass up the chain, which makes future approvals easier to get.
- Make sure you share whatever you’ve learned. If nobody has established a formal way of doing that, do it directly. Create a PDF or send an email.
- Don’t fuck with production! Experimentation doesn’t mean bringing down core systems. Roll it out starting in dev, then test etc. If those environments don’t exist, then you’ve just found your first experiment. Go build them.
Tips for Management
- Try to keep the terror out of your eyes as the team starts making changes. It’s going to be painful at the start, but the team needs to feel empowered and that you have their back.
- People will make mistakes and break non-production systems. It’s your job to make sure there is no blowback for that.
- If breaking the build in development is a critical issue, then 1) It’s not non-production and 2) it highlights you need a new solution allowing for multiple environments
- Don’t build a bureaucracy to manage the experimentation. This is meant to be small, and fast changes.
- Make sure the team’s have a way to share knowledge
- Share the positive results of your team’s experimentation and give them the credit. Talk about how the change in culture is leading to positive outcomes.