Q&A: Eric Mizell explains the reliability challenges of DevOps

In this Q&A, Eric Mizell, OverOps' vice president of solution engineering, discusses the challenges of DevOps at speed. When developers can deploy code with the touch of a button, it's as important as ever for organizations to establish code accountability and tight feedback loops. Ultimately, teams can tackle the challenges of DevOps with the right visibility, data and context.

How can organizations go from releasing once every six months, to once a month and on down the line?

Eric Mizell: How do I get things moving faster starts with culture. It's a cultural problem. And culture starts with ... 'Hey, everybody has to be onboard, right?' I think one of the biggest things we're running into is accountability -- a code accountability problem of who's responsible for taking care of code across the development lifecycle. It starts with this culture of code accountability.

The days of writing in code and it goes through manual QA cycles and we do a release every six months and there's a freeze cycle -- those are fading out. I want to check in my code; it automatically builds; we automatically create my total infrastructure. Cloud has given way to auto-creation of my entire ecosystem in a tap of a finger. Tests are running. Code's deployed, and the challenges are now, 'How do I know if I'm promoting good code?'

How can teams handle both the quality and velocity challenges of DevOps?

Eric Mizell

Mizell: What I'm starting to see a lot of is development teams are responsible for innovation, and then they throw it over to these [reliability] teams to then deploy, manage, fix the bugs. That's the mantra and what's happening. The biggest challenge for [reliability teams] is they don't know what's being built. They don't know the product. They don't know the software. So they don't have visibility. They don't have context.

We have this notion of reliability tradeoffs, where there's this Venn diagram of speed, rate at which I can develop software, complexity; how complex the software is; and then the quality of code. And, really, you get two. If I go fast and the complexity is high, my quality will go down. If I want higher quality and high complexity, then my speed tends to go down, and I can't develop as fast. Same thing with high quality, high speed. My complexity will go down. Those challenges are very real, so DevOps is trying to get the best of those three.

At what point in the DevOps process does OverOps come in?

What's missing [in CI/CD] is a feedback loop back to developers as quick as possible.

Mizell: We're part of the software development lifecycle. I see a lot of developers using it on their desktop, as they check in code to make sure they didn't break anything. Code gets checked in, it's auto-deployed and as test cases get run -- the manual, the JUnit, the regression, the load testing -- OverOps is monitoring those apps and at the end of the cycle, which tells you how many errors you had, how many are new, how many have resurfaced. We're able to classify and give someone context -- real numbers as to what's going on.

What's missing [in CI/CD] is a feedback loop back to developers as quick as possible.

Why are feedback loops so important in DevOps?

Mizell: Being a software engineer, one of the biggest challenges is being able to recreate an issue. When I went into the office for a heavy code day, I would grab a soda and a candy bar. And I think that's what you see at any of these startups and a lot of these West Coast companies. They have snacks in the office to keep people there. You keep your developers sugared up. It's really weird how it worked, but my brain worked at 100 times what it normally works at, and I could write some amazing code -- in my opinion. And then, the next day, I kid you not, when something goes wrong, I look at the code and think, 'What did I write?' Because my brain was in a different state. And this is standard for every dev.

And so what happens is a lot of errors will get reported, whether it's through QA, through production or a different region, and developers can sometimes get defensive; I know, I've been. Because you built the code off the way a spec was written and think, 'Well, it works for me.'

And that becomes a challenge from a QA perspective. And then they start arguing. What OverOps gives you is a clear picture of why. You can actually see the data that was moving through the system at the time the error occurred. And that's where all devs are challenged.

'I didn't expect it to be that. How did it get to be that? It can't be that.'

It doesn't matter what you thought, expected or it shouldn't be. It is, because I have a snapshot that shows me. This is the data.