Third wave of Automation

SysAdminDay aka System Administrator Appreciation Day is on the Last Friday in July. This year (2018) was on the 27th. I was mostly working inside of one of our datacenters.

The "clouds" have physical servers, storage filers, network & fiber optic cables and as much as power cables as everything else. And there are people (we used to call them sysadmins) that are managing the "cloud"!

If you have such people in your teams and companies just bring them cake! As Dylan Moran best said it:

"Cake is the language of love."

Having said that, sysadmins are mostly lazy human beings!

But not lazy as you think. They can spent a large amount of time to develop something, just to use it the next time. This is the definition of automation in layman's terms. In modern times and especially in code development this is called: DRY or Don't Repeat Yourself .

I usually go with these simple three rules:

• Run once, do it manual.
• Run twice, see a pattern.
• Run third time? Automate it.

First Wave of Automation

So what is the "first wave"? First wave is when you are scripting your daily jobs. Like creating a user into 1000 machines (physical & virtual) with different operating systems. You are not going to login to every server to manually create a user. This is not efficient of your skills and time. You will script this, make tests (trial & error) through the process, noticing what is missing and every time adding a few lines of code. So in the end, you have your "user_add_remove.sh" that you can run from your machine to create or delete users.

Scripting is the first wave.

You are starting to see that you need a version control system, a test-bed machine, a wiki page, a manual page (plz document your work), you have to inform your colleagues of this awesome script.

Five Rules to remember

I usually trying to make a point with the below five rules:

1. Where is your code (need to be on a version control system) ?
2. Where is your wiki page (need to write the story) ?
3. Where is your help/manual page (need to know what this script will do, show me a usage, perhaps some examples) ?
4. Where is your email (informing all your colleagues and manager, also tracking your job) ?
5. Where is the print-screen (a visualization is more helpful to memorize) ?

Second Wave of Automation

What about the second wave? Second wave is when you stopped scripting your way out of everything but rather now trying to use a configuration management tool, like ansible, or puppet, or salt (saltstack), chef, cfengine, capistrano, juju, name your poison - we are all friends here!

Why you want to do that? Cause other people have already solved your problems (in some way). Why not use their knowledge to your benefit? We are using ansible for automation, configuration management, provisioning & deployment (in some cases) !!!

I know it is hard to migrate your entire legacy infrastructure, code and scripts to this new shiny thing. Guess what? It's not that hard! you just need to buy some time and work on this. Just remember your work is now a culture change.

Configuration management software is the second wave.

As your organization grows your people is most likely to change teams. Your new people should be able to maintain all the legacy custom scripts (some of them are 20 years perl thousand lines scripts) and run their own from scratch.

Using a configuration management tool, you now can add/remove users by just running a playbook via a simple command. Perhaps you need to extend this playbook in the future but there is no need of keeping legacy custom scripts in your infrastructure.

Second and a half Wave of Automation

CI/CD aka Continuous Integration & Continuous Deployment tools are now part of our life. We are using gitlab to create new custom rpm packages in our production, every time there is a new upstream version. And ansible to manually deployment them! Make your developers and sysadmins aka your devops people happy. Now the code is being tested before hand in containers or disposable virtual machines. Win-Win.

Third Wave of Automation

What about containers, docker, kubernetes, hybrid and private clouds like openstack and vmware? Are they part of automation or they are part of infrastructure? What about amazon AWS or azure or google cloud computing? We are scripting our way through all these parts. Is that automation?

Actually Yes, it is automation. But not in a traditional way. You can still work with Virtualization, Cloud (private or public) and still run a combination of first or second wave of automation.

To simple put, the third wave of automation is when you automate your infrastructure. When you are dealing with all these moving parts as lego bricks in your test-bed enviroment and you deploy the whole application in production as one: app & infra as one.

Infrastructure as Code

Projects like terraform, cloudformation, ansible tower are the tools of the trade in this case. You are building lego bricks, artifacts with your CI (continuous integration) tool. Your app is now together with your infrastructure, and your code is the orchestrator of both. And not only your app, your security is already build-in, your testing is on your CI, the entire code for both your app and infra is on your version control system.

Every time a developer makes a change a new infrastructure with the new app is being deployed in production with a rolling release process!

Immutable infrastructure is the third wave of automation.

We are now moving artifacts, not code!

Misconceptions

By moving from one wave of automation to another you don't have to ditch all your previous work. You need to be able to see the big picture. Every wave solves previous problems but also brings new ones. The most difficult of all is when you need to change your ways. Change the way you work, new ideas is not the problem, business culture is.

Culture

There are a few known methodologies to follow: kata, lean, agile (scrum, kanban) DevOps, NoOps are some of them. All of them are having a common thing in their base and this is culture change. We are now moving into a new era, from creating custom scripts to scripting entire infrastructures.

So don't forget:

"Automate All The Things!"