The New Systems Engineer

Titles mean something to me. They tell me about the nature of the work being done. They tell me about the organization’s values. They tell me where an organization is at on their journey to a technical and cultural revolution. Unfortunately, they mean very little to most organizations.

While this has always happened, it became even more so apparent with the rampant abuse of the term “DevOps Engineer”. How someone can be an engineer over a bunch of philosophies is mystifying to me and only proves the problem of vague titles. If the nature of your work means nothing to you and only the dollars matter, you probably don’t care.

This led me to think about what an organization is actually looking for when they say “DevOps Engineer”.

Release Engineer

A Release Engineer is literally just someone who coordinates releases. They are a functional silo that goes down a checklist and makes sure the I’s are dotted and the T’s are crossed. CI & CD have automated much of this job away, thus what companies are usually looking for is someone who has a functional understanding of CI pipelines and deployment methodologies. Since pipelines are key, basic scripting is another key requirement, as is understanding programmatic API’s - though it seems organizations mainly focus on REST.

Operations Engineer

The Operations Engineer is nothing new, generally, these people have been systems types or middleware admins responsible for deploying and sunsetting other peoples code. Basically, they are the manager of the latter part of the SDLC, and somehow, sometimes even the owners of it.

Middleware Admin

These folks manage one-off applications. It was a prioritization of the specialization in an individual application rather than a concept or an idea. I see very few of these jobs available today, but if they are, they are usually thrown under the title of “DevOps Engineer”.

These are old roles seeking a new brand without a thorough scrubbing of the skills and requirements to perform them. The commonality in all of them is they promote the old silo-style of work that DevOps has sought to rid the world of.

The Rise and Fall of the Systems Engineer

A long, long time ago in an industry far, far away a Systems Engineer had a definition. Frequently they knew C or C++, a common requirement was the ability to debug and contribute to the kernel or systems tools. Unfortunately, over much of tech and throughout my career I’ve met very few people who bore the title that could do as much. Even I lacked much of these abilities for much of my career.

I cannot accurately tell the story of how things transitioned from primarily Systems Engineering to what I consider more “Systems Administration” or Middleware admins. Anecdotally, I would probably blame the conglomeration of Network Operations Centers, where systems teams grew. Why have a team of Systems Engineers when you can have a ragtag group of Systems Administrators who could quickly patch systems and move on? As a Network Engineer, this is what I witnessed, which resulted in stories like the Bastard Operator from Hell (BOFH) when scale began to weigh on these teams.

The New Systems Engineer

Look! There, I said it.

Before I get started I want to clarify something:

I am in no way trying to gatekeep this industry. I am not trying to say this is what you have to know otherwise you’re not actually a Systems Engineer. Instead, I am trying to establish Systems Engineering as both a role and a skillset. The skillset is the basis for many other roles, which I will introduce later.

One might ask why I’m not saying, “back to the basics” or something of that nature. Indeed, rather than returning to old Systems Engineering, where the maintainers of systems were the same people who contributed to the kernel, I do think some things have indeed changed.

Understanding of Linux Concepts and Capabilities

Namely, most companies don’t need someone who can contribute to the kernel. Instead, companies seem to value a thorough understanding of *Nix in order to pursue organizational goals of performance, reliability, or security. This doesn’t mean that contributing to the kernel is irrelevant - in fact, it’s very relevant - but I think that it’s more appropriate to file it under a new title now.

There is value in understanding lxc, KVM, namespaces, cgroups, chroots, jails (knowing some things about FreeBSD is cool too!), and permissions.

Networking

I’m not saying every Systems Engineer needs to be a Network Engineer, however, networking is important to understand as well. Some useful bodies of knowledge:

  • IP addressing
  • Subnets
  • Popular protocol mechanics
  • Routing

All of these will help when both troubleshooting and architecting complex systems. They also assist the Systems Engineer in tasks such as Software Defined Networking without having to export huge bodies of work to a specialized Network Engineer.

Writing code

It is still important that systems engineers know how to code though. Some great language choices might be Python, Go, Rust, and Bash. Go and Rust are great for producing binaries as they’re compiled languages.

Go has continually shown to be very effective when writing asynchronous apps or wrappers while Rust is emerging as a replacement for C++.

Python is great for its utility as an analytics language and even for times when scripting would do.

Bash and the Bourne Shell are still needed in much of the container world where your tools might be limited.

All of these languages contribute to some aspect of building automation, tooling, or maintaining systems which are the key.

Delivery of Artifacts and Automation

Increasingly, systems components are being treated as artifacts - just like pieces of software essentially. They’re packaged up into a virtual machine or container image and deployed using some automation. Thus, it’s increasingly important for Systems Engineers to know and understand the concepts of immutability and idempotency.

Being able to apply the SDLC to systems processes and artifacts is key to being able to deliver reliable systems and automation.

The Emergence of New Roles

As the requirements for Systems Engineering evolved, I think the roles that follow it did too. The Operations Engineer is moving towards philosophies like Production Engineering or Site Reliability Engineering, or more basically, operations through the lens of a software engineer. These roles, I believe, involve some additional specialized skills but start at the base of either Systems Engineering or Software Engineering.

Distributed Systems have given rise to Infrastructure Engineers and Distributed Systems Engineers who build platforms for deploying software onto. Their primary goal is abstracting infrastructure away from software developers in order to make delivering software effortless for software teams.

Release Engineering and Middleware administration are now just side effects of writing good process and code. No one person should be in charge of it, rather the organization should be encouraged to boldly uphold a set of standards and self-manage accordingly.

Conclusion

This isn’t really a call to action, but rather, what I hope is the start to a discussion about consistency. I am sure that some will argue they’re all different companies, thus all their requirements are different, but I think there is a way to carve out some consistency in the skillset.

My hope is a clear skillset paves the way for more Systems Engineers, and more Systems Engineers means more folks saddling the new roles I talked about as well.