Yeah, that's meMatt Ouille Site Reliability Engineer A software engineer focused on infrastructure automation, Linux, and service reliability.

Why what Trustico did is so bad

If you haven’t heard, last Wednesday it was revealed that Trustico, a large reseller of TLS certificates, violated customer trust by emailing another reseller 23,000 of their customer private keys. That’s really the gist of what they did, what these articles don’t really cover is the underlying reasons why this is directly bad and why what it means is even worse.

There’s this really sweet document called the (“Baseline Requirements for the Issuance/Management of Publicly-Trusted Certificates”)[]. In that document there’s section 6.1.2 which states:

Parties other than the Subscriber SHALL NOT archive the Subscriber Private Key without authorization by the Subscriber. If the CA or any of its designated RAs generated the Private Key on behalf of the Subscriber, then the CA SHALL encrypt the Private Key for transport to the Subscriber. If the CA or any of its designated RAs become aware that a Subscriber’s Private Key has been communicated to an unauthorized person or an organization not affiliated with the Subscriber, then the CA SHALL revoke all certificates that include the Public Key corresponding to the communicated Private Key.

This helps spell out and dispel some things. CA’s are allowed to generate keys on behalf of the customer, however, they are not allowed to archive said keys for any reason. If the CA generates the key on behalf of the customer, they must encrypt said key for transmission to the customer.

What I generally regard as safe

I don’t let a CA generate private keys for me. Private keys are used for encryption, but what they really are is a method of identity. There’s three important pieces to secure TLS communication: the private key, the public key, and a CSR (Certificate Signing Request). These three things form a trust relationship between a well known Certificate Authority and something on the internet. In laymens terms the CA is trusted to have a root private key and to use that to sign off on relationships with endpoints on the internet given some burden of proof.

The public, private keys, and CSR should be generated on whatever network you intend to assume trust for. The real point here is that the private key never leaves.

Next the CSR is transmitted to the CA. The CSR contains some metadata and the public key encoded in ASN.1. The CA verifies any information they feel they need to verify to establish trust, sign the certificate, and transmit it back to you. The CA should never actually know your private key, they only establish trust.


TLS issuance, and largely cryptography in general, was made from the ground up to be automated. There’s a couple major failures:

  1. A CEO thought it was a good idea to transmit private keys and CSR’s (which you now know contain public keys, am I right?) across email. Whether it was encrypted or not, this action violates the trust of every customer they’ve had. CA’s jobs are to establish trust, and the reason they’re given the trust to do that is because people trust their processes.

  2. Private keys were stored in the first place. We know from our Baseline Requirements documents that this isn’t acceptable under any terms.

Action Items

  1. Transmitting private keys is dangerous. The Baseline Requirements should, in my opinion, be updated to get rid of CA generated private keys.

  2. The CEO of Trustico does not understand the underlying technology or premises of the company he runs. He should be terminated immediately.

  3. Each major CA should be evaluated by a third party to assess the CA’s adherence to the Baseline Requirements and CA’s outside of the Baseline Requirements should be forced to notify current and future customers of this without explanation. I say without explanation, because Trustico said they needed to maintain private keys in order to revoke them. I’m not sure what world they live in given this statement.

That’s all my ramblings for now, hopefully it’s helped some people, thanks for reading!

DevOps and SRE in the Enterprise

At start ups we have the luxury of starting from the ground up. Philosophical and cultural revolutions are always easier to consider when simply nothing exists. This is a common complaint of large enterprises when they examine DevOps, SRE, or Production Engineering for their own organizations. Subsequently they end up morphing the core beliefs of those philosophies to fit their existing culture. While this can be done it really results in a lot of confusion and frustration especially at the ground level where, in the aforementioned disciplines, most of the work and decision making takes place. The question remains what does a properly scaled Enterprise grade version of DevOps looks like?

Using return values inside CloudFormation UserData

I've been working on a project that uses CloudFormation exclusively, so I don't get to do variable interpolation that's as simple as Terraform makes it. Thus, I've had to do some improvising when generating files based off my infrastructure orchestration.

Testing Jekyll with Travis CI

I've finally settled with the fact that Jekyll is going to be my mainstay for a while. It's got a lot of features I like some of which are subsequent to being a static site generator, others being thoughtful implementations.

Taking Security a Step Further with VPC Endpoints

Ever since I wrote S3 Bucket Security and Best Practices I’ve been playing with how to extend security within a given AWS account.

Policies and KMS are great but to me assurance comes in threes. That said I don’t see VPC Endpoints discussed very much when people are talking about S3 Bucket Security, especially for something that’s served over a public API!

Variations in High Availability

High availability is a term thrown around quite a bit these days. Many professionals conflate high availability with the idea of a theoretical 100% availability. I'll be the barer of bad news in saying such a thing rarely exists, is even harder/more expensive to obtain, and often not worth it. Rather, I encourage teams to identify what high availability means to them. This article will be an exercise in exactly that.

Variations in High Availability feature imagePhoto Credit: PerconaDB

A Big Thank You

I owe everyone a tremendous debt of gratitude. I’ve been posting on this blog for quite some time. I’ve made some transitions from Ghost, to Wordpress, and now to Jekyll. In that time I also tried out some SaaS platforms like Medium and Blogger but I’d rather not remember those times. Thus, you see the articles that are on my current blog - I’ve lost a lot but I’ve gained a lot. Just like myself my blog had to evolve over time. One thing has remained fairly consistent though: why I write.

S3 Bucket Security and Best Practices

In this write up I explore some of the intricacies of Amazon S3's permission matrix. I go over best practices, ACL's, Policies, KMS, and Server Side Encryption. Read on for more!

Docker and Docker-Compose for the developing mind

Introducing docker and docker-compose to developers for developing powerful local development environments and easy to deploy production environments.

Docker commands every dev should know

Docker is a really awesome containerization platform. It dutifully simplifies LXC (Linux Containers) and enables developers to develop faster. That said, at times Docker can be a tad confusing and things can get out of hand quickly if you’re not up to speed on your docker commands. These are all commands that I keep up to date on my GitHub gists page for docker, so I use them regularly as well.

Pointers in Go

I've been learning Go recently and I've been having some issues solidifying my understanding of pointers, so I thought I'd write a quick blog post explaining what I've learned and how to easily understand it. Click the title to read on!

“The Most Forgotten Pattern”

Developers love consistent ways of solving repeating problems, but the most consistent problem of them all is never solved repeatedly. Every time we sit down to solve problems with software we apply design patterns to overcome situations that would be contentious if we hadn’t already solved them years ago. There’s no point in reinventing the wheel, right? So why are we sitting here planning development of software for months on end still?

The switch to Jekyll and updates

2017 has been really good to me and I’ve had the opportunity to grow a lot both as as person and in my career. I’m currently writing this post sitting on my back porch watching my dogs play.

That said, earlier versions of my website were always based on Wordpress. While Wordpress is a great functional FOSS blogging platform, it was really a bit bloated for my needs. As a developer, and someone who is trying to make more efficient use of their time, I was looking for something that got down to the nuts and bolts of what I wanted without sacrificing too much functionality.

In that I found Jekyll and GitHub pages. I kind of wrote this off as a poor man’s solution to a common problem, however, I’ve discovered the elegance in what Jekyll provides. Jekyll is categorized as a Static Site Generator written in Ruby and extended through plugins and API driven services such as Disqus. You can checkout Jekyll here.

I use a Mac and haven’t done a lot of Ruby focused projects so I was really unaware of how to start. What I’ve figured out though is that between RVM and HomeBrew you can get a sustainable environment going. I could probably also have written a Dockerfile to do a lot of the dependency management for me, but I’m only using one gem and that’s Jekyll itself.

As for deployment, I already pay for GitHub. I was a little disappointed to discover GitHub pages doesn’t support a CDN proxy to run SSL on custom domains but I looped in CloudFlare for that. All in all, a blog post is just a commit away and I can use Atom as a post editor.

While I’ve been writing a lot of specific ‘getting started’ tutorials lately I think it’s time that I start writing more Site Reliaibility Engineering focused work. I currently work in an AWS environment but dumping my virtual machine provider frees me up to do some AWS and GCP tutorials. Who knows, maybe I’ll get froggy and do some Azure too.

Test Driven Infrastructure Basics

Today I’m going to go over the basics of Test Driven Infrastructure, what it means, how to do it, when it applies, and why. In this tutorial I’m going to use Chef, but you can use whatever you want.

User error caused a massive S3 blackout

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems. source:

Is this where we echo one of the great pillars of Linux? With great power comes great responsibility.

SendGrid DNS White Labeling and CloudFront for Secure Click Tracking Links

At StarLeaf we had a need to secure our SendGrid click tracking links, unfortunately our provider, SendGrid, had no way of sending HTTPS traffic with their in place white labeling solution. This is how we solved that.

Starting a Career in DevOps

A few years ago I got out of the military as a radio technician, but before I got out I had a ominous conversation with a long time friend explaining that I thought the future of an IT career lied in a mix of systems, programming and virtualization. At the time I really knew nothing about virtualization, had really only web and some Perl/Python experience and a life long love of Linux. Since then, my experience has turned it into a beautiful well-rounded career that’s only growing.

Basic Ubuntu Security

Especially in 2017 everyone should be concerned about security. You don’t need to be a genius or completely paranoid in order to avoid most potentially compromising situations. Follow these instructions and you’ll have a basic understanding of what it is to secure your brand new, vanilla Ubuntu server. That said, if you use the web or have a publicly accessible service then there is always a chance you will be compromised. You cannot predict application patches with vulnerabilities or the colorful attempts a hacker that is specifically targeting you may employ. You may simply prep with your best effort.