AWS Morning Brief
aws-section-divider
Audio Icon
Whiteboard Confessional: The Rise and Fall of the T-Shaped Engineer
About Corey Quinn
Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links
Transcript
Corey Quinn: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. CHAOSSEARCH. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at CHAOSSEARCH.io.

For a long time now, I’ve been a believer in the idea of the T-shaped engineer. And what I mean by that is that you should be broad across a wide variety of technologies, but deep in one or two very specific areas. So, it looks a bit like a T, or an inverted T, depending upon how you wind up visualizing that. I’m describing this with words. I don’t have a whiteboard in front of me. Use your imagination, you’ll be okay. The point being is that whenever you’re working in a new environment, or on a new problem, having a broad base of technologies of which you’re aware, is incredibly useful to fall back upon. Now, the reason to be super deep in one or two areas, is that specialization is generally what lets people charge more for various services. People want to hire domain-specific expertise for an awful lot of problems that they want to get solved. So, having something that you can bring into job interviews and more or less mop the floor with people asking questions around that domain is an incredibly valuable thing to have.

But that has some other consequences too. And that’s what today’s episode of The Whiteboard Confessional is talking about. Back in my first Unix admin job, I busily began upgrading a whole lot of the infrastructure and ripping out very early Red Hat Enterprise Linux and CentOS version 4 systems and replacing them with the one true operating system, which, of course, is FreeBSD. And I had a litany of explanation as to why it was the best option, what it could do for various problems, and why there was just absolutely no comparison between FreeBSD and anything else. I could justify it super easily, and the real defense mechanism here was that people get really, really, really tired of talking to zealots, so no one kept questioning me. They just basically said, “Fine, whatever,” and got out of the way. Years later, I decided to focus on something that wasn’t an esoteric operating system to go super deep in, and that’s right, I picked SaltStack, which is configuration management done right, tied to remote execution. 

I’d worked with Puppet, I’d tolerated CFEngine, but I had a bunch of angry loud opinions about it and SaltStack was absolutely the way and the light. So, in the company I was working at at that time, I rolled it out everywhere, and our entire provisioning and configuration management process was run through SaltStack. And I could come up with a whole litany of reasons why this was the right answer, and that no one else was going to be able to come close to what the ideal correctness that SaltStack provided. And people eventually stopped arguing with me, because they had better things to do than argue with a zealot about which configuration management system was the right one to go with. I’ve also talked on previous episodes of the show about using ClusterSSH. And this was before I discovered the religion that was configuration management. 

It was the right answer, because rather than having to run a for loop with shell scripting, which was suboptimal for a wide variety of reasons, and I would explain to everyone why it was suboptimal. So, again, they shrugged, got out of the way and let me use ClusterSSH. And a similar pattern happened when I was working with large scale storage. NetApp was the right answer for all of our enterprise storage needs because let’s face it, it wasn’t my money. And when it comes to NFS, even today, they are head and shoulders above anything else in the space. And then eventually, it turned to AWS. And for a while, I want to say around 2014, 2015, I would tell you why AWS was the right answer for every problem. What challenge are you trying to work with? Well, AWS has an answer for that, because of course, they do. Their product strategy is, “yes”. Now, what do all of those independent stories have in common? Great question. Let’s talk about that. But first…

In the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider CHAOSSEARCH. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to CHAOSSEARCH.io. And my thanks to them for sponsoring this incredibly depressing podcast. 

The problem is, is that everything I just mentioned, was a pet technology, or a pet company. Something that I had taken the time to get deep in and learn. And therefore, it became my de facto answer for anything that even remotely looked like that problem. It’s like that old adage, “When the only tool you have is an axe, every problem looks like hours of fun, regardless of whether hitting something with a piece of metal that’s sharpened on one side is actually what you’re looking to do right now.” The problem with FreeBSD, for example, was that, terrific, yeah, it served an awful lot of valuable purposes, but no one else in the team knew how it worked. 

So, when I left that job, they had to rip it out and replace it. And that was not easy or fun, because I was about at good as documenting back then as you’d probably think I would be. The problem with SaltStack is that, to some extent, not everyone was on board with the idea of that particular tool. So, people were trying to use Ansible for some things Puppet for something else, and then SaltStack for everything else. So, you wind up with dueling configuration management systems, ClusterSSH is wrong. It’s not the right answer for anything. The problem was shell scripting wasn’t shell scripting itself, but that I was really bad at shell scripting. 

But, by casting my own shortcomings as this universal truth, it wound up very quickly becoming this accepted truism that I’m still stamping out and I hear about today. NetApps are great technology, but centralized storage is not really the right answer, and if it is, you’re probably looking at an object store in 2020. And when it comes to AWS, well, I’ll be honest with you, I don’t really have a strong cloud vendor preference. My default advice is, go with the one you’re currently using. I’m most familiar with AWS, but that doesn’t mean that it’s the right answer or wrong answer for your specific tool or your specific business problem.

The problem, as a result, becomes that a single person, in all these stories that was me, they champion this technology, and then they steamroll everyone else in the group until people get tired of arguing and let them do what they want. Or worse, they outright don’t even ask, they just go ahead and do it. And, “Surprise, we’re a FreeBSD shop now.” “Well, that doesn’t sound ideal. I don’t know if we want to be a FreeBSD shop.” “Well, yes, we’ve already deployed it to physical servers that are currently in 18 different buildings, so if you want to go rip it out and replace it with our special standard Linux distribution, you go on right ahead.” It doesn’t work. And I encourage you at this point, and this is where I start winning friends, look around your environment and see if this has happened to you. 

The most obvious example of this in the modern zeitgeist is Kubernetes. Take a look at your current Kubernetes environment and ask yourself a seriously tough question. Namely, how many people within your organization need to quit on the spot before you have a serious operational problem tied to the care and feeding of Kubernetes? In some very interesting shops, the answer is one. In some cases, the answer is zero because they already have an operational problem on top of Kubernetes, where an outside consultancy or someone who is no longer there has handled the configuration and care and feeding of their environment, and now it’s more or less become oral tradition of how you work with the existing Kubernetes install. The people don’t tend to understand it. People don’t tend to necessarily understand why things are the way that they are. And Heaven forbid it comes time to fix it. 

I’m also not picking specifically on Kubernetes. Serverless has the exact same problems. I’ve wound up in a serverless rat’s nest, trying to figure out what’s going on, where, how or why it was built, and that’s in my own environment where I’m the only person that builds all of these things. It turns out that the worst developer I’ve ever met is me three years ago when I have to go back and resurrect that monster’s code. It’s terrible. It’s awful. And I have to imagine that I will be saying the same thing about present-day me three years from now. I’m not saying don’t use these technologies. I’m not saying don’t enjoy the technologies that you’re using. But remember, you don’t work for the vendor in question. You aren’t being paid, most of us, by AWS to wind up championing what they do. And if something else emerges rather than your chosen technology, keep an open mind. Don’t crap on things just because it’s different. Don’t be down on technologies, just because it’s not what you already know, give it a fair shake, then crap on it from a position of knowing why it’s terrible. 

Most of the things that I make fun of in the various snark analyses that I do are around things that I think are terrible because I tried to use them, and I see where the sharp edges are. Making fun of something from a position of ignorance is both super easy and super fraught because it’s very easy to be told, you don’t actually know what you’re doing, and come to find out, you’ve been inadvertently making fun of a whole lot of other people’s hard work. And that’s not a great position to be in, and it’s not fair to them. So, my ask to you, as you think about this, is remember, there are other people who also have their strong angry loud opinions. Do you want to be subject to what they unilaterally decide? I’d suspect not. A little empathy goes a long way, particularly in these times where virtually all communication is not face-to-face. 

Thanks for listening. I’m Cloud Economist Corey Quinn. This is the AWS Morning Brief: Whiteboard Confessional. 

Thank you for joining us on Whiteboard Confessional. If you have terrifying ideas, please reach out to me on twitter at @quinnypig and let me know what I should talk about next time.

Announcer: This has been a HumblePod production. Stay humble.
View Full TranscriptHide Full Transcript