Whiteboard Confessional: On Getting Fired

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

This episode is sponsored by a personal favorite: Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front end frameworks to figure out or access controls to manage; just ship the tools that will move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas: buttons, checkboxes, tables, etc. Then I can wire all of those things up to queries with all kinds of different parameters, post, get, put, delete, etc. It all connects to virtually every database natively, or you can do what I did and build a whole crap ton of lambda functions, shove them behind some API’s gateway and use that instead. It speaks MySQL, Postgres, Dynamo—not Route 53 in a notable oversight; but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces. And I don't know frontend; that's the most important part here: Retool is transformational for those of us who aren't front end types. It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com/lastweekinaws. That's retool.com/lastweekinaws, and tell them Corey sent you because they are about to be hearing way more from me.

Today's episode of the AWS Morning Brief: Whiteboard Confessional was supposed to be about a zero-day that I was disclosing. Cooler heads have prevailed and we will talk about that next week instead, once I've finished some conversations with the company in question. Sorry to disappoint you all, but I have something you might enjoy instead.

So, today I want to talk about getting fired, which is one of my personal specialties. I'm not kidding when I tell people that a primary driver of starting my own consultancy was to build a company wherein I could not be ejected on the spot by surprise. Since I can't be fired anymore, let's talk about the mechanics of getting fired from someone who's been through it, just so folks get a better perspective on this. In the United States, our worker protections are basically non-existent compared to most civilized countries. Barring a contract or collective bargaining agreement to the contrary, you can be fired in the United States for any reason or no reason, except based upon membership in a protected class. 

So, to be clear, my personality is certainly justification enough to fire me. I say this for our listeners in other countries who hear I was fired and equate that to a moral failing. “What’d you do, rob the cash register?” No, I'm just me; I'm difficult to work with; I'm expensive to manage, and my personality is exactly what you would expect it to be based upon this podcast. The way the firing usually works is that you get an unexpected meeting request with your boss. “Hey, can we chat?” 

Those meetings are so unnerving that even that intro leaves scars years later, my business partner and I—both of us can't be fired clearly. But we still get nervous when we tell each other, “Hey, we need to talk in an hour.” We have instituted an actual policy against this at our company, just due to the collective trauma that so many of us have gone through with those, “Is this how I get fired?” moments. So, you have an unplanned meeting with your boss. Nine times out of 10—or more: 99 times out of 100 that's fine—it’s no big deal: it’s about something banal. 

But on this meeting, you walk in and surprise, there's someone from human resources there too, and they don't offer you coffee. First. I want to say the idea of calling people resources is crappy. HR—whatever you want to call it: people ops—but regardless, they're there; they're certainly not smiling, and they don't offer you coffee. 

And that's the tell. When you're invited to a meeting that you weren't expecting and no one gives you coffee, it is not going to be a happy meeting. They usually have a folder sitting there on the table in front of them that has a whole bunch of paperwork in it. There's the, “This is the NDA that you signed, when you started your job here; it's still enforceable: We're reminding you of it paperwork.” There's a last paycheck and a separate paycheck of your cashed out vacation time in jurisdictions where that gets paid out, like California. And often, there's another contract there. This is called a severance agreement. The company is going to pay you some fixed amount of money in return for absolving them of any civil claims that you may have had during the course of your employment. I'm not your attorney, but let me tell you what the right answer here is. 

Whatever you do, do not sign that contract in that room, in that moment. You've just been blindsided; you don't have a job anymore; you're most definitely not at your best. And you're certainly going to be in no position to carefully read a nuanced legal document prepared by your employer’s attorney designed to constrain your future behavior. They may say, “Take all the time you want,” or they may imply they can't give you your last paycheck until you sign it. The Department of Labor would like a word with them if that's the case because that's not legal. 

Thank them, leave with your head held high and bask for a moment in the freeing sense of no longer having any obligation to your now ex-employer. All the projects you had in flight, let them go. All the things you needed to tackle; the office politics: not your problem anymore. You're free. Now, in the next day or two, have an employment attorney read through that agreement and give their advice. 

Usually, there's a payment of some varying amount of money, and in return for that payment, you agree to a few things. You'll waive them of civil liability for any claims you may have, and in many cases, you're going to be asked to agree to a non-disparagement clause. That means that you agree not to say anything critical or disparaging about your previous employer. There's also a separate NDA as a part of these things that is going to be preventing you from disclosing the existence of that agreement. This incidentally is why I have nothing disparaging or critical to say about a number of my previous employers. In addition to crapping on your old job being a generally terrible look, it might lead to having to return a small to mid-sized check. Now, let's get to the fun part of the story—by which I mean the technical bits—after this.

This episode is sponsored in part by N2WS. You know what you care about? Many things, but never backups. At least until right after you really, really, really needed to care about backups. That's what N2WS does for your AWS account. It allows you to cycle backups through different storage tiers; you can back things up cost-effectively, and safely. For a limited time, N2WS is offering you $100 in AWS credits for setting up their free trial, and I encourage you to give it a shot. To learn more, visit snark.cloud/n2ws. That's snark.cloud/n2ws. 

So, I have a weird thing that I've noticed when getting fired from companies. My job was invariably always either running the ops team, or on the ops team and, in a couple places, being the entire ops team. That meant that I had full access to basically everything: the full production environment, the AWS account, the secret store, the shared password manager, the domain registration account that the founder used, all of it. So, for companies that aren't practiced at exercising a rigorous termination policy and procedure—which in my case was most of them because I worked in small business—an awful lot of things have to change, and quickly. 

You can't fire someone and then walk around for a couple days with them still having access to your systems. That is terrible policy; not everyone's going to be level headed; and you never know when someone's going to do something ill-advised. Now, as you might imagine, when you're working inside of a small company, and there's an infinite amount of work to do, building out a policy by which you quickly and efficiently lock someone out—who has full access to pretty much everything—isn't the first thing you focus on until one day suddenly it absolutely is because someone just got fired. Now, before I was the greatest cloud influencer in the world, remember, if you will, that I started my career as a grumpy Unix sysadmin. That means that despite all of my modern sensibilities here in 2020, I still had a personal server or two hanging out somewhere that was completely disconnected from everything corporate. That's where I kept all of my stuff that I cared about: my side projects, etc. That server in various forms has saved my bacon more than once because one of the things I always taught it to do, out of habit, when I started a new ops job was to keep an eye on my employer’s public sites. 

There are certain classes of systems or network failure where everything—even the monitoring systems themselves—break. I always viewed that personal system there—with a cron job that would ping things with Nagios or whatnot—as sort of a watchtower of last resort. I never put anything confidential onto it, for obvious reasons. It was strictly looking at public-facing endpoints. There were no credentials. It wasn't logging in. It wasn't a great check. But, “Hey, is the site up?” is more or less what it was looking at. 

There have been two occasions in my career during which time that system caught outages that other things didn't. Because frankly, monitoring is freakin’ terrible, and sometimes that's what it takes is a third party that's not connected to anything. Multiple times during my career, either when being fired in that meeting or shortly after that meeting, that system sent me it's, “Hey, your website's down, genius,” message. Now, I don't have inside info of what happened next because, again, I was fired. But what I strongly suspect happened was that companies were doing the right thing by revoking all of my credentials and all of the shared credentials, but when they did it, they did so unevenly, and would, for example, reset a database password before teaching the application about the new one, and then the whole site goes down. 

I do want to point out for a minute here that I have a functioning sense of ethics. I don't actually know for a fact that my previous employers ever turned off my accounts, or that this is what caused the next day or two of sporadic outages of their public-facing websites in some cases. I'm theorizing wildly here. I mean, what was I going to do, log in with those now dead credentials—or try to log in and check? Yeah, that is an ethical breach. Don't do that. 

So, the trick and the takeaway here is, you should have a plan for what it looks like when someone leaves the company before you need to figure it out in a hurry and probably take the website down. And credit where due: there's a villain in every story, and in this case, it's me. I admit that the lack of a policy and procedure in place to safely rotate credentials when firing someone does, as an ops person, fall squarely on my shoulders. I failed in that aspect of my job, so I guess it's a good thing that I was fired.

This has been the AWS Morning Brief: Whiteboard Confessional. I am Cloud Economist Corey Quinn. And if you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. Whereas if you've hated it, please leave a five-star review on Apple Podcasts and tell me another reason I should have been fired.

Thank you for joining us on Whiteboard Confessional. If you have terrifying ideas, please reach out to me on twitter at @quinnypig and let me know what I should talk about next time.

Announcer: This has been a HumblePod production. Stay humble.