Whiteboard Confessional: Help, I’ve Lost My MFA Device!

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

This episode is sponsored by a personal favorite: Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front end frameworks to figure out or access controls to manage; just ship the tools that will move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas: buttons, checkboxes, tables, etc. Then I can wire all of those things up to queries with all kinds of different parameters, post, get, put, delete, etc. It all connects to virtually every database natively, or you can do what I did and build a whole crap ton of lambda functions, shove them behind some API’s gateway and use that instead. It speaks MySQL, Postgres, Dynamo—not Route 53 in a notable oversight; but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces. And I don't know frontend; that's the most important part here: Retool is transformational for those of us who aren't front end types. It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com/lastweekinaws. That's retool.com/lastweekinaws, and tell them Corey sent you because they are about to be hearing way more from me.

Welcome to the AWS Morning Brief: Whiteboard Confessional. Today I want to talk about infosec. Specifically, an aspect of infosec that I think is not given proper attention, namely two-factor auth. Now, two-factor auth is important to enable but first, back up a second. Use a password manager with strong passwords for all of your stuff. Those are table stakes at this point. 

Now, most password managers will offer to also store your multi-factor auth codes, your OTP tokens, etcetera. I'm not a big fan of that because it feels to me, perhaps incorrectly, like I'm collapsing multiple factors back down into that same factor. Someone gets access to my password manager, worst-case scenario, I’m potentially hosed. That's not great. Now, the password managers will argue that this isn't technically true, yada, yada. I'm old fashioned. I'm grumpy. I'm an old Unix systems administrator that had certain angry loud opinions, so I'm going to keep using separate tools for managing passwords, as well as getting in as a second factor. May I also point out that SMS is terrible as far as a second factor. Don't use it if you possibly can, for reasons that go well beyond the scope of this show: we're not that kind of podcast. 

Now, let's talk about what happens if you, for one reason or another, lose your MFA device, or the app on your phone because this happened to a certain business partner of mine named Mike Julian. Now, Mike wound up getting a new phone, which is great because his was something from the Stone Age presumably some kind of Nokia candy bar phone. I hear someone dropped one of those things once the last time they were in mass sale and accidentally killed the dinosaurs. So, that's the kind of era of phone he was upgrading from to, I think, the iPhone SE, but don't quote me on that. I don't tend to pay attention to his taste in electronics. Personally, I question his taste in business partners, but that's all right; he signed on the dotted line; he stuck with me now. 

So, he inadvertently wound up losing access to all of his old MFA tokens and having to get them re-added in other places. Some systems worked super well for this. It was a matter of, “Oh, I'll just use my backup codes,” which he kept like a good responsible person. It let him in, he would then be able to regenerate backup codes, change over the device and everything was glory. For others, he wasn't so lucky and had to phone in and get a reset after identity verification. So, now he didn't have his multi-factor device, so it would fall back to using SMS because it had his cell phone. And he could not disable that with some environments. So, that becomes an attack vector, if you're able to compromise an SMS number which, surprise, is not that hard if you put some effort into it. 

This, of course, does bring us to Amazon. Mike needed to reset his Amazon MFA token. Now, when I say Amazon, I don't mean AWS. I mean, Amazon, and I'm going to go back and forth as I go down the story a little bit. So, this is an Amazon retail account, not an AWS account. And it turns out when you Google how to reset your Amazon MFA token, all the results are about AWS. 

So, “Okay, that's interesting,” says Mike. He Googles effectively to remove all results from aws.amazon.com. Cool. Now all the results are about things that are not Amazon stuff. Not anything helpful. So, there's no documentation in Google for any of this as applies to Amazon retail, it may as well not exist as a problem. This is less than ideal from Mike's perspective. He was able to reset his AWS multi-factor auth for the AWS account—that's for the same email address tied to that amazon.com account, but AWS and Amazon have completely separate MFA infrastructures. 

So, this is fascinating. He posts on Twitter, which is the number one way to get help when you have an Amazon issue and you run a company devoted to making fun of Amazon, and AWS support chimes in because they're helpful. Someone else says, “I've been trying to solve this problem for 10 years and got nowhere. Good luck, Godspeed.” And it seemed odd because it's an Amazon retail problem. Why is AWS chiming in? And this leads to a phone call. Mike finally winds up getting on a series of phone calls with AWS support. 

Let me handwave past the boring part. More or less, no one knew what the story was here. It goes back and forth, and back and forth. It turns out that if you have an AWS account and an amazon.com account tied to the same email address, there's a little known secret that is kept secret not just from customers, but from support on both sides in many cases, where if you disable MFA for your amazon.com account, the AWS token for your linked AWS account now takes over on the amazon.com account, because everything is terrible, and broken, and awful. 

And it was a living nightmare to sort that out. If someone is listening to this, ideally this may save you if you're still suffering from the Underpants Problem. For those who are not familiar, the Underpants Problem is when your $4 billion startup unicorn has all of its infrastructure running in an AWS account that is still linked to the amazon.com account that your founder uses to buy underpants. That's why it's called the Underpants Problem. This was not, however, the worst problem that Mike encountered while replacing his many, many MFA tokens. But first:

This episode is sponsored in part by N2WS. You know what you care about? Many things, but never backups. At least until right after you really, really, really needed to care about backups. That's what N2WS does for your AWS account. It allows you to cycle backups through different storage tiers; you can back things up cost-effectively, and safely. For a limited time, N2WS is offering you $100 in AWS credits for setting up their free trial, and I encourage you to give it a shot. To learn more, visit snark.cloud/n2ws. That's snark.cloud/n2ws.

I'm not going to name the vendor because I've reached out to them, and had a long conversation with their CSO, and identified a whole bunch of failures here and it's not constructive to name them, so I'm not going to. But the story is instructive. For one of our business service vendors, Mike called up support as you’re want to do. The answer then was, “Oh, okay. What's your name?” Mike gave his name. It's Mike Julian, for those following along at home. “Great. What's your email address?” Was the second question, which, again, for those following along at home is not that hard to wind up figuring out, if you have ever emailed with someone at this company. Spoiler: it might be “Mike at,” similar to the fact that I'm “Corey at,” and the third question was—just kidding. There was no third question. “There you go. I have disabled the MFA portion of your account. Have a nice day.” 

There was also no email that went out to other admins, owners, etcetera, of this account. So, it was a pure just take someone at their word when they call in, and then go ahead and disable that for the asking. You begin to see the problem here, because the vendor in question has a bunch of sensitive data that we kind of have to give them, given the nature of what they do, and it's not something that anyone really has much of a choice in once you pick a vendor. So, it became a difficult situation. 

So, I lashed out on Twitter, as I tend to do, without naming names; then I calmed down a little bit. Pro tip: don't make decisions when you're angry. You almost never make the right one. And then I had a conversation with their CSO because it turns out that when you're loud and obnoxious and have your own podcast, you could get in front of all kinds of folks. 

And I learned a couple of things. First, there was another form of authentication. Namely, they were able to figure out that the phone number that Mike was calling in from was in fact associated with the account and the way that they wind up doing phone number identification, it's not impossible, but it's not trivial to spoof it in that respect and, frankly, the answer then became that someone in their support center did not follow the process and procedure that they were supposed to. I validated this was in fact, a teachable moment and not, I'm going to get someone fired by mistake moment, and they had the right answers, and I genuinely believe that they were sincere and correct in this. 

But the lesson that we take away from this, it goes well beyond MFA, and goes well beyond security, and goes into a human problem, for which there is as of yet, no patch. And that simply is, is that no matter what you do for policies and procedures, and what you build, and how your security flow works, if someone can choose to disregard it and just bypass all of the authentication and the checks, you need to have a plan for that. Expect people to inherently do the wrong thing. Make it hard for people to do the wrong thing; make it easy for them to do the right thing and follow up on these things. Credit where due, they were able to pull the recording in question and validate exactly what happened and why. 

Note also that when an angry customer calls up, “I'm locked out of my account. And I hate having to call in.” There's a human-nature piece where you want to make that person happy. Because, sure, people might go on the internet and tell lies, but call you up on the phone? Who would do such a thing? That's not reasonable. That's not something that anyone would do here on my society. So, understand that there are a lot of challenges here that need to be worked through. There are a lot of process flows. And training and drilling people on procedures is important. And no, firing people who get it wrong is never a valid answer, because that doesn't go super-well. It winds up incentivizing people to hide mistakes. Don't do that. This has been sort of a letting down of my originally promised zero-day disclosure on this. This is why, incidentally, it's best not to make tweets or podcasts while angry.

This has been the AWS Morning Brief: Whiteboard Confessional. I am Cloud Economist Corey Quinn fixing AWS bills here or wherever I happen to find them. And if you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. Whereas if you've hated this podcast, please leave a five-star review on Apple Podcasts anyway, and tell me what I should be using for an MFA provider instead.

Thank you for joining us on Whiteboard Confessional. If you have terrifying ideas, please reach out to me on twitter at @quinnypig and let me know what I should talk about next time.

Announcer: This has been a HumblePod production. Stay humble.