Ask Me Anything with Corey Quinn

• lastweekinaws.com/disclosures: https://lastweekinaws.com/disclosures
• duckbillgroup.com: https://duckbillgroup.com

Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: As businesses consider automation to help build and manage their hybrid cloud infrastructures, deployment speed is important, but so is cost. Red Hat Ansible Automation Platform is available in the AWS Marketplace to help you meet your cloud spend commitments while delivering best-of-both-worlds support.

Corey: Well, all right. Thank you all for coming. Let’s begin and see how this whole thing shakes out, which is fun and exciting, and for some godforsaken reason the lights like to turn off, so we’re going to see if that continues. I’ve been doing Screaming in the Cloud for about, give or take, 500 episodes now, which is more than a little bit ridiculous. And I figured it would be a nice change of pace if I could, instead of reaching out and talking to folks who are innovative leaders in the space and whatnot, if I could instead interview my own favorite guest: myself.

Because the entire point is, I’m usually the one sitting here asking questions, so I’m instead going to now gather questions from you folks—and feel free to drop some of them into the comments—but I’ve solicited a bunch of them, I’m going to work through them and see what you folks want to know about me. I generally try to be fairly transparent, but let’s have fun with it. To be clear, if this is your first exposure to my Screaming in the Cloud podcast show, it’s generally an interview show talking with people involved with the business of cloud. It’s not intended to be snarky because not everyone enjoys thinking on their feet quite like that, but rather a conversation of people about what they’re passionate about. I’m passionate about the sound of my own voice. That’s the theme of this entire episode.

So, there are a few that have come through that are in no particular order. I’m going to wind up powering through them, and again, throw some into the comments if you want to have other ones added. If you’re listening to this in the usual Screaming in the Cloud place, well, send me questions and I am thrilled to wind up passing out more of them. The first one—a great one to start—comes with someone asked me a question about the video feed. “What’s with the Minecraft pickaxe on the wall?” It’s made out of foam.

One of my favorite stories, and despite having a bunch of stuff on my wall that is interesting and is stuff that I’ve created, years ago, I wrote a blog post talking about how machine learning is effectively selling digital pickaxes into a gold rush. Because the cloud companies pushing it are all selling things such as, you know, they’re taking expensive compute, large amounts of storage, and charging by the hour for it. And in response, Amanda, who runs machine learning analyst relations at AWS, sent me that by way of retaliation. And it remains one of my absolute favorite gifts. It’s, where’s all this creativity in the machine-learning marketing? No, instead it’s, “We built a robot that can think. But what are we going to do with it now? Microsoft Excel.” Come up with some of that creativity, that energy, and put it into the marketing side of the world.

Okay, someone else asks—Brooke asks, “What do I think is people’s biggest misconception about me?” That’s a good one. I think part of it has been my misconception for a long time about what the audience is. When I started doing this, the only people who ever wound up asking me anything or talking to me about anything on social media already knew who I was, so I didn’t feel the need to explain who I am and what I do. So, people sometimes only see the witty banter on Twitter and whatnot and think that I’m just here to make fun of things.

They don’t notice, for example, that my jokes are never calling out individual people, unless they’re basically a US senator, and they’re not there to make individual humans feel bad about collectively poor corporate decision-making. I would say across the board, people think that I’m trying to be meaner than I am. I’m going to be honest and say it’s a little bit insulting, just from the perspective of, if I really had an axe to grind against people who work at Amazon, for example, is this the best I’d be able to do? I’d like to think that I could at least smack a little bit harder. Speaking of, we do have a question that people sent in in advance.

“When was the last time that Mike Julian gave me that look?” Easy. It would have been two days ago because we were both in the same room up in Seattle. I made a ridiculous pun, and he just stared at me. I don’t remember what the pun is, but I am an incorrigible punster and as a result, Mike has learned that whatever he does when I make a pun, he cannot incorrige me. Buh-dum-tss. That’s right. They’re no longer puns, they’re dad jokes. A pun becomes a dad joke once the punch line becomes a parent. Yes.

Okay, the next one is what is my favorite AWS joke? The easy answer is something cynical and ridiculous, but that’s just punching down at various service teams; it’s not my goal. My personal favorite is the genie joke where a guy rubs a lamp, Genie comes out and says, “You can have a billion dollars if you can spend $100 million in a month, and you’re not allowed to waste it or give it away.” And the person says, “Okay”—like, “Those are the rules.” Like, “Okay. Can I use AWS?” And the genie says, “Well, okay, there’s one more rule.” I think that’s kind of fun.

Let’s see, another one. A hardball question: given the emphasis on right-sizing for meager cost savings and the amount of engineering work required to make real architectural changes to get costs down, how do you approach cost controls in companies largely running other people’s software? There are not as many companies as you might think where dialing in the specifics of a given application across the board is going to result in meaningful savings. Yes, yes, you’re running something in hyperscale, it makes an awful lot of sense, but most workloads don’t do that. The mistakes you most often see are misconfigurations for not knowing this arcane bit of AWS trivia, as a good example. There are often things you can do with relatively small amounts of effort. Beyond a certain point, things are going to cost what they’re going to cost without a massive rearchitecture and I don’t advise people do that because no one is going to be happy rearchitecting just for cost reasons. Doesn’t go well.

Someone asks, “I’m quite critical of AWS, which does build trust with the audience. Has AWS tried to get you to market some of their services, and would I be open to do that?” That’s a great question. Yes, sometimes they do. You can tell this because they wind up buying ads in the newsletter or the podcast and they’re all disclaimed as a sponsored piece of content.

I do have an analyst arrangement with a couple of different cloud companies, as mentioned lastweekinaws.com/disclosures, and the reason behind that is because you can buy my attention to look at your product and talk to you in-depth about it, but you cannot buy my opinion on it. And those engagements are always tied to, let’s talk about what the public is seeing about this. Now, sometimes I write about the things that I’m talking about because that’s where my mind goes, but it’s not about okay, now go and talk about this because we’re paying you to, and don’t disclose that you have a financial relationship.

No, that is called fraud. I figure I can sell you as an audience out exactly once, so I better be able to charge enough money to never have to work again. Like, when you see me suddenly talk about multi-cloud being great and I became a VP at IBM, about three to six months after that, no one will ever hear from me again because I love nesting doll yacht money. It’ll be great.

Let’s see. The next one I have on my prepared list here is, “Tell me about a time I got AWS to create a pie chart.” I wish I’d see less of it. Every once in a while I’ll talk to a team and they’re like, “Well, we’ve prepared a PowerPoint deck to show you what we’re talking about.” No, Amazon is famously not a PowerPoint company and I don’t know why people feel the need to repeatedly prove that point to me because slides are not always the best way to convey complex information.

I prefer to read documents and then have a conversation about them as Amazon tends to do. The visual approach and the bullet lists and all the rest are just frustrating. If I’m going to do a pie chart, it’s going to be in service of a joke. It’s not going to be anything that is the best way to convey information in almost any sense.

“How many internal documents do I think reference me by name at AWS,” is another one. And I don’t know the answer to documents, but someone sent me a screenshot once of searching for my name in their Slack internal nonsense thing, and it was about 10,000 messages referenced me that it found. I don’t know what they were saying. I have to assume, on some level, just something that does a belt feed from my Twitter account where it lists my name or something. But I choose to believe that no, they actually are talking about me to that level of… of extreme.

Let’s see, let’s turn back to the chat for a sec because otherwise it just sounds like I’m doing all prepared stuff. And I’m thrilled to do that, but I’m also thrilled to wind up fielding questions from folks who are playing along on these things. “I love your talk, ‘Heresy in the Church of Docker.’ Do I have any more speaking gigs planned?” Well, today’s Wednesday, and this Friday, I have a talk that’s going out at the CDK Community Day.

I also have a couple of things coming up that are internal corporate presentations at various places. But at the moment, no. I suspect I’ll be giving a talk if they accept it at SCALE in Pasadena in March of next year, but at the moment, I’m mostly focused on re:Invent, just because that is eight short weeks away and I more or less destroy the second half of my year because… well, holidays are for other people. We’re going to talk about clouds, as Amazon and the rest of us dance to the tune that they play.

“Look in my crystal ball; what will the industry look like in 5, 10, or 20 years?” Which is a fun one. You shouldn’t listen to me on this. At all. I was the person telling you that virtualization was a flash in the pan, that cloud was never going to catch on, that Kubernetes and containers had a bunch of problems that were unlikely to be solved, and I’m actually kind of enthused about serverless which probably means it’s going to flop.

I am bad at predicting overall trends, but I have no problem admitting that wow, I was completely wrong on that point, which apparently is a rarer skill than it should be. I don’t know what the future the industry holds. I know that we’re seeing some AI value shaping up. I think that there’s going to be a bit of a downturn in that sector once people realize that just calling something AI doesn’t mean you make wild VC piles of money anymore. But there will be use cases that filter out of it. I don’t know what they’re going to look like yet, but I’m excited to see it.

Okay, “Have any of the AWS services increased costs in the last year? I was having a hard time finding historical pricing charts for services.” There have been repricing stories. There have been SMS charges in India that have—and pinpointed a few other things—that wound up increasing because of a government tariff on them and that cost was passed on. Next February, they’re going to be charging for public IPV4 addresses.

But those tend to be the exceptions. The way that most costs tend increase have been either, it becomes far cheaper for AWS to provide a service and they don’t cut the cost—data transfer being a good example—they’ll also often have stories in that they’re going to start launching a bunch of new things, and you’ll notice that AWS bills tend to grow in time. Part of that growth, part of that is just cruft because people don’t go back and clean things up. But by and large, I have not seen, “This thing that used to cost you $1 is now going to cost you $2.” That’s not how AWS does pricing. Thankfully. Everyone’s always been scared of something like that happening. I think that when we start seeing actual increases like that, that’s when it’s time to start taking a long, hard look at the way that the industry is shaping up. I don’t think we’re there yet.

Okay. “Any plans for a Last Week in Azure or a Last Week in GCP?” Good question. If so, I won’t be the person writing it. I don’t think that it’s reasonable to expect someone to keep up with multiple large companies and their releases. I’d also say that Azure and GCP don’t release updates to services with the relentless cadence that AWS does.

The reason I built the thing to start with is simply because it was difficult to gather all the information in one place, at least the stuff that I cared about with an economic impact, and by the time I’d done that, it was, well, this is 80% of the way toward republishing it for other people. I expected someone was going to point me at a thing so I didn’t have to do it, and instead, everyone signed up. I don’t see the need for it. I hope that in those spaces, they’re better at telling their own story to the point where the only reason someone would care about a newsletter would be just my sarcasm tied into whatever was released. But that’s not something that I’m paying as much attention to, just because my customers are on AWS, my stuff is largely built on AWS, it’s what I have to care about.

Let’s see here. “What do I look forward to at re:Invent?” Not being at re:Invent anymore. I’m there for eight nights a year. That is shitty cloud Chanukah come to life for me. I’m there to set things up in advance, I’m there to tear things down at the end, and I’m trying to have way too many meetings in the middle of all of that. I am useless for the rest of the year after re:Invent, so I just basically go home and breathe into a bag forever.

I had a revelation last year about re:Play, which is that I don’t have to go to it if I don’t want to go. And I don’t like the cold, the repetitive music, the giant crowds. I want to go read a book in a bathtub and call it a night, and that’s what I hope to do. In practice, I’ll probably go grab dinner with other people who feel the same way. I also love the Drink Up I do there every year over at Atomic Liquors. I believe this year, we’re partnering with the folks over at RedMonk because a lot of the people we want to talk to are in the same groups.

It’s just a fun event: show up, let us buy you drinks. There’s no badge scan or any nonsense like that. We just want to talk to people who care to come out and visit. I love doing that. It’s probably my favorite part of re:Invent other than not being at re:Invent. It’s going to be on November 29th this year. If you’re listening to this, please come on by if you’re unfortunate enough to be in Las Vegas.

Someone else had a good question I want to talk about here. “I’m a TAM for AWS. Cost optimization is one of our functions. What do you wish we would do better after all the easy button things such as picking the right instance and family, savings plans RIs, turning off or delete orphan resources, watching out for inefficient data transfer patterns, et cetera?” I’m going to back up and say that you’re begging the question here, in that you aren’t doing the easy things, at least not at scale, not globally.

I used to think that all of my customer engagements would be, okay after the easy stuff, what’s next? I love those projects, but in so many cases, I show up and those easy things have not been done. “Well, that just means that your customers haven’t been asking their TAM.” Every customer I’ve had has asked their TAM first. “Should we ask the free expert or the one that charges us a large but reasonable fixed fee? Let’s try the free thing first.”

The quality of that advice is uneven. I wish that there were at least a solid baseline. I would love to get to a point where I can assume that I can go ahead and be able to just say, “Okay, you’ve clearly got your RI stuff, you’re right-sizing, you’re deleting stuff you’re not using, taken care of. Now, let’s look at the serious architecture stuff.” It’s just rare that I get to see it.

“What tool, feature, or widget do I wish AWS would build into the budget console?” I want to be able to set a dollar figure, maybe it’s zero, maybe it’s $20, maybe it is irrelevant, but above whatever I set, the account will not charge me above that figure, period. If that means they have to turn things off if that means they had to delete portions of data, great. But I want that assurance because even now when I kick the tires in a new service, I get worried that I’m going to wind up with a surprise bill because I didn’t understand some very subtle interplay of the dynamics. And if I’m worried about that, everyone else is going to wind up getting caught by that stuff, too.

I want the freedom to experiment and if it smacks into a wall, okay, cool. That’s $20. That was worth learning that. Whatever. I want the ability to not be charged unreasonable overages. And I’m not worried about it turning from 20 into 40. I’m worried about it turning from 20 into 300,000. Like, there’s the, “Oh, that’s going to have a dent on the quarterlies,” style of [numb 00:16:01]—

All right. Someone also asked, “What is the one thing that AWS could do that I believe would reduce costs for both AWS and their customers. And no, canceling re:Invent doesn’t count.” I don’t think about it in that way because believe it or not, most of my customers don’t come to me asking to reduce their bill. They think they do at the start, but what they’re trying to do is understand it. They’re trying to predict it.

Yes, they want to turn off the waste in the rest, but by and large, there are very few AWS offerings that you take a look at and realize what you’re getting for it and say, “Nah, that’s too expensive.” It can be expensive for certain use cases, but the dangerous part is when the costs are unpredictable. Like, “What’s it going to cost me to run this big application in my data center?” The answer is usually, “Well, run it for a month, and then we’ll know.” But that’s an expensive and dangerous way to go about finding things out.

I think that customers don’t care about reducing costs as much as they think; they care about controlling them, predicting them, and understanding them. So, how would they make things less expensive? I don’t know. I suspect that data transfer if they were to reduce that at least cross-AZ or eliminate it ideally, you’d start seeing a lot more compute usage in multiple AZs. I’ve had multiple clients who are not spinning things up in multi-AZ, specifically because they’ll take the reliability trade-off over the extreme cost of all the replication flowing back and forth. Aside from that, they mostly get a lot of the value right in how they price things, which I don’t think people have heard me say before, but it is true.

Someone asked a question here of, “Any major trends that I’m seeing in EDP/PPA negotiations?” Yeah, lately, in particular. Used to be that you would have a Marketplace as the fallback, where it used to be that 50 cents of every dollar you spent on Marketplace would count. Now, it’s a hundred percent up to a quarter of your commit. Great.

But when you have a long-term commitment deal with Amazon, now they’re starting to push for all—put all your other vendors onto the AWS Marketplace so you can have a bigger commit and thus a bigger discount, which incidentally, the discount does not apply to Marketplace spend. A lot of folks are uncomfortable with having Amazon as the middleman between all of their vendor relationships. And a lot of the vendors aren’t super thrilled with having to pay percentages of existing customer relationships to Amazon for what they perceive to be remarkably little value. That’s the current one.

I’m not seeing generative AI play a significant stake in this yet. People are still experimenting with it. I’m not seeing, “Well, we’re spending $100 million a year, but make that 150 because of generative AI.” It’s expensive to play with gen-AI stuff, but it’s not driving the business spend yet. But that’s the big trend that I’m seeing over the past, eh, I would say, few months.

“Do I use AWS for personal projects?” The first problem there is, well, what’s a personal project versus a work thing? My life is starting to flow in a bunch of weird different ways. The answer is yes. Most of the stuff that I build for funsies is on top of AWS, though there are exceptions. “Should I?” Is the follow-up question and the answer to that is, “It depends.”

The person is worrying about cost overruns. So, am I. I tend to not be a big fan of uncontrolled downside risk when something winds up getting exposed. I think that there are going to be a lot of caveats there. I know what I’m doing and I also have the backstop, in my case, of, I figure I can have a big billing screw-up or I have to bend the knee and apologize and beg for a concession from AWS, once.

It’ll probably be on a billboard or something one of these days. Lord knows I have it coming to me. That’s something I can use as a get-out-of-jail-free card. Most people can’t make that guarantee, and so I would take—if—depending on the environment that you know and what you want to build, there are a lot of other options: buying a fixed-fee VPS somewhere if that’s how you tend to think about things might very well be a cost-effective for you, depending on what you’re building. There’s no straight answer to this.

“Do I think Azure will lose any market share with recent cybersecurity kerfuffles specific to Office 365 and nation-state actors?” No, I don’t. And the reason behind that is that a lot of Azure spend is not necessarily Azure usage; it’s being rolled into enterprise agreements customers negotiate as part of their on-premises stuff, their operating system licenses, their Office licensing, and the rest. The business world is not going to stop using Excel and Word and PowerPoint and Outlook. They’re not going to stop putting Windows on desktop stuff. And largely, customers don’t care about security.

They say they do, they often believe that they do, but I see where the bills are. I see what people spend on feature development, I see what they spend on core infrastructure, and I see what they spend on security services. And I have conversations about budgeting with what are you doing with a lot of these things? The companies generally don’t care about this until right after they really should have cared. And maybe that’s a rational effect.

I mean, take a look at most breaches. And a year later, their stock price is larger than it was when they dispose the breach. Sure, maybe they’re burning through their ablated CISO, but the business itself tends to succeed. I wish that there were bigger consequences for this. I have talked to folks who will not put specific workloads on Azure as a result of this. “Will you talk about that publicly?” “No, because who can afford to upset Microsoft?”

I used to have guests from Microsoft on my show regularly. They don’t talk to me and haven’t for a couple of years. Scott Guthrie, the head of Azure, has been on this show. The problem I have is that once you start criticizing their security posture, they go quiet. They clearly don’t like me.

But their options are basically to either ice me out or play around with my seven seats for Office licensing, which, okay, whatever. They don’t have a stick to hit me with, in the way that they do most companies. And whether that’s true or not that they’re going to lash out like that, companies don’t want to take the risk of calling Microsoft out in public. Too big to be criticized as sort of how that works.

Let’s see, someone else asks, “How can a startup get the most out of its startup status with AWS?” You’re not going to get what you think you want from AWS in this context. “Oh, we’re going to be a featured partner so they market us.” I’ve yet to hear a story about how being featured by AWS for something has dramatically changed the fortunes of a startup. Usually, they’ll do that when there’s either a big social mission and you never hear about the company again, or they’re a darling of the industry that’s taking the world by fire and they’re already [at 00:22:24] upward swing and AWS wants to hang out with those successful people in public and be seen to do so.

The actual way that startup stuff is going to manifest itself well for you from AWS is largely in the form of credits as you go through Activate or one of their other programs. But be careful. Treat them like actual money, not this free thing you don’t have to worry about. One day they expire or run out and suddenly you’re going from having no dollars going to AWS to ten grand a month and people aren’t prepared for that. It’s, “Wait. So you mean this costs money? Oh, my God.”

You have to approach it with a sense of discipline. But yeah, once you—if you can do that, yeah, free money and a free cloud bill for a few years? That’s not nothing. I also would question the idea of being able to ask a giant company that’s worth a trillion-and-a-half dollars and advice for how to be a startup. I find that one’s always a little on the humorous side myself.

“What do I think is the most underrated service or feature release from 2023? Full disclosures, this means I’ll make some content about it,” says Brooke over at AWS. Oh, that’s a good question. I’m trying to remember when various things have come out and it all tends to run together. I think that people are criticizing AWS for charging for IPV4 an awful lot, and I think that that is a terrific change, just because I’ve seen how wasteful companies are with public IP addresses, which are basically an exhausted or rapidly exhausting resource.

And they just—you spend tens or hundreds of thousands of these things and don’t use reason to think about that. It’ll be one of the best things that we’ve seen for IPV6 adoption once AWS figures out how to make that work. And I would say that there’s a lot to be said for since, you know, IPV4 is exhausted already, now we’re talking about can we get them on the secondary markets, you need a reasonable IP plan to get some of those. And… “Well, we just give them the customers and they throw them away.” I want AWS to continue to be able to get those for the stuff that the rest of us are working on, not because one big company uses a million of them, just because, “Oh, what do you mean private IP addresses? What might those be?” That's part of it.

I would say that there’s also been… thinking back on this, it’s unsung, the compute optimizer is doing a lot better at recommending things than it used to be. It was originally just giving crap advice, and over time, it started giving advice that’s actually solid and backs up what I’ve seen. It’s not perfect, and I keep forgetting it’s there because, for some godforsaken reason, it’s its own standalone service, rather than living in the billing console where it belongs. But no one’s excited about a service like that to the point where they talk about or create content about it, but it’s good, and it’s getting better all the time. That’s probably a good one. They recently announced the ability for it to do GPU instances which, okay great, for people who care about that, awesome, but it’s not exciting. Even I don’t think I paid much attention to it in the newsletter.

Okay, “Does it make economic sense to bring your own IP addresses to AWS instead of paying their fees?” Bring your own IP, if you bring your own allocation to AWS, costs you nothing in terms of AWS costs. You take a look at the market rate per IP address versus what AWS costs, you’ll hit break even within your first year if you do it. So yeah, it makes perfect economic sense to do it if you have the allocation and if you have the resourcing, as well as the ability to throw people at the problem to do the migration. It can be a little hairy if you’re not careful. But the economics, the benefit is clear on that once you account for those variables.

Let’s see here. We’ve also got tagging. “Everyone nods their heads that they know it’s the key to controlling things, but how effective are people at actually tagging, especially when new to cloud?” They’re terrible at it. They’re never going to tag things appropriately. Automation is the way to do it because otherwise, you’re going to spend the rest of your life chasing developers and asking them to tag things appropriately, and then they won’t, and then they’ll feel bad about it. No one enjoys that conversation.

So, having derived tags and the rest, or failing that, having some deployment gate as early in the process as possible of, “Oh, what’s the tag for this?” Is the only way you’re going to start to see coverage on this. And ideally, someday you’ll go back and tag a bunch of pre-existing stuff. But it’s honestly the thing that everyone hates the most on this. I have never seen a company that says, “We are thrilled with our with our tag coverage. We’re nailing it.” The only time you see that is pure greenfield, everything done without ClickOps, and those environments are vanishingly rare.

“Outside a telecom are customers using local zones more, or at all?” Very, very limited as far as what their usage looks like on that. Because that’s… it doesn’t buy you as much as you’d think for most workloads. The real benefit is a little more expensive, but it’s also in specific cities where there are not AWS regions, and at least in the United States where the majority of my clients are, there is not meaningful latency differences, for example, from in Los Angeles versus up to Oregon, since no one should be using the Northern California region because it’s really expensive. It’s a 20-millisecond round trip, which in most cases, for most workloads, is fine.

Gaming companies are big exception to this. Getting anything they can as close to the customer as possible is their entire goal, which very often means they don’t even go with some of the cloud providers in some places. That’s one of those actual multi-cloud workloads that you want to be able to run anywhere that you can get a baseline computer up to run a container or a golden image or something. That is the usual case. The rest are, for local zones, is largely going to be driven by specific one-off weird things. Good question.

Let’s see, “Is S3 intelligent tiering good enough or is it worth trying to do it yourself?” Your default choice for almost everything should be intelligent tiering in 2023. It winds up costing you more only in very specific circumstances that are unlikely to be anything other than a corner case for what you’re doing. And the exceptions to this are, large workloads that are running a lot of S3 stuff where the lifecycle is very well understood, environments where you’re not going to be storing your data for more than 30 days in any case and you can do a lifecycle policy around it. Other than those use cases, yeah, the monitoring fee is not significant in any environment I’ve ever seen.

And people view—touch their data a lot less than they believe. So okay, there’s a monitoring fee for object, yes, but it also cuts your raw storage cost in half for things that aren’t frequently touched. So, you know, think about it. Run your own numbers and also be aware that first month as it transitions in, you’re going to see massive transition charges per object, but wants it’s an intelligent tiering, there’s no further transition charges, which is nice.

Let’s see here. “We’re all-in on serverless”—oh good, someone drank the Kool-Aid, too—“And for our use cases, it works great. Do I find other customers moving to it and succeeding?” Yeah, I do when they’re moving to it because for certain workloads, it makes an awful lot of sense. For others, it requires a complete reimagining of whatever it is that you’re doing.

The early successes were just doing these periodic jobs. Now, we’re seeing full applications built on top of event-driven architectures, which is really neat to see. But trying to retrofit something that was never built with that in mind can be more trouble than it’s worth. And there are corner cases where building something on serverless would cost significantly more than building it in a server-ful way. But its time has come for an awful lot of stuff. Now, what I don’t subscribe to is this belief that oh, if you’re not building something serverless you’re doing it totally wrong. No, that is not true. That has never been true.

Let’s see what else have we got here? Oh, “Following up on local zones, how about Outposts? Do I see much adoption? What’s the primary use case or cases?” My customers inherently are coming to me because of a large AWS bill. If they’re running Outposts, it is extremely unlikely that they are putting significant portions of their spend through the Outpost. It tends to be something of a rounding error, which means I don’t spend a lot of time focusing on it.

They obviously have some existing data center workloads and data center facilities where they’re going to take an AWS-provided rack and slap it in there, but it’s not going to be in the top 10 or even top 20 list of service spend in almost every case as a result, so it doesn’t come up. One of the big secrets of how we approach things is we start with a big number first and then work our way down instead of going alphabetically. So yes, I’ve seen customers using them and the customers I’ve talked to at re:Invent who are using them are very happy with them for the use cases, but it’s not a common approach. I’m not a huge fan of the rest.

“Someone said the Basecamp saved a million-and-a-half a year by leaving AWS. I know you say repatriation isn’t a thing people are doing, but has my view changed at all since you’ve published that blog post?” No, because everyone’s asking me about Basecamp and it’s repatriation, and that’s the only use case that they’ve got for this. Let’s further point out that a million-and-a-half a year is not as many engineers as you might think it is when you wind up tying that all together. And now those engineers are spending time running that environment.

Does it make sense for them? Probably. I don’t know their specific context. I know that a million-and-a-half dollars a year to—even if they had to spend that for the marketing coverage that they’re getting as a result of this, makes perfect sense. But cloud has never been about raw cost savings. It’s about feature velocity.

If you have a data center and you move it to the cloud, you’re not going to recoup that investment for at least five years. Migrations are inherently expensive. It does not create the benefits that people often believe that they do. That becomes a painful problem for folks. I would say that there’s a lot more noise than there are real-world stories [hanging 00:31:57] out about these things.

Now, I do occasionally see a specific workload that is moved back to a data center for a variety of reasons—occasionally cost but not always—and I see proof-of-concept projects that they don’t pursue and then turn off. Some people like to call that a repatriation. No, I call it as, “We tried and it didn’t do what we wanted it to do so we didn’t proceed.” Like, if you try that with any other project, no one says, “Oh, you’re migrating off of it.” No, you’re not. You tested it, it didn’t do what it needed to do. I do see net-new workloads going into data centers, but that’s not the same thing.

Let’s see. “Are the talks at re:Invent worth it anymore? I went to a lot of the early re:Invents and haven’t and about five years. I found back then that even the level 400 talks left a lot to be desired.” Okay. I’m not a fan of attending conference talks most of the time, just because there’s so many things I need to do at all of these events that I would rather spend the time building relationships and having conversations.

The talks are going to be on YouTube a week later, so I would rather get to know the people building the service so I can ask them how to inappropriately use it as a database six months later than asking questions about the talk. Conference-ware is often the thing. Re:Invent always tends to have an AWS employee on stage as well. And I’m not saying that makes these talks less authentic, but they’re also not going to get through slide review of, “Well, we tried to build this onto this AWS service and it was a terrible experience. Let’s tell you about that as a war story.” Yeah, they’re going to shoot that down instantly even though failure stories are so compelling, about here’s what didn’t work for us and how we got there. It’s the lessons learned type of thing.

Whenever you have as much control as re:Invent exhibits over its speakers, you know that a lot of those anecdotes are going to be significantly watered down. This is not to impugn any of the speakers themselves; this is the corporate mind continuing to grow to a point where risk mitigation and downside protection becomes the primary driving goal.

Let’s pull up another one from the prepared list here. “My most annoying, overpriced, or unnecessary charge service in AWS.” AWS Config. It’s a tax on using the cloud as the cloud. When you have a high config bill, it’s because it charges you every time you change the configuration of something you have out there. It means you’re spinning up and spinning down EC2 instances, whereas you’re going to have a super low config bill if you, you know, treat it like a big dumb data center.

It’s a tax on accepting the promises under which cloud has been sold. And it’s necessary for a number of other things like Security Hub. Control Towers magic-deploys it everywhere and makes it annoying to turn off. And I think that that is a pure rent-seeking charge because people aren’t incurring config charges if they’re not already using a lot of AWS things. Not every service needs to make money in a vacuum. It’s, “Well, we don’t charge anything for this because our users are going to spend an awful lot of money on storing things in S3 to use our service.” Great. That’s a good thing. You don’t have to pile charge upon charge upon charge upon charge. It drives me a little bit nuts.

Let’s see what else we have here as far as questions go. “Which AWS service delights me the most?” Eesh, depends on the week. S3 has always been a great service just because it winds up turning big storage that usually—used to require a lot of maintenance and care into something I don’t think about very much. It’s getting smarter and smarter all the time. The biggest lie is the ‘Simple’ in its name: ‘Simple Storage Service.’ At this point, if that’s simple, I really don’t want to know what you think complex would look like.

“By following me on Twitter, someone gets a lot of value from things I mention offhandedly as things everybody just knows. For example, which services are quasi-deprecated or outdated, or what common practices are anti-patterns? Is there a way to learn this kind of thing all in one go, as in a website or a book that reduces AWS to these are the handful of services everybody actually uses, and these are the most commonly sensible ways to do it?” I wish. The problem is that a lot of the stuff that everyone knows, no, it’s stuff that at most, maybe half of the people who are engaging with it knew.

They find out by hearing from other people the way that you do or by trying something and failing and realizing, ohh, this doesn’t work the way that I want it to. It’s one of the more insidious forms of cloud lock-in. You know how a service works, how a service breaks, what the constraints are around when it starts and it stops. And that becomes something that’s a hell of a lot scarier when you have to realize, I’m going to pick a new provider instead and relearn all of those things. The reason I build things on AWS these days is honestly because I know the ways it sucks. I know the painful sharp edges. I don’t have to guess where they might be hiding. I’m not saying that these sharp edges aren’t painful, but when you know they’re there in advance, you can do an awful lot to guard against that.

“Do I believe the big two—AWS and Azure—cloud providers have agreed between themselves not to launch any price wars as they already have an effective monopoly between them and [no one 00:36:46] win in a price war?” I don’t know if there’s ever necessarily an explicit agreement on that, but business people aren’t foolish. Okay, if we’re going to cut our cost of service, instantly, to undercut a competitor, every serious competitor is going to do the same thing. The only reason to do that is if you believe your margins are so wildly superior to your competitors that you can drive them under by doing that or if you have the ability to subsidize your losses longer than they can remain a going concern. Microsoft and Amazon are—and Google—are not in a position where, all right, we’re going to drive them under.

They can both subsidize losses basically forever on a lot of these things and they realize it’s a game you don’t win in, I suspect. The real pricing pressure on that stuff seems to come from customers, when all right, I know it’s big and expensive upfront to buy a SAN, but when that starts costing me less than S3 on a per-petabyte basis, that’s when you start to see a lot of pricing changing in the market. The one thing I haven’t seen that take effect on is data transfer. You could be forgiven for believing that data transfer still cost as much as it did in the 1990s. It does not.

“Is AWS as far behind in AI as they appear?” I think a lot of folks are in the big company space. And they’re all stammering going, “We’ve been doing this for 20 years.” Great, then why are all of your generative AI services, A, bad? B, why is Alexa so terrible? C, why is it so clear that everything you have pre-announced and not brought to market was very clearly not envisioned as a product to be going to market this year until 300 days ago, when Chat-Gippity burst onto the scene and OpenAI [stole a march 00:38:25] on everyone?

Companies are sprinting to position themselves as leaders in the AI space, despite the fact that they’ve gotten lapped by basically a small startup that’s seven years old. Everyone is trying to work the word AI into things, but it always feels contrived to me. Frankly, it tells me that I need to just start tuning the space out for a year until things settle down and people stop describing metric math or anomaly detection is AI. Stop it. So yeah, I’d say if anything, they’re worse than they appear as far as from behind goes.

“I mostly focus on AWS. Will I ever cover Azure?” There are certain things that would cause me to do that, but that’s because I don’t want to be the last Perl consultancy is the entire world has moved off to Python. And effectively, my focus on AWS is because that’s where the painful problems I know how to fix live. But that’s not a suicide pact. I’m not going to ride that down in flames.

But I can retool for a different cloud provider—if that’s what the industry starts doing—far faster than AWS can go from its current market-leading status to irrelevance. There are certain triggers that would cause me to do that, but at the time, I don’t see them in the near term and I don’t have any plans to begin covering other things. As mentioned, people want me to talk about the things I’m good at not the thing that makes me completely nonsensical.

“Which AWS services look like a good idea, but pricing-wise, they’re going to kill you once you have any scale, especially the ones that look okay pricing-wise but aren’t really and it’s hard to know going in?” CloudTrail data events, S3 Bucket Access logging any of the logging services really, Managed NAT Gateways in a bunch of cases. There’s a lot that starts to get really expensive once you hit certain points of scale with a corollary that everyone thinks that everything they’re building is going to scale globally and that’s not true. I don’t build things as a general rule with the idea that I’m going to get ten million users on it tomorrow because by the time I get from nothing to substantial workloads, I’m going to have multiple refactors of what I’ve done. I want to get things out the door as fast as possible and if that means that later in time, oh, I accidentally built Pinterest. What am I going to do? Well, okay, yeah, I’m going to need to rebuild a whole bunch of stuff, but I’ll have the user traffic and mindshare and market share to finance that growth.

Early optimization on stuff like this causes a lot more problems than it solves. “Best practices and anti-patterns in managing AWS costs. For context, you once told me about a role that I had taken that you’d seen lots of companies tried to create that role and then said that the person rarely lasts more than a few months because it just isn’t effective. You were right, by the way.” Imagine that I sometimes know what I’m talking about.

When it comes to managing costs, understand what your goal is here, what you’re actually trying to achieve. Understand it’s going to be a cross-functional work between people in finance and people that engineering. It is first and foremost, an engineering problem—you learn that at your peril—and making someone be the human gateway to spin things up means that they’re going to quit, basically, instantly. Stop trying to shame different teams without understanding their constraints.

Savings Plans are a great example. They apply biggest discount first, which is what you want. Less money going out the door to Amazon, but that makes it look like anything with a low discount percentage, like any workload running on top of Microsoft Windows, is not being responsible because they’re always on demand. And you’re inappropriately shaming a team for something completely out of their control. There’s a point where optimization no longer makes sense. Don’t apply it to greenfield projects or skunkworks. Things you want to see if the thing is going to work first. You can optimize it later. Starting out with a, ‘step one: spend as little as possible’ is generally not a recipe for success.

What else have we got here? I’ve seen some things fly by in the chat that are probably worth mentioning here. Some of it is just random nonsense, but other things are, I’m sure, tied to various questions here. “With geopolitics shaping up to govern tech data differently in each country, does it make sense to even build a globally distributed B2B SaaS?” Okay, I’m going to tackle this one in a way that people will probably view as a bit of an attack, but it’s something I see asked a lot by folks trying to come up with business ideas.

At the outset, I’m a big believer in, if you’re building something, solve it for a problem and a use case that you intrinsically understand. That is going to mean the customers with whom you speak. Very often, the way business is done in different countries and different cultures means that in some cases, this thing that’s a terrific idea in one country is not going to see market adoption somewhere else. There’s a better approach to build for the market you have and the one you’re addressing rather than aspirational builds. I would also say that it potentially makes sense if there are certain things you know are going to happen, like okay, we validated our marketing and yeah, it turns out that we’re building an image resizing site. Great. People in Germany and in the US all both need to resize images.

But you know, going in that there’s going to be a data residency requirement, so architecting, from day one with an idea that you can have a partition that winds up storing its data separately is always going to be to your benefit. I find aligning whatever you’re building with the idea of not being creepy is often a great plan. And there’s always the bring your own storage approach to, great, as a customer, you can decide where your data gets stored in your account—charge more for that, sure—but then that na—it becomes their problem. Anything that gets you out of the regulatory critical path is usually a good idea. But with all the problems I would have building a business, that is so far down the list for almost any use case I could ever see pursuing that it’s just one of those, you have a half-hour conversation with someone who’s been down the path before if you think it might apply to what you’re doing, but then get back to the hard stuff. Like, worry on the first two or three steps rather than step 90 just because you’ll get there eventually. You don’t want to make your future life harder, but you also don’t want to spend all your time optimizing early, before you’ve validated you’re actually building something useful.

“What unique feature of AWS do I most want to see on other cloud providers and vice versa?” The vice versa is easy. I love that Google Cloud by default has the everything in this project—which is their account equivalent—can talk to everything else, which means that humans aren’t just allowing permissions to the universe because it’s hard. And I also like that billing is tied to an individual project. ‘Terminate all billable resources in this project’ is a button-click away and that’s great.

Now, what do I wish other cloud providers would take from AWS? Quite honestly, the customer obsession. It’s still real. I know it sounds like it’s a funny talking point or the people who talk about this the most under the cultists, but they care about customer problems. Back when no one had ever heard of me before and my AWS Bill was seven bucks, whenever I had a problem with a service and I talked about this in passing to folks, Amazonians showed up out of nowhere to help make sure that my problem got answered, that I was taken care of, that I understood what I was misunderstanding, or in some cases, the feedback went to the product team.

I see too many companies across the board convinced that they themselves know best about what customers need. That occasionally can be true, but not consistently. When customers are screaming for something, give them what they need, or frankly, get out of the way so someone else can. I mean, I know someone’s expecting me to name a service or something, but we’ve gotten past the point, to my mind, of trying to do an apples-to-oranges comparison in terms of different service offerings. If you want to build a website using any reasonable technology, there’s a whole bunch of companies now that have the entire stack for you. Pick one. Have fun.

We’ve got time for a few more here. Also, feel free to drop more questions in. I’m thrilled to wind up answering any of these things. Have I seen any—here’s one that about Babelfish, for example, from Justin [Broadly 00:46:07]. “Have I seen anyone using Babelfish in the wild? It seems like it was a great idea that didn’t really work or had major trade-offs.”

It’s a free open-source project that translates from one kind of database SQL to a different kind of database SQL. There have been a whole bunch of attempts at this over the years, and in practice, none of them have really panned out. I have seen no indications that Babelfish is different. If someone at AWS works on this or is a customer using Babelfish and say, “Wait, that’s not true,” please tell me because all I’m saying is I have not seen it and I don’t expect that I will. But I’m always willing to be wrong. Please, if I say something at some point that someone disagrees with, please reach out to me. I don’t intend to perpetuate misinformation.

“Purely hypothetically”—yeah, it’s always great to ask things hypothetically—“In the companies I work with, which group typically manages purchasing savings plans, the ops team, finance, some mix of both?” It depends. The sad answer is, “What’s a savings plan,” asks the company, and then we have an educational path to go down. Often it is individual teams buying them ad hoc, which can work, cannot as long as everyone’s on the same page. Central planning, in a bunch of—a company that’s past a certain point in sophistication is where everything winds up leading to.

And that is usually going to be a series of discussions, ideally run by that group in a cross-functional way. They can be cost engineering, they can be optimization engineering, I’ve heard it described in a bunch of different ways. But that is—increasingly as the sophistication of your business and the magnitude of your spend increases, the sophistication of how you approach this should change as well. Early on, it’s the offense of some VP of engineering at a startup. Like, “Oh, that’s a lot of money,” running the analyzer and clicking the button to buy what it says. That’s not a bad first-pass attempt. And then I think getting smaller and smaller buys as you continue to proceed means you can start to—it no longer becomes the big giant annual decision and instead becomes part of a frequently used process. That works pretty well, too.

Is there anything else that I want to make sure I get to before we wind up running this down? To the folks in the comments, this is your last chance to throw random, awkward questions my way. I’m thrilled to wind up taking any slings, arrows, et cetera, that you care to throw my way a going once, going twice style. Okay, “What is the most esoteric or shocking item on the AWS bill that you ever found with one of your customers?” All right, it’s been long enough, and I can say it without naming the customer, so that’ll be fun.

My personal favorite was a high five-figure bill for Route 53. I joke about using Route 53 as a database. It can be, but there are better options. I would say that there are a whole bunch of use cases for Route 53 and it’s a great service, but when it’s that much money, it occasions comment. It turned out that—we discovered, in fact, a data exfiltration in progress which made it now a rather clever security incident.

And, “This call will now be ending for the day and we’re going to go fix that. Thanks.” It’s like I want a customer testimonial on that one, but for obvious reasons, we didn’t get one. But that was probably the most shocking thing. The depressing thing that I see the most—and this is the core of the cost problem—is not when the numbers are high. It’s when I ask about a line item that drives significant spend, and the customer is surprised.

I don’t like it when customers don’t know what they’re spending money on. If your service surprises customers when they realize what it costs, you have failed. Because a lot of things are expensive and customers know that and they’re willing to take the value in return for the cost. That’s fine. But tricking customers does not serve anyone well, even your own long-term interests. I promise.

“Have I ever had to reject a potential client because they had a tangled mess that was impossible to tackle, or is there always a way?” It’s never the technology that will cause us not to pursue working with a given company. What will is, like, if you go to our website at duckbillgroup.com, you’re not going to see a ‘Buy Here’ button where you ‘add one consulting, please’ to your shopping cart and call it a day.

It’s a series of conversations. And what we will try to make sure is, what is your goal? Who’s aligned with it? What are the problems you’re having in getting there? And what does success look like? Who else is involved in this? And it often becomes clear that people don’t like the current situation, but there’s no outcome with which they would be satisfied.

Or they want something that we do not do. For example, “We want you to come in and implement all of your findings.” We are advisory. We do not know the specifics of your environment and—or your deployment processes or the rest. We’re not an engineering shop. We charge a fixed fee and part of the way we can do that is by controlling the scope of what we do. “Well, you know, we have some AWS bills, but we really want to—we really care about is our GCP bill or our Datadog bill.” Great. We don’t focus on either of those things. I mean, I can just come in and sound competent, but that’s not what adding value as a consultant is about. It’s about being authoritatively correct. Great question, though.

“How often do I receive GovCloud cost optimization requests? Does the compliance and regulation that these customers typically have keep them from making the needed changes?” It doesn’t happen often and part of the big reason behind that is that when we’re—and if you’re in GovCloud, it’s probably because you are a significant governmental entity. There’s not a lot of private sector in GovCloud for almost every workload there. Yes, there are exceptions; we don’t tend to do a whole lot with them.

And the government procurement process is a beast. We can sell and service three to five commercial engagements in the time it takes to negotiate a single GovCloud agreement with a customer, so it just isn’t something that we focused. We don’t have the scale to wind up tackling that down. Let’s also be clear that, in many cases, governments don’t view money the same way as enterprise, which in part is a good thing, but it also means that, “This cloud thing is too expensive,” is never the stated problem. Good question.

“Waffles or pancakes?” Is another one. I… tend to go with eggs, personally. It just feels like empty filler in the morning. I mean, you could put syrup on anything if you’re bold enough, so if it’s just a syrup delivery vehicle, there are other paths to go.

And I believe we might have exhausted the question pool. So, I want to thank you all for taking the time to talk with me. Once again, I am Cloud Economist Corey Quinn. And this is a very special live episode of Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review wherever you can—or a thumbs up, or whatever it is, like and subscribe obviously—whereas if you’ve hated this podcast, same thing: five-star review, but also go ahead and leave an insulting comment, usually around something I’ve said about a service that you deeply care about because it’s tied to your paycheck.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.