The Appalachian Cloud Trail: Hiking, Cloud Economics, and Finding Perspective

Transcript

Caleb: I think honestly I love hiking. Um, I've been section hiking the Appalachian Trail and Corey joined me on this current leg. And I think what I love most about it is the community

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn, and today's guest has had just about enough of me. Caleb Hurd is many things. He's periodically a cloud economist here at Duckbill, but he was also my hiking buddy for a 60-mile section hike of the Appalachian Trail. Caleb, thank you for continuing to speak with me.

Caleb: You're welcome, Corey. You're welcome. It's a pleasure to be here, and, uh, I would say six days on the trail with you is very much like s- Screaming in the Cloud and Last Week in AWS, in that you don't really know why you do it, but then at the end of it you feel like you should have done it and you appreciate it

Corey: This episode is sponsored in part by my day job, Duckbill.

Do you have a horrifying AWS bill? That can mean a lot of things. Predicting what it's going to be, determining what it should be, negotiating your next long-term contract with AWS, or just figuring out why it increasingly resembles a phone number, but nobody seems to quite know why that is. To learn more, visit duckbillhq.com.

Remember, you can't duck the Duckbill bill, which my CEO reliably informs me is absolutely not our slogan.

Thank you. I, I still remember a few of the things you said along the way. Uh, for example, y- you confessed that you thought that a lot of my jokes on Twitter were AI-driven until you started talking to me and realized, no, I talk exactly like my Twitter account does.

Yeah, pretty much. Sarcasm remains my first language spoken at home. Yeah, we wound up doing, uh, what was it? 10 miles a day on average for six days, uh, at the end of which everything hurt. It was fun. Uh, carrying, carrying obsessively about how much everything we carried weighed was certainly a new experience.

And, and then talking to folks who were doing three, four times our mileage, good for them, good for them. I, I, I choose happiness in life.

Caleb: Yeah, I don't think managing ounces that you carry in your pack is a new experience for you at all because that's exactly what you do tuning AWS bills. Um, I feel like managing weight is like you load everything in you need, and then halfway through the trip you start throwing it off the trail, not that we would ever litter, and you end up at the end of it with what you exactly needed.

Corey: Oh, it was... It's great too. My back is super strong from carrying AWS marketing all these years. So I don't know why I was so focused on being ultra-lightweight. Y- you had a knack, though, for finding very expensive things along the trail. Just people had littered, and you, well, you picked them up, and one, for, to be a, a good citizen.

Like, you found a headlamp that was not inexpensive and also a pound and a half just because someone hated themselves. I can see why they wanted to leave that behind. Please don't litter in the woods. But yeah, you, you had a knack for being very lucky around things like that.

Caleb: Yeah. It's true. It's true. And for the listeners, uh, for the podcast, day one, I was convinced that Corey had memorized all of his jokes and AI-generated content.

Uh, day two, I started to suspect it might actually be coming out of him naturally. Day three through six, I realized this is actually just how Corey talks, and in fact, AI is the one catching up, and he probably doesn't generate any of his content

Corey: Not really. Sometimes I'll workshop titles, and that's, that's about enough.

Honestly, in this case, I don't even need to. The title writes itself. Appalachian Cloud Trail.

Caleb: It's beautiful. A wonderful podcast title.

Corey: I mean, the best part of it was we didn't have to think about clouds for the entire trip. The internet can't find us there, and y- you learn a lot about yourself when you're, you know, huddled in a tent in a rainstorm, or in your case, trying fruitlessly to convince me to switch to a hammock all the time, or, you know, when my sleeping pad exploded under me at 10:30 at night.

Uh, then as I'm lying there in the cold, dark rocks, I get to hear the American singing retrievers 100 yards away. Uh, the only difference between them and coyotes is my unwillingness to admit when I've made a terrible mistake. And it, it was just a, a series of fun things that at the time didn't seem like fun, but now I'm nostalgic about them because anything's better than dealing with AWS bills.

Caleb: I don't disagree, and what, what really the highlight of the trip, and you probably should take a step back, and we should, we should walk them through it in chronological order, but, uh, was being saved by Shepherd, our, uh, our shuttle driver.

Corey: Yeah. Why don't you begin that story? 'Cause otherwise it looks like I'm dunking on you, and it's more fun if you dunk on yourself.

Please, take it away. We'll get to the cloud stuff. Don't worry. But I want... I... This is the story worth telling.

Caleb: It's, it's phenomenal. It's, there's the three part, and so I'll try to go through it rapidly, and you, uh, jump in when I miss a detail. But we started off parking my vehicle at the bottom, and then we had a shuttle driver drive us to the top, and his name was Tom.

Corey: The bottom, top. There was a lot of uphill in, in a lot of our hiking. It was... There... We, we had what? Something like a mile in elevation?

Caleb: Oh, it was, it was something crazy. I don't even remember. It was, it was, it was significant, and every day was hard. It... There m- we might've had one flat day out of the six.

Like, it was pretty wild.

Corey: Or one flat night when my sleeping pad exploded, but we'll get there.

Caleb: Please continue. We definitely had a flat night. So, uh, a couple days in, I don't know, night two, I can't remember which exact night, I slept great. I heard nothing other than the, the coyotes, which apparently are, uh, not coyotes.

But apparently in the middle of the night, Corey's sleeping pad blew up, and early in the night, too. It was, like, 11:00 PM or something like that.

Corey: Eight-inch blowout. We're not talking a slow leak that you get to find. We're talking all the air left in about a second and a half. Uh, in fact, the next morning you asked if I had gone to sleep with ice skates on.

Caleb: That's right. It was pretty wild. It was an unrepairable, uh, thing. So of course Corey, uh, he had a great night. He managed to turn his backpack into a, a sleeping bag.

Corey: Oh, I s- I left a review on the Durston subreddit. Uh, it was the Kwaka, uh, 55, and I said this is a solid 8 out of 10 as a backpack, uh, 2 out of 10 as a sleeping pad.

Caleb: That's exactly right. So we were trying to figure out the next day how to not have Corey sleeping on the ground, and I was like, "You know what? I should text Tom, our shuttle driver," 'cause-

Corey: And oh, I wanna be clear. Before, before this point, you were extraordinarily kind. You're like, "Well, you should've woken me up."

Like, why? Like, like my... One of our... Like, we both have young kids. Part of the reason we wanted to go get lost in the woods. It's like I... Like, they're, like, standing by the side of your hammock. "My blanket fell on the floor. I had a bad dream. I want some water." Like, no. I... There's nothing you can do. I'm just gonna...

Like, I'm already not getting a great night's sleep. Why do I make it both of us? Because I'm not a jerk.

Caleb: That's true. He did let me sleep, but, um, that's, that's accurate. So the next day, me with my full night's rest in a beautiful hammock, by the way, which you should switch to sometime, Corey. We woke up and...

Well, he- Corey didn't wake up. He was already awake. We texted Tom, and basically we're like, "Hey, could you shuttle us into this town nearby, Hot Springs?" And we can buy a sleeping pad at, you know, wherever- 'Cause we were shooting two nights

Corey: out, and I wasn't gonna do that to my back ...

Caleb: right. No. That's, that's not, that's not how...

Nobody wants to do that for multiple nights in a row. And, uh, Tom is like, "Hey, um, why don't I just run you one out?" And so next thing you know, we hike to a road, and our shuttle driver runs us a sleeping pad out that Corey, uh, uses the rest of the trip. It was phenomenal. So that was, tha- those are the first two parts of the story that involved Tom.

Corey: I liked it because it d- it wasn't as lightweight, but it also didn't explode. Uh, w- we'll get there.

Caleb: There is an advantage to a sleeping pad that functions. And then after we rolled into Hot Springs, slightly before we rolled into Hot Springs, I realized that I had left my phone at the campsite where we had just walked out of, and there was no way that I was gonna hike, you know, multiple miles back to go retrieve it.

I started doing the math in my head of, like, "How much do phones cost these days?" So we ended up at a hostel to do laundry. It's a VM

Corey: iron. What's, that doesn't sound like a bad thing to litter in the woods. And maybe we're uplifting the technological development of the local squirrel population. Why not?

Caleb: That's right. That's right. No, I was leaving a relic to be discovered by future generations up there. But anyway, so we mention it to Tom, who was at the hostel. He happened to be one of the people running the hostel, and, uh, his trail name is Shepherd, which we quickly discovered why it was Shepherd. Uh, and he's like, "You know what?

Let me just trail run back. I can just trail run back and get your phone for you." And next thing you know, this guy, who's an absolute legend, trail runs back to, uh, where we camped at, retrieves my phone, and runs back before the sun even sets. It was wild.

Corey: It's super easy to picture this guy because picture a guy that can just run up the side of a mountain like that to find a phone with weak-ass directions for where this is, find it, and come back.

That's exactly what he looks like. Um, but it was great. He's like, "So how will I know what's your campsite?" "Well, there's a tree that looks like it's about to give out at any moment and obliterate the fire pit," which, okay, great. He knows nothing about us, whether we know what we're talking about in the woods, and he's, he's walking around the area.

And then he sees that giant tree about to obliterate a fire. He's like, "Yep, this is definitely the site. Holy crap." Yeah, like, we, you started setting up. You saw it from the other side, and you said, "Yeah, well, let's move about 20 feet that way. Uh, choose life. Fine." A- and he just did th- he, he was, he pulled a rabbit out of a hat for us.

It was the, it was the weirdest thing And you could, like for us it's this legendary story, and for him it was any given Tuesday.

Caleb: Yeah. I think honestly I love hiking. Um, I've been section hiking the Appalachian Trail, and Corey joined me on this current leg. And I think what I love most about it is the community.

Like, just people are willing to just jump in and help. If you need something out on the trail, you can't just go to a city, you know, to a town casually and grab it. People, you know, I... For example, um, there was a guy passing us whose water filter gave out, and so Corey and I, Corey and I both have water filters.

His is better than mine. I won't admit that. Don't tell Corey that. It'll get to his head. And so it was like, "You know what? I can use that water filter. I'll give you my water filter." But, and it's not just me. Like, the whole trail has that communal feeling to it, which makes the whole experience even that much better.

Corey: Yeah, we ran to an old guy with a dog who was, uh, three miles from water. We gave him the last of ours, 'cause we're, we're young enough that we could, we could still deal with getting there, getting a little dehydrated, and then just drinking three gallons of it. Um, we want... Yeah, it's, it's the thing you do.

Like, who needs it more?

Caleb: Exactly. Exactly. I have a similar feeling doing, uh, cloud economy work in that there's so few people in this space, especially ones that have heavy duty engineering backgrounds, and it's a similar community where everybody just jumps in. Like, there's a lot, uh, there's a lot of sharing of knowledge, there's a lot of opinions, there's a lot of esoteric information that's out there, and every time I come across somebody in my space that works in this field, they just freely give that knowledge.

It's one of the things I enjoy about working in this space.

Corey: Yeah. Uh, uh, one thing that I found, the, the reason I like doing stuff like this is it's been a month since we got back, and I'm sti- it's weird how quickly you get accustomed to it, but y- but I'm also making it a point to notice it now. I dunno if you're listening to this, uh, you're just hearing my voi- my dulcet tones in a mic, but if you're watching this on the YouTubes, uh, this is my first recording from our new Duckbill office in downtown, and I'm sitting in a comfy chair at a desk with a sparkle water in my hand, and everything is comfortable, everything is clean.

Uh, I don't have to dig a hole if I wanna poop. I mean, I can, but that's generally frowned upon in the office environment. Uh, uh, yeah, it... And I don't have to care how much something weighs. It's if I'm gonna... Am I gonna take this home tonight or not? Well, weight isn't really a factor on that. So it, it's just, it's a different way of living on some point.

At some point it's, it makes you more thankful for what you've got. Like, I, I still wander the grocery store in awe compared to how p- how much of a pain in the neck it is to feed yourself in the woods.

Caleb: Yeah, no, I think that's a great... Like, perspective's everything, and there sh- it's so easy to go through life fast.

I think the, the best way I can describe coming off the trail, and it being a month in, this is probably, this feeling's already passed for me. Um, but the first probably solid week, uh, when I come back from a long hike, everything feels fast. Meetings feel fast, conversations feel fast, cars feel fast. You know, even schedule, like even my calendar, everything just feels fast.

And when you, when you're out on the trail, you, everything's serial. You have, like, one task to do There's no multitasking and, and everything happens in order, and your brain can follow it in order. And it's a real reminder that, like, cognitive overload in your day-to-day, it's a real thing. Like, simplifying what you have in front of you, uh, and, and doing it serially really is an effective strategy.

Corey: Yeah. What, what's also wild is I, I then got to spend a whole bunch of time harassing you after the fact with, uh, references to the callback jokes we had on it. Like, we, we decided that everything was heavy. Why don't we just build gear full of helium, uh, so that it actually has a negative weight, and that means you could take a lot more with it.

And like three days later, I'm at home and I sit bolt upright and text you. It's, "I figured it out. We're gonna sell this, and we're gonna brand it, and here's how. Most sleeping bags are filled with down. These are the only ones that are filled with up." And you had the courtesy of not responding to that, and that was great.

Uh, uh, thanks to the magic of AI image editing, I wound up finding, uh, like I did some rain gear that looked like Rudolph, and I called it Rudolph the Red-Nosed Reindeer, and that was awesome. I wound up with a blister on my heel when I switched away from toe socks because it turns out I am lactose intolerant.

Ah.

Caleb: That was, that was one of my favorites.

Corey: It was great. It was just great being able to throw the dumb jokes at someone who will absolutely smack them back.

Caleb: No, it was, it was... We had a phenomenal time. It was, uh, it really was... It's something I wish I could do more of, like, and not just me, I wish, I wish everybody, I wish there was, like, a public holiday where everybody went and spent a week on the trail, 'cause I think we would all view life differently.

Corey: Right, and what do you do when you're trying to climb up a mountain that never seems to freaking end? Well, you come up with trail facts about Tom. For example, that, like, he doesn't use GPS positioning himself, the satellites just realign to match his trail out of respect. Uh, he passes the white blaze, they repaint themselves.

You know, usual stuff, one would expect, Chuck Norris fact style, only Tom is still alive, 'cause that man will never die.

Caleb: Well, and that's true. I re- I refuse to believe those headlines. Um, another example of having time is we were headed up one particularly bad, uh, climb at the end of the day, so we were both exhausted, and this was towards the end of the hike, so we were even more exhausted, and I think I turned to Corey, and it was like, "I think we've got about an hour of climbing.

You should tell me a joke," and he proceeded to tell me the world's longest joke. And it, I think it is titled The World's Longest Joke, and I kid you not, it was at least, what? An hour in the telling, Corey? Oh, yeah, it had to be. Yeah, and I didn't feel one single step. Uh, like, time just basically teleported me to the top, uh, when he finally got to the punchline, which was very much worth it, by the way.

Corey: Exactly. Maybe one of these days when I have nothing better to do, I'll record an hour-long podcast of just that joke.

Caleb: I, you know, I'm here for it, 100%. I would, uh, I would be your audience member. I could hop on, and, uh, laugh and interject appropriately.

Corey: Yeah. It, it was an awful lot of fun, just doing something completely different.

And then we went back to work and worried about cloud finance and the rest. And one of the reasons I love working with you is that you've been doing this almost as long as I have. Uh, you were the guy that went on stage a while back and talked about putting Kubernetes clusters in every Chick-fil-A restaurant.

I don't know if that's the secret ingredient in their sandwiches or not, but good job. That's, uh, sort of everything tastes better with Kubernetes. Like, maybe it's like trail hunger, where you're, uh, you're hungry, so everything tastes better. Uh, with Kubernetes, it's the, it's the suffering that infuses the sandwich that makes everyone just enjoy it more.

Maybe that's the knack.

Caleb: That's right. It's like, uh, lab-grown diamonds versus real diamonds. It is the suffering that makes the, uh, that makes the diamond.

Corey: Jesus. Is there a con- is that a conflict diamond? Well, not until I fight you for it. Yeah, exactly.

Caleb: That's funny. No, so, uh, yeah, it's... I've been doing this for a long time.

I think I, I really enjoy doing cloud economy work. Uh, I, I do consulting for Duckbill. Um, I also have a full-time job, keeps me busy too. And o- one of the reasons I enjoy it so much is because, um, in order to be really good at it, you have to have walked the gauntlet of racking and stacking as an engineer so you know op-prem pieces.

You have to walk the gauntlet of operational work, keeping systems alive, because cost is only one element of consideration for an organization, and if you bust through the door and you're like, "Hey, these are the f- you know, you could, you could save a ton of money if you could just turn production off between the hours of 11:00 and 6:00 AM," people are gonna look like at you like you're crazy.

Like-

Corey: The United States Social Security Administration basically does, but that's neither here nor there. It's some sort of mainframe backup job that means that they have actual, I'm not kidding, business hours for their website.

Caleb: I don't doubt that in the least. I mean, you know what? Maybe they're actually ahead of the curve.

Maybe everybody should take their example. Maybe it's not them needing to come- The

Corey: computers are tired, Caleb. Let them sleep.

Caleb: Yeah, you know, everything comes back around, right? Maybe we should all regress to 1980s models. That's, that's one of the things I've been helping people with recently is convincing them not to go back to on-prem.

Corey: Oh, it's funny. We've had a number of clients approach us to this, and I, I swear every one of them is gonna think that we're calling them out, but no, no, this is a larger trend. And one of the fun things that you and I got to do because we're hiking through the woods with not much else to talk about at one point, was try to out trauma each other with what's gonna suck the most that people don't see coming about building out a, a data center.

And we talked about things like rack nuts, and bad batches of cables, and Dell screwing things up, as is their nature, and Cisco doing the same, as is their nature. And then you have remote hands, which are great, and you have smart hands, which says something really insulting about remote hands by a juxtaposition there.

And, and all the, the trauma and the rest, and we mo- and, like, the speed of Jason, Jason APIs, uh, which is when you have a guy named Jason who opens a ServiceNow ticket via fax and eventually implements it. And I'm not saying that, that on-prem is always a bad idea, but the companies that have never done it before are opening up a bag of pain that the clouds have done such a good job abstracting away that people don't realize it's there or believe that it exists to the degree that we're talking about it.

You and I collectively have damn near 30 years of experience working in data centers and the like. Uh, the fact that we do not work there anymore is not because we are stupid

This episode is sponsored by my own company, Duckbill. Having trouble with your AWS bill? Perhaps it's time to renegotiate a contract with them.

Maybe you're just wondering how to predict what's going on in the wide world of AWS. Well, that's where Duckbill comes in to help. Remember, you can't duck the Duckbill bill, which I am reliably informed by my business partner is absolutely not our motto. To learn more, visit duckbillhq.com.

Caleb: Yeah, I mean, at the end of the day, like at least your cloud bill edges up over time, and you get to investigate it and figure out why, and then troubleshoot it and lower that bill.

But with data centers, like you can wake up tomorrow and, you know, it can be a, a catastrophic issue or outage and or a-

Corey: Through no fault of your own. Well, well clouds take outages too. Yeah, but the difference there is, one, everyone is gonna be down at the same time, and two, if it's a foundational infrastructure issue, they have a team of the best people in the world that are working to fix that faster than you will.

It's... There, there's no if, ands, or but around that. It's a different universe.

Caleb: Yeah. Yeah. My DR strategy if us-east-1 goes down is I'm gonna go, uh, grab Corey, and we're gonna shelter it out in the Appalachian Mountains in a cabin.

Corey: And or depending on whether the world is still survivable at that point, we will also make all the money consulting people with very expensive problems all of a sudden because people generally don't plan...

Well, like we've seen the regional outages in the Middle East. Yeah, geopolitics matter. If, if we see a us-east-1, uh, regional outage, uh, that says something concerning from a physical infrastructure layer after it goes about a week 'cause this is about 100 square miles in northern Virginia we're talking about.

Caleb: Yeah. Actually, you got a good point. If us-east-1 was down for more than a week, you would see Duckbill become- Pacific

Corey: Coast Trail. Pacific Coast Trail. Yeah, we don't wanna be on the Appalachian at that point.

Caleb: Yeah, exactly. Or and or once we come back from the trail, we'll, we'll retool Duckbill to focus on migrating people out of the cloud onto on-prem.

Corey: Ducks are migratory. I mean, you know, arguably we could make that argument. But, uh, more duck less bill now.

Caleb: More duck less bill. Amazing. So what was your, uh, what was your favorite part of, uh, of hiking with me, Corey?

Corey: It's weird because at the time, the thing you look forward to the most is we hit the halfway point and managed to eat like 5,000 calories in a meal.

That was awesome. Uh, there was one, uh, place in Hot Springs that was dialing in their pizza oven, and it was... They were so good at... They... There was a sign that said, uh, "Best pizza in town." Uh, that was a fucking lie. Uh, I love the fact that both of us were starving and still wouldn't eat that piece of crap, and then we went to the actual pizzeria, and it was freaking incredible.

Everything else at that place was amazing, but it was, uh, but it was very much a eh. Curiously, the two of us got to meet up with Chris, uh, my podcast producer. So it was, it was fun being able to sit down and talk a little bit of shop about this. We tried to convince him to use Kubernetes for something, mostly as a prank, and yeah, it was fun just being able to let go of a lot of the work stuff.

It, it was also nice to not have to worry about the complex things and instead worry about the simple things that are a little bit more survival-oriented. Like, "Okay, I don't like the look of those clouds. We should probably find a decent camping spot so we're not surprised off the side of the mountain.

Uh, uh, wow, we're getting a little low on water. How do we believe that, uh, that next water source ahead is gonna be sufficient, or should we use this crappy one while we've got it?" Uh, there's, it's... These are decisions that are great. Frankly, you also sleep like the dead after your second night out there just because you're bone tired.

Caleb: Yeah, even, even if it is on a backpack.

Corey: Yeah, you know, we take what we can get. Uh, I, I did send... To their credit, uh, it was a NEMO Tensor Elite. This thing's about a $250 sleeping pad that weighs about as much as a postage stamp, and it's, it's a great pad, but it's delicate. And to their credit, they replaced the entire thing with a, uh, position of just write the RMA number on it in permanent ink.

Great, we're mailing you a new one. Sorry for the trouble, which was super appreciated. But also now the problem I've got is one of these already failed me once. Would I trust it as my only sleeping option going forward? I don't know. It's a, there's a question of faith. Uh, and this of course is similar to cloud providers.

When, uh, you know how a cloud provider fails and how they handle that failure from a business perspective. Um, when you're dealing with a new provider, well, you don't know how that's going to break, and furthermore, there's a terrific opportunity for them to either gain or lose you as a customer forever.

Uh, I'm saying nice things about NEMO. Like, their Tensor product line is pretty great. It's delicate, but it worked super well. If they'd been a jerk to me, oh, that would be the headline instead, and I would've been tearing into them, but they did the right thing There's a... People, I think cloud providers forget the human component.

Like, Google can't wait to not talk to humans as fast as humanly possible

Caleb: Do I get, uh, do I get to split the, uh, product placement fee for Nemo there?

Corey: You know, I, I miss the days when I used to sponsor this with something other than Duckbill itself, uh, just because it's great. Hey, why not? It's... At some point you start to wonder, like, whether the audience is no longer aligned.

But well, you, you went from ad- advertising enterprise SaaS and observability products to camping gear. Do... Bad business or do you know something we don't? Like, eh, you never know. No, but it's nice not to have to be Cloud Billy Mays, uh, pitching for something different every week.

Caleb: Yeah. No, it was a great time, and there's so many parallels between, um, you know, camping and doing cloud economist work from, you know, planning and managing the load you're gonna carry to, you know, going through AWS bills and trying to understand services and how that impacts organizations, and the communal aspect of the trail versus the communal aspect of helping these organizations.

You know, everybody piling and helping. Some people being resistant to, you know, some people being like, "You're here to just to make me look bad." It's like, "No, no, I'm not here to make you look bad. I really am here to try to help the organization."

Corey: Well, you have to be careful how far you stretch that analogy to the breaking point.

Otherwise, it starts to sound like a really bad long form LinkedIn post. Uh, what being brutally surprised by a raccoon taught me about B2B sales.

Caleb: Like, I- Uh, I mean, I... You know, I could see that. I could see being brutally surprised by a raccoon helping me with B2B sales. I think I would hate both things just as equally much.

Corey: No, raccoons are great, except by surprise.

Caleb: That's right. Oh, man. But, um, but yeah, it was, it was, it was a good time

Corey: So I, I guess my question now is since we're, we're back and ostensibly having to think about work again, what do you see that's exciting coming down the pike right now as far as the wide world of cloud economics?

I, I have a list of my own, but, uh, this is not one of those areas that's right or wrong.

Caleb: Yeah, it's a great question. One of the things that I've seen emerging that I have mixed feelings about, I used to have stronger feelings about it, but now I think there's arguments that could be made for both sides, is I'm starting to see serverless, uh, within the AWS ecosystem.

And when I say the word serverless, I'm referring to, uh, Lambda, I'm referring to Mongo or, uh, DynamoDB and other serverless functions that don't require constant loads being run, even like Aurora Serverless to a certain extent. I'm starting to see these functions start to become more part of the main production systems that I find.

Corey: Yeah. This, this is gonna sound weird. I, I wanna, I wanna flag this out because companies have been using these things for 10 years at this point. We are not late to, we are not late to the adoption curve, but what we're seeing is this, say, being a significant part of the core production application line, a significant driver of spend.

Uh, yes, it was replacing batch jobs a long time ago. Now it's in the critical path in a way that it wasn't previously in large enterprises. Yeah, small companies have been doing this forever, but that's, that's the beautiful part of being a small company. Uh, you can put a funny animal on your website, "Oops, we're down," and there's the raccoon looking embarrassed.

And great, it comes back up an hour later, things are generally okay. Enterprises don't have that luxury.

Caleb: Which I'm a big fan of. Like, the small projects that I built for myself, for example, I built them all serverless 'cause I don't know what kind of load I'm gonna get. I wanna be able to scale up if it, whatever I'm working on goes bananas, and I wanna be able to not pay a bunch of money if it doesn't So it's kind of an ideal solution for a lot of things I build.

Um, I think for the first time about a month ago, I came across a customer who probably 90% of their bill, and they were a fairly significant spend per year, annual spend per year, but probably 80 to 90% of their services were all serverless technologies. They didn't... Uh, I think ECS was probably the closest thing that I would say was, like, constant.

They had, like, a minimum number of ECS jobs that they had kind of tuned their application down to for some things. It was interesting 'cause evaluating that customer, there were areas that I could give advice on, you know, how they were doing CloudTrail and CloudWatch and some CSP pieces, et cetera. But, uh, I couldn't really tune down their CPU and memory loads, um, because, because they were so s- heavy on serverless, they were already using pretty much exactly what they needed to.

Now, the negative downside of that is, um, because people aren't, haven't historically been running most of their loads in these particular technologies, A, the one-to-one ratio of serverless is more expensive. Like, if you took the per call cost for, you know, Lambda or the per call cost for, um, DynamoDB, and you compared it to running your own large cluster doing that thing, y- it's always gonna be a two to three, sometimes four to one ratio as far as dollars to dollars Where that's a little deceptive is that people don't run perfectly efficient solutions when they manage themselves If you have perfect

Corey: foreknowledge and can run it exactly 100% with no overhead, and your people's time to manage that cluster is free, yeah, sure, maybe, but we're talking math, not reality.

Caleb: Yeah, so what was really interesting was, like, where I started, started to see it break down a little bit is the tooling isn't there yet to really manage serverless technology at scale. So if you've got thousands and thousands of Lambdas, tuning that, managing it, controlling it, you kinda destroy the 80/20 rule, right?

So the nice thing about traditional ways of solving cloud problems is the top 20% of your services are the heavy hitters, they're expensive. Um, you... If you tune those top 20%, you're gonna get 80% of your bill impacted, so that's- that's sort of the 80/20 rule.

Corey: Don't optimize the tiny thing that gets fired once a month.

Maybe optimize the thing that gets fired 18 times per user request. And also question your architecture if that's what it's doing.

Caleb: Also true. But the, um, but the serverless approach tends to kind of peanut butter your code, repos, services, et cetera, and stretch it out, and so you no longer really have that 80/20 rule.

If you wanna go tune something, you really have to go and revisit 1,000 Lambdas or 1,000 different services to go, you know, change how you're doing a particular thing. So I think there's a sprawl problem that a lot of people experienced when containers first hit the market, that people are finally figuring out, like, that sprawl and making things too easy and too e- easy to manage will translate into making more services, which creates sprawl.

So I think my opinion with this trend, I really like it. As a cloud economist, I love the idea that people are choosing a technology that is efficient and cost-effective, but I also think, as a technologist that's run these systems before, I think you're gonna see people start hitting a very similar sprawl problem that they hit with containers.

All

Corey: this has happened before, and it, it doesn't... The, the rise of vibe coding is gonna make this worse instead of better. Like, "Oh, you have some bad code? We have a solution to that. It's called a lot more code." And it builds on top of everything. It doesn't get additive. It doesn't refactor on its own. It's let's continue to build, build, build, build, build.

The foundation could be rotten.

Caleb: Yeah. Observability, uh, in those sorts of environments becomes a lot harder, too. I mean, one of the upsides of a monolith is that, you know, you instrument it to write to logs in a opinionated way, and it doesn't matter what somebody does in the future, for the most part, you're gonna be able to go get that information, that observability, and go drill down into it.

But when you've got 2,000 Lambdas that were written over a period of five years by, you know, 15 different people-

Corey: Coverage is uneven. It's weird that the, some of the best environments you see are the most... Some of the best environments you see architecturally are some of the newest just because people have, people have gotten into it, uh, not having to learn the early mistakes the rest of us did

Caleb: Yeah, again, I, I strongly agree with that.

But I think there's... Every new trend, every new technology, there is some downside that we just don't see yet. I, I haven't seen, in the last 10 years in this field, I haven't seen any trend emerge that doesn't have some sort of payoff later that you just don't know yet. So me personally, if I was at the helm of a very large enterprise organization, I, I would be fine being m- maybe not a late adopter, but I wouldn't wanna be the tip of the spear on a lot of these new technologies.

Corey: No, spend your innovation tokens where it matters, on the thing your company actually does. Uh, Caleb, I, I wanna thank you for taking the time to speak with me. If people wanna learn more, where's the best place for them to find you?

Caleb: Yeah, you can hit me up on LinkedIn. Um, I think it's... Let me actually look up my LinkedIn.

I should have this memorized here. It's Caleb R Hird, H-I-R-D. Um, but yeah, they can hit me up on LinkedIn.

Corey: Awesome. Or find you on the trail. Caleb, thank you for putting up with me as long as you have and not murdering me out in the middle of the woods. It's been, it's an absolute pleasure.

Caleb: Yeah. You too, Corey.

It's always fun.

Corey: Caleb Hird, cloud economist here at Duckbill. I am Corey Quinn, chief cloud economist here at Duckbill, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry, insulting comment that then will be gathered, printed out, and hand-delivered to us on the trail by Tom.