Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored in part by Catchpoint look, 80% of performance and availability issues don't occur within your application code in your data center itself. It occurs well outside those boundaries. So it's difficult to understand what's actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course validate how reachable their application is. And of course, how happy their users are. It helps you get visible and to reach a bit availability, performance, reliability, of course, absorbency. Cause we'll throw that one in too. And it's used by a bunch of interns and companies you may have heard of like, you know, Google, Verizon, Oracle, but don't hold that against them. And many more. To learn more, visit www.catchpoint.com and tell them Cory sent you wait for the wince.

Corey: This episode is brought to you by Trend Micro Cloud One™. A security services platform for organizations building in the Cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? “I'm glad we have Trend Micro Cloud One™, a security services platform for organizations building in the Cloud,” or, “Hey, bad news. It's going to be a few more weeks. I kind of forgot about that security thing.” I thought so. Trend Micro Cloud One™ is an automated, flexible all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline, and access your cloud environments sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One™ a security services platform for organizations building in the Cloud. Whew. At trendmicro.com/screaming.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by someone who needs no introduction, so I'm going to give her as little of one as possible. Charity Majors founder and CTO of Honeycomb, but famous on the internet long before that. Welcome to the show.

Charity: Thank you, it's nice to be here. We've been trying to do this for a while.

Corey: We have, and it seems it's scheduling never quite works out until suddenly everyone is trapped at home, and eventually we both run out of excuses.

Charity: Yes. Well done.

Corey: So, what are you up to these days?

Charity: Well, I feel like I've turned into a den mother for a bunch of very anxious camp-goers. Through this whole COVID thing, I have always hated repeating myself, and that's something that I feel like once you go into management, you kind of have to get over. But during the pandemic, repeating myself, honestly, has come to feel like my one and only job.

Corey: In what sense?

Charity: “Everything's going to be okay. You're doing everything we could have asked of you. You're doing all right. Everything you're doing is enough. You are enough. You are beautiful. Please go home, get some sleep, have some soup. Take care of yourself. Take care of your loved ones. Yes, you're working hard enough. Yes, you are doing your best. Your best is enough.” You know, that kind of thing. 

I feel like something that I kind of said off the cuff in our all-hands a few weeks ago was just that brain weasels love authority figures and they love reassurances from authority figures. And it's so true. We're adults, we can be as anti-authoritarian as we want to be, it's still—there's something very reassuring—when we're very anxious about everything, there's something reassuring about somebody in a position of authority just telling you that you're good enough and everything's okay. And for the first time in my life, I find myself awkwardly in that position of authority. And so, just trying to use it to reassure people as much as possible that what they're doing is good, they are fine and that they can take a break, seems to be my main job right now.

Corey: Management seems to be one of those interesting areas of expertise because, to be blunt, a lot of the people who write these books and tell you how to be a good manager—

Charity: I hope “expertise” was said with scare quotes.

Corey: Oh, absolutely it was because all of these people you read, “Oh, you're going to give me advice on how to manage? Great, what do you do?” “Oh, you haven't managed anyone in 20 years. Instead, you've been writing books on how to manage people.”

Charity: Yes. Or, or better yet, they are a VP of something-something, and then you hear around the grapevine that they are terrible at their job.

Corey: Exactly. I can think of several people off the cuff that we are not naming because—

Charity: No, no, of course not.

Corey: —eh, lawsuits are passé.

Charity: Yes.

Corey: But you're right. I see something similar to this when I was getting started in freelancer circles, where people are focusing on specific areas that they want to tackle. It’s, “Oh, I'm going to learn how to do positioning as a freelancer,” and it turns into, “Huh, it seems I can make more money teaching other freelancers how to do positioning,” for example, “Or how to do sales than doing those things myself.”

Charity: I wish that people were legally mandated to present context with their advice. This advice is true within the narrow circumstances of these things that I've experienced before. Because people sound very authoritative. They're like, “You should do this.” And they don't bother to mention that they come from some completely different context, or we're talking to completely different people. Context is everything when it comes to advice.

Corey: Absolutely. I mean, one of the things I've stayed away from for a long time has been management advice, simply because my only management training was having a bunch of really terrible managers, and then doing the exact opposite—

Charity: Mm-hm.

Corey: —which it turns out gets you further than you might expect. But then I started this thing, and when I took on a business partner, Mike, I stopped managing anyone because what I do requires me to be more than a little bit self-promotional, and that is the recipe for a terrible manager in my experience. You have to build other people up, but I have to be the loud obnoxious one all the time.

Charity: And you hate that, don't you?

Corey: I can do it, but it's one of those things where I can't do it while simultaneously lifting up others.

Charity: You do it and you're very good at it.

Corey: Well, thank you, I appreciate that. I am the loudest and most obnoxious out of the entire flock.

Charity: Hearts.

Corey: Exactly. So, one of the questions I see is given that you have been able to walk a tighter line than I have because you still manage teams and you tend to focus on the empathy piece quite a bit, but you also have never been one to shy away from sharing your opinion online in various ways that I hold a deep personal affinity for. Some might call them bombastic; I call them enthusiastic.

Charity: Sure. We only speak in grand statements. Yeah, well, I mean, part of it is I don't really manage a team right now. I manage three people: Liz, Emily, and Megan. They've graduated, they don't need my bull-[BLEEP], you know? 

So, I honestly feel like if we're talking to engineering managers here—context, right?—I would say that your job is to craft a team. Your job is to craft and coach a team. You're not hiring individuals, you're not hiring people. You're curating a team. And the kind of just emotional intensity that you need to have with that team, you're in it with them every day. And I'm not doing that right now. 

And so most of the, when I'm giving my quote-unquote, “advice,” it's mostly me thinking back to the days where I was doing that and drawing on my experiences there. That plus I don't really like to talk about the people that I’m working with now. It feels kind of rude and kind of non-consensual to show our dirty laundry. I'll talk about them someday, but right now, I honestly try not to talk very much about people at Honeycomb.

Corey: Well, you also have entered one of those very interesting and rarefied positions where the people who report to you are themselves senior enough and have the context to be able to need guidance in a very different way, versus junior people or folks who are new to the workforce in one form or another, often don't have that. And the way to manage those people, in my experience, has been radically different.

Charity: Yeah, because I don't really have to think about the careers of the people who report to me much. I trust them for that. They're in a good spot. It's much less of a here, help them grow in their skill set to get to senior engineer, and then help them be effective in a large [unintelligible], and it’s much more just, they're grown. They're baked. They're cool, so I need to just give them the information that they need. 

But managing managers is very different than managing engineers or managing ICs of any sort. The visual that I always get is, when you're doing the work, you're assembling a basket with your hands. When you're managing people, you're walking up and down watching people assemble baskets, and you're inspecting, and peering. When you're managing managers, you're standing over on the other side of the pier, squinting and trying to see how the baskets are coming out, and then waving at them with flags to try and signal what they should be telling their team to do. You're so far removed from the actual being able to do the thing that it's terrifying, honestly. 

It's terrifying and it's a whole different skill set that involves a lot of letting go of the ego. We spend our whole career trying to get good at this skill set, right? Being good at engineering, being good at getting things done, making things, and then we get drafted into management, and it's not even, like, a sidestep career-wise, it's like you're starting over at zero. But you still get to sit there and enviously watch the people doing the thing that you love to do. And you're trying to wall that part of yourself off, and I don't know, it's weird, man. I don't understand why people want to be managers.

Corey: My honest assessment of why people want management is that they're told that that's the thing you do next. It's what when you've conquered the IC scale, then you get promoted to management, but it's a parallel track. It's not something in most jobs that should be considered a promotion.

Charity: Or even more perniciously, they feel disempowered. You know, what did Daniel Pink say that we want out of life? We want autonomy, mastery, and meaning. That’s what we want from our work. So, many people show up to work every day, and maybe mastery, maybe they're getting mastery, but they may or may not have meaning, but autonomy, they don't have it. And they want it. They crave it because we all do. 

And they're told, or they intuit for themselves, that the way that you get more control over what you're doing, you should go into management. And this is so… I mean, this is why I went into management. I wanted a bigger say over what I was doing. No shame. I'm not sharing to shame anyone who's done any of these things. 

I think that even wanting more power, I think that it’s better to be honest about these impulses within ourselves and deal with them in a straightforward way than it is to learn to puppet that, “Well, I just want to help other people, or—” it's better to be honest with yourself about why you want something because I don't think there's a wrong reason.

Corey: I would like more money and the ability to tell people what to do.

Charity: You know, that's fine. That said, you're going to be in for a rude awakening when you realize that managers don't actually get to tell people what to do, at least not in healthy organizations.

Corey: Oh yes, but what made you think that most organizations are healthy ones?

Charity: Oh, there's the rub. And so there's always this back and forth between why do you give people advice for how to be a manager, but it very quickly segues into how healthy is your organization? How healthy can you make it? How many other people are there who feel like you? Because here's the thing, your company doesn't exist: you show up every day and create it with your colleagues. There's nothing that exists other than that. We have all of the power that we need in our hands to change things radically if you just convince people that it's worth doing, and enough of the people show up and create it this different way tomorrow, right?

Corey: Absolutely.

Charity: Which is a terrifying and awesome thing to realize.

Corey: That's quite the thought to wrap my head around, but you're right.

Charity: And so many organizations where the only way to proceed is to go into management. Well, what if there could be parallel tracks? What if there could be technical tracks that go hand in hand with managerial tracks. But the way that you need to sustain this is it can't just be managers giving power to engineers. It has to be engineers embodying that power and taking it for themselves, too. Have you ever been on teams where managers want to give engineers more power and they won't take it? Because I have.

Corey: It's fascinating to watch that dynamic play out and how the organization responds to that tells you an awful lot about that company.

Charity: It really does. Now, in my mind's eye, I have this ideal, the way that engineering organizations should operate. This is a creative profession. Yes, there's a high basic skill-level for entry, but we create things, and managing a large codebase is fundamentally a creative act, which means that we're going to do our best work when we are empowered, and inspired, and motivated. So, I feel like power given to managers should be limited and enumerated. Less King George and more articles of the Constitution, where it's like, okay, we're not supposed to bow and scrape to our leaders anymore. 

They serve us, and we are giving them these articulated powers, and they have authority and power and responsibility to do these limited things insofar as they are doing them on behalf of the company as a whole. I think that's the appropriate healthy attitude towards management because there is this drift, this power will tend to drift towards management over time unless you're consciously pushing it back out to the ICs. Because so much of power is information flow, and managers, they are absolutely privy to private information, and they spend their days talking to people instead of heads down in code. So, power is going to drift towards them unless you actively push it out.

Corey: And something you said just there's there does resonate in that we have a high basic skill level required for these jobs, but if you look at a lot of historical companies, by the time that someone is coming in making, say, six figures at a manufacturing plant, they've been there for 20 years, they’ve worked their way up. Now, a new grad gets that and a lot of assumptions in a lot of these cultures is baked in that by the time you're paying someone a certain amount of money they, for example, have the insight, wisdom, and political savvy not to call their boss an asshole in the all-hands meeting. That does not always play out the way that one would hope.

Charity: No, it doesn't. Something I've been thinking about a lot lately is the way we humans, we’re hierarchical beings. If there's a ladder there, we instantly want to climb it. And there's this woman, Molly, who I worked with, she was an engineer, she graduated, she got offered a manager's job pretty quickly, she climbed a ladder, she had a series of high profile executive jobs. 

Twenty years she worked this industry before she suddenly realized she hated it. She was miserable. All she wanted to do is write code, and she was jealous every day of the people that she worked with who were working software engineers. She's like, “Why did I climb this ladder for 20 years? I hate everything about this.” And I find that so sad. And there's so much pathos to that because we have this assumption—just like you said—that if there's a hierarchy than the person at the top of the hierarchy, is superior to all who are below them, in knowledge, in judgment, in experience, and all these things when in fact, I can now say from experience, it's just a different kind of work. 

Yes, when you have an organization of a certain size—it's like a human data structure, basically, that you need for an org chart because you need some people to have their heads up scanning the horizon and thinking about that one to two-year, five-year journey, and you need people down in the mid-range with their heads at that 6- to 12-month range, and you need people down in the weeds, but they're just different kinds of work. They're not better or worse than each other. And some of us are made happy by some types of this work, and not other types, and if we could just strip it of all of the baggage of power and hierarchy, if Molly could just work on the things that brought her joy, this is the amazing thing that we take for granted this industry. We get to work on things that bring us joy, every day. This in and of itself, sets us above 99 percent of all humans who have ever lived, and yet we throw away this joy and meaning that we found to chase the being better than the people around us. And I feel like happiness should be given a greater weight.

Corey: In what you might be forgiven for mistaking for a blast from the past, today I want to talk about New Relic. They seem to be a relatively legacy monitoring company, and I would have agreed with that assessment up until relatively recently. But they did something a little out there: they reworked everything. They went open source, they made it so you can monitor your whole stack in one place and, most notably from my perspective, they simplified their pricing into something that is much more affordable for almost everyone. There's even a free tier with one user and 100 gigs per month, totally free. Check it out at newrelic.com.

Corey: You make excellent points, and I think it's time for a second rant on a different topic if you'll indulge me yet further. Namely, this for a lot of people relatively new to the workforce has been their first downturn. The last one was 10 years ago and change, and there have been a lot of people entering the workforce since then. What are you seeing as far as the downturn’s impact on Cloud and cloud-like services?

Charity: Yeah. So, a little bit of background. I love running mail servers.

Corey: I started my career that way, and I'm right there with you. And then it—ehh, Postfix was great.

Charity: Right? At Linden Lab, I was the only one who could get IMAP working. We ran our own trainer and spam assassin, and ClamAV, like antivirus—

Corey: Dovecot, or one of the older ones like Courier?

Charity: Yeah. All that.

Corey: Excellent.

Charity: The first job I had in industry was actually writing QMail spam filters, the first spam filters in QMail.

Corey: Ohh, you were on that side of the world.

Charity: Yeah. So, I love that [BLEEP], and when it was—oh, [BLEEP], how many years ago? A lot of years ago. I was still a teenager—and they wanted to move our mail to Google, and I was like, “No.” And I wrote this long and passionate email about why it was terrible to outsource our email to Google. And fundamentally, I still wanted to be able to grep my mail spool. Anyway, TL;DR we moved to Google, I quickly saw the light, and the world has been outsourcing more and more ever since.

Corey: Right. Personally, I finally shut off my mail server in my rack and moved it to Google, and suddenly I had hours a week back that I was spending playing whack-a-mole with spammers.

Charity: I know, it's amazing. When you free up time from the optional things to work on the things that actually matter to you, it's a good trend, I'm all for it. So, the last time we had a downturn, 10 years ago, we saw a lot more of this. And the way I feel about it is, God knows I have tried—I see people out there going, “I’m going to hire an observability team, and they're going to build it from scratch for us.” And I'm just like, “Oh, you sweet summer child.” 

But because we are not very good at quantifying the costs in human terms, this keeps happening. It’s kind of like a vanity project, almost, where people would just be like—you know, they want to have more teams underneath them, or they want to feel like their workload is so special, and so one of a kind, and so bespoke, and so they need to do all these things themselves. And they don't think critically about the cost involved. Well, during downturns, all of that changes, and all of a sudden, people are looking at the fact that they have a hiring freeze for the foreseeable future, if not letting some percentage of their team go, and they're still going to be responsible for the same amount of work, if not more, and suddenly the value of focusing on their core business initiatives—the things that are differentiators for them as business—becomes manifestly clear. And the argument for outsourcing major parts of their business—like, email is still very core to everyone's business, right?—becomes very, very compelling. 

And so these great waves happen of retrenchment, where people move things to the Cloud, move things to other providers, and then when the money comes back, they don't get reversed because it was always a stronger argument all along. It's just that we don't really know how to assign actual dollars and numbers to it, so it's still a very manager-driven instinct, intuition-driven process of allocating people to effort. But what we've seen at Honeycomb is very much this: that a lot of companies were, kind of, in the—like, large, mid-size tech companies large enough to maybe start to preen and go, “Oh, we're so special,” but actually not—those people were planning on going and hiring teams to do observability, and instead, they're coming to us. And they're especially coming to us because we're very much associated with the next way of doing things, and new things get a lot of heat—sometimes deservedly. God knows I've [BLEEP] all over serverless enough—but one thing they do tend to be is cheaper.

Corey: By a lot.

Charity: By a lot. Orders of magnitude.

Corey: Oh, yeah, the most expensive thing at any company never shows up in the AWS bill, and that is the engineering time gone into wrangling these services that show up on your AWS bill.

Charity: Yes, thank you. Absolutely true. And the newer ways of doing things are cheaper from a financial perspective. And it's almost like, we don't have to think about that while times are good. We think about all the other things—because think about all of the factors that go into making a technical or architectural decision. There's so many. What does your team like? What languages do you know? What stuff do you have provisioned? What other stuff do you already have written? All this stuff, and it can become very muddled. And in downturns, the pure cost of doing business, and writing code, and shipping value to customers, stands out.

Corey: And that's what it comes down to is value to customers, on some level. Most of us have this insane thing beaten into our skulls, where—we talked about learning these things, and how we start out going down these paths. We’re generally hobbyists playing around, and our time is free; computers cost money. That's no longer true at all in any professional context.

Charity: No. But because we love what we do, it still feels free to us.

Corey: Exactly. So, “Oh, why would I pay extra for that managed service? I can build it myself on top of EC2.” And then people set out to do it.

Charity: “And I enjoy it. And I know all the ins and outs,” and blah, blah, blah, blah, blah. Yeah, exactly.

Corey: Right. And half the time, I come in and I see an environment like that, and I understand intrinsically why it is the way that it is, and how it came to be that way, but it's not a good idea. And trying to convince folks of this is sort of an uphill battle because people have to learn this on their own.

Charity: I mean, yes, and no. I'll give the old guard a little bit of credit in that the costs of learning a new system can be quite high. I do think these need to be, kind of, step functions. I think it's fine to invest in technology, and a architecture model, and a set of technologies for a while until they start to—you know, you kind of want to standardize on a golden path until they start to show their age, which could be three years, right? [laughs]. But every three to five years, you're going to need to renew and refresh it radically from scratch.

Corey: And that's an awful lot of…

Charity: It's kind of the ‘optimize locally versus optimize globally’ tension.

Corey: Yeah. And the problem, of course, is I look at this, how in the world, am I going to be able to intelligently advise companies to do this on a one-off basis when I see this pattern happening again, and again, and again, everywhere? It feels in some cases like I'm shouting into the wind.

Charity: Yeah, well, I think you have been because they've had more money than sense. And now for a time they won't.

Corey: And we've always been waiting for the next correction. I don’t think we expected it to look quite like this. But now it's a question of, “Okay, if we have to reduce headcount or stall on hiring, which things can we transition to a managed provider that yes, will cost us more in raw infrastructure terms, but reduce the expensive engineering time by 10x?”

Charity: Yes. Some reporter was asking me, “Are people going to be backing away from the Cloud?” I'm like, “No,” [laughs]. “No, the sticker cost it is, like, a 10th of the actual cost.” Most of the cost is an engineering time and an opportunity time: you not doing the things that could have made your business succeed because you're doing the [BLEEP] infrastructure.

Corey: Exactly. “Will people be backing away from the Cloud?” “Well, not smart, people.”

Charity: [laughs]. Not smart people. Exactly. And serverless is an order of magnitude or more cheaper than even containers. Now, you can't use that for everything, and maybe it didn't make sense for a long time to rewrite parts of your infra in that, but we wrote our database in serverless, basically. It’s S3 files and Lambda functions, and it fell in cost by an order of magnitude.

Corey: And that is something I'm finding that’s—it—just people look at cost through the wrong lens. They're worried about lock-in. Well, great. So, you're going to build this thing—

Charity: It's easy to get hire.

Corey: Oh, yeah.

Charity: It’s easier to get headcount. So.

Corey: And finally, we're starting to see that change a bit—I wish it were happier circumstances—but we see Kubernetes, too, where it feels like all these people—like, that's been sort of the refuge for engineers who want to build their own cloud provider, but don't actually work at a cloud provider, so here we go.

Charity: I have been calling Kubernetes resume-driven development because nobody needs it but all the engineers think they need it on their resume in order to get jobs. So…

Corey: And if you're working on Kubernetes all the time, who's minding the store?

Charity: I know. What the [BLEEP] are you even doing? You know, funny story. Actually got leaked a couple of slide decks from startups: founders who are out there pitching basically Honeycomb for Kubernetes, or observability for Kubernetes, which is missing the point in such an enormous way because the point is supposed to be that it's boring. The thing that serves the code is supposed to be as boring as possible. It's inside your code and inside the systems that is supposed to be interesting.

Corey: Exactly. If you get excited about the things that Honeycomb does, for example, past a certain point, you probably should apply to work there.

Charity: Yeah. Well, lots of people do. But anyway.

Corey: [laughs]. You should not try and roll it yourself from first principles inside of your own company.

Charity: Yeah.

Corey: There’s, like, five companies in the world that might want to do something like that intelligently, and the rest are, again, trying to reinvent things from first principles that they best should not.

Charity: Sadly, all the people who are out there hiring, quote-unquote “observability teams” are hiring to staff them with the last generation of time series database folks who are just like, “Oh, this is a nail? I have a hammer.” And I think they're going to be not too happy with the results.

Yeah, I don't know. I think that some downward pressure is—you know, creativity loves constraints. And I feel like for too long in engineering, we haven't had constraints in terms of the resources. And I don't want to minimize the amount of pain that's out there right now, or people losing their jobs. That sucks a lot, and I feel for you. 

Engineers, I think you're going to be okay. I mean, I think the other part of this downturn is folks are going to distributed teams in a real way, which is exciting. Engineers are going to be able to get jobs for the most part. And I think it's good to have some downward pressure on costs. But every crisis is an opportunity, and I think that's it companies that have learned these lessons well, and have not forgotten them between downturns have been the companies that you really look to who are doing things well, who are doing things in ways that are sustainable, who don't have to make many corrections when these things hit.

Corey: And I hope that that winds up, ideally, breeding a culture that's a bit more aligned around this going forward.

Charity: I would hope so.

Corey: I don't want to go back to the way things were and seeing these ridiculous things that are pitched with absolutely no business model.

Charity: I know. I know. At some point engineers, we have to come into the fold as a part of the business. Engineering exists to serve the business. We exist to build things for users. We don't exist for our own entertainment. There are plenty of ways you can write code for fun if that's what you were trying to do, but at work, we have a responsibility to not just take these things into account but to do better at learning how to verbalize them, how to spread them as our values, how to make them something that is not just passed along as an oral tradition the way I feel like it is right now, but actually, professional standard.

Corey: Wouldn't that be the dream for all of us?

Charity: We're moving in that direction. It's just a question of how quickly.

Corey: We certainly seem to be. So, thank you so much for taking the time to chat with me about this. If people care more about what you have to say, where can they find you?

Charity: Oh, gosh, I have no idea. [laughs].

Corey: Right?

Charity: I'm all over the internet. You can find me, my personal blog is at charity.wtf. Honeycomb is at honeycomb.io. Our blog on the Honeycomb site has a lot of observability stuff. And basically I'm talking [BLEEP] all over, so.

Corey: Which is delightful and entertaining. If you enjoy the theme of my nonsense, I definitely recommend you check out Charity, should you not be fortunate enough to be already aware of it.

Charity: We have similar nonsense themes, it is true.

Corey: Well, thank you once again, I really do appreciate it.

Charity: Thanks, Corey. It's been fun.

Corey: Of course. Charity Majors, CTO and co-founder of Honeycomb. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts, whereas if you've hated it, please leave a five-star review on Apple Podcasts along with a comment telling me which system you're building yourself from first principles.

Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at ScreamingintheCloud.com, or wherever fine snark is sold.

This has been a HumblePod production. Stay humble.