Liz is a developer advocate, labor and ethics organizer, and Site Reliability Engineer (SRE) with 16+ years of experience. She is an advocate at Honeycomb for the SRE and Observability communities, and previously was an SRE working on products ranging from the Google Cloud Load Balancer to Google Flights.



Links Referenced

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Are you better than the average bear with AWS? If you're listening to this podcast, the answer is almost certainly yes. Want to turn those skills into money? If you're US-based and have an AWS certification, sign up as an expert on AWS IQ today, and help customers with their problems., visit snark.cloud/IQ to learn more.

Corey: This episode is sponsored in part by our good friends over at ChaosSearch, which is a fully managed log analytics platform that leverages your S3 buckets as a data store with no further data movement required. If you're looking to either process multiple terabytes into petabyte scale of data a day or a few hundred gigabytes, this is still economical and worth looking into .You don't have to manage Elasticsearch yourself. If your ELK stack is falling over, take a look at using ChaosSearch for Log Analytics. Now, if you do a direct cost comparison, you're going to save 70 to 80 percent on the infrastructure costs, which does not include the actual expense of paying infrastructure people to mess around with running Elasticsearch themselves. You can take it from me or you can take it from many of their happy customers, but visit chaossearch.io today to learn more.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week once again by Liz Fong-Jones. Liz, welcome to the show.

Liz: Thank you for having me on.

Corey: So, you were one of a very small group of people who was on this show before it launched: you generally want to make sure you can have more than two or three conversations before launching what purports to be a weekly podcast, and you were generous enough to put up with my bumbling, dropping the microphone, et cetera, when I had no idea what I'm doing.

Liz: And to add a complication, I was working for a giant multinational cloud provider at the time that closely scrutinized everything I said.

Corey: Yes, which makes it extra fun. These days, however, I'm still fumbling but I don't drop the microphone because now I can afford a microphone stand. Other than that, a few things have changed. You are now no longer at said large company with incredible degrees of scrutiny. You're instead a principal developer advocate at Honeycomb, a company that everyone loves, or should love if they've heard of you, which again, they really should have by now.

Liz: We're pretty polarizing. Some people hate us.

Corey: So, tell me more about that. Who would hate Honeycomb, and on what axis?

Liz: I think that the large APM vendors are very, very scared of us, and therefore they are trying desperately to undercut our messaging. We also have a very colorful CTO who loves to make provocative statements on Twitter that push the envelope, and people tend to give her a lot of hate for that, as well. So, we're kind of in this interesting position of being now this scrappy startup that everyone loves to cheer for or hate, as opposed to the giant cloud company that people mostly hate on.

Corey: So, one thing I want to call out about Honeycomb is that I've worked with you folks in a variety of different ways, all public. One, you folks have sponsored this and other various productions over the last few years that I'm involved with, which is always appreciated. We love our sponsors; that's how that works. Second, you folks have been a public reference client for us at The Duckbill Group. You had some fantastic stuff going on in your AWS account that to be very honest, we learned an awful lot from as we went through it, and it really is nice to have folks who will get up and say, “Yes, in fact, we will talk in-depth about the things that we were able to do.” It's surprisingly challenging to find that in this space.

Liz: Yeah, definitely. Honeycomb has a culture of transparency with our customers, which means that we can talk about things like, “Hey, here are our outages. Here are cost savings.” These are things that we don't necessarily have to play super close to the chest.

Corey: And the last thing, which is not nearly as well known, but I've never for a minute forgotten it, is back when it occurred to me, “Hey, I should write a sarcastic insulting newsletter about AWS every week,” Charity Majors, your founder and CTO, wound up tweeting about it before I launched. And before that, I was on track to have a couple of people who I wasn't a blood relative of, sign up for the newsletter. And after she did it, over 500 people signed up. The first issue went to exactly 550 people. Now it's 20,000, but I have never forgotten that, and that was one of those transformative inflection points. And it didn't take much work on her part. It was a tweet that was it. It's the sort of thing that people will do without don't even really thinking about it. But it mattered a lot to me, and I've been effectively trying to repay that forward for the last couple of years as a result.

Liz: That's so awesome to hear when things that we do have a really positive impact upon people's lives, for sure.

Corey: So, let's get in a little bit to your perspective on, well, for lack of a better term, the cloud wars. You were at Google Cloud for a long time, and now you're in an AWS shop. You were one of the folks early on who left Google based upon, effectively, its labor practices. Is that right?

Liz: Yes, both its labor and its product ethics practices. And unfortunately, I can't say that AWS’s practices are much better in that department: they do the same kind of collaboration with the federal government on defense matters, they definitely do a lot of the supporting indirectly of Palantir, so there's no big winner among the cloud providers for ethics issues. But a matter of how I spend my precious labor versus who I pay for commodities and services, I think those are two very different axes.

Corey: My opinion on this is it's easy for me as someone who owns a small business that has no illusions of becoming a venture-backed monstrosity that takes over the world, and a $500 million exit is a bad exit. That's not the track that I'm on, so I have the luxury of being able to pick and choose who I do business with. So, when people like to rejoinder things I say with, “Oh, so your company's hands are completely clean?” Well, yeah. We don't do unethical things, but we're also, at time of this recording, seven people. So, I understand that I have the luxury of being able to take those positions. I'm not convinced it's possible to become a large, multinational publicly-traded company without, for lack of a better term, committing atrocities along the way.

Liz: This is the joys of existing under late-stage capitalism.

Corey: Indeed, and we hold this up as the one true path forward.

Liz: Yeah, right. Like, you know, you kind of can't throw stones from glass houses, but you can at least try to make those glass houses better. That's how I tend to approach things.

Corey: One thing I find really interesting is looking over time, as people's bios change as they change companies, now the one that you submitted for the show says that you're a developer advocate, labor and ethics organizer, and SRE with 16 years of experience. Labor and ethics organizer is one of those interesting things in a bio, specifically because you normally don't see it. With most companies looking at most folks who are on the job market, having something out there, you may as well put a racist slur into your bio, for as hireable as that's going to be in an awful lot of shops given our current approach. The fact that you're out there with something that is more or less poison from a lot of these big companies’ perspectives, is fascinating, and I really wish we saw more of it. The ethics and labor part, not the racist slur part.

Liz: Yeah, no kidding. I think that part of why I do that is to set an example using my financial privilege. I definitely had the financial privilege of when Google actually valued my labor and ethics organizing, which was five years ago it did, they paid me a lot of money to continue doing it. And therefore, I can choose to say no to any job that I want. I'm also very highly in demand, right? So, that kind of combination of financial privilege, and also having this profile in the community means that I have that capability to be this walking advertisement for, “Yes, I am going to make your workforce more ethical. Yes, I am going to organize your workforce and you have to be okay with that if you're going to hire me.”

Corey: On that note, if it's not too far of a bridge, how are you finding the, I guess, labor organizing effort at Honeycomb, given that you are not that much bigger than The Duckbill Group, last time I checked?

Liz: I think that it's definitely—companies reach an inflection point when they get past 50 employees, where it's no longer everyone closely knows each other; everyone no longer necessarily closely trusts management. At the moment, we are in a position where people feel like they can get issues addressed by talking directly to executives. And I think that part of my goal is to ensure that the right structures exist to support that feedback loop as the company grows. So, that's kind of the overarching lens that I'm looking at it through, with the understanding that we saw what happened with Kickstarter, we saw what happened with companies like Lanetix, we saw what happened to NPM. God bless the hearts of the employees at NPM who tried to unionize. The instant that your board or funders get wind that you are trying to unionize, using the word union, they will have no qualms about crushing your company. And I think that's something that I'm trying to be very careful around.

Corey: It turns out that a lot of what I thought I stood for isn't actually what I stand for, once I was able to refine that, and have a little bit of space. Easy example: when I thought that I was starting my company, my entire position was, well, I can't ever be political on Twitter—

Liz: [laughs].

Corey: —because if I do that, all I'm going to do is potentially alienate half of my potential customer base. And on some level, there is validity to that line of thought. On the other, things that I'm currently political about, for example, kids in cages. That's one of those things where if you're on the other side of that issue, I don't want your business to be very honest with you. I'm not saying that there's this very high moral bar that every company has to wind up passing. I'm not sitting here calculating out what the pay differential is between their CEO and their low paid employees—thought maybe there's an argument that I should be—there's a spectrum there. But at some point, you have to take a stand for something, or you're ultimately willing to do business with absolutely anyone that pays you. And down that path, lay things that are fundamentally monstrous.

Liz: Yeah, I think it's this interesting dilemma where boards certainly want maximum profitability unless you're a B Corp. Executives are torn between their own personal values and what the board is mandating that they do, but employees can exercise their conscience in a way that executives are not necessarily as beholden to. And you think that means that employee organizing is the best bulwark against companies doing an unethical thing because they kind of have a lot more flexibility to push back against the board in a way that executives can't do by themselves.

Corey: One of the most interesting aspects of smaller companies is how accessible everyone has to be out of necessity, for lack of a better term. And when I was at large companies, as one case through acquisition, I took that with me, and when someone who was a SVP or equivalent, said, “Oh, I have an open-door policy,” I took them at their word and I would come in and talk to him about things that were on my mind—

Liz: Ohhh.

Corey: —and they were always surprised that I’d done it. Because it’s, “Well, don't offer if you're not serious.” Now, yes, there's a tremendous amount of privilege, me as a cishet white dude being able to walk into a room and worst case, people think I'm being an aggressive go-getter and not stepping outside of my lane, as much. I definitely had that come back on me, once I did start stepping outside my lane, but I took advantage of that, and it turns out when people say that they don't actually mean it.

Liz: No, they get upset that you went over their head. Or they get upset that you're bugging them about something trivial.

Corey: Yeah, it's psychotic in some ways. So, moving back a little bit to the area of cloud, what's your take these days on AWS and/or GCP, since I'm going to bet that those are both companies and offerings you have opinions on?

Liz: Yes, I feel like I have a reasonably—well, no one has a good understanding of AWS’s is offering, let's be clear—but I have a competent understanding of both AWS and GCP’s offerings, and, you know, we are an AWS customer; we are going to be an AWS customer for the foreseeable future, but I also continue to follow what's happening over in GCP-land. And I think what's particularly interesting is incumbent effects are really, really serious. I think that, kind of, price is an area that AWS is putting pressure on GCP on, and I think that no amount of technical superiority of the GCP offering is going to be able to compensate for those factors, at least in the near term. So, what do I mean by these things? Well, I think that on the cost dynamic front, what we’re seeing is that the introduction of Gravitron2 on AWS has been a game-changer because it has meant that people can run the same workload at 40 percent off, 50 percent off. 

And that was one of the ways in which GCP had previously been able to win things was that, “Hey, Google has more efficient data centers, and therefore has a ability to offer more competitive pricing.” But that, kind of, no longer is something that is possible, as long as people are willing to recompile their applications for ARM. Google has no answer to that. Google only recently has introduced POWER on GCP. Which, you know, “Would you like a $20,000 per hour instance?” “Why, yes, you can have your IBM Power of $20,000 per hour instance.” Like that’s, kind of, been their only foray into non-x86. Google has made forays into, you know, you can run your giant SAP database. Yes, you can run your giant IBM mainframe power architecture workload on GCP. But Google has not done anything that, at least it's publicly visible, on the front of non-x86 architectures for general use, for general compute workloads. 

So, that means that unless they catch up, they're going to be at a cost disadvantage for people who care primarily about cost. The other area that I was talking about was incumbency effects where we are a data ingest provider. We ingest a lot of data, which means it's super cheap for us to ingest data because it's free in a lot of cases, although you helped us discover some cases in which it wasn’t, but it means that someone who is on GCP and is trying to send telemetry data to Honeycomb is paying a dramatically increased cost in order to egress that data over the public internet to Amazon. And I think that if we were in the inverse situation of Honeycomb being hosted on GCP, AWS customers would balk at the idea of paying eight cents per gigabyte to egress the data over the public internet article. And I think that that is a significant issue where these walled gardens in the form of eight-cent per gigabyte taxes really, really add up, and mean that you as a service provider, have to locate yourself where a majority of your customers are, or else you're going to wind up running multiple environments or having your customers foot the bill for paying through the nose.

Corey: There's a lot to unpack there. So, first, I think that Gravitron2 is fascinating. When I first heard about it, I thought, “Oh, yeah, that’s not for me. All the stuff I use is x86, why in the world would I want to use a different architecture?” Yeah, it turns out that there were exactly two command-line tools that I used that did not offer a already compiled ARM binary, and it was mostly due to oversights in both cases. One of them was an actual AWS-provided tool. Both of them do now, and from my perspective, there is no functional difference for anything I've ever written or built, and that's neat. So, it becomes a straight cost optimization story, as well as using something that's nifty and from the future. To be blunt it gives me optimism that at the time that we record this, there's been no formal announcement about Apple switching to ARM, but that no longer seems as far away as it once would have, to me.

Liz: Yeah, definitely all of the tech rumors are saying that it's going to happen. It will be a surprise if it doesn't happen.

Corey: At this point. Yes. Bloomberg does not generally make things up, unless they're talking about the big hack story a couple of years ago.

Liz: [laughs].

Corey: For those who weren't paying attention, that was that Amazon, Apple, and a couple of other companies had been compromised by microscopic implants in their mainboard. The sort of stuff that your racist uncle will send you on Facebook. And it was never substantiated; every named source repudiated what was said, and Bloomberg has never retracted the story. So, it’s, “Cool, I like to go on the internet and write fanfiction, too, sometimes, but I usually do it on Twitter, not Bloomberg, the cover story.” It was one of the most bizarre things, and it really damaged my impression of Bloomberg as a journalistic outlet at the time and still has echoes today.

Liz: Yeah. So, we were talking about how ARM is a game-changer, and it definitely is. We definitely were some of the earliest adopters of Gravitron2. We’re definitely cited in the press release, and it's been super, super exciting for us as a startup that now cares very, very much about our runway, right, series A in the middle of a pandemic recession. These things really, really add up and matter.

Corey: One other thing that we talked about a second ago was data transfer and it's ridiculous edge cases. One of my favorite things that has happened this year was both Zoom and Eight by Eight have done large public deals with Oracle Cloud. First, Oracle Cloud’s public data transfer pricing is one cent per gigabyte or less, depending. Whereas AWS’s starts at nine cents, so there's a massive cost differential there. Now that's fine. First, would I trust Oracle Cloud for something? Well, I don't know, but an 8 to 10x cheaper data transfer story, well, for some workloads, you have my attention, and I'm willing to do an experiment and see.

Liz: Yeah, for some workloads, the data transfer is the majority of the cost. It's not the compute.

Corey: For me, at least the funny part, though, was watching Amazon lose its collective mind when this happened. It's strange because I don't get the distinct impression, so far, and maybe I'm wrong on this, that Oracle is a serious, large-scale competitive threat to AWS, but for some reason, with that particular company, Amazon loses its mind. Andy Jassy—the AWS CEO—always takes a swipe at Oracle during his keynote talks. It feels like it's a also-ran company in the modern era. I do not understand this level of pathological hatred for Oracle. Now I get it from Oracle customers, don't get me wrong, but by the same token, airing that in public seems like a very odd choice. And whenever I see AWS get that animated about something, I start paying attention. So, I started digging into Oracle Cloud, and please don't tear me to pieces for this, Internet, but it's not all terrible.

Liz: I personally, because of my distaste for Oracle's business tactics, I've never tried Oracle Cloud. But I think in the broader networking space, definitely my experience from having worked in the past with Google Cloud’s networking, say what you will, it’s had some prominent ingest layer outages in the past couple of years, but overall, Google Cloud’s network, from a latency perspective, is significantly better than Amazon's Ingress, than Oracle's Ingress. I will take Google Cloud’s user-facing network over any other cloud. There's a reason that that is a premium feature. And then they offered, hey, by the way, if you want this same shitty level of reliability you get out of every cloud provider, we’ll discount it by, like, half. So, I think that that's this interesting thing where, if you care about quality of service, you're going to pay the same price for egress that you would at Amazon, and you'll get a much better latency experience. Or you could pay less, and get the same experience as on other clouds.

Corey: This episode is brought to you by Trend Micro Cloud One™. A security services platform for organizations building in the Cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? I'm glad we have Trend Micro Cloud One™ a security services platform for organizations building in the Cloud, or, “Hey, bad news. It's going to be a few more weeks. I kind of forgot about that security thing.” I thought so. Trend Micro Cloud One™ is an automated, flexible all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline, and access your cloud environments sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One™ a security services platform for organizations building in the Cloud. Whew. At trendmicro.com/screaming.

Corey: With AWS, it's, “We have one network, it's awesome, and you will pay a premium for it.” Well, great; I have workloads where I don't necessarily care how reliable it is, or even necessarily how fast it is. I want to get this data from here to over there, and I don't care if it takes you a month to do it. It just needs to get there at some point. Don't charge me through the nose for it. And unlike almost everything else in AWS, there is no dial on that.

Liz: Yeah, I think it's this dilemma of some things are latency-sensitive user-facing interactive service workloads, and others are not. Making that kind of prioritization is an important thing that people are realizing they have the need for, that I think GCP was ahead of the game there. Although GCP originally released the premium only and then went to premium and standard.

Corey: There's a lot to admire technologically about GCP.

Liz: There's a ton to admire technologically, it is built so solidly. I really, really respect the engineers who built it.

Corey: I feel like they're being let down on some level by the business leaders, to be very direct with you. Every time I talk to a customer who's eyeing GCP, they raise the same concern: “What if Google gets bored, or changes the pricing model in a way that dramatically impacts our business?” And every time I bring this up to people at GCP, generally, the response is usually something delivered in an accent heavy with condescension. And it's, oh, I clearly don't understand the grand vision. And maybe; maybe I don't, but I'm passing this on from customers I talk to who likewise don't understand it, and yelling at people does not turn them into customers. Believe me, I've tried it.

Liz: I think that from the strategic perspective, Google has invested so much into GCP that it's a “can't fail” project. I think that it is what Google's leaders are looking to as its next several billion-dollar business line, you know, separate from ads. So, I'm less worried about the company deciding it’s bored and it going away. When I am worried about is speed of execution. As of when I left Google, there were several hundred people working on Stackdriver. 

And it's like, Honeycomb is running circles around some of this with a dozen engineers, and its own dedicated sales and marketing team. Anything that this kind of velocity penalty of having everything be big company, whereas you can trust it with—Amazon, sometimes left hand doesn't talk to right hand, but they're at least a little bit more agile about launching stuff, and I feel like Google's kind of mired a lot in bureaucratic shit these days and kind of overbuilding these teams, and I worry about the squandering of resources, there.

Corey: It's one of the strangest things I've noticed about Google, is it's difficult to understand what it is that drives them in, I guess, anything beyond the, “This is technically fascinating,” or, “We found a new way to show ads that people don't want to see to them.” There's a lot there. I joke, conversely, that AWS has a product strategy consisting of a post-it note that says, “yes” on it. I will say that AWS is absolutely leaving Google in the dirt with respect to excelling at suing former employees under the auspices of non-compete agreements. It's clear that if Google wants to be serious about being a world-class cloud provider, they've got to start suing a lot more people who got them where they are under incredibly onerous, overbroad agreements, especially to keep their existing employees in line.

Liz: Yeah, in terms of keeping existing employees in line, like, going back to the first subject that we talked about. What Diane Greene saw was the potential to have a multi-billion dollar business doing business with the US Department of Defense. And employees said, “Hell, no, we're not doing that.” Google did retaliate against employees, but Google also did show Diane Greene in the door, in the end. She's no longer an employee of Google. Thomas Kurian is leading the organization now. 

But I think the problem was that all those eggs were in that one basket. That Diane Greene put all those eggs into that one basket—or I guess, into two baskets: defense business, and online e-retailers who don't want to pad Amazon's margins. Those are the two big investments that Diane Greene made as leader of GCP. And I think that Google now is having to pursue, “Okay, what is a plan for revenue look like, that is a little bit more diversified?”

Corey: That's really a sad thing to see on some level. It feels like, on the one hand, everyone's competing to see who can come up with more ridiculous things that don't really seem to move the needle. Serverless space is ripe with this. Kubernetes, eh, kind of. But if I look at—

Liz: Kubernetes, I think, was a strategic coup. It was successful, right? It forced people to cloud-neutral their workloads. Sure, you might still be tied to RDS. But—

Corey: I’m talking the ecosystem. Everything tied into Kubernetes, here so you can run Kubernetes on your Raspberry Pi. While you're already up, make—

Liz: Oh, goodness. [laughs]. God, the proliferation of Amazon product names, it's hilarious.

Corey: My opinion on Kubernetes on a cynical level is that it was, in some ways, an effort by Google to help the rest of the world write software in ways that were more Google-y.

Liz: In ways that were more Google-y, and that were less Amazon-specific, right? I think it was a successful project in that way.

Corey: You're not wrong. And a lot of the tenets that it gets at are extraordinarily valuable for companies to consider. My concern is it's still finicky, it's difficult, it’s an awful lot of painful things that need to be aligned just so, and it doesn't make it a fit for every workload, but just serverless one of its biggest challenges is its entire, I guess, hype-monster that lives behind it, pushing it for, whatever your problem is that I'm not listening to, here's the technology that will solve it for you.

Liz: And indeed, right? Like, Honeycomb is not a production Kubernetes user. That's why because we focus on utilizing whole machines, and running on whole machines because that's how our workload is. It's a very homogenous workload. And I think that the things that people under-discuss about Kubernetes is, when is it right? When is it wrong? And where Kubernetes is right is where you're trying to bin pack a heterogeneous workload. And there are a lot of organizations that do not fit that mold, and I think that those organizations are trying to adopt it just because it's flashy, and they're seeing mixed results.

Corey: One thing that I find neat about Honeycomb is on some level, you could be accused of some of the same things, whereas you're fundamentally trying to get people to change how they think about workloads in production. Now, it doesn't have the same force of a foundation that can more or less vote people off the island if it doesn't want to, behind it. But it's definitely gaining mindshare among people who are generally known for having made a career out of making good decisions. Tell me a little bit more about that.

Liz: Yeah, I think the discussions about test in production, and discussions of push on Fridays and discussions of instrument your code rather than monitoring what the CPU usage is on one machine, we're moving towards our opinions being a lot more mainstream rather than heretical, so I think that that's been really, really exciting to see, but it's gotten to the point where the large incumbent vendors are now kind of bandwagoning, and saying, “Oh, we do that too,” by bolting products together, by remarketing things, by trying to redefine observability. I think that that’s… imitation is a form of flattery, and I have to remind myself of that every single day. But I think the way that you get there is by demonstrating to people so they see it with their own eyes, with their own hands-on keyboards, with their hands on their braille readers. Is that observability is a better strategy that gives meaningfully better business results? If you don't have the results to show for its hype, I think that getting out of that hype-cycle into here are the concrete results that people are seeing, I think that's what's actually moving the needle is not just us talking about it, but people talking about their own experiences using Honeycomb; with their own experiences using distributed tracing, using wide events using ask-any-question-based observability tooling, even if it's not Honeycomb’s.

Corey: At the time that we're recording this, I spent some time last night doing a Twitter thread, as happens when I get bored, and I've scrolled to the end of the internet. It’s, “Okay, give me an AWS service and I'll explain it sensibly.” Someone said X-Ray, and it's someone who saw what Honeycomb was doing and wanted to do the same thing as an Amazon product. Unfortunately, their understanding was limited to yelling at people to do things differently, and not understanding the why behind it. And when I look through a lot of the tracing stories that are out there, it feels like there's a conceptual gap between where people are, and understanding the value of it. 

I've long said that one of Honeycombs biggest challenges is describing what they do to someone who does not themselves have an SRE background because if you wind up explaining this to someone who is making your coffee because, I don't know, you insisted on paying for your coffee with Bitcoin, and during the 15-minute transaction settlement process, you're now going to yell at them about observability, then it doesn't wind up having any impact because it sounds like easy problems to anyone who's not trying to solve them themselves.

Liz: Yeah, I think the challenges of explaining why the previous approaches didn't work, that's the real challenge is explaining to people why—oh, isn't it just as simple as doing x? The soundbite for why it matters is easy. The soundbite is, “We help your developers waste less time, and help your site be down less.” But the problem is people are used to doing it the old ways and don't understand that it's too costly to do it the old ways, that it's too slow, or that you just flat out can't understand your systems anymore. And I think that that's really been the challenge because people look at these disjointed tools that people have put together to try to solve the problem using their existing solutions, and they're like, “There's no way that you can make this work.” People have been burned by years and years and years of really bad distributed tracing, really bad instrumentation, with high overhead that's hard for your developers to understand, and then really poor analytical tools. And then when you're like, “Hey, if you just use the right backend data store, that's a column store, all these problems reduce to being very simple.” And people are like, “What planet are you from?” I think that's the challenge is explaining to people that this isn't science fiction, this is real.

Corey: One of the hardest parts for me is, I guess, getting people to internalize that the things that we talk about on Twitter do in fact have bearing in reality. And these lessons are all hard-won, but a lot of people will not appreciate the value of a lesson until they experience the pain themselves.

Liz: I know. I really wish people would learn from other people's hard experiences, rather than having to go through that pain themselves. I watch people try to scale up systems based entirely off of metrics, and fail. And it's like, “Can you not learn from the 100 other people who also fell down this rabbit hole? Like, seriously?” People repeating the same mistakes over and over. And some lessons, I guess, have to be learned the hard way. But other people are more receptive, and those are the people that are going to have more success because they listened early.

Corey: One last thing I want to call out because I generally refuse to be on podcasts with you unless I'm able to tell this story because it was such a transformative moment for me. One of the first times we ever spent time together was at SREcon EMEA—that was when I was just learning to live-tweet events in a humorous fashion before that became a standard bit—and someone got up and gave a talk from some giant multinational insurance company, or bank or whatnot, about their DevOps transformation, and I started dunking on what the company was doing, and you called me out for that and said that I was punching down. And sure the usual response is getting called out, at least for me at least, a flash of defensiveness that I've learned to ignore and move past because it's not productive, but then it was, how is that even possible? How could I be dunking, or punching down at a giant company? You were right, it is possible to do that. And when I dunk on companies, I try and do it not from a position of power. There are times when I will be actively insulting: AWS suing former employees, I will go after them with a vengeance for that because honestly, the people who work there deserve better.

Liz: That's what it is. It's about the difference between the corporate entity and the employees. I think that you can definitely wind up accidentally punching down at the employees when you're meaning to punch up at the company. And I think that's where we have to come at things from, is like, when you look, for instance, at Google having this reputation for being condescending. I think a lot about the fact that my colleagues who are in Google DevRel all wanted the best for users. That the engineers on product engineering teams all wanted the best for users. And that overall the thing that we shouldn't be criticizing is not the individual employee’s intentions, it's the system that, for instance, prevents feedback from being heard, or people feeling like it's hopeless to raise the issue of fixing individual bugs.

Corey: It's easy to lose sight of the fact that there are people behind these things. As my platform has broadened and gotten bigger. I have to be more constrained in what I say about new AWS service launches. Making fun of names is usually safe because no one spent 18 months at Amazon naming Systems Manager Session Manager. If they did, maybe they should feel crappy, but that can't be said for building the service. If you put blood, sweat, tears, passion into something and the first thing that happens is some jackass on the internet starts dunking on it, and everyone thinks they’re being hilarious, that doesn't feel good, and I don't want people to read my material and feel bad as a result, with a few very specific exceptions.

Liz: Yeah. Humor; not being a jerk.

Corey: Yeah, it’s, if the joke requires you to crap on someone, maybe the joke’s not that funny. So, where can people find you if they want to hear more about what you have to say about life, the universe, ethics, labor organizing, or anything else that crosses your mind?

Liz: People can find me at honeycomb.io/liz, at lizthegrey.com, or @lizthegrey on Twitter, spelled with an E rather than A, although I should probably register that domain name, too.

Corey: That would not be a terrible idea. As always, thank you so much for taking the time out of your day to speak with me. It's appreciated.

Liz: It's always fun talking to you, Corey.

Corey: It really is, isn't it? Liz Fong-Jones, principal developer advocate at Honeycomb. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. Whereas if you've hated this episode, please leave a five-star review on Apple Podcasts along with a comment telling me why observability is overrated, and prefer dead reckoning instead.

Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at ScreamingintheCloud.com, or wherever fine snark is sold.

This has been a HumblePod production. Stay humble.