Creating Value in Incident Management with Robert Ross

Episode Summary

Robert Ross, CEO and Co-Founder at FireHydrant, joins Corey on Screaming in the Cloud to discuss how being an on-call engineer fighting incidents inspired him to start his own company. Robert explains how FireHydrant does more than just notify engineers of an incident, but also helps them to be able to effectively put out the fire. Robert tells the story of how he “accidentally” started a company as a result of a particularly critical late-night incident, and why his end goal at FireHydrant has been and will continue to be solving the problem, not simply choosing an exit strategy. Corey and Robert also discuss the value and pricing models of other incident-reporting solutions and Robert shares why he feels surprised that nobody else has taken the same approach FireHydrant has. 

Episode Show Notes & Transcript

About Robert

Robert Ross is a recovering on-call engineer, and the CEO and co-founder at FireHydrant. As the co-founder of FireHydrant, Robert plays a central role in optimizing incident response and ensuring software system reliability for customers. Prior to founding FireHydrant, Robert previously contributed his expertise to renowned companies like Namely and Digital Ocean. 

Links Referenced:


Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Developers are responsible for more than ever these days. Not just the code they write, but also the containers and cloud infrastructure their apps run on. And a big part of that responsibility is app security — from code to cloud. That’s where Snyk comes in. Snyk is a frictionless security platform that meets teams where they are, automating application security controls across their existing tools, workflows, and the AWS application stack — including seamless integrations with AWS CodePipeline, Amazon EKS, Amazon Inspector and several others. I’m a customer myself. Deploy on AWS. Secure with Snyk. Learn more at That’s S-N-Y-K-dot-C-O/scream.

Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. And this featured guest episode is brought to us by our friends at FireHydrant and for better or worse, they’ve also brought us their CEO and co-founder, Robert Ross, better known online as Bobby Tables. Robert, thank you for joining us.

Robert: Super happy to be here. Thanks for having me.

Corey: Now, this is the problem that I tend to have when I’ve been tracking companies for a while, where you were one of the only people that I knew of at FireHydrant. And you kind of still are, so it’s easy for me to imagine that, oh, it’s basically your own side project that turned into a real job, sort of, side hustle that’s basically you and maybe a virtual assistant or someone. I have it on good authority—and it was also signaled by your Series B—that there might be more than just you over there now.

Robert: Yes, that’s true. There’s a little over 60 people now at the company, which is a little mind-boggling for me, starting from side projects, building this in Starbucks to actually having people using the thing and being on payroll. So, a little bit of a crazy thing for me. But yes, over 60.

Corey: So, I have to ask, what is it you folks do? When you say ‘fire hydrant,’ the first thing that I think I was when I was a kid getting yelled at by the firefighter for messing around with something I probably shouldn’t have been messing around with.

Robert: So, it’s actually very similar where I started it because I was messing around with software in ways I probably shouldn’t have and needed a fire hydrant to help put out all the fires that I was fighting as an on-call engineer. So, the name kind of comes from what do you need when you’re putting out a fire? A fire hydrant. So, what we do is we help people respond to incidents really quickly, manage them from ring to retro. So, the moment you declare an incident, we’ll do all the timeline tracking and eventually help you create a retrospective at the very end. And it’s been a labor of love because all of that was really painful for me as an engineer.

Corey: One of the things that I used to believe was that every company did something like this—and maybe they do, maybe they don’t—I’m noticing these days an increasing number of public companies will never admit to an incident that very clearly ruined things for their customers. I’m not sure if they’re going to talk privately to customers under NDAs and whatnot, but it feels like we’re leaving an era where it was an expectation that when you had a big issue, you would do an entire public postmortem explaining what had happened. Is that just because I’m not paying attention to the right folks anymore, or are you seeing a downturn in that?

Robert: I think that people are skittish of talking about how much reliability they—or issues they may have because we’re having this weird moment where people want to open more incidents like the engineers actually want to say we have more incidents and officially declare those, and in the past, we had these, like, shadow incidents that we weren’t officially going to say it was an incident, but was a pretty big deal, but we’re not going to have a retro on it so it’s like it didn’t happen. And kind of splitting the line between what’s a SEV1, when should we actually talk about this publicly, I think companies are still trying to figure that out. And then I think there’s also opposing forces. We talk to folks and it’s, you know, public relations will sometimes get involved. My general advice is, like, you should be probably talking about it no matter what. That’s how you build trust.

It’s trust, with incidences, lost in buckets and gained back in drops, so you should be more public about it. And I think my favorite example is a major CDN had a major incident and it took down, like, the UK government website. And folks can probably figure out who I’m talking about, but their stock went up the next day. You would think that a major incident taking down a large portion of the internet would cause your stock to go down. Not the case. They were on it like crazy, they communicated about it like crazy, and lo and behold, you know, people were actually pretty okay with it as far as they could be at the end of the day.

Corey: The honest thing that really struck me about that was I didn’t realize that CDN that you’re referencing was as broadly deployed as it was. took some downtime as a result of this.

Robert: Yeah.

Corey: It’s, “Oh, wow. If they’re in that many places, I should be taking them more seriously,” was my takeaway. And again, I don’t tend to shame folks for incidents because as soon as you do that, they stopped talking about them. They still have them, but then we all lose the ability to learn from them. I couldn’t help but notice that the week that we’re recording this, so there was an incident report put out by AWS for a Lambda service event in Northern Virginia.

It happened back in June, we’re recording this late in October. So, it took them a little bit of time to wind up getting it out the door, but it’s very thorough, very interesting as far as what it talks about as far as their own approach to things. Because otherwise, I have to say, it is easy as a spectator slash frustrated customer to assume the absolute worst. Like, you’re sitting around there and like, “Well, we have a 15-minute SLA on this, so I’m going to sit around for 12 minutes and finish my game of solitaire before I answer the phone.” No, it does not work that way. People are scrambling behind the scenes because as systems get more complicated, understanding the interdependencies of your own system becomes monstrous.

I still remember some of the very early production engineering jobs that I had where—to what you said a few minutes ago—oh, yeah, we’ll just open an incident for every alert that goes off. Then we dropped a [core switch 00:05:47] and Nagio sent something like 8000 messages inside of two minutes. And we would still, 15 years later, not be done working through that incident backlog had we done such a thing. All of this stuff gets way harder than you would expect as soon as your application or environment becomes somewhat complicated. And that happens before you realize it.

Robert: Yeah, much faster. I think that, in my experience, there’s a moment that happens for companies where maybe it’s the number of customers you have, number of servers you’re running in production, that you have this, like, “Oh, we’re running a big workload right now in a very complex system that impacts people’s lives, frankly.” And the moment that companies realize that is when you start to see, like, oh, process change, you build it, you own it, now we have an SRE team. Like, there’s this catalyst that happens in all of these companies that triggers this. And it’s—I don’t know, from my perspective, it’s coming at a faster rate than people probably realize.

Corey: From my perspective, I have to ask you this question, and my apologies in advance if it’s one of those irreverent ones, but do you consider yourself to be an observability company?

Robert: Oh, great question. No. No, actually. We think that we are the baton handoff between an observability tool and our platform. So, for example, we think that that’s a good way to kind of, you know, as they say, monitor the system, give reports on that system, and we are the tool that based on that monitor may be going off, you need to do something about it.

So, for example, I think of it as like a smoke detector in some cases. Like, in our world, like that’s—the smoke detector is the thing that’s kind of watching the system and if something’s wrong, it’s going to tell you. But at that point, it doesn’t really do anything that’s going to help you in the next phase, which is managing the incident, calling 911, driving to the scene of the fire, whatever analogies you want to use. But I think the value-add for the observability tools and what they’re delivering for businesses is different than ours, but we touch each other, like, very much so.

Corey: Managing an incident when something happens and diagnosing what is the actual root cause of it, so to speak—quote-unquote, “Root cause.” I know people have very strong opinions on—

Robert: Yeah, say the word [laugh].

Corey: —that phrase—exactly—it just doesn’t sound that hard. It is not that complicated. It’s, more or less, a bunch of engineers who don’t know what they’re actually doing, and why are they running around chasing this stuff down is often the philosophy of a lot of folks who have never been in the trenches dealing with these incidents themselves. I know this because before I was exposed to scale, that’s what I thought and then, oh, this is way harder than you would believe. Now, for better or worse, an awful lot of your customers and the executives at those customers did, for some strange reason, not come up through production engineering as the thing that they’ve done. They are executives, so it feels like it would be a challenging conversation to have with them, but one thing that you’ve got in your back pocket, which I always love talking to folks about, is before this, you were an engineer and then you became a CEO of a reasonably-sized company. That is a very difficult transition. Tell me about it.

Robert: Yeah. Yeah, so a little of that background. I mean, I started writing code—I’ve been writing code for two-thirds of my life. So, I’m 32 now; I’m relatively young. And my first job out of high school—skipping college entirely—was writing code. I was 18, I was working in a web dev shop, I was making good enough money and I said, you know what? I don’t want to go to college. That sounds—I’m making money. Why would I go to college?

And I think it was a good decision because I got to be able—I was right kind of in the centerpiece of when a lot of really cool software things were happening. Like, DevOps was becoming a really cool term and we were seeing the cloud kind of emerge at this time and become much more popular. And it was a good opportunity to see all this confluence of technology and people and processes emerge into what is, kind of like, the base plate for a lot of how we build software today, starting in 2008 and 2009. And because I was an on-call engineer during a lot of that, and building the systems as well, that I was on call for, it meant that I had a front-row seat to being an engineer that was building things that was then breaking, and then literally merging on GitHub and then five minutes later [laugh], seeing my phone light up with an alert from our alerting tool. Like, I got to feel the entire process.

And I think that that was nice because eventually one day, I snapped. And it was after a major incident, I snapped and I said, “There’s no tool that helps me during this incident. There’s no tool that kind of helps me run a process for me.” Because the only thing I care about in the middle of the night is going back to bed. I don’t have any other priority [laugh] at 2 a.m.

So, I wanted to solve the problem of getting to the fire faster and extinguishing it by automating as much as I possibly could. The process that was given to me in an outdated Confluence page or Google Doc, whatever it was, I wanted to automate that part so I could do the thing that I was good at as an engineer: put out the fire, take some notes, and then go back to bed, and then do a retrospective sometime next day or in that week. And it was a good way to kind of feel the problem, try to build a solution for it, tweak a little bit, and then it kind of became a company. I joke and I say on accident, actually.

Corey: I’ll never forget one of the first big, hairy incidents that I had to deal with in 2009, where my coworker had just finished migrating the production environment over to LDAP on a Thursday afternoon and then stepped out for a three-day weekend, and half an hour later, everything started exploding because LDAP will do that. And I only had the vaguest idea of how LDAP worked at all. This was a year into my first Linux admin job; I’d been a Unix admin before that. And I suddenly have the literal CEO of the company breathing down my neck behind me trying to figure out what’s going on and I have no freaking idea of myself. And it was… feels like there’s got to be a better way to handle these things.

We got through. We wound up getting it back online, no one lost their job over it, but it was definitely a touch-and-go series of hours there. And that was a painful thing. And you and I went in very different directions based upon experiences like that. I took a few more jobs where I had even worse on-call schedules than I would have believed possible until I started this place, which very intentionally is centered around a business problem that only exists during business hours. There is no 2 a.m. AWS billing emergency.

There might be a security issue masquerading as one of those, but you don’t need to reach me out of business hours because anything that is a billing problem will be solved in Seattle’s timeline over a period of weeks. You leaned into it and decided, oh, I’m going to start a company to fix all of this. And okay, on some level, some wit that used to work here, wound up once remarking that when an SRE doesn’t have a better idea, they start a monitoring company.

Robert: [laugh].

Corey: And, on some level, there’s some validity to it because this is the problem that I know, and I want to fix it. But you’ve differentiated yourself in a few key ways. As you said earlier, you’re not an observability company. Good for you.

Robert: Yeah. That’s a funny quote.

Corey: Pete Cheslock. He has a certain way with words.

Robert: Yeah [laugh]. I think that when we started the company, it was—we kind of accidentally secured funding five years ago. And it was because this genuinely was something I just, I bought a laptop for because I wanted to own the IP. I always made sure I was on a different network, if I was going to work on the company and the tool. And I was just writing code because I just wanted to solve the problem.

And then some crazy situation happened where, like, an investor somehow found FireHydrant because they were like, “Oh, this SRE thing is a big space and incidents is a big part of it.” And we got to talking and they were like, “Hey, we think what you’re building is valuable and we think you should build a company here.” And I was—like, you know, the Jim Carrey movie, Yes Man? Like, that was kind of me in that moment. I was like, “Sure.” And here we are five years later. But I think the way that we approached the problem was let’s just solve our own problem and let’s just build a company that we want to work at.

And you know, I had two co-founders join me in late 2018 and that’s what we told ourselves. We said, like, “Let’s build a company that we want to work for, that solves problems that we have had, that we care about solving.” And I think it’s worked out, you know? We work with amazing companies that use our tool—much to their chagrin [laugh]—multiple times a day. It’s kind of a problem when you build an incident response tool is that it’s a good thing when people are using it, but a bad thing for them.

Corey: I have to ask of all of the different angles to approach this from, you went with incident management as opposed to focusing on something that is more purely technical. And I don’t say that in any way that is intended to be sounding insulting, but it’s easier from an engineering mind to—having been one myself—to come up with, “Here’s how I make one computer talk to his other computer when the following event happens.” That’s a much easier problem by orders of magnitude than here’s how I corral the humans interacting with that computer’s failure to talk to another computer in just the right way. How did you get onto this path?

Robert: Yeah. The problem that we were trying to solve for it was the getting the right people in the room problem. We think that building services that people own is the right way to build applications that are reliable and stable and easier to iterate on. Put the right people that build that software, give them, like, the skin in the game of also being on call. And what that meant for us is that we could build a tool that allowed people to do that a lot easier where allowing people to corral the right people by saying, “This service is broken, which powers this functionality, which means that these are the people that should get involved in this incident as fast as possible.”

And the way we approached that is we just built up part of our functionality called Runbooks, where you can say, “When this happens, do this.” And it’s catered for incidents. So, there’s other tools out there, you can kind of think of as, like, we’re a workflow tool, like Zapier, or just things that, like, fire webhooks at services you build and that ends up being your incident process. But for us, we wanted to make it, like, a really easy way that a project manager could help define the process in our tool. And when you click the button and say, “Declare Incident: LDAP is Broken,” and I have a CEO standing behind me, our tool just would corral the people for you.

It was kind of like a bat signal in the air, where it was like, “Hey, there’s this issue. I’ve run all the other process. I just need you to arrive at and help solve this problem.” And we think of it as, like, how can FireHydrant be a mech suit for the team that owns incidents and is responsible for resolving them?

Corey: There are a few easier ways to make a product sound absolutely ridiculous than to try and pitch it to a problem that it is not designed to scale to. What is the ‘you must be at least this tall to ride’ envisioning for FireHydrant? How large slash complex of an organization do you need to be before this starts to make sense? Because I promise, as one person with a single website that gets no hits, that is probably not the best place for—

Robert: Probably not.

Corey: To imagine your ideal user persona.

Robert: Well, I’m sure you get way more hits than that. Come on [laugh].

Corey: It depends on how controversial I’m being in a given week.

Robert: Yeah [laugh].

Corey: Also, I have several ridiculous, nonsense apps out there, but honestly, those are for fun. I don’t charge people for them, so they can deal with my downtime till I get around to it. That’s the way it works.

Robert: Or, like, spite-visiting your website. No it’s—for us, we think that the ‘must be this tall’ is when do you have, like, sufficiently complicated incidents? We tell folks, like, if you’re a ten-person shop and you have incidents, you know, just use our free tier. Like, you need something that opens a Slack channel? Fine. Use our free tier or build something that hits the Slack API [unintelligible 00:18:18] channel. That’s fine.

But when you start to have a lot of people in the room and multiple pieces of functionality that can break and multiple people on call, that’s when you probably need to start to invest in incident management. Because it is a return on investment, but there is, like, a minimum amount of incidents and process challenges that you need to have before that return on investment actually, I would say, comes to fruition. Because if you do think of, like, an incident that takes downtime, or you know, you’re a retail company and you go down for, let’s say, ten minutes, and your number of sales per hour is X, it’s actually relatively simple for that type of company to understand, okay, this is how much impact we would need to have from an incident management tool for it to be valuable. And that waterline is actually way—it’s way lower than I think a lot of people realize, but like you said, you know, if you have a few 100 visitors a day, it’s probably not worth it. And I’ll be honest there, you can use our free tier. That’s fine.

Corey: Which makes sense. It’s challenging to wind up-sizing things appropriately. Whenever I look at a pricing page, there are two things that I look for. And incidentally, when I pull up someone’s website, I first make a beeline for pricing because that is the best way I found for a lot of the marketing nonsense words to drop away and it get down to brass tacks. And the two things I want are free tier or zero-dollar trial that I can get started with right now because often it’s two in the morning and I’m trying to see if this might solve a problem that I’m having.

And I also look for the enterprise tier ‘contact us’ because there are big companies that do not do anything that is not custom nor do they know how to sign a check that doesn’t have two commas in it. And whatever is between those two, okay, that’s good to look at to figure out what dimensions I’m expected to grow on and how to think about it, but those are the two tent poles. And you’ve got that, but pricing is always going to be a dark art. What I’ve been seeing across the industry. And if we put it under the broad realm of things that watch your site and alert you and help manage those things, there are an increasing number of, I guess what I want to call component vendors, where you’ll wind up bolting together a couple dozen of these things together into an observability pipeline-style thing, and each component seems to be getting extortionately expensive.

Most of the wake-up-in-the-middle-of-the-night services that will page you—and there are a number of them out there—at a spot check of these, they all cost more per month per user than Slack, the thing that most of us to end up living within. This stuff gets fiendishly expensive, fiendishly quickly, and at some point, you’re looking at this going, “The outage is cheaper than avoiding the outage through all of these things. What are we doing here?” What’s going on in the industry, other than ‘money printing machine stopped going brrr’ in quite the same way?

Robert: Yeah, I think that for alerting specifically, this is a big part of, like, the journey that we wanted to have in FireHydrant was like, we also want to help folks with the alerting piece. So, I’ll focus on that, which is, I think that the industry around notifying people for incidents—texts, call, push notifications, emails, there’s a bunch of different ways to do it—I think where it gets really crazy expensive as in this per-seat model that most of them seem to have landed on. And we’re per-seat for, like, the core platform of FireHydrant—so you know, before people spite-visit FireHydrant, look at our pricing pitch—but we’re per-seat there because the value there is, like, we’re the full platform for the service catalog retrospectives, Runbooks, like, there’s a whole other component of FireHydrant—status pages—but when it comes to alerting, like, in my opinion, that should be active user for a few reasons. I think that if you’re going to have people responding to incidents and the value from us is making sure they get to that incident very quickly because we wake them up in the middle of the night, we text them, we call them we make their Hue lights turn red, whatever it is, then that’s, like, the value that we’re delivering at that moment in time, so that’s how we should probably invoice you.

And I think that what’s happened is that the pricing for these companies, they haven’t innovated on the product in a way that allows them to package that any differently. So, what’s happened, I think, is that the packaging of these products has been almost restrictive in the way that they could change their pricing models because there’s nothing much more to package on. It’s like, cool there’s an alerting aspect to this, but that’s what people want to buy those tools for. They want to buy the tool so it wakes them up. But that tool is getting more expensive.

There was even a price increase announced today for a big one [laugh] that I’ve been publicly critical of. That is crazy expensive for a tool that texts you and call you. And what peo—what’s going on now are people are looking, they’re looking at the pricing sheet for Twilio and going, “What the heck is going on?” Like, I—to send a text on Twilio in the United States is fractions of a penny and here we are paying $40 a user for that person to receive six texts that month because of a webhook that hit an HCP server and, like, it’s supposed to call that person? That’s kind of a crazy model if you think about it. Like, engineers are kind of going, “Wait a minute. What’s up here?” Like, and when engineers start thinking, “I could build this on a weekend,” like, something’s wrong, like, with that model. And I think that people are starting to think that way.

Corey: Well engineers, to be fair, will think that about an awful lot of stuff.

Robert: Anything. Yeah, they [laugh]—

Corey: I’ve heard it said about Dropbox, Facebook, the internet—

Robert: Oh, Dropbox is such a good one.

Corey: BGP. Yeah okay, great. Let me know how that works out for you.

Robert: What was that Dropbox comment on Hacker News years ago? Like, “Just set up NFS and host it that way and it’s easy.” Right?

Corey: Or rsync. Yeah—

Robert: Yeah, it was rsync.

Corey: What are you going to make with that? Like, who’s going to buy that? Like, basically everyone for at least a time.

Robert: And whether or not the engineers are right, I think is a different point.

Corey: It’s the condescension dismissal of everything that isn’t writing the code that really galls, on some level.

Robert: But I think when engineers are thinking about, like, “I could build this on a weekend,” like, that’s a moment that you have an opportunity to provide the value in an innovative, maybe consolidated way. We want to be a tool that’s your incident management ring to retro, right? You get paged in the middle of the night, we’re going to wake you up, and when you open up your laptop, groggy-eyed, and like, you’re about to start fighting this fire, FireHydrant’s already done a lot of work. That’s what we think is, like, the right model do this. And candidly, I have no idea why the other alerting tools in this space haven’t done this. I’ve said that and people tend to nod in agreement and say like, “Yeah, it’s been—it’s kind of crazy how they haven’t approached this problem yet.” And… I don’t know, I want to solve that problem for folks.

Corey: So, one thing that I have to ask, you’ve been teasing on the internet for a little bit now is something called Signals where you are expanding your product into the component that wakes people up in the middle of the night, which in isolation, fine, great, awesome. But there was a company whose sole stated purpose was to wake people up in the middle of the night, and then once they started doing some business things such as, oh I don’t know, going public, they needed to expand beyond that to do a whole bunch of other things. But as a customer, no, no, no, you are the thing that wakes me up in the middle of the night. I don’t want you to sprawl and grow into everything else because if you’re going to have to pick a vendor that claims to do everything, well, I’ll just stay with AWS because they already do that and it’s one less throat to choke. What is that pressure that is driving companies that are spectacular at the one thing to expand into things that frankly, they don’t have the chops to pull off? And why is this not you doing the same thing?

Robert: Oh, man. The end of that question is such a good one and I like that. I’m not an economist. I’m not—like, that’s… I don’t know if I have a great comment on, like, why are people expanding into things that they don’t know how to do. It seems to be, like, a common thing across the industry at a certain point—

Corey: Especially particularly generative AI. “Oh, we’ve been experts in this for a long time.” “Yeah, I’m not that great at dodgeball, but you also don’t see me mouthing off about how I’ve been great at it and doing it for 30 years, either.”

Robert: Yeah. I mean, there was a couple ads during football games I watched. I’m like, “What is this AI thing that you just, like, tacked on the letter X to the end of your product line and now all of a sudden, it’s AI?” I have plenty of rants that are good for a cocktail at some point, but as for us, I mean, we knew that we wanted to do alerting a long time ago, but it does have complications. Like, the problem with alerting is that it does have to be able to take a brutal punch to the face the moment that AWS us-east-2 goes down.

Because at that moment in time, a lot of webhooks are coming your way to wake somebody up, right, for thousands of different companies. So, you do have to be able to take a very, very sufficient amount of volume instantaneously. So, that was one thing that kind of stopped us. In 2019 even, we wrote a product document about building an alerting tool and we kind of paused. And then we got really deep into incident management, and the thing that makes us feel very qualified now is that people are actually already integrating their alerting tools into FireHydrant today. This is a very common thing.

In fact, most people are paying for a FireHydrant and an alerting tool. So, you can imagine that gets a little expensive when you have both. So, we said, well, let’s help folks consolidate, let’s help folks have a modern version of alerting, and let’s build on top of something we’ve been doing very well already, which is incident management. And we ended up calling it Signals because we think that we should be able to receive a lot of signals in, do something correct with them, and then put a signal out and then transfer you into incident management. And yeah, we’re are excited for it actually. It’s been really cool to see it come together.

Corey: There’s something to be said for keeping it in a certain area of expertise. And people find it very strange when they reach out to my business partner and me asking, okay, so are you going to expand into Google Cloud or Azure or—increasingly, lately—Datadog—which has become a Fortune 500 board-level expense concern, which is kind of wild to me, but here we are—and asking if we’re going to focus on that, and our answer is no because it’s very… well, not very, but it is relatively easy to be the subject matter expert in a very specific, expensive, painful problem, but as soon as you start expanding that your messaging loses focus and it doesn’t take long—since we do you view this as an inherent architectural problem—where we’re saying, “We’re the best cloud engineers and cloud architects in the world,” and then we’re competing against basically everyone out there. And it costs more money a year for Accenture or Deloitte’s marketing budget than we’ll ever earn as a company in our entire lifetime, just because we are not externally boosted, we’re not putting hundreds of people into the field. It’s a lifestyle business that solves an expensive, painful problem for our customers. And that focus lends clarity. I don’t like the current market pressure toward expansion and consolidation at the cost of everything, including it seems, customer trust.

Robert: Yeah. That’s a good point. I mean, I agree. I mean, when you see a company—and it’s almost getting hard to think about what a company does based on their name as well. Like, names don’t even mean anything for companies anymore. Like Datadog has expanded into a whole lot of things beyond data and if you think about some of the alerting tools out there that have names of, like, old devices that used to attach to our hips, that’s just a different company name than what represents what they do.

And I think for us, like, incidents, that’s what we care about. That’s what I know. I know how to help people manage incidents. I built software that broke—sometimes I was an arsonist—sometimes I was a firefighter, it really depends, but that’s the thing that we’re going to be good at and we’re just going to keep building in that sphere.

Corey: I think that there’s a tipping point that starts to become pretty clear when companies focus away from innovating and growing and serving customers into revenue protection mode. And I think this is a cyclical force that is very hard to resist. But I can tell even having conversations like this with folks, when the way that a company goes about setting up one of these conversations with me, you came by yourself, not with a squadron of PR people, not with a whole giant list of talking points you wanted to go to, just, “Let’s talk about this stuff. I’m interested in it.”

As a company grows, that becomes more and more uncommon. Often, I’ll see it at companies a third the size of yours, just because there’s so much fear around everything we say must be spoken in such a way that it could never be taken in a negative way against us. That’s not the failure mode. The failure mode is that no one listens to you or cares what you have to say. At some point, yeah, I get the shift, but damned if it doesn’t always feel like it’s depressing.

Robert: Yeah. This is such great questions because I think that the way I think about it is, I care about the problem and if we solve the problem and we solve it well and people agree with us on our solution being a good way to solve that problem, then the revenue, like, happens because of that. I’ve gotten asked from, like, from VCs and customers, like, “What’s your end goal with FireHydrant as the CEO of the company?” And what they’re really asking is, like, “Do you want to IPO or be acquired?” That’s always a question every single time.

And my answer is, maybe, I don’t know, philosophical, but it’s, I think if we solve the problem, like, one of those will happen, but that’s not the end goal. Because if I aim at that, we’re going to come up short. It’s like how they tell you to throw a ball, right? Like they don’t say, aim at the glove. They say, like, aim behind the person.

And that’s what we want to do. We just want to aim at solving a problem and then the revenue will come. You have to be smart about it, right? It’s not a field of dreams, like, if you build it, like, revenue arrives, but—so you do have to be conscious of the business and the operations and the model that you work within, but it should all be in service of building something that’s valuable.

Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where should they go to find you, other than, you know, to their most recent incident page?

Robert: [laugh]. No, thanks for having me. So, to learn more about me, I mean, you can find me on Twitter on—or X. What do we call it now?

Corey: I call it Twitter because I don’t believe in deadnaming except when it’s companies.

Robert: Yeah [laugh]. if you want to find me there. If you want to learn more about FireHydrant and what we’re doing to help folks with incidents and incident response and all the fun things in there, it’s or, but we’ll redirect you to dot com.

Corey: And we will, of course, put a link to all of that in the [show notes 00:33:10]. Thank you so much for taking the time to speak with me. It’s deeply appreciated.

Robert: Thank you for having me.

Corey: Robert Ross, CEO and co-founder of FireHydrant. This featured guest episode has been brought to us by our friends at FireHydrant, and I’m Corey Quinn. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that will never see the light of day because that crappy platform you’re using is having an incident that they absolutely do not know how to manage effectively.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit to get started.

Newsletter Footer

Get the Newsletter

Reach over 30,000 discerning engineers, managers, enthusiasts who actually care about the state of Amazon’s cloud ecosystems.

"*" indicates required fields

This field is for validation purposes and should be left unchanged.
Sponsor Icon Footer

Sponsor an Episode

Get your message in front of people who care enough to keep current about the cloud phenomenon and its business impacts.