- Cribl: https://cribl.io
- Cribl sandbox: https://sandbox.cribl.io
- Cribl.cloud: https://cribl.cloud
- Jobs: https://cribl.io/jobs
Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.io
Corey: And now for something completely different!
Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. My guest this week for this promoted episode is Clint Sharp, the CEO and co-founder of a company called Cribl. Clint, thank you for joining me, and let’s get the big question out of the way first: what is Cribl?
Clint: Yeah, so Cribl makes a stream processing engine for log and metric data. And that sounds really dry and boring, but what it really means is, we help connect, in the observability and security world, lots of log and metric sources, so you can take stuff from anywhere and put it to anywhere. And you can think of it like ETL or you can think of it like middleware; it sits there in this particular space, and it’s built for SRE and security people.
Clint: Yeah, so what’s interesting about enterprises is they care about money, and then they don’t care about money. And so it’s a really good way to get a meeting. We definitely do help people save a ton of money, but ultimately, I think what the value people get out of the product is helping connect all the things that they have. And so one of the biggest problems that we see in the spaces is, “Hey, I have all these agents deployed.” Maybe it’s Fluentd or Fluent Bit, or Elastic Beats or Splunk’s Forwarder.
Clint: Well, the data is the same. And it starts there because we’re collecting log data, and that log data may go into a SIEM tool, and people are using that to try to understand their security posture, and malicious actors, and threats. Oh, and by the way, that same log data is also used for understanding the performance and availability of your systems. The same type of metric data is used in both, the same type of catalogs that say, hey, what is my inventory, and what assets do I have, and where are they deployed? And all of that is relevant for both sides.
And the tooling often ends up being very similar, if not identical. And I used to work in Splunk many years ago; that’s a tool that’s well known for being popular in both camps. And so I developed this decade-long perspective of like, man, I’d show up and actually, they’re sitting right next to each other; there’s DevOps—DevSecOps now, which are now trying to marry those things. And so certainly, there’s just a ton of overlap.
Great, but it is a 40% raise so I’m going to be quiet about the purity of titles and take the money was my approach back then. And now there are 10 or 15 different ways you can refer to people who are more or less doing the same job and there’s no consistency between company to company in many respects. They almost become buzzwords and trite at some point, but it’s easier than trying to have a 15-minute conversation in response to, “So, what do you do at whatever company you work at?”
Clint: Well, also the grizzled sysadmin persona very much now a security person as well, right? So, you know, coming out of that sysadmin lineage, now I have to learn a whole bunch of new words, and security very much as a discipline, what I would criticize as saying, is very gatekeeper-y in terms of, “Okay, we’re going to come up with their own vernacular so that we know that you’re not one of us.” That’s one of my big criticisms of security. But the skill set, the same people who were sysadmin 20 years ago are definitely becoming security specialists, they’re becoming SREs. And so if you share the same lineage, then you’re really not all that different.
But at the same time, then when you look at observability, it’s the same problem; I need to be able to ask and answer arbitrary questions of data. And security detection is fundamentally the same problem, I have all this data that’s being egressed from my complex systems, all my endpoints, all of my VMs, my containers, all of my infrastructure, all my applications, and I need to be able to detect when someone is doing something wrong, like, some malicious actor is doing something wrong. Tell me that’s not observability.
So, someone creates something, someone accesses something, et cetera, et cetera. That’s useful when you need that, but if you’re going to take action based on that, you want to know sooner rather than later. Same story with any sort of monitoring tool that, “Oh, yeah, the site’s taking an outage and our system will let us know in only 20 short minutes.” Oh, I assure you customers will tell us long before then.
Corey: And honestly, in the full sweep of time, you really care about that one specific CloudTrail thing, but it’s the needle in the haystack.
Clint: And so AWS, this is a constant conflict between people who have to observe and secure systems need all the data because I may not know in advance what question I want to ask, but at the same time, I do know that not all of that is necessarily interesting right now, and so there’s a fundamental tension between, okay, the developer says, “Well, look. You can’t ask a question of data that’s not there, so I’m going to put everything in the log. Literally every byte of data, everything that I could ever think of, I’m going to put in that log.”
And then the receiver of that says like—I’ll give a good example. We’ve been talking about EDR. CrowdStrike EDR logs, phenomenal data source, have a ton of really interesting information about the security of your endpoints, and they also have an extra 100 fields that nobody gives a crap about. So, what do I do with that data? Do I pay to ingest all that data because all my vendors are charging me based off the bytes of data that are going into their platforms? And so there’s a real optimization potential there to have a really strong opinion on what good data is.
Corey: Part of the problem, too, is that you absolutely want the totality of everything captured around the specific event you care about. But by and large, we’ve all been in environments where we have a low-traffic app, and we see giant piles of web server logs. “Okay, great. Let’s take a look at what those web server logs are.” And by volume, it’s 98% load balancer health checks showing up.
It seems to me there might either be a way to strip them out entirely or alternately express those in a way that is a lot more compact and doesn’t fill things out. I still feel like there’s some terrible company somewhere where their entire way of getting signal from noise is to pay a whole bunch of interns to read the entire log by hand. I like to imagine that is me speaking hyperbolically, but I’m kind of scared it’s not.
Clint: Yeah. And then the question is, well, then how do I achieve a goal of actually getting the right data to the right place? So, that’s something that we help out about. I think that the—I feel a lot for the persona of this kind of sysadmin, this type of security person because they’re caught in this tension: like, do I go write code? My skill set as an SRE or my skill set as a security person is being an expert in the data itself.
I know that event is good, and I know that event is bad. Am I also supposed to be a person who then needs to go write a bunch of pipelines and Lambda functions, and how do I actually achieve the goal because there’s always way more demand than there is capacity to be able to onboard all of this data. So fundamentally, how do we get the right thing to the right place?
Corey: That’s, on some level, a serious problem. I will say that looking at what you do and how you do it, you take a whole bunch of different disparate data sources, and then effectively reduce all of those into passing through the Cribl log stream, and then sending the data out to exactly where it needs to go. And I have to imagine that when you talk about what you’re doing to typical VCs and whatnot, their question is, “Ah, but what if AWS launches a thing to do that?” To which I can only assume that your response must have been, “You’re right, if AWS does learn to speak coherently and effectively across all of their internal service teams, we’re going to have a serious problem.” At which point, I can only imagine that your VCs threw back their heads, you shared a happy laugh, and then they handed you another $200 million, which you have just raised. Congratulations, by the way.
Clint: Thank you so much. It’s, you know, people say a lot of times in startup-land, like, “Oh, we shouldn’t celebrate the fundraising.” I’ll tell you, as a person who’s done it a few times, I celebrate. That’s a shitload of work.
Okay, that seems like a different path. And there are advantages and disadvantages to every approach you can take on this. I mean, yeah, no shade here on how you decide to build out a technology company using VC-backed up resourcing, which is a sensible way to do it, but it’s a different style. And the sheer amount of work that very clearly goes into raising a fundraising round is just staggering to me. And that’s for seed-level rounds; I can only imagine down the path. This is not your first round.
And so really, this concept of a lake is like, well, look, I can finally at least put it somewhere as an insurance policy and make sure that’s actually going to be relevant. And then eventually what’s going to be happening is people are going to go help you make use of that data—and we will as well—be going out there to help you take petabytes and petabytes and petabytes of logs data, metric data, trace data, observability data and give you the ability to analyze that effectively.
So, you want to retain it, particularly if you have compliance obligations. There’s—theoretically at least—business value locked up in those things and you need to be able to access that in a reasonable way. And anytime I see tooling that winds up billing based upon amount of data stored in it, so just cut retention significantly. It feels like it cuts against the grain of what they’re trying to do.
Clint: I mean, yeah, retention, I mean, especially for security people—this is the difference between security and operations because operations is like, “Last 24 hours a data, I need. Pretty much after that, give me some aggregated statistics and I’m good.” Security people want full-fidelity data dating back years. But I think one of the other important concepts that we haven’t seen in the industry, and part of what we’re trying to change is, you know, I put data into a tool today. It’s that tool’s data, right?
So—and it doesn’t matter which tool it is that I’m put—they’re all the same. But fundamentally, I put data into a metrics or time-series database and put data into a logging tool, and that data is now owned by that vendor. And the big difference that we see in the concept of a lake is raw data at rest in S3 buckets—or other object storage depending on your cloud provider, depending on who, on-prem, is providing you that interface—in a way in which I can choose in the future, what tool is going to use to analyze that and I’m no longer locked in. And I think that’s really what we’ve been trying to advocate as an industry is that every enterprise I’ve talked to has everything. They’ve got one of every single tool and none of them are going away.
There is no such thing as a single pane of glass; that’s a myth that we’ve been talking about for 30 freaking years and it’s just never actually going to happen. And so really, what you need to be able to do is integrate things better and just make sure that people can actually use the tool that they want to use to analyze the data in the way that they see fit, and not be bound by the decision that was made six months ago as to which tool to put it in.
Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.
And we got a quote that was more than the valuation of the company at the time. Because it seems like their biggest market headwind at the time was the rise of democracy basically making monarchies go out of fashion, and there were fewer princesses that we could kidnap for ransom in order to pay the Splunk bill. And, to their credit, they reached out every quarter and said, “Oh, have your needs change any?” “No, we have not massively inflated the value of this company so we can afford your bill. Thank you for asking.”
But the problem that I had is when I pushed back on them on this—because it’s not just one of those make fun of it and move on stories because Splunk was at the time very much the best-of-breed answer here—their response was, “Oh, just go ahead and log less and that brings your bill back into something that’s a lot more cohesive and understandable.”
Clint: Which destroys the utility of the whole tool to begin with.
Corey: Exactly. The entire reason to have a tool like that is to go through vast quantities of data and extract meaning from it. And if you’re not able to do that because you have less data, it completely defeats the value proposition of what it is you’re bringing to the table. Because in the security space, in many ways in the observability space, and certainly in my world of the cost optimization space, it’s an optimization story. It does not speed your time to market, it does not increase revenue in almost every case, so it’s always going to be a trailing function behind things that do.
Corey: Exactly. And I have to assume on some level, this was one of those, “Okay, first start using it. You’ll see how valuable it becomes, and then you’ll start logging more data.” But it didn’t feel right because it’s either being disingenuous, or it’s saying that, “Oh, don’t worry. You’ll find the money somehow.”
Which is not true in that scenario. Now, they’ve redone their pricing multiple times since then. There are other entrants in the market that help us look at data in a bunch of different ways, but across the board, it’s frustrating seeing that there are all these neat tools that I wanted to use and I was perfectly positioned to use back then, and now nine years later, when someone says, “Oh, we use Splunk.” My immediate instinctive reaction is, “Oh, wow. You must have a lot of money to spend on services.” Which is not necessarily even close to reality in some cases, but first impressions like that really stick around a long time.
Clint: Oh, absolutely. They stick around often because they’re reaffirmed multiple times throughout [laugh] people’s continued interactions. And I think there’s just really a fundamental tension in the marketplace where the value proposition is massive amounts of data. And massive is different, depending on the size of your organization: if you’re a big Fortune 100, massive might be, you know, a 100 petabytes at rest and a petabyte a day of data moving; or for you, massive might be a terabyte a day moving, and maybe a 50 terabytes at rest. But—and by the way, that’s not going down.
So, some of the bigger trends that we’re seeing with the advent of zero trust, with the advent of remote work, with just in general growth of cloud containerized workloads, microservices, people are seeing a lot more data today than they were seeing two years ago, three years ago. And by the way, it’s not like IT went from 2% of the budget to 10% of the budget. The budget’s the same, so I got to do more with less. And it’s a tension between data growth and cost and capacity. And so we got to get smarter.
Corey: I like the fact that you’re saying that you have to get smarter as you think about this from a tool perspective of being able to serve your customers, as opposed to a lot of tooling out there seems to inherently and intrinsically take the world view—and I don’t know if this is an actual choice or just an unfortunate side effect—of, “Yeah, we have to educate our customers because right now, our customers are fairly dumb and we’d like it if they were smarter. If you were smart enough to appreciate how we do things, then things will go super well.” And I always found that to be a condescending attitude that doesn’t serve customers super well. And it also leaves a lot of money on the table because for better or worse, you have to meet customers where they are: at their level of understanding, at their expression of the problem. And I’ve talked to a number of folks over at Cribl and, similar to certain large cloud providers, one of the things that you focus on is the customer; it’s clearly a value of the company. How do you think about that?
It’s like, well, do people actually want to buy it? Is this thing actually solving real problems? And so from the beginning, me and my co-founders, we just wanted to make sure that the concept of the customer was embedded at the core of the company. And every time that an employee at Cribl is interacting and talking about what should we do next, and what features should we build, and how should we market, and how should we sell, let’s make sure the customer is there. Customers first always is the value, including in how we sell.
We actively leave money on the table when it’s not in the customer’s right interest because we know that we want them to come back and buy from us again, later. When we market, we try to make sure that we’re speaking to our customers in a language that is their language. When we’re building a product, we use the product, we try to make sure that this is actually everyday, we don’t look at, hey, it needs to look like this and have these features to meet these criteria and be called this. It’s just like, “Well, does it actually help the customer solve a real problem for them? If so, let's build it. And if not, then who gives a [BLEEP]?”
Corey: Exactly. It’s understanding what your customers’ pain points are. I mean, I ran into some similar problems when I was starting my consultancy where I—it turns out that I knew people who were more or less top of their class when it came to AWS bill understanding, reconciliation, and the rest. And those are the people I reached out to because I assumed that they knew what they’re doing. There must be lots of people like them, everyone must be like these folks.
And I talked to them about how they looked at their AWS bill. And, okay, “They said I would—I’d love to hire you to come in and do this as a consultant, but I would expect this, this, this, this, and this.” And, “Okay, I better come loaded for bear.” And so I did. And it turns out there’s a lot more people out there who have never heard of a savings plan or a reserved instance before or, “Wait. You mean continues to charge me even after I’m still using it if I don’t turn it off?” Yes, that is generally how it works.
Clint: Software is fundamentally a people business and when you end up implementing a tool—what’s become fascinating to me as I’ve become the CEO of this company, rather than just kind of a product guy, so now I’ve had to sell it and I’ve had to market it, and I had to start very much from scratch, is that this stuff doesn’t just get implemented by magic; even if they download the tool and is the easiest to use tool that you’ve ever used, they still don’t have the time to learn all the details and intricacies of your product, and so hey, they actually want some professional services people to come and go install that; they want a salesperson to help them understand the value. I know a lot of people, especially coming from my background in, like, SRE or sysadmin from when I was doing it, kind of, “Oh, salespeople.” But, like, they do a real job; they help you articulate the value of this thing so that your bosses understand what you’re actually buying. The sales engineers help you understand what those features are. And so having a customer-aligned company means that every interaction that they have with you needs to be a really, really great interaction so that they want to interact with you again because fundamentally, even though the bits are really awesome and they solve this really awesome technology challenge, nobody really cares about it.
Ultimately, they’re buying from people, they’re implementing software built by people, and they’re calling for support—which is another important part—from people who fundamentally care about them as well. So, in every interaction, fundamentally software is a people business, and you got to have the best people and the people that care.
Corey: I wish more people took that philosophy because, frankly, it’s missing from an awful lot of different expressions of what companies do. It’s oh, if we can make the code just a little bit smarter, a little bit more predictive, then we never have to talk to the customer at all. It’s, “No. You shouldn’t write a line of anything before doing a whole bunch of customer research to validate that your understanding of the problem space aligns with theirs.”
Clint: A good way to find out that doesn’t work is to fail for a while, too. So, [laugh] so we did our fair share of that, too, and kind of pontificating and trying to figure out what we thought was best at the market, and it turned out that really what you needed to be able to do was to work closely with customers and understand their problems and tightly pair that sales cycle, that marketing messaging, that product all towards customer pain. And if you do that, customers are great because they see the people who care, and they will reward you by becoming your customer and continuing to advocate for you and talk about you. And it’s so rewarding if you can take the right perspective.
Corey: So, we’ve covered a fair number of things: your philosophy on the world of security versus observability; we’ve talked about meeting customers where they are; we’ve talked about AWS being so inept at communicating internally and cross-functionally that you’re able to raise staggeringly large rounds, and we’ve talked about, I guess, how we wind up viewing the world of log collection, for lack of a better term. If people want to learn more about what you’re up to, and how you get there, where can they find you?
Corey: Oh, I don’t have to speak to a salesperson?
Clint: No, you don’t have to talk to anybody. You can download the bits, you can try our cloud product for free at cribl.cloud. We are all about making sure that engineers can get access to the product before you have to talk to us. And if you think that’s valuable, if this helps you solve a problem, then and only then should you engage with us and we’ll see if we can figure out a way to sell you some software.
Corey: Customer-focused. I’m also going to take a spot check here. I’m going to guess that given your recent funding news, you’re also aggressively hiring.
Clint: We are hiring across every function, and if you are interested in working for our customers-first software company and this sounds refreshing, please check out cribl.io/jobs, and we’ve hiring everywhere.
Corey: I can endorse. We used to hang out, back before you wound up starting this place, and you were kicking around this idea of, “I have an idea for a company,” and my general perception is, “Eh, I don’t know. Doesn’t sound like it has legs to me.” And well, here we are. I sure can pick them. Badly. Clint, thank you so much for taking the time to speak with me.
Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.