The Future Is Time Series Data with Russ Savage

About Russ Savage
Russ Savage is a Product Manager at InfluxData where he focuses on enabling DevOps for teams using InfluxDB and the TICK Stack. He has a background in computer engineering and has been focused on various aspects of enterprise data for the past 10 years. Russ has previously worked at Cask Data, Elastic, Box, and Amazon. When Russ is not working at InfluxData, he can be seen speeding down the slopes on a pair of skis.


Links Referenced

Transcript

Announcer: Hello and welcome to Screaming in the Cloud, with your host cloud economist, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Russ Savage, the director of product management at InfluxData, which I am under no circumstances to refer to as Quinnflux Data or Russ will presumably hit me with a belt. Russ, welcome to the show.

Russ: Thanks Corey. Glad to be here.

Corey: About a month or so ago at this point we had, effectively one of your co-founders, Paul on the show and talked for about half an hour about some of the ins and outs of Influx and how one might use a time series database, what a time series database might be. And that got a decent enough reception that we decided, "Hey, who else can we have on?" So congratulations. You're in the hot seat. But unlike some software products, people generally don't emerge into the world fully formed like from the forehead of some ancient God. How long have you been at Influx and where were you before?

Russ: Yeah, I've been at Influx for over two years now. I've been in the data space for a long time. I came from a small Hadoop startup that got acquired by Google and then I was previously at Elasticsearch.

Corey: That sounds like an interesting and winding road. It sounds like you did fundamentally what everyone was trying to do now and get out of Hadoop, and you were able to do it by changing companies. Some folks, not so lucky. But from my naive understanding, I mean for God's sake, my favorite database is Route 53 so I have other problems that I have to work through. But Hadoop seemed like it was a big thing and then suddenly it wasn't a thing anymore. And I don't see new Hadoop projects. They all feel somewhat legacy. Does that match your experience or am I just talking about things I don't understand or quite possibly both?

Russ: Well, I think my personal opinion is I think Hadoop solves Hadoop scale problems really well. I don't know if as many companies have Hadoop scale problems as they think. What's happening is computers are becoming more powerful. Individual computers are becoming more powerful. So you can do more with less. And so I think for a ton of the problems that people are facing now, you can run that on much smaller systems. You don't need a 50 or a 500 node Hadoop cluster to do that. Also, the complexity of setting up and maintaining Hadoop was very high. I think with these smaller systems, it's much easier to get them set up and keep them running a lot less maintenance overhead than you have with a Hadoop cluster.

Corey: It always was interesting watching people come out with big data problems and "Oh great, so where's your data?" And they pull a thumb drive out of their pocket, and you don't have a big data problem. Alternately, you can create big data problems if you're a terrible enough programmer. I mean my only debug strategy is print lines. So my logs are enormous because I never go back and fix anything. So every time someone has a request, I get 8,000 lines of logs. So yeah, there's always something people can do to turn it into a big data problem.

Russ: Yeah, I think a lot of those problems stem, they actually turn into larger than Excel problems or larger than what I can run on my local machine problems. But that doesn't mean that they're Hadoop problems.

Corey: That seems to be the point that the world has reached consensus around as well. So you wound up leaving a Hadoop startup and finding yourself effectively selling open source time series databases, which step one, selling open source software has always been a bit of an interesting challenge for some folks. Can we talk a little bit about that, and I guess first, what inspired you to come to InfluxData? And secondly, how are you, I guess engaging with the community when you do have a service or a software that you are attempting successfully, it turns out, to sell?

Russ: Yeah, great question. So my background is as a developer, and I love to write code, not production code, but a lot of code regardless. And I think my passion for open source, I love the community that evolves around open source projects. And so when I was looking for a new role, I was looking specifically for companies that were strong in the open-source communities. And so InfluxData is a great, great company for that.

So we have a set of products, open-source products that we use as the basis for our cloud and enterprise offerings. I really love the fact that there's so much capability in our open-source tool that the individual developer or the small team can really, it's not just demoware it's not just shareware. You can actually get real workloads done in the open-source tool. And then when you're getting more value out of open source or you need some support or you want to create a team dedicated to managing it, we can help you do that easier.

Corey: And I do want to point out, otherwise Lord knows, I will get letters, that as you're mentioning is I pull up the InfluxDB open-source repository on GitHub or GitHub, depending upon pronunciation choice and yes indeed it is licensed under the MIT license, an actual open source license. None of these nonsense "Oh you can use the source for anything unless you make money on it. And then we're coming for you" as some companies in this space recently have been doing as a, I guess a hedge against cloud providers. It's interesting that you folks A, haven't felt the need to do that. And B, have continued to engage in good faith with your own user community to work with people who are very clearly passionate fans of what you're doing.

Russ: Yeah, I agree. And I really, I applaud our leadership for doing that. Paul Dix, our CTO, is very passionate about open source and very passionate about licensing and he tends to write a lot of blogs on that topic. That was also really important that we're a true open source company, not one in name only.

Corey: I really think that can't be stressed enough. If I were going to be building something to give to the world or put out there as a product, I have a lot of decisions to make. And on the one hand, there is some compelling value to having something to be open source, but that does come with costs. And I think these ideas of just try to get the good parts of the open-source world - community engagement, free promotion, et cetera, et cetera - but not any of the downsides of well no one can ever offer this as a service except us, it just rings hollow and feels transparent. I mean, I'm sympathetic to business model challenges, but suddenly moving the goalposts on existing communities really breaks the social contract that open source communities have come to expect.

Russ: Yeah, I agree. And the problem that I see, so I've been on the other side and I've done evaluations of different software and looked at... Long term, you're thinking, you're bringing in third-party software into your company. Your company is going to be around for a long time and you want to make sure that you're not bringing in something that you're going to have to replace in a few years. And I think the real key with open source is you can take that technology and really embrace it and really take it to the next level, really expand upon it.

And when you're bringing open source software into your company, you're thinking about the long term horizon. And if you have companies that are making changes or deciding to suddenly change the way their licensing or change that, it really calls into jeopardy your long term vision and it's a risk.

Corey: I think that understanding that open source is a double-edged sword in some ways and it is a approach to solving certain problems, but it's not a strategy in itself. It is something that a number of somewhat naive founders have gotten trapped in in the past. Of course, here we are now where you're very clearly doing well. I've talked to a number of Influx customers that we'll talk about in a bit, but it's very clear that you're doing something right. And believe me, when I have people on this show, one of the first things I do is start Googling, all right, who hates them? And I can't really find anyone saying negative things about Influx. Believe me, I've looked. Am I just looking in the wrong places or does the community actually seem to like you folks?

Russ: Well, I talk to the community every day and I have a ton of positive interactions with them and we're always working to bring them closer and bring them into our discussions as we move as a company. And so I think that community engagement is really important, and it also drives a ton of transparency in the organization. Right. So Telegraf is one of our projects. It's a data collection agent. And all of our discussion happens in the public with our community. I think that's really important and it makes people want to be a part of that community. And we want people to want to be a part of our community and that's what we could strive for.

Corey: That tends to be a hot button that gets a lot of people riled up. Let's change gears a bit and talk about something else that I've been using to successfully annoy people for a couple of years now. I have been a long fan of the idea that multi-cloud as a best practice, is stupid. If you're building something and you want to be able to magically deploy it to AWS and GCP and Azure and Oracle Cloud because you lost a bet somewhere, then you're slowing yourself down and effectively trading feature velocity for a level of faux agnosticism that you're not necessarily going to ever take advantage of.

And I stand by that statement, but what people then don't listen to is paragraph two after that, which is, here's a list of exceptions. First and foremost among that list of exceptions is who are your customers? If your customers are attempting to service their own customers in a variety of different locations, they have to meet them where they are.

"Our service is awesome, but you have to move over to this cloud provider" is absolutely not going to happen. You're not going to be able to serve their needs. And in turn, telling customers that they should migrate their own work, their own stuff and become multi-cloud themselves, if they listened to previous episodes of this podcast, they're going to say, "Oh wait, we've heard about this. It's stupid."

Now because of you being who you are and having to service a wide variety of different customers, you fundamentally need to have an offering that spans beyond a single provider.

Russ: No, I completely agree with you. I think one of the things that you look at is we are a platform company. And we want our users to build on our platform. And when you're running a platform company and you want users to build on top of it, you need to make that platform available where your users are. And the reality is is that different companies use different clouds for different reasons. I don't think that one company is going to... It makes sense for one company to run in multiple clouds, in most cases. We have some customers who are running in Azure Cloud for specific reasons that we're not really... We don't really care what those reasons are. We just know that they can only run in a specific cloud and that's where we want to be. And so, yeah, we 100% want to run our infrastructure in as many clouds as our customers need.

Corey: And that's sort of a piece that often gets lost in nuanced conversations about this. When customers have needs, you've got to be able to meet those needs rather than condescendingly telling them that they're wrong all the time, unlike certain cloud companies we can all think of, but not name. The fun part where that also expands beyond just the idea of multiple clouds, it's easy to also sit here and say, "Oh, and hybrid is dumb, too. You shouldn't have any on-premise physical data centers." Oh, that's adorable. Great theory. But here in practice, we have this thing called legacy, which engineers here is old and broken, but here in the real world I hear as things that make money.

So I don't see hybrid going away anytime soon. And unlike a lot of different platform companies we could name, you have viable options for folks who are running on-prem. Was that something you planned from day one? Did that accidentally happen along the way as you were building this stuff out?

Russ: Yeah. So we have on-prem and we have cloud. And I think the interesting thing there is a lot of companies will tell you that they're all cloud. But if you actually dig down into their engineering organizations or dig down into the development areas, you'll actually find a ton of quote unquote “on-prem development.” We’ve got people writing applications against on-prem versions of software and they want to make sure that those applications run, whether it's on-prem, whether it's in the cloud, wherever.

And so I think one of the big things that we've focused on is, again, the idea of a platform and being able to develop against that platform and having a common set of APIs, whether you're running this stuff on-prem and you're building out development as a proof of concept to make sure that it's working for you, or you're running this on a cloud and you want to service a global customer base, or if you're an IoT shop and you're running in an oil rig in the middle of the ocean and you don't always have cloud connectivity. I think, again, it's another question of where are our customers running their infrastructure and since we're an infrastructure company we want to be where they are.

Corey: But something you just mentioned is fascinating. The idea of oil rigs for instance. People tend to not necessarily have a lot of exposure to that type of environment, but to my understanding, a lot of them tend to live in the middle of the ocean where internet connectivity, shall we say, comes at a premium. And having something cloud-hosted just simply isn't an option there for anything that needs to be even slightly performant. So there really is no alternative short of asking AWS to build a region in your house, effectively. I've asked, they won't. US Bathtub One will not exist this year, but we're optimistic for 2020.

There are use cases that the cloud cannot, with current technology, address. And I think that there's this certain willingness of the part of the cloud native sort to turn our noses up at that type of use case. But things like that are important and customers are there with problems. Making sure that you can address what those customers are up to and meet them where they are is something that I think an awful lot of companies just sort of tiptoe past and don't really investigate in the name of ideological purity.

Russ: Yeah. I think you're right. And I think we see at least in our customer base, basically where the application or where the infrastructure is deployed, kind of serves for different use cases. And so you look at running something on the edge or running something on-prem, on an oil rig for example, right, you're basically... Who's your customer in that regard? It's the individual operator of the system on the platform looking at real time data of how different gauges, how different pressures are operating. Right? And that's valuable and it gives you a lot of capabilities and insights.

But in order to get long term trends or in order to do large scale analysis across hundreds of platforms all over the world, you want to send that data into the cloud for processing. And that data that's in the cloud is going to be massive aggregated data across many different platforms. And the audience for that is going to be a data scientist who's building out a machine learning algorithm that can then be pushed down into the oil rig. Right? And so you get this system where you're actually, you're running your software, you're running the same software, but you're addressing two different needs and two different business needs.

Corey: I guess I have the advantage on some folks because though I've never worked on an oil rig, I did grow up in rural Maine. And you have about the same level of internet connectivity in some of those places as you would on an oil rig. So running things in my living room was always what I did growing up. There was always the one room that was 20 degrees hotter than anywhere else because that was my makeshift data center with crappy old computers. And it was terrible, but it was also a half step above IBM cloud, so there's that.

And eventually the world modernized. Internet came to rural Maine. But I remember those days where you're generating more data locally than you're ever going to be able to shove into, well, however large a pipe is to get it to a cloud provider. So, especially when you're in the business of doing data collection and aggregation, which fundamentally is what a lot of databases are, whether they're time series or not, it needs to be as close to what's generating those inputs as possible, I'd imagine.

Russ: Yeah, and especially true for time series data where time is important and latencies are important and milliseconds matter, right? And so we get into a scenario... Our database is one of the few that can do at the nanosecond level. And while we're not recommending that people monitor everything at the nanosecond level, it does help if you can trigger an event and then you start collecting data a lot more frequently than you normally would so that you can see exactly what's going on during that event. And then you go back to one second intervals after it's finished.

And so I think putting your database close to where you're located and reducing that latency gives you an advantage in your response time to the specific problem that comes up.

Corey: Oh, absolutely. Anyone who wants to experience this themselves can wind up trying something at an AWS account, and then just start a stopwatch and see how long it takes that event to show up and either CloudWatch Logs or far worse Amazon CloudTrail. They put the eventual in eventual consistency. There's value to getting rapid response that could inform what's going on. I mean there are use cases where that simply won't do. Imagine having significant latency and a self-driving car for example. At that point, oh, you should have stopped at that light four lights ago. Not a great plan.

Russ: Right, yet, the latencies on on-prem and when you're close to the source are very different than the latencies than in cloud. And so again, it's different use cases that you're solving and it's different dimensions where you're analyzing the data.

Corey: Our biases tend to inform how we wind up looking at different tools and how they might factor into our own use cases. So rather than continuing to assume that everyone would use a time series database like I would, because I have a SysAdmin background, what kind of customers do you have? What are they doing with a time series database that isn't, for example, just monitoring whether a computer is up or not?

Russ: Yeah, great question. So to be clear, a lot of our customers are monitoring whether a computer is up or not. That's just a huge use case for us, and-

Corey: It seems a common pain point.

Russ: Yeah, it turns out it's hard to keep those things running. No. So, we have a ton of use cases around time. And our belief is if you actually look at the data that's being collected and the data that's out there, most of the data that you're seeing is actually more valuable when you look at it through the lens of time and you look at it, how things trend over time.

So even if you think of traditional data stores, if you add a time element to it, you suddenly start getting more insights than you could without. And we've got use cases everywhere. So for example, we were talking about oil platforms earlier, right? We have a company named Equinor who is using our platform for monitoring those oil platforms in the Norwegian continental shelf. And it's really exciting and really interesting to hear some of the stuff that comes out of there. But basically the ability to gather that data in real time and make decisions on it was really important for them.

And when you think about... We consider this an IoT use case. A lot of people when they think about IoT, they think about you're looking at temperature sensors or you're looking at different air quality measures. But a ton of our industrial IoT customers are monitoring large, large, massive oil rigs or drilling platforms or things you wouldn't typically come to the top of your mind when you're talking about IoT, but very, very important use cases.

Corey: So what use cases do you have that are outside of the IoT space? I mean as much fun as it is to talk about A, oil rigs and B, refrigerators that tell me when the milk is expiring, what other use cases do you see from customers?

Russ: Yeah, IoT use cases are really fun to talk about because they're so varied and widely used. But we have people using us for a ton of different things. So as I mentioned before, we have customers that are managing data centers with tens of thousands of machines. And we even work with NASA who's monitoring their infrastructure for launching satellites.

Corey: And at least until it launches, it presumably has a better connectivity to the internet than it does once it's in space. Although, one starts to wonder.

Russ: Exactly. I never know, you never know what Elon's up to.

Corey: The problem I run into always is that I tend to think of these things just in the context of what I've used time series databases for before. Now, the challenge of course is I was a network admin once upon a time when the closest thing we had to a time series database was RRDtool or MRTG that wrapped RRDtool. And as a result, oh time series databases. Those are those things that are terrible and cause all matter of problems and its primary use case is to be embedded in cacti, whose intern primary use case is to sit in the corner until someone breaks into it and uses it as an attack platform. That may not be entirely hypothetical.

But increasingly, the capabilities have dramatically changed. And being able to see such, I guess, different capabilities now, I look back at the outages of yesteryear, for lack of a better term, and it would have been so much easier to diagnose some of these outages with the right tooling, as it turns out. Unfortunately, it is a time series database is not a time traveling database, so I don't think you can go back and necessarily help me with those problems in the past.

But for other people who I guess who are stuck in that outmoded view of time series, what's changed? How would you describe the advancements? Because I'm sort of going to assume just on a lark here that you didn't declare InfluxDB feature complete in 2013 and the rest has just been maintenance releases.

Russ: No, definitely not. I think the key fundamental difference between a generic general database and a time series database is you can turn a generic database into a time series database with enough manpower, with a big enough team. I think the question-

Corey: Oh, with a dumb enough perspective, you can turn DNS into a database as well. I've done it. But everyone looks at you with this horrified look on their face and then you get asked to leave the interview immediately.

Russ: Right, right, exactly. It's basically, it's about what you ultimately want to accomplish and where you want to put your resources. Right? Everything, and I'm a product manager, so I'm always thinking about resources and priority. But if you're a company and you are solving a specific business need, does it make sense to actually spend 50% of your time taking a generic database and turning it into a time series database so then you can solve your problem? Or would it make sense to go out and get a best-in-class time series database so that you can focus on the thing that's actually going to make to drive capabilities of your business? Right?

And so when I think of the reasons why you would choose a time series database, behind the scenes, we've done a ton of legwork and a ton of the optimizations that you would have to do yourself if you were returning a generic database into time series. And so that gives you a great starting point and allows you to focus on the business problem that you want to solve instead of these infrastructure issues with creating a time series database.

Corey: What's interesting to me, if I take a look through the Influx suite of tooling, for lack of a better term, you have a hosted version that people can run wherever, you have an enterprise offering, you have cloud-hosted versions. What is the decision matrix for people who are trying to figure out which version would make sense?

Russ: Great question. So we talk to our customers a lot. And basically we kind of break it down into two different paths. Between those paths are different decision points that you can make depending on what you want to solve to cross the divide, right? And so on the on-prem scenario, you've got the open-source instance that you can run locally and build your application against. You have an enterprise instance of on-prem that you can host yourself if you need more power or high availability or security. Then we also have the equivalent in cloud. So in cloud, we have a free tier that lets you get started quickly and build your application against. And then if you want to host all your infrastructure in cloud, we then offer that pay-as-you-go service in cloud.

And so basically it depends on your use case. It depends on your infrastructure. It depends on how you want to run things. But either way, whether you're on-prem and want to manage all the infrastructure yourself or you don't want to manage any infrastructure and you want us to do that for you, we have offerings for both.

Corey: One of the, I guess, surest signs of a fanatic is when they have absolutely nothing negative whatsoever to say in any context about a given topic. You don't strike me as someone who falls into that category. So I have to ask, from a high level, what unsolved problems are there today in the world of time series databases?

Russ: Oh, that's a tough one. Unsolved problems. So the thing that I see when I talk to customers is, and you're starting to see this a ton in the marketplace, is I think everybody had this dream and bringing this back to Hadoop is this like this data lake, which actually turned out to be a data swamp as people realized, but you've got these specialized databases and I 100% agree that you should use the right tool for the job. And so I think the specialized databases make a ton of sense.

But I think the issue that you're seeing is you're actually starting to create these individual silos across different storage mechanism, different databases, different tools. And one of the struggles that people have is connecting all of that stuff together, right?

And so you see a ton of companies coming up that are basically just connecter companies that bring data from A to B and C to B and all of these things together. And so I think that's a huge struggle in specialized databases and in time series. And so that's something where you'll see a lot of development in our data manipulation language called Flux that actually will make it much easier to bring that data from those disparate databases, from those disparate locations and run analysis that you couldn't really do before.

We see a ton of people who in the past have basically had to build applications on top of multiple platforms to bring data from all of those platforms and combine them, and our goal is to bring that application logical closer to the data store and so that you can then leverage the capabilities of the community and not have to build everything yourself. And so we're starting to see areas in Flux where people are connecting to different data sources, bringing that information that has typically been siloed in individual tools into the same place and running that analysis.

Corey: I have some level of contempt for companies that are entirely built around solving a particular pain point or a problem that are effectively one feature release away from another service or product that puts them completely out of business from a model perspective. I'm not going to necessarily name names, but if your entire company's job is to take data in one format and translate it to another, if the source company releases one day of feature of, "Oh by the way, you can now query this and get a JsonResult instead and that destroys your company," maybe when you're raising giant piles of VC money, thinking through how unassailable of a moat you've built really winds up being worth pursuing.

It definitely seems, from conversations I had with you and with others, that this is a serious... It's not a problem that you've built things around. It's an entire field of inquiry where you might have heard of time series database and are scratching the surface a little bit, but the further you get into it, the more you realize there's an entire sector here. There's an entire ecosystem that is dealing with painful problems.

I mean, we've gotten to a point now where, I wouldn't have believed this was possible, you can commit the Cardinal sin of posting a graph on the internet and not label your axes, but you can generally assume with a time series driven graph that X is time and it moves from left to right. That alone is an industry win. Congratulations. You have finally gotten statisticians and high school math teachers to stop screaming about that one. That alone says that there's a lot more to it than that.

I think my rubric for determining whether something's important or not based upon a 10th grade experience I had might be slightly flawed, but I'd also think that this is something that is starting to rise in awareness. People are starting to understand that there's something there. And again, I keep looking for dirt on you people and I'm having a lot of trouble finding it.

Russ: I appreciate it. I think the one thing that I like to focus on again, I really love the community that we've built over the years and the trust that we've built in that community. And so I think if you're honest and you're doing good work and you're being transparent about the work you're doing, then it'll attract good people. And so I'm again going back to our original conversation, that's one of the main reasons I came here in the first place, and it's the reason I'm still here now, so.

Corey: Well thank you so much for taking the time to speak with me. If people want to learn more about what you're up to, where can they find you?

Russ: If you want to learn more about InfluxData, influxdata.com. We've got a free signup for our Cloud 2 offering. You can take it for a spin and see if it meets your needs, and that's a great place to learn more.

Corey: Terrific. Thanks again for taking the time to speak with me today. I appreciate it.

Russ: Thanks, Corey. Thanks for having me.

Corey: Russ Savage, Director of Product Management at InfluxData. I'm Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this episode, please leave it a five-star review in Apple Podcasts. If you hated this episode, please leave it a five-star review in Apple Podcasts and in the comments, ask Russ to hit me with his belt.

Announcer: This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com or wherever fine snark is sold.

Announcer: This has been a HumblePod Production. Stay humble.