Let Your Backups Help you Sleep with Simon Bennett

Episode Summary

Backups matter, or at least they should, and if you’ve ever lost your data, you’d be inclined to agree. Simon Bennett, CEO and Founder of SnapShooter, has found a way to address those issues. SnapShooter backups for your servers, databases, and website, and makes it easy to do to boot. Simon talks about their offerings, notably that they’ll help you sleep well at night. Simon expands on the needs to make data safe and secure for his customers so that they can go back to doing what they do best. Their actual work, instead of fretting about data. Simon talks about areas where cloud providers fail in the backup space, the flexibility he gives his customers, how he prevents the customers having to deal with the finicky parts of backups, and more!

Episode Show Notes & Transcript

About Simon
Founder and CEO of SnapShooter a backup company 

Links Referenced:

Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.


Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that’s where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats’s snark.cloud/D-U-P-L-O-C-L-O-U-D.


Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.


Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. One of the things that I learned early on in my career as a grumpy Unix systems administrator is that there are two kinds of people out there: those who care about backups an awful lot, and people who haven’t lost data yet. I lost a bunch of data once upon a time and then I too fell on the side of backups are super important. Here to talk with me about them a bit today is Simon Bennett, founder and CEO of SnapShooter.com. Simon, thanks for joining me.


Simon: Thanks for having me. Thank you very much.


Corey: It’s fun to be able to talk to people who are doing business in the cloud space—in this sense too—that is not venture-backed, that is not, “Well, we have 600 people here that are building this thing out.” And similar to the way that I handle things at The Duckbill Group, you are effectively one of those legacy things known as a profitable business that self-funds. What made you decide to pursue that model as opposed to, well, whatever the polite version of bilking venture capitalists out of enormous piles of money for [unintelligible 00:01:32]?


Simon: I think I always liked the idea of being self-sufficient and running a business, so I always wanted to start a physical business when I was younger, but when I got into software, I realized that that’s a really easy way, no capital needed, to get started. And I tried for years and years to build products, all of which failed until finally SnapShooter actually gained a customer. [laugh].


Corey: “Oh, wait, someone finally is paying money for this, I guess I’m onto something.”


Simon: Yeah.


Corey: And it’s sort of progressed from there. How long have you been in business?


Simon: We started in 2017, as… it was an internal project for a company I was working at who had problems with DigitalOcean backups, or they had problems with their servers getting compromised. So, I looked at DigitalOcean API and realized I could build something. And it took less than a week to build a product [with billing 00:02:20]. And I put that online and people started using it. So, that was how it worked.


Every other product I tried before, I’d spent months and months developing it and never getting a customer. And the one time I spent less than [laugh] less than a week’s worth of evenings, someone started paying. I mean, admittedly, the first person was only paying a couple of dollars a month, but it was something.


Corey: There’s a huge turning point where you just validate the ability and willingness for someone to transfer one dollar from their bank account to yours. It speaks to validation in a way that social media nonsense generally doesn’t. It’s the oh, someone is actually willing to pay because I’m adding value to what they do. That’s no small thing.


Simon: Yeah. There’s definitely a big difference between people saying they’re going to and they’d love it, and actually doing it. So.


Corey: I first heard about you when Patrick McKenzie—or @patio11, as he goes by on Twitter—wound up doing a mini-thread on you about, “I’ve now used SnapShooter.com for real, and it was such a joy, including making a server migration easier than it would otherwise have been. Now, I have automatically monitored backups to my own S3 account for a bunch of things, which already had a fairly remote risk of failure.” And he keeps talking about the awesome aspects of it. And okay, when Patrick says, “This is neat,” that usually means it’s time for me to at least click the link and see what’s going on.


And the thing that jumped out at me was a few things about what it is that you offer. You talk about making sure that people can sleep well at night, that it’s about why backups are important, about—you obviously check the boxes and talk about how you do things and why you do them the way that you do, but it resonates around the idea of helping people sleep well at night. Because no one wants to think about backups. Because no one cares about backups; they just care an awful lot about restores, usually right after they should have cared about the backups.


Simon: Yeah. This is actually a big problem with getting customers because I don’t think it’s on a lot of people’s minds, getting backups set up until, as you said in the intro, something’s gone wrong. [laugh]. And then they’re happy to be a customer for life.


Corey: I started clicking around and looking at your testimonials, for example, on your website. And the first one I saw was from the CEO of Transistor.fm. For those who aren’t familiar with what they do, they are the company that hosts this podcast. I pay them as a vendor for all the 
back issues and whatnot.


Whenever you download the show. It’s routing through their stuff. So yeah, I kind of want them to have backups of these things because I really don’t want to have all these conversations [laugh] again with everyone. That’s an important thing. But Transistor’s business is not making sure that the data is safe and secure; it’s making podcasts available, making it easy to publish to them.


And in your case, you’re handling the backup portion of it so they can pay their money and they set it up effectively once—set it and forget it—and then they can go back to doing the thing that they do, and not having to fuss with it constantly. I think a lot of companies get it wrong, where they seem to think that people are going to make sustained, engaged efforts in whatever platform or tool or service they build. People have bigger fish to fry; they just want the thing to work and not take up brain sweat.


Simon: Yeah. Customers hardly ever log in. I think it’s probably a good sign when they don’t have to log in. So, they get their report emails, and that’s that. And they obviously come back when they got new stuff to set up, but from a support point of view is pretty, pretty easy, really, people don’t—[laugh] constantly on there.


Corey: From where I sit, the large cloud providers—and some of the small ones, too—they all have backup functionality built into the offering that they’ve got. And some are great, some are terrible. I assume—perhaps naively—that all of them do what it says on the tin and actually back up the data. If that were sufficient, you wouldn’t have any customers. You clearly have customers. What is it that makes those things not work super well?


Simon: Some of them are inflexible. So, some of the providers have built-in server backups that only happen weekly, and six days of no backups can be a big problem when you’ve made a mistake. So, we offer a lot of flexibility around how often you backup your data. And then another key part is that we let you store your data where you want. A lot of the providers have either vendor lock-in, or they only store it in themselves. So… we let you take your data from one side of the globe to the other if you want.


Corey: As anyone who has listened to the show is aware, I’m not a huge advocate for multi-cloud for a variety of excellent reasons. And I mean that on a per-workload basis, not, “Oh, we’re going to go with one company called Amazon,” and you use everything that they do, including their WorkMail product. Yeah, even Amazon doesn’t use WorkMail; they use Exchange like a real company would. And great, pick the thing that works best for you, but backups have always been one of those areas.


I know that AWS has great region separation—most of the time. I know that it is unheard of for there to be a catastrophic data loss story that transcends multiple regions, so the story from their side is very often, oh, just back it up to a different region. Problem solved. Ignoring the data transfer aspect of that from a pricing perspective, okay. But there’s also a risk element here where everyone talks about the single point of failure with the AWS account that it’s there, people don’t talk about as much: it’s your payment instrument; if they suspend your account, you’re not getting into any region.


There’s also the story of if someone gets access to your account, how do you back that up? If you’re going to be doing backups, from my perspective, that is the perfect use case, to put it on a different provider. Because if I’m backing up from, I don’t know, Amazon to Google Cloud or vice versa, I have a hard time envisioning a scenario in which both of those companies simultaneously have lost my data and I still care about computers. It is very hard for me to imagine that kind of failure mode, it’s way out of scope for any disaster recovery or business continuity plan that I’m coming up with.


Simon: Yeah, that’s right. Yeah, I haven’t—[laugh] I don’t have that in my disaster recovery plan, to be honest about going to a different cloud, as in, we’ll solve that problem when it happens. But the data is, as you say, in two different places, or more. But yeah, the security one is a key one because, you know, there’s quite a lot of surface area on your AWS account for compromising, but if you’re using either—even a separate AWS account or a different provider purely for storage, that can be very tightly controlled.


Corey: I also appreciate the idea that when you’re backing stuff up between different providers, the idea of owning both sides of it—I know you offer a solution where you wind up hosting the data as well, and that has its value, don’t get me wrong, but there are also times, particularly for regulated industries, where yeah, I kind of don’t want my backup data just hanging out with someone else’s account with whatever they choose to do with it. There’s also the verification question, which again, I’m not accusing you of in any way, shape, or form of being nefarious, but it’s also one of those when I have to report to a board of directors of like, “Are you sure that they’re doing what they say they’re doing?” It’s a, “Well, he seemed trustworthy,” is not the greatest answer. And the boards ask questions like that all the time. Netflix has talked about this where they backup a rehydrate-the-business level of data to Google Cloud from AWS, not because they think Amazon is going to disappear off the face of the earth, but because it’s easier to do that and explain it than having to say, “Well, it’s extremely unlikely and here’s why,” and not get torn to pieces by auditors, shareholders, et cetera. It’s the path of least resistance, and there is some validity to it.


Simon: Yeah, when you see those big companies who’ve been with ransomware attacks and they’ve had to either pay the ransom or they’ve literally got to build the business from scratch, like, the cost associated with that is almost business-ending. So, just one backup for their data, off-site [laugh] they could have saved themselves millions and millions of pounds. So.


Corey: It’s one of those things where an ounce of prevention is worth a pound of cure. And we’re still seeing that stuff continue to evolve and continue to exist out in the ecosystem. There’s a whole host of things that I think about like, “Ooh, if I lost, that would be annoying but not disastrous.” When I was going through some contractual stuff when we were first setting up The Duckbill Group and talking to clients about this, they would periodically ask questions about, “Well, what’s your DR policy for these things?” It’s, “Well, we have a number of employees; no more than two are located in the same city anywhere, and we all work from laptops because it is the 21st century, so if someone’s internet goes out, they’ll go to a coffee shop. If everyone’s internet goes out, do you really care about the AWS bill that month?”


It’s a very different use case and [unintelligible 00:11:02] with these things. Now, let’s be clear, we are a consultancy that fixes AWS bills; we’re not a hospital. There’s a big difference in the use case and what is acceptable in different ways. But what I like is that you have really build something out that lets people choose their own adventure in how managed they want it to be, what the source is, what the target should be. And it gives people enough control but without having to worry about the finicky parts of aligning a bunch of scripts that wind up firing off in cron jobs.


Simon: Yeah. I’d say a fair few people run into issues running scripts or, you know, they silently fail and then you realize you haven’t actually been running backups for the last six months until you’re trying to pull them, even if you were trying to—


Corey: Bold of you to think that I would notice it that quickly.


Simon: [laugh]. Yeah, right. True. Yeah, that’s presuming you have a disaster recovery plan that you actually test. Lots of small businesses have never even heard of that as a thing. So, having as us, kind of, manage backups sort of enables us to very easily tell people that backups of, like—we couldn’t take the backup. Like, you need to address this.


Also, to your previous point about the control, you can decide completely where data flows between. So, when people ask us about what’s GDPR policies around data and stuff, we can say, “Well, we don’t actually handle your data in that sense. It goes directly from your source through almost a proxy that you control to your storage.” So.


Corey: The best answer: GDPR is out of scope. Please come again. And [laugh] yeah, just pass that off to someone else.


Simon: In a way, you’ve already approved those two: you’ve approved the person that you’re managing servers with and you’ve already approved the people that are doing storage with. You kind of… you do need to approve us, but we’re not handling the data. So, we’re handling your data, like your actual customer; we’re not handling your customer’s customer’s data.


Corey: Oh, yeah. Now, it’s a valuable thing. One of my famous personal backup issues was okay, “I’m going to back this up onto the shared drive,” and I sort of might have screwed up the backup script—in the better way, given the two possible directions this can go—but it was backing up all of its data and all the existing backup data, so you know, exponential growth of your backups. Now, my storage vendor was about to buy a boat and name it after me when I caught that. “Oh, yeah, let’s go ahead and fix that.”


But this stuff is finicky, it’s annoying, and in most cases, it fails in silent ways that only show up as a giant bill in one form or another. And not having to think about that is valuable. I’m willing to spend a few hours setting up a backup strategy and the rest; I’m not willing to tend it on an ongoing basis, just because I have other things I care about and things I need to get done.


Simon: Yeah. It’s such a kind of simple and trivial thing that can quickly become a nightmare [laugh] when you’ve made a mistake. So, not doing it yourself is a good [laugh] solution.


Corey: So, it wouldn’t have been a @patio11 recommendation to look at what you do without having some insight into the rest of the nuts and bolts of the business and the rest. Your plans are interesting. You have a free tier of course, which is a single daily backup job and half a gig of storage—or bring your own to that it’s unlimited storage—


Simon: Yep. Yeah.


Corey: Unlimited: the only limits are your budget. Yeah. Zombo.com got it slightly wrong. It’s not your mind, it’s your budget. And then it goes from Light to Startup to Business to Agency at the high end.


A question I have for you is at the high end, what I’ve found has been sort of the SaaS approach. The top end is always been a ‘Contact Us’ form where it’s the enterprise scope of folks where they tend to have procurement departments looking at this, and they’re going to have a whole bunch of custom contract stuff, but they’re also not used to signing checks with fewer than two commas in them. So, it’s the signaling and the messaging of, “Reach out and talk to us.” Have you experimented with that at all, yet? Is it something you haven’t gotten to yet or do you not have interest in serving that particular market segment?


Simon: I’d say we’ve been gearing the business from starting off very small with one solution to, you know, last—and two years ago, we added the ability to store data from one provider to a different provider. So, we’re sort of stair-stepping our way up to enterprise. For example, at the end of last year, we went and got certificates for ISO 27001 and… one other one, I can’t remember the name of them, and we’re probably going to get SOC 2 at some point this year. And then yes, we will be pushing more towards enterprises. We add, like, APIs as well so people can set up backups on the fly, or so they can put it as part of their provisioning.


That’s hopefully where I’m seeing the business go, as in we’ll become under-the-hood backup provider for, like, a managed hosting solution or something where their customers won’t even realize it’s us, but we’re taking the backups away from—responsibility away from businesses.


Corey: For those listeners who are fortunate enough to not have to have spent as long as I have in the woods of corporate governance, the correct answer to, “Well, how do we know that vendor is doing what they say that they’re doing,” because the, “Well, he seemed like a nice guy,” is not going to carry water, well, here are the certifications that they have attested to. Here’s copies under NDA, if their audit reports that call out what controls they claim to have and it validates that they are in fact doing what they say that they’re doing. That is corporate-speak that attests that you’re doing the right things. Now, you’re going to, in most cases, find yourself spending all your time doing work for no real money if you start making those things available to every customer spending 50 cents a year with you. So generally, the, “Oh, we’re going to go through the compliance, get you the reports,” is one of the higher, more expensive tiers where you must spend at least this much for us to start engaging down this rabbit hole of various nonsense.


And I don’t blame you in the least for not going down that path. One of these years, I’m going to wind up going through at least one of those certification approaches myself, but historically, we don’t handle anything except your billing data, and here’s how we do it has so far been sufficient for our contractual needs. But the world’s evolving; sophistication of enterprise buyers is at varying places and at some point, it’ll just be easier to go down that path.


Simon: Yeah, to be honest, we haven’t had many, many of those customers. Sometimes we have people who come in well over the plan limits, and that’s where we do a custom plan for them, but we’ve not had too many requests for certification. But obviously, we have the certification now, so if anyone ever [laugh] did want to see it under NDA, we could add some commas to any price. [laugh].


Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.


Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.


To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.


Corey: What I like as well is that you offer backups for a bunch of different things. You can do snapshots from, effectively, every provider. I’m sorry, I’m just going to call out because I love this: AWS and Amazon LightSail are called out as two distinct things. And Amazonians will say, “Oh, well, under the hood, they’re really the same thing, et cetera.” Yeah, the user experience is wildly different, so yeah, calling those things out as separate things make sense.


But it goes beyond that because it’s not just, “Well, I took a disk image. There we go. Come again.” You also offer backup recipes for specific things where you could, for example, back things up to a local file and external storage where someone is. Great, you also backup WordPress and MongoDB and MySQL and a whole bunch of other things.


A unified cloud controller, which is something I have in my house, and I keep thinking I should find a way to back that up. Yeah, this is great. It’s not just about the big server thing; it’s about having data living in managed services. It’s about making sure that the application data is backed up in a reasonable, responsible way. I really liked that approach. Was that an evolution or is that something you wound up focusing on almost from the beginning?


Simon: It was an evolution. So, we started with the snapshots, which got the business quite far to be honest and it was very simple. It was just DigitalOcean to start with, actually, for the first two years. Pretty easy to market in a way because it’s just focused on one thing. Then the other solutions came in, like the other providers and, you know, once you add one, it was easy to add many.


And then came database backups and file backups. And I just had those two solutions because that was what people were asking for. Like, they wanted to make sure their whole server snapshot, if you have a whole server snapshot, the point in time data for MySQL could be corrupt. Like, there could be stuff in RAM that a MySQL dump would have pulled out, for example. Like… there’s a possibility that the database could be corrupt from a snapshot, so people were asking for a bit of, more, peace of mind with doing proper backups of MySQL.


So, that’s what we added. And it soon became apparent when more customers were asking for more solutions that we really needed to, like, step back and think about what we’re actually offering. So, we rebuilt this whole, kind of like, database engine, then that allowed us to consume data from anywhere. So, we can easily add more backup types. So, the reason you can see all the ones you’ve listed there is because that’s kind of what people have been asking for. And every time someone comes up with a new, [laugh], like, a new open-source project or database or whatever, we’ll add support, even ones I’ve never heard of before. When people ask for some weird file—


Corey: All it takes is just waiting for someone to reach out and say, hey, can you back this thing up, please?


Simon: Yeah, exactly, some weird file-based database system that I’ve never ever heard of. Yeah, sure. Just give us [laugh] a test server to mess around with and we’ll build, essentially, like, we use bash in the background for doing the backups; if you can stream the data from a command, we can then deal with the whole management process. So, that’s the reason why. And then, I was seeing in, like, the Laravel space, for example, people were doing MySQL backups and they’d have a script, and then for whatever reason, someone rotated the passwords on the database and the backup script… was forgotten about.


So, there it is, not working for months. So, we thought we could build a backup where you could just point it at where the Laravel project is. We can get all the config we need at the runtime because it’s all there with the project anyway, and then thus, you never need to tell us the password for your database and that problem goes away. And it’s the same with WordPress.


Corey: I’m looking at this now just as you go through this, and I’m a big believer in disclaiming my biases, conflicts of interest, et cetera. And until this point, neither of us have traded a penny in either direction between us that I’m ever aware of—maybe you bought a t-shirt or something once upon a time—but great, I’m about to become a customer of this because I already have backup solutions for a lot of the things that you currently support, but again, when you’re a grumpy admin who’s lost data in the past, it’s, “Huh, you know what I would really like? That’s right, another backup.” And if that costs me a few hundred bucks a year for the peace of mind is money well spent because the failure mode is I get to rewrite a whole lot of blog posts and re-record all podcasts and pay for a whole bunch of custom development again. And it’s just not something that I particularly want to have to deal with. There’s something to be said for a holistic backup solution. I wish that more people thought about these things.


Simon: Can you imagine having to pull all the blog posts off [unintelligible 00:22:19]? [laugh]—


Corey: Oh, my got—


Simon: —to try and rebuild it.


Corey: That is called the crappiest summer internship someone has ever had.


Simon: Yeah.


Corey: And that is just painful. I can’t quite fathom having to do that as a strategy. Every once in a while some big site will have a data loss incident or go out of business or something, and there’s a frantic archiving endeavor that happens where people are trying to copy the content out of the Google Search Engine’s cache before it expires at whatever timeline that is. And that looks like the worst possible situation for any sort of giant backup.


Simon: At least that’s one you can fix. I mean, if you were to lose all the payment information, then you’ve got to restitch all that together, or anything else. Like, that’s a fixable solution, but a lot of these other ones, if you lose the data, yeah, there’s no two ways around it, you’re screwed. So.


Corey: Yeah, it’s a challenging thing. And it’s also—the question also becomes one of, “Well, hang on. I know about backups on this because I have this data, but it’s used to working in an AWS environment. What possible good would it do me sitting somewhere else?” It’s, yeah, the point is, it’s sitting somewhere else, at least in my experience. You can copy it back to that sort of environment.


I’m not suggesting this is a way that you can run your AWS serverless environment on DigitalOcean, but it’s a matter of if everything turns against you, you can rebuild from those backups. That’s the approach that I’ve usually taken. Do you find that your customers understand that going in or is there an education process?


Simon: I’d say people come for all sorts of reasons for why they want backup. So, having your data in two places for that is one of the reasons but, you know, I think there’s a lot of reasons why people want peace of mind: for either developer mistakes or migration mistakes or hacking, all these things. So, I guess the big one we come up with a lot is people talking about databases and they don’t need backups because they’ve got replication. And trying to explain that replication between two databases isn’t the same as a backup. Like, you make a mistake you drop—[laugh] you run your delete query wrong on the first database, it’s gone, replicated or not.


Corey: Right, the odds of me fat-fingering an S3 bucket command are incredibly likelier than the odds of AWS losing an entire region’s S3 data irretrievably. I make mistakes a lot more than they tend to architecturally, but let’s also be clear, they’re one of the best. My impression has always been the big three mostly do a decent job of this. The jury’s still out, in my opinion, on other third-party clouds that are not, I guess, tier one. What’s your take?


Simon: I have to be careful. I’ve got quite good relationships with some of these. [laugh].


Corey: Oh, of course. Of course. Of course.


Simon: But yes, I would say most customers do end up using S3 as their storage option, and I think that is because it is, I think, the best. Like, is in terms of reliability and performance, some storage can be a little slow at times for pulling data in, which could or could not be a problem depending on what your use case is. But there are some trade-offs. Obviously, S3, if you’re trying to get your data back out, is expensive. If you were to look at Backblaze, for example, as well, that’s considerably cheaper than S3, especially, like, when you’re talking in the petabyte-scale, there can be huge savings there. So… they all sort of bring their own thing to the table. Personally, I store the backups in S3 and in Backblaze, and in one other provider. [laugh].


Corey: Oh, yeah. Like—


Simon: I like to have them spread.


Corey: Like, every once in a while in the industry, there’s something that happens that’s sort of a watershed moment where it reminds everyone, “Oh, right. That’s why we do backups.” I think the most recent one—and again, love to them; this stuff is never fun—was when that OVH data center burned down. And OVH is a somewhat more traditional hosting provider, in some respects. Like, their pricing is great, but they wind up giving you what amounts to here as a server in a rack. You get to build all this stuff yourself.


And that backup story is one of those. Oh, okay. Well, I just got two of them and I’ll copy backups to each other. Yeah, but they’re in the same building and that building just burned down. Now, what? And a lot of people learned a very painful lesson. And oh, right, that’s why we have to do that.


Simon: Yeah. The other big lesson from that was that even if the people with data in a different region—like, they’d had cross-regional backups—because of the demand at the time for accessing backups, if you wanted to get your data quickly, you’re in a queue because so many other people were in the same boat as you’re trying to restore stored backups. So, being off-site with a different provider would have made that a little easier. [laugh].


Corey: It’s a herd of elephants problem. You test your DR strategy on a scheduled basis; great, you’re the only person doing it—give or take—at that time, as opposed to a large provider has lost a region and everyone is hitting their backup service simultaneously. It generally isn’t built for that type of scale and provisioning. One other question I have for you is when I make mistakes, for better or worse, they’re usually relatively small-scale. I want to restore a certain file or I will want to, “Ooh, that one item I just dropped out of that database really should not have been dropped.” Do you currently offer things that go beyond the entire restore everything or nothing? Or right now are you still approaching this from the perspective of this is for the catastrophic case where you’re in some pain already?


Simon: Mostly the catastrophic stage. So, we have MySQL [bin logs 00:27:57] as an option. So, if you wanted to do, like, a point-in-time of store, which… may be more applicable to what you’re saying, but generally, its whole, whole website recovery. For example, like, we have a WordPress backup that’ll go through all the WordPress websites on the server and we’ll back them up individually so you can restore just one. There are ways that we have helped customers in the past just pull one table, for example, from a backup.


But yeah, we geared towards, kind of, the set and the forget. And people don’t often restore backups, to be honest. They don’t. But when they do, it’s obviously [laugh] very crucial that they work, so I prefer to back up the whole thing and then help people, like, if you need to extract ten megabytes out of an entire gig backup, that’s a bit wasteful, but at least, you know, you’ve got the data there. So.


Corey: Yeah. I’m a big believer in having backups in a variety of different levels. Because I don’t really want to do a whole server restore when I remove a file. And let’s be clear, I still have that grumpy old Unix admin of before I start making changes to a file, yeah, my editor can undo things and remembers that persistently and all. But I have a disturbing number of files and directories whose names end in ‘.bac’ with then, like, a date or something on it, just because it’s—you know, like, “Oh, I have to fix something in Git. How do I do this?”


Step one, I’m going to copy the entire directory so when I make a pig’s breakfast out of this and I lose things that I care about, rather than having to play Git surgeon for two more days, I can just copy it back over and try again. Disk space is cheap for those things. But that’s also not a holistic backup strategy because I have to remember to do it every time and the whole point of what you’re building and the value you’re adding, from my perspective, is people don’t have to think about it.


Simon: Yes. Yeah yeah yeah. Once it’s there, it’s there. It’s running. It’s as you say, it’s not the most efficient thing if you wanted to restore one file—not to say you couldn’t—but at least you didn’t have to think about doing the backup first.


Corey: I really want to thank you for taking the time out of your day to talk to me about all this. If people want to learn more for themselves, where can they find you?


Simon: So, SnapShooter.com is a great place, or on Twitter, if you want to follow me. I am @MrSimonBennett.


Corey: And we will, of course, put links to that in the [show notes 00:30:11]. Thank you once again. I really appreciate it.


Simon: Thank you. Thank you very much for having me.


Corey: Simon Bennett, founder and CEO of SnapShooter.com. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve 
enjoyed this episode, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this episode, please leave a five-star review on your podcast platform of choice, along with an angry insulting comment that, just like your backup strategy, you haven’t 
put enough thought into.


Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.


Announcer: This has been a HumblePod production. Stay humble.

Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that’s where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats’s snark.cloud/D-U-P-L-O-C-L-O-U-D.

Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. One of the things that I learned early on in my career as a grumpy Unix systems administrator is that there are two kinds of people out there: those who care about backups an awful lot, and people who haven’t lost data yet. I lost a bunch of data once upon a time and then I too fell on the side of backups are super important. Here to talk with me about them a bit today is Simon Bennett, founder and CEO of SnapShooter.com. Simon, thanks for joining me.

Simon: Thanks for having me. Thank you very much.

Corey: It’s fun to be able to talk to people who are doing business in the cloud space—in this sense too—that is not venture-backed, that is not, “Well, we have 600 people here that are building this thing out.” And similar to the way that I handle things at The Duckbill Group, you are effectively one of those legacy things known as a profitable business that self-funds. What made you decide to pursue that model as opposed to, well, whatever the polite version of bilking venture capitalists out of enormous piles of money for [unintelligible 00:01:32]?

Simon: I think I always liked the idea of being self-sufficient and running a business, so I always wanted to start a physical business when I was younger, but when I got into software, I realized that that’s a really easy way, no capital needed, to get started. And I tried for years and years to build products, all of which failed until finally SnapShooter actually gained a customer. [laugh].

Corey: “Oh, wait, someone finally is paying money for this, I guess I’m onto something.”

Simon: Yeah.

Corey: And it’s sort of progressed from there. How long have you been in business?

Simon: We started in 2017, as… it was an internal project for a company I was working at who had problems with DigitalOcean backups, or they had problems with their servers getting compromised. So, I looked at DigitalOcean API and realized I could build something. And it took less than a week to build a product [with billing 00:02:20]. And I put that online and people started using it. So, that was how it worked.

Every other product I tried before, I’d spent months and months developing it and never getting a customer. And the one time I spent less than [laugh] less than a week’s worth of evenings, someone started paying. I mean, admittedly, the first person was only paying a couple of dollars a month, but it was something.

Corey: There’s a huge turning point where you just validate the ability and willingness for someone to transfer one dollar from their bank account to yours. It speaks to validation in a way that social media nonsense generally doesn’t. It’s the oh, someone is actually willing to pay because I’m adding value to what they do. That’s no small thing.

Simon: Yeah. There’s definitely a big difference between people saying they’re going to and they’d love it, and actually doing it. So.

Corey: I first heard about you when Patrick McKenzie—or @patio11, as he goes by on Twitter—wound up doing a mini-thread on you about, “I’ve now used SnapShooter.com for real, and it was such a joy, including making a server migration easier than it would otherwise have been. Now, I have automatically monitored backups to my own S3 account for a bunch of things, which already had a fairly remote risk of failure.” And he keeps talking about the awesome aspects of it. And okay, when Patrick says, “This is neat,” that usually means it’s time for me to at least click the link and see what’s going on.

And the thing that jumped out at me was a few things about what it is that you offer. You talk about making sure that people can sleep well at night, that it’s about why backups are important, about—you obviously check the boxes and talk about how you do things and why you do them the way that you do, but it resonates around the idea of helping people sleep well at night. Because no one wants to think about backups. Because no one cares about backups; they just care an awful lot about restores, usually right after they should have cared about the backups.

Simon: Yeah. This is actually a big problem with getting customers because I don’t think it’s on a lot of people’s minds, getting backups set up until, as you said in the intro, something’s gone wrong. [laugh]. And then they’re happy to be a customer for life.

Corey: I started clicking around and looking at your testimonials, for example, on your website. And the first one I saw was from the CEO of Transistor.fm. For those who aren’t familiar with what they do, they are the company that hosts this podcast. I pay them as a vendor for all the back issues and whatnot.

Whenever you download the show. It’s routing through their stuff. So yeah, I kind of want them to have backups of these things because I really don’t want to have all these conversations [laugh] again with everyone. That’s an important thing. But Transistor’s business is not making sure that the data is safe and secure; it’s making podcasts available, making it easy to publish to them.

And in your case, you’re handling the backup portion of it so they can pay their money and they set it up effectively once—set it and forget it—and then they can go back to doing the thing that they do, and not having to fuss with it constantly. I think a lot of companies get it wrong, where they seem to think that people are going to make sustained, engaged efforts in whatever platform or tool or service they build. People have bigger fish to fry; they just want the thing to work and not take up brain sweat.

Simon: Yeah. Customers hardly ever log in. I think it’s probably a good sign when they don’t have to log in. So, they get their report emails, and that’s that. And they obviously come back when they got new stuff to set up, but from a support point of view is pretty, pretty easy, really, people don’t—[laugh] constantly on there.

Corey: From where I sit, the large cloud providers—and some of the small ones, too—they all have backup functionality built into the offering that they’ve got. And some are great, some are terrible. I assume—perhaps naively—that all of them do what it says on the tin and actually back up the data. If that were sufficient, you wouldn’t have any customers. You clearly have customers. What is it that makes those things not work super well?

Simon: Some of them are inflexible. So, some of the providers have built-in server backups that only happen weekly, and six days of no backups can be a big problem when you’ve made a mistake. So, we offer a lot of flexibility around how often you backup your data. And then another key part is that we let you store your data where you want. A lot of the providers have either vendor lock-in, or they only store it in themselves. So… we let you take your data from one side of the globe to the other if you want.

Corey: As anyone who has listened to the show is aware, I’m not a huge advocate for multi-cloud for a variety of excellent reasons. And I mean that on a per-workload basis, not, “Oh, we’re going to go with one company called Amazon,” and you use everything that they do, including their WorkMail product. Yeah, even Amazon doesn’t use WorkMail; they use Exchange like a real company would. And great, pick the thing that works best for you, but backups have always been one of those areas.

I know that AWS has great region separation—most of the time. I know that it is unheard of for there to be a catastrophic data loss story that transcends multiple regions, so the story from their side is very often, oh, just back it up to a different region. Problem solved. Ignoring the data transfer aspect of that from a pricing perspective, okay. But there’s also a risk element here where everyone talks about the single point of failure with the AWS account that it’s there, people don’t talk about as much: it’s your payment instrument; if they suspend your account, you’re not getting into any region.

There’s also the story of if someone gets access to your account, how do you back that up? If you’re going to be doing backups, from my perspective, that is the perfect use case, to put it on a different provider. Because if I’m backing up from, I don’t know, Amazon to Google Cloud or vice versa, I have a hard time envisioning a scenario in which both of those companies simultaneously have lost my data and I still care about computers. It is very hard for me to imagine that kind of failure mode, it’s way out of scope for any disaster recovery or business continuity plan that I’m coming up with.

Simon: Yeah, that’s right. Yeah, I haven’t—[laugh] I don’t have that in my disaster recovery plan, to be honest about going to a different cloud, as in, we’ll solve that problem when it happens. But the data is, as you say, in two different places, or more. But yeah, the security one is a key one because, you know, there’s quite a lot of surface area on your AWS account for compromising, but if you’re using either—even a separate AWS account or a different provider purely for storage, that can be very tightly controlled.

Corey: I also appreciate the idea that when you’re backing stuff up between different providers, the idea of owning both sides of it—I know you offer a solution where you wind up hosting the data as well, and that has its value, don’t get me wrong, but there are also times, particularly for regulated industries, where yeah, I kind of don’t want my backup data just hanging out with someone else’s account with whatever they choose to do with it. There’s also the verification question, which again, I’m not accusing you of in any way, shape, or form of being nefarious, but it’s also one of those when I have to report to a board of directors of like, “Are you sure that they’re doing what they say they’re doing?” It’s a, “Well, he seemed trustworthy,” is not the greatest answer. And the boards ask questions like that all the time. Netflix has talked about this where they backup a rehydrate-the-business level of data to Google Cloud from AWS, not because they think Amazon is going to disappear off the face of the earth, but because it’s easier to do that and explain it than having to say, “Well, it’s extremely unlikely and here’s why,” and not get torn to pieces by auditors, shareholders, et cetera. It’s the path of least resistance, and there is some validity to it.

Simon: Yeah, when you see those big companies who’ve been with ransomware attacks and they’ve had to either pay the ransom or they’ve literally got to build the business from scratch, like, the cost associated with that is almost business-ending. So, just one backup for their data, off-site [laugh] they could have saved themselves millions and millions of pounds. So.

Corey: It’s one of those things where an ounce of prevention is worth a pound of cure. And we’re still seeing that stuff continue to evolve and continue to exist out in the ecosystem. There’s a whole host of things that I think about like, “Ooh, if I lost, that would be annoying but not disastrous.” When I was going through some contractual stuff when we were first setting up The Duckbill Group and talking to clients about this, they would periodically ask questions about, “Well, what’s your DR policy for these things?” It’s, “Well, we have a number of employees; no more than two are located in the same city anywhere, and we all work from laptops because it is the 21st century, so if someone’s internet goes out, they’ll go to a coffee shop. If everyone’s internet goes out, do you really care about the AWS bill that month?”

It’s a very different use case and [unintelligible 00:11:02] with these things. Now, let’s be clear, we are a consultancy that fixes AWS bills; we’re not a hospital. There’s a big difference in the use case and what is acceptable in different ways. But what I like is that you have really build something out that lets people choose their own adventure in how managed they want it to be, what the source is, what the target should be. And it gives people enough control but without having to worry about the finicky parts of aligning a bunch of scripts that wind up firing off in cron jobs.

Simon: Yeah. I’d say a fair few people run into issues running scripts or, you know, they silently fail and then you realize you haven’t actually been running backups for the last six months until you’re trying to pull them, even if you were trying to—

Corey: Bold of you to think that I would notice it that quickly.

Simon: [laugh]. Yeah, right. True. Yeah, that’s presuming you have a disaster recovery plan that you actually test. Lots of small businesses have never even heard of that as a thing. So, having as us, kind of, manage backups sort of enables us to very easily tell people that backups of, like—we couldn’t take the backup. Like, you need to address this.

Also, to your previous point about the control, you can decide completely where data flows between. So, when people ask us about what’s GDPR policies around data and stuff, we can say, “Well, we don’t actually handle your data in that sense. It goes directly from your source through almost a proxy that you control to your storage.” So.

Corey: The best answer: GDPR is out of scope. Please come again. And [laugh] yeah, just pass that off to someone else.

Simon: In a way, you’ve already approved those two: you’ve approved the person that you’re managing servers with and you’ve already approved the people that are doing storage with. You kind of… you do need to approve us, but we’re not handling the data. So, we’re handling your data, like your actual customer; we’re not handling your customer’s customer’s data.

Corey: Oh, yeah. Now, it’s a valuable thing. One of my famous personal backup issues was okay, “I’m going to back this up onto the shared drive,” and I sort of might have screwed up the backup script—in the better way, given the two possible directions this can go—but it was backing up all of its data and all the existing backup data, so you know, exponential growth of your backups. Now, my storage vendor was about to buy a boat and name it after me when I caught that. “Oh, yeah, let’s go ahead and fix that.”

But this stuff is finicky, it’s annoying, and in most cases, it fails in silent ways that only show up as a giant bill in one form or another. And not having to think about that is valuable. I’m willing to spend a few hours setting up a backup strategy and the rest; I’m not willing to tend it on an ongoing basis, just because I have other things I care about and things I need to get done.

Simon: Yeah. It’s such a kind of simple and trivial thing that can quickly become a nightmare [laugh] when you’ve made a mistake. So, not doing it yourself is a good [laugh] solution.

Corey: So, it wouldn’t have been a @patio11 recommendation to look at what you do without having some insight into the rest of the nuts and bolts of the business and the rest. Your plans are interesting. You have a free tier of course, which is a single daily backup job and half a gig of storage—or bring your own to that it’s unlimited storage—

Simon: Yep. Yeah.

Corey: Unlimited: the only limits are your budget. Yeah. Zombo.com got it slightly wrong. It’s not your mind, it’s your budget. And then it goes from Light to Startup to Business to Agency at the high end.

A question I have for you is at the high end, what I’ve found has been sort of the SaaS approach. The top end is always been a ‘Contact Us’ form where it’s the enterprise scope of folks where they tend to have procurement departments looking at this, and they’re going to have a whole bunch of custom contract stuff, but they’re also not used to signing checks with fewer than two commas in them. So, it’s the signaling and the messaging of, “Reach out and talk to us.” Have you experimented with that at all, yet? Is it something you haven’t gotten to yet or do you not have interest in serving that particular market segment?

Simon: I’d say we’ve been gearing the business from starting off very small with one solution to, you know, last—and two years ago, we added the ability to store data from one provider to a different provider. So, we’re sort of stair-stepping our way up to enterprise. For example, at the end of last year, we went and got certificates for ISO 27001 and… one other one, I can’t remember the name of them, and we’re probably going to get SOC 2 at some point this year. And then yes, we will be pushing more towards enterprises. We add, like, APIs as well so people can set up backups on the fly, or so they can put it as part of their provisioning.

That’s hopefully where I’m seeing the business go, as in we’ll become under-the-hood backup provider for, like, a managed hosting solution or something where their customers won’t even realize it’s us, but we’re taking the backups away from—responsibility away from businesses.

Corey: For those listeners who are fortunate enough to not have to have spent as long as I have in the woods of corporate governance, the correct answer to, “Well, how do we know that vendor is doing what they say that they’re doing,” because the, “Well, he seemed like a nice guy,” is not going to carry water, well, here are the certifications that they have attested to. Here’s copies under NDA, if their audit reports that call out what controls they claim to have and it validates that they are in fact doing what they say that they’re doing. That is corporate-speak that attests that you’re doing the right things. Now, you’re going to, in most cases, find yourself spending all your time doing work for no real money if you start making those things available to every customer spending 50 cents a year with you. So generally, the, “Oh, we’re going to go through the compliance, get you the reports,” is one of the higher, more expensive tiers where you must spend at least this much for us to start engaging down this rabbit hole of various nonsense.

And I don’t blame you in the least for not going down that path. One of these years, I’m going to wind up going through at least one of those certification approaches myself, but historically, we don’t handle anything except your billing data, and here’s how we do it has so far been sufficient for our contractual needs. But the world’s evolving; sophistication of enterprise buyers is at varying places and at some point, it’ll just be easier to go down that path.

Simon: Yeah, to be honest, we haven’t had many, many of those customers. Sometimes we have people who come in well over the plan limits, and that’s where we do a custom plan for them, but we’ve not had too many requests for certification. But obviously, we have the certification now, so if anyone ever [laugh] did want to see it under NDA, we could add some commas to any price. [laugh].

Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.

Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.

To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.

Corey: What I like as well is that you offer backups for a bunch of different things. You can do snapshots from, effectively, every provider. I’m sorry, I’m just going to call out because I love this: AWS and Amazon LightSail are called out as two distinct things. And Amazonians will say, “Oh, well, under the hood, they’re really the same thing, et cetera.” Yeah, the user experience is wildly different, so yeah, calling those things out as separate things make sense.

But it goes beyond that because it’s not just, “Well, I took a disk image. There we go. Come again.” You also offer backup recipes for specific things where you could, for example, back things up to a local file and external storage where someone is. Great, you also backup WordPress and MongoDB and MySQL and a whole bunch of other things.

A unified cloud controller, which is something I have in my house, and I keep thinking I should find a way to back that up. Yeah, this is great. It’s not just about the big server thing; it’s about having data living in managed services. It’s about making sure that the application data is backed up in a reasonable, responsible way. I really liked that approach. Was that an evolution or is that something you wound up focusing on almost from the beginning?

Simon: It was an evolution. So, we started with the snapshots, which got the business quite far to be honest and it was very simple. It was just DigitalOcean to start with, actually, for the first two years. Pretty easy to market in a way because it’s just focused on one thing. Then the other solutions came in, like the other providers and, you know, once you add one, it was easy to add many.

And then came database backups and file backups. And I just had those two solutions because that was what people were asking for. Like, they wanted to make sure their whole server snapshot, if you have a whole server snapshot, the point in time data for MySQL could be corrupt. Like, there could be stuff in RAM that a MySQL dump would have pulled out, for example. Like… there’s a possibility that the database could be corrupt from a snapshot, so people were asking for a bit of, more, peace of mind with doing proper backups of MySQL.

So, that’s what we added. And it soon became apparent when more customers were asking for more solutions that we really needed to, like, step back and think about what we’re actually offering. So, we rebuilt this whole, kind of like, database engine, then that allowed us to consume data from anywhere. So, we can easily add more backup types. So, the reason you can see all the ones you’ve listed there is because that’s kind of what people have been asking for. And every time someone comes up with a new, [laugh], like, a new open-source project or database or whatever, we’ll add support, even ones I’ve never heard of before. When people ask for some weird file—

Corey: All it takes is just waiting for someone to reach out and say, hey, can you back this thing up, please?

Simon: Yeah, exactly, some weird file-based database system that I’ve never ever heard of. Yeah, sure. Just give us [laugh] a test server to mess around with and we’ll build, essentially, like, we use bash in the background for doing the backups; if you can stream the data from a command, we can then deal with the whole management process. So, that’s the reason why. And then, I was seeing in, like, the Laravel space, for example, people were doing MySQL backups and they’d have a script, and then for whatever reason, someone rotated the passwords on the database and the backup script… was forgotten about.

So, there it is, not working for months. So, we thought we could build a backup where you could just point it at where the Laravel project is. We can get all the config we need at the runtime because it’s all there with the project anyway, and then thus, you never need to tell us the password for your database and that problem goes away. And it’s the same with WordPress.

Corey: I’m looking at this now just as you go through this, and I’m a big believer in disclaiming my biases, conflicts of interest, et cetera. And until this point, neither of us have traded a penny in either direction between us that I’m ever aware of—maybe you bought a t-shirt or something once upon a time—but great, I’m about to become a customer of this because I already have backup solutions for a lot of the things that you currently support, but again, when you’re a grumpy admin who’s lost data in the past, it’s, “Huh, you know what I would really like? That’s right, another backup.” And if that costs me a few hundred bucks a year for the peace of mind is money well spent because the failure mode is I get to rewrite a whole lot of blog posts and re-record all podcasts and pay for a whole bunch of custom development again. And it’s just not something that I particularly want to have to deal with. There’s something to be said for a holistic backup solution. I wish that more people thought about these things.

Simon: Can you imagine having to pull all the blog posts off [unintelligible 00:22:19]? [laugh]—

Corey: Oh, my got—

Simon: —to try and rebuild it.

Corey: That is called the crappiest summer internship someone has ever had.

Simon: Yeah.

Corey: And that is just painful. I can’t quite fathom having to do that as a strategy. Every once in a while some big site will have a data loss incident or go out of business or something, and there’s a frantic archiving endeavor that happens where people are trying to copy the content out of the Google Search Engine’s cache before it expires at whatever timeline that is. And that looks like the worst possible situation for any sort of giant backup.

Simon: At least that’s one you can fix. I mean, if you were to lose all the payment information, then you’ve got to restitch all that together, or anything else. Like, that’s a fixable solution, but a lot of these other ones, if you lose the data, yeah, there’s no two ways around it, you’re screwed. So.

Corey: Yeah, it’s a challenging thing. And it’s also—the question also becomes one of, “Well, hang on. I know about backups on this because I have this data, but it’s used to working in an AWS environment. What possible good would it do me sitting somewhere else?” It’s, yeah, the point is, it’s sitting somewhere else, at least in my experience. You can copy it back to that sort of environment.

I’m not suggesting this is a way that you can run your AWS serverless environment on DigitalOcean, but it’s a matter of if everything turns against you, you can rebuild from those backups. That’s the approach that I’ve usually taken. Do you find that your customers understand that going in or is there an education process?

Simon: I’d say people come for all sorts of reasons for why they want backup. So, having your data in two places for that is one of the reasons but, you know, I think there’s a lot of reasons why people want peace of mind: for either developer mistakes or migration mistakes or hacking, all these things. So, I guess the big one we come up with a lot is people talking about databases and they don’t need backups because they’ve got replication. And trying to explain that replication between two databases isn’t the same as a backup. Like, you make a mistake you drop—[laugh] you run your delete query wrong on the first database, it’s gone, replicated or not.

Corey: Right, the odds of me fat-fingering an S3 bucket command are incredibly likelier than the odds of AWS losing an entire region’s S3 data irretrievably. I make mistakes a lot more than they tend to architecturally, but let’s also be clear, they’re one of the best. My impression has always been the big three mostly do a decent job of this. The jury’s still out, in my opinion, on other third-party clouds that are not, I guess, tier one. What’s your take?

Simon: I have to be careful. I’ve got quite good relationships with some of these. [laugh].

Corey: Oh, of course. Of course. Of course.

Simon: But yes, I would say most customers do end up using S3 as their storage option, and I think that is because it is, I think, the best. Like, is in terms of reliability and performance, some storage can be a little slow at times for pulling data in, which could or could not be a problem depending on what your use case is. But there are some trade-offs. Obviously, S3, if you’re trying to get your data back out, is expensive. If you were to look at Backblaze, for example, as well, that’s considerably cheaper than S3, especially, like, when you’re talking in the petabyte-scale, there can be huge savings there. So… they all sort of bring their own thing to the table. Personally, I store the backups in S3 and in Backblaze, and in one other provider. [laugh].

Corey: Oh, yeah. Like—

Simon: I like to have them spread.

Corey: Like, every once in a while in the industry, there’s something that happens that’s sort of a watershed moment where it reminds everyone, “Oh, right. That’s why we do backups.” I think the most recent one—and again, love to them; this stuff is never fun—was when that OVH data center burned down. And OVH is a somewhat more traditional hosting provider, in some respects. Like, their pricing is great, but they wind up giving you what amounts to here as a server in a rack. You get to build all this stuff yourself.

And that backup story is one of those. Oh, okay. Well, I just got two of them and I’ll copy backups to each other. Yeah, but they’re in the same building and that building just burned down. Now, what? And a lot of people learned a very painful lesson. And oh, right, that’s why we have to do that.

Simon: Yeah. The other big lesson from that was that even if the people with data in a different region—like, they’d had cross-regional backups—because of the demand at the time for accessing backups, if you wanted to get your data quickly, you’re in a queue because so many other people were in the same boat as you’re trying to restore stored backups. So, being off-site with a different provider would have made that a little easier. [laugh].

Corey: It’s a herd of elephants problem. You test your DR strategy on a scheduled basis; great, you’re the only person doing it—give or take—at that time, as opposed to a large provider has lost a region and everyone is hitting their backup service simultaneously. It generally isn’t built for that type of scale and provisioning. One other question I have for you is when I make mistakes, for better or worse, they’re usually relatively small-scale. I want to restore a certain file or I will want to, “Ooh, that one item I just dropped out of that database really should not have been dropped.” Do you currently offer things that go beyond the entire restore everything or nothing? Or right now are you still approaching this from the perspective of this is for the catastrophic case where you’re in some pain already?

Simon: Mostly the catastrophic stage. So, we have MySQL [bin logs 00:27:57] as an option. So, if you wanted to do, like, a point-in-time of store, which… may be more applicable to what you’re saying, but generally, its whole, whole website recovery. For example, like, we have a WordPress backup that’ll go through all the WordPress websites on the server and we’ll back them up individually so you can restore just one. There are ways that we have helped customers in the past just pull one table, for example, from a backup.

But yeah, we geared towards, kind of, the set and the forget. And people don’t often restore backups, to be honest. They don’t. But when they do, it’s obviously [laugh] very crucial that they work, so I prefer to back up the whole thing and then help people, like, if you need to extract ten megabytes out of an entire gig backup, that’s a bit wasteful, but at least, you know, you’ve got the data there. So.

Corey: Yeah. I’m a big believer in having backups in a variety of different levels. Because I don’t really want to do a whole server restore when I remove a file. And let’s be clear, I still have that grumpy old Unix admin of before I start making changes to a file, yeah, my editor can undo things and remembers that persistently and all. But I have a disturbing number of files and directories whose names end in ‘.bac’ with then, like, a date or something on it, just because it’s—you know, like, “Oh, I have to fix something in Git. How do I do this?”

Step one, I’m going to copy the entire directory so when I make a pig’s breakfast out of this and I lose things that I care about, rather than having to play Git surgeon for two more days, I can just copy it back over and try again. Disk space is cheap for those things. But that’s also not a holistic backup strategy because I have to remember to do it every time and the whole point of what you’re building and the value you’re adding, from my perspective, is people don’t have to think about it.

Simon: Yes. Yeah yeah yeah. Once it’s there, it’s there. It’s running. It’s as you say, it’s not the most efficient thing if you wanted to restore one file—not to say you couldn’t—but at least you didn’t have to think about doing the backup first.

Corey: I really want to thank you for taking the time out of your day to talk to me about all this. If people want to learn more for themselves, where can they find you?

Simon: So, SnapShooter.com is a great place, or on Twitter, if you want to follow me. I am @MrSimonBennett.

Corey: And we will, of course, put links to that in the [show notes 00:30:11]. Thank you once again. I really appreciate it.

Simon: Thank you. Thank you very much for having me.

Corey: Simon Bennett, founder and CEO of SnapShooter.com. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this episode, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this episode, please leave a five-star review on your podcast platform of choice, along with an angry insulting comment that, just like your backup strategy, you haven’t put enough thought into.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Announcer: This has been a HumblePod production. Stay humble.

Newsletter Footer

Get the Newsletter

Reach over 30,000 discerning engineers, managers, enthusiasts who actually care about the state of Amazon’s cloud ecosystems.

"*" indicates required fields

This field is for validation purposes and should be left unchanged.
Sponsor Icon Footer

Sponsor an Episode

Get your message in front of people who care enough to keep current about the cloud phenomenon and its business impacts.