AWS Storage Day 2020 Part 2

Transcript
Corey: Gravitational is now Teleport because when way more people have heard of your product than your company, maybe that’s a sign it’s a time to change your branding. Teleport enables engineers to quickly access any computing resource, anywhere on the planet. You know, like VPNs were supposed to do before we all started working from home, and the VPNs melted like glaciers. Teleport provides a unified access plane for developers and security professionals seeking to simplify secure access to servers, applications, and data across all of your environments without the bottleneck and management overhead of traditional VPNs. This feels to me like it’s a lot like the early days of HashiCorp’s Terraform. My gut tells me this is the sort of thing that’s going to transform how people access their cloud services and environments. To learn more, visit goteleport.com.

Pete: Hello, and welcome to AWS Morning Brief. I am Pete Cheslock, and I'm also here, again, with Jesse DeRose. Hey, Jesse, how's it going?

Jesse: Not too bad. Thanks for having me.

Pete: It is part two of AWS Storage Day. If you haven't had the chance to listen to last week's episode, Jesse and I dove into some of the new features really focusing on what we would think is the biggest feature of AWS Storage Day, which was the S3 Intelligent Tiering. Go back and listen to it if you didn't hear about it. But essentially, Amazon keeps extending out features [00:01:34 unintelligible] this Intelligent Tiering platform. And we talked a little bit about it last week. 

But there were a lot of announcements as part of Storage Day, some pretty impressive, and some that were maybe a little underwhelming. We'll let you be the judge of that because some of these things could be incredibly important for you as—maybe—someone who operates on Amazon. So, now what we're going to do is we're going to dive into some of the other features, not only additional interesting S3 features, but there were a lot of new features announced around EBS, and EFS, and FSx, and all of the different ways that you can interact with AWS storage. I don't want to call it the biggest feature of this section because I think—let's be honest—they're all equally meh features, right, Jesse?

Jesse: Yeah.

Pete: I think that's going to be the common thread. Again, you might look at some of these features and go, “Finally, my life is so much better because they've announced this feature.” But I got to say, outside of Intelligent Tiering, Storage Day felt a little weak. But let's dive in anyway. S3 Replication; if you are replicating your data from one S3 bucket to another bucket, another region, which maybe you need to do for compliance reasons, disaster recovery reasons, some of the new features they added are around replication metrics and notifications. 

Now, previously, these metrics and notifications were only available if you used the Time Control Replication, and that is a additional charge to get a predictable SLA for your data to be backed up. They made these metrics now available for anyone, so that's actually awesome to hear that they’ve really just extended that out and are kind of giving you something for free. Additionally, they now replicate delete markers, which I swear I looked at a bunch of documents to understand better what delete markers mean, and the best I got to it, I don't actually really understand the problem from before, other than as you delete a version of something in the source, the delete marker moves over. But then maybe the previous versions are in the destination. That was my gist of it, Jessie, what was your gist of that one?

Jesse: Yeah, I struggled a little bit with some of these previously because S3 replication always felt like this magical hand-wavy feature where you turned it on and then just waited, and eventually your objects would show up in your destination bucket or destination folder. But there wasn't really any clear path to what was going on behind the scenes. So, I'm really excited to see that now these metrics and notifications are available to everyone, not just to folks who were using the Replication Time Control feature, and allows everybody to more easily understand how their data is replicating between S3 buckets behind the scenes. So, I feel good about this one. I feel like this is definitely a step in the right direction. I'm really excited to see that this is now broadly available for everybody that's using S3. I think it will make using S3 Replication easier for a lot of folks who need it for business purposes or any other use case.

Pete: Yeah, absolutely. Another really awesome feature—I was actually excited for this because, of course, it must affect me in my day-to-day—S3 object ownership is now available for all the Amazon regions and amazingly supported by CloudFormation, which I feel like is always an afterthought. But what this allows you to do is you can use this feature too, when you upload files, it'll make sure that the ownership is assumed by the bucket you've uploaded it into. And so this gets around a lot of hairy issues that come into S3 permissioning, IAM permissioning. I mean, S3 permissioning, in general, predates IAM. I don't know how many people actually know that. And I think because of it, there are some really gnarly edge cases people run into, and this is a big problem solver.

Jesse: I am really, really excited about this feature release, I cannot say how many times we've run into this edge case with some of our internal tooling because we have effectively copied or synced data from a client's S3 bucket into our S3 bucket, and we don't gain ownership. And that becomes such a permissioning headache to be able to do anything with that data once we have it in our S3 bucket. So, I'm really excited to see that object ownership is now not only a first-class citizen but now is also built into and supported by AWS CloudFormation.

Pete: Yeah, absolutely. Another new feature: it has to do with Outpost actually, and you can get S3 on Outposts now which, that's truly amazing if you think about it. Now, I don't know of anyone who actually is using Outposts, and I would love to chat with someone who can, if they're even allowed to, or if they're stuck under an NDA. But what an Outpost allows you to do is essentially purchase a rack of AWS; it's a rack of servers and storage with Amazon APIs. If you really just think about that for a second, that's pretty impressive. 

And if you are going to do hybrid cloud, and you have maybe some data locality requirements like you really need data in a specific location and that's not a region that Amazon supports, or you have data centers, or there's always some requirements, you can now get S3 on there. And they said that they can support 48 or 96 terabytes of S3 capacity per Outpost. What that actually means—like, is that a rack? Is that a whole rack? Is that just a single S3 configuration? Hard to really know. There's no API to go and provision an Outpost yet.

Jesse: Yeah, I'm really curious about this one to see how folks end up using it because I'm super excited that this is a feature that's now available. I love the idea of Outposts, even though it may not be a business use case for us internally. But I'm really curious to see how this changes the game in terms of object availability closer to the edge, closer to different locations for not just availability, but also for legal requirements for data storage around where you can or need to store data for compliance purposes.

Pete: Look. I'll be honest, I know that we will have made it as a business when we get an Outpost shipped to Corey’s house, so that we can put The Duckbill Group static website on an S3 bucket in Corey’s house, that's just how you've made it. 

Jesse: But honestly, I have to say that I still prefer a Duckbill website status page that is manually updated by our intern, Fred, on an hourly basis. And so I don't know if we'll ever be able to move away from that model.

Pete: It's true. It is serverless, so we do like to be really progressive in our usage of serverless there. But I think that gist that Storage Day really talked about when it came to S3 is, to use the right storage class for your workload. 

Jesse: Absolutely.

Pete: Amazon gives you so many different types of storage class tiers that it's almost criminal to just use S3 Standard for everything. We see it, right, Jesse? We see this all the time.

Jesse: Yeah, all the time. So, many folks turn on S3 storage, and put their objects in S3, and call it a day, and walk away. But there's so much functionality available beneath the surface in the different S3 tiers that can be leveraged. And we highly, highly, highly recommend finding the right tier for you. We highly recommend leveraging those tiers to optimize the amount of money you're saving for object storage and S3. 

And to be clear, we understand that you may not be able to spend tons of time looking at the access patterns for your object data, so you may not want to spend the engineering overhead to move data into these different tiers, but ultimately then, you can turn on S3 Intelligent Tiering and which will automatically analyze those patterns for you, and move objects into the correct tiers accordingly. Or you could turn on S3 Analytics, which will also do all that work for you, and then make recommendations that you can choose to implement to move your S3 data into different tiers.

Pete: I think it's important to call out, too that it's a little surprising, but also maybe, I guess, I'm not surprised—and that's a weird statement to say—but when you use Amazon storage, you just push data and you forget about it. You don't have to think about it. You don't need an administrator. I mean, Jesse, you and I are both former sysadmins, we've managed NAS boxes, and SANs before. Can you imagine a world—I mean, you don't have to because we're there—that you would be storing petabytes of data without a administrator of some things in those systems in order to administer that infrastructure? 

And I think that's kind of what we're seeing is that we're just seeing that lack of ownership and that no one really owns the S3 storage because you don't need it anymore. But because of that, people end up just saying, “Well, I’ll use Standard and call it a day.” And you don't really notice it until it becomes one of your top three line items. And you're like, “Whoa, how did our S3 storage approach six and seven figures? That seems like we should look into it.”

Jesse: Yeah, to me, I'm almost thinking of a metaphor related to a DBA, somebody who effectively will optimize your database usage and your database storage in such a way that you are optimizing your spend, which a lot of companies don't spend money on because they put a lot of things into storage in a database, and then they forget about it in a similar way that they push their object’s data into S3 and forget about it. I'm not recommending that you hire somebody specifically to do analysis of your S3 object data, but I think that it is something that is worth investigating. Even if all you do is turn on S3 Intelligent Tiering, or turn on S3 Analytics and then implement some of the lifecycle policies and recommendations that that feature makes, it's still worth your time because you will end up saving money optimizing that spend.

Pete: Yeah. And that's why we keep going back to Intelligent Tiering because it's the best way to save money in a passive—wait, you don't have to think about it. As your data gets older and unused, it automatically costs less. And that's what is so compelling of a service of a storage tier, that if you don't have the time, just leverage Intelligent Tiering. Again, there's some caveats, and we called that out in the last one, and you definitely should think about that, but when it comes to saving money, if you can just sit back and let it happen, there's not much better than that.

Jesse: Absolutely.

Corey: This episode is sponsored in part by ChaosSearch. Now their name isn’t in all caps, so they’re definitely worth talking to. What is ChaosSearch? A scalable log analysis service that lets you add new workloads in minutes, not days or weeks. Click. Boom. Done. ChaosSearch is for you if you’re trying to get a handle on processing multiple terabytes, or more, of log and event data per day, at a disruptive price. One more thing, for those of you that have been down this path of disappointment before, ChaosSearch is a fully managed solution that isn’t playing marketing games when they say “fully managed.” The data lives within your S3 buckets, and that’s really all you have to care about. No managing of servers, but also no data movement. Check them out at chaossearch.io and tell them Corey sent you. Watch for the wince when you say my name. That’s chaossearch.io.

Pete: So, that pretty much is it for the S3 changes and the new features there. Of course, re:Invent is coming up soon, so we'll have to hear some of the other cool stuff that's coming out. But there were some EBS announcements, although only really one main one that I could see, and that is a cost savings. So, there's a sc1—the cold hard drive volume type—which is a really low-cost magnetic type storage that you can use for any of your sequential workloads, or Hadoop clusters, or log processing, just those large scale but sequential non-random I/O type of usage. And the price of those, they dropped it pretty dramatically for Amazon: 40 percent—four-zero percent. That—I'm trying to remember when we've seen some really big cost-cutting measures by Amazon. I mean, in the early days, it was like every year there was like another price cut, but I don't feel like we see this as much anymore.

Jesse: Now, I think that this is something that does not happen nearly as often as we would like it to. There's definitely price reductions over time as older hardware is phased out and newer hardware is phased in, but I can't remember the last time that we saw such a dramatic price cut and not just a dramatic price cut, but a dramatic price cut across all regions where EBS is available.

Pete: Yeah. And so begs the question, why was there such a big price cut? Was there a low adoption? Did people think it was too expensive? Did some large customer use a ton of these and then turn them off one day, and now you're getting some EBS on the cheap? We don't know.

Jesse: We don't know. 

Pete: But I do remember, a very long time ago—it's probably one of my favorite EBS stories—was in the very early days of EBS, it was around 2010 or so, I remember working at a company; we had about 350 terabytes of unattached EBS storage, which I have heard, very informally, that that was a multiple percent of global EBS capacity at the time. And it was sitting, unattached, in an account because we didn't clean up after our testing. And we would provision 16 one-terabyte volumes to test out our striping setup. And then one day we went and deleted them, and I think—it's been a decade now, and I still think about that because either the EBS team was supremely happy that I just gave them back a ton of capacity, or really sad that I just gave them back a ton of capacity. 

Jesse: [laugh].

Pete: I don't recall any big price cuts afterwards, so I don't think it was too bad. Next on the list was some EFS announcements. And I got to say, they talked about the history of EFS announcements through the year, and in many ways, I think that was what we noticed about Storage Day was not necessarily a here are all these great announcements we held onto for the day, but really, let's summarize all of the hard work that was put into place over the last year. 

Jesse: Absolutely.

Pete: And so that's why in a lot of ways, I think Jessie and I are both, kind of like—thought this was very underwhelming, but it's because our noses are so close to Amazon, we look at the blog constantly, we obviously follow Corey’s newsletter. It is a requirement of Duckbill: you have to all sit around together and read it out loud. We take turns; it's fun—

Jesse: It’s great.

Pete: And because of that, we see these features come out all the time. And we know that we are an outlier. Not everyone has that ability. And so that is where—they spent a lot of time talking about these new features, and with EFS, they talked about features like they've added support for Fargate earlier in the year, for ECS, for Lambda, for EKS—Fargate on EKS. But they did just add one additional feature that kind of feels like maybe it wasn't a feature. I don't know, Jessie. What was this feature? And how amazing was it?

Jesse: Yeah, drumroll please: you now can directly create and attach EFS to EC2 instances at launch, through the EC2 Console Wizard. 

Pete: So, basically, the EC2 console team discovered a brand new service.

Jesse: And I think that's the big story here is that all of a sudden, this new feature is available. But let's talk about that because it's not really moving any major needles. Is it really the super innovative things that we're used to from AWS? No. I'm thankful that this is now available, and I'm thankful that this is a feature that we can leverage starting today, but and similar to what you just said, Pete, it just feels like the logical next thing to do, but I don't understand why it's part of—like, why is it getting its own announcement in Storage Day?

Pete: Yeah, exactly. I think looking at how this was before, you'd have to go into EFS, you'd create a file system, and then you go into EC2 and attach it, maybe after the fact, or maybe it was even an extra step in there. So, look, kudos. They’re removing a step or two steps into the process, and anytime you can do that, that's great. Of course, my sysadmin, automate everything, curmudgeon self says, why are you in the console anyway?

Jesse: Oh, yeah.

Pete: That that part of me just got—it was like, “Come on. We've automated this. Hopefully, this shouldn't be an issue.” But the other part that, I guess, annoys me a little bit to this is, have you spun up an EC2 server via the console recently? I call it the Christmas tree, the Christmas tree application. 

And what happens is, the Amazon engineering team keeps putting ornaments on it; eventually, that tree is going to fall over. I don't know when, but the number of settings and tabs that you have to go through to get an instance, this is why services like DigitalOcean exist and are doing so well. Just give me a server and get out of my way. But the number of things that you might have to answer has got to be measured in the hundreds at this point when provisioning EC2. But now it's in—add one more to it because now you can turn on EFS, as well. 

So, there were a lot of talk as well about FSx. That's a fully managed Windows file server. For a lot of enterprises, that's probably a big feature that they announced different user quotas and bandwidth quotas, but I think the biggest thing that we really were seeing was further integration with the Amazon AWS backup services with broader storage services, just adding more support for that. Which makes a ton of sense because if you're using a lot more storage services, if Amazon is providing you with more storage services, and you're any sort of business that has to backup that data, having an integrated way of doing that makes a lot of sense. And as we know, Amazon builds for the customers, right? They build what the customers ask for.

Jesse: Yeah, I'm really excited to see these new features released. It definitely feels like a step in the right direction, and it definitely feels like the correct way to help customers manage their backups across various different AWS services. I'm looking forward to seeing more usage—or more features of AWS backup in the future.

Pete: Absolutely. That pretty much does it. There is a whole slew of different services. You can actually go to the Amazon Twitch site to actually watch these videos. It's kind of nice background, you know, listen in. There were some questions that were asked along the way. I mean, all in all, I think it was a interesting presentation, talking about some stuff. And feels like maybe they're holding on to the really good stuff for re:Invent.

Jesse: Yeah.

Pete: Only time will tell. I think we'll look forward to what they announce later. But really, some interesting features. But I think at the end of the day, personally, a lot of this stuff just felt like a summarization of the year and not really brand new announcements. What were your thoughts, Jesse?

Jesse: I struggled with the same thing. A lot of these felt like logical next steps for releases, especially in terms of the new S3 tiers and the S3 metrics being available. But overall, I felt like so much of the information that was shared was just data without a story; without use cases. It was very difficult for me to understand, why is this an important thing that I should be celebrating alongside you? Why is this a feature that all of our customers are going to sing your praises for? 

Pete: Yeah, I think at the end of the day, the real winner of this one is the AWS marketing team because they have this whole day of stuff that got us to watch it, and also to talk about it after the fact in multiple podcast formats. So, kudos to the Amazon marketing team.

Jesse: I will say the one thing that I did appreciate is, AWS did comment several times about their goal to focus on helping customers more during the pandemic. They did recognize that customers using AWS today need to be able to do more with less, or do more with the same amount of resources or the same amount of spend that they have now, given current economic times and given current restrictions in this pandemic. So, I really do appreciate and want to send kudos to AWS for acknowledging their customers pain points on that and giving more to customers from that. I just wish that it was a cleaner overall thread and cleaner overall story throughout the entire day of announcements.

Pete: Yeah, I think that's really what I missed on it as well is, I like the storytelling; I like to better understand the problems that people are facing, and then how this new feature is going to solve these problems. And that's kind of what it lacked; it kind of lacked the story behind it. So, we'll see what happens at re:Invent, and mostly just wait to hear what other awesome features that they've got in the bag that they're waiting to show us. 

Well, if you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice, and tell us what is your favorite AWS storage service. Thanks again.

Announcer: This has been a HumblePod production. Stay humble.