AWS Wishlist and Chrismahanukwanzakah Part 2

Transcript
Corey: When you think about feature flags (and you should), you should also be thinking of LaunchDarkly. LaunchDarkly is a feature management platform that lets all your teams safely deliver and control software through feature flags by separating code deployments from feature releases at massive scale (and small-scale too), LaunchDarkly enables you to innovate faster, increase developer, happiness (which is more important than you think), and drive transformation throughout your organization.


LaunchDarkly enables teams to modernize faster. Awesome companies have used them, large, small, and everything in between. Take a look at launchdarkly.com to learn more and tell them that I sent you. My thanks again for their sponsorship of this episode.

Pete: Hello and welcome to the AWS Morning Brief. I am Pete Cheslock.

Jesse: I'm Jesse DeRose.

Pete: We are welcomed yet again with Amy Negrette.

Amy: Hello.

Pete: We are here. We made it. It is actually 2021.

Jesse: I can tell you flying cars: definitely a thing. World peace: we're close, we're so close.

Pete: We're so close. Well, guess what? We made it, we survived 2020. And with it, we brought with us part two of the #awswishlist. So, this is where we went through—especially as leading up to re:Invent and getting through re:Invent—we went through and looked at the Twitter hashtag of #awswishlist so that we could pick out some of our favorite things, some #awswishlist items that we think are important to us, or just interesting in their own right. We'll include the link to these tweets in the [00:01:57 show notes]. 

So definitely go check that out, and you can check out the conversation, or maybe follow some of that to see when things actually come around. But yeah, we'll just walk through some of the things we found that were pretty interesting and chat about why we hope Amazon includes them into a future release. So, one thing that I saw which I thought was pretty interesting because I run into this problem also, is a way of downloading data from various third party locations directly into S3, Dynamo, or some sort of data store location. Essentially, it'd be awesome to just completely get rid of having services around, or Fargates, or Lambdas set up for downloading data from places that—how cool would it be? And this is, again, not an enterprise-y type feature, but just, like, a personal thing of how cool would it be to be, like, I want to take this ISO from a place and just put a URL in S3 and say, “Put that thing in this thing,” and call it a day. So, again, a personal complaint of mine plus, also, someone else tweeted it, so there's two people out there that want this—at least—so therefore Amazon, you got to build it for me.

Amy: Those are the rules.

Pete: Those are the rules. Right. Right, Amy, those are the rules. 

Jesse: And I feel like, let's be honest, that ISO that you want to download anyway is probably living in S3 somewhere else anyhow. So, it's just moving bucket to bucket.

Pete: Someone has that, you know, Slackware ISO that I've been looking for, from, you know, 2001. It's in someone else's bucket; just let me have it myself. Exactly. Amy, what did you find in your discovery of the #awswishlist hashtag?

Amy: This is a thing that I think really should be on any of these on-demand pay-as-you-go services because AWS really targets those [00:03:48 unintelligible] markets for a lot of their serverless deployments. And this actually came from one of my friends who had this problem on Twitter, where you need to be able to set a maximum on on-demand spend, let's say in his case, Dynamo. So, you don't hypothetically build in a loop and spend a whole bunch of money. 

Pete: Yeah.

Amy: And really, it should be in anything that does that. If it's not telling you something where I'm only wanting to run this much because it's on-demand, then you should be able to control that spend somehow.

Pete: And with the—what is it—millisecond billion on Lambda, you can get really granular bills for your poorly architected Lambda functions. 

Jesse: I feel like computers are the best because they'll do exactly what you want them to do, except for when they do what you tell them to do and not what you actually want them to do, and that drives me absolutely insane. So, I'm with you. I think that this is a great opportunity.

Amy: That problem will be solved when the robots take over.

Pete: [laugh]. One of my favorite discoveries of doing our kind of Duckbill cost optimizations where we dive into people's spend and help them architect things new was finding a Lambda function that was taking longer and longer to execute—meaning, costing more money—by putting more and more data into a poorly configured Dynamo table that was also causing it to take longer and longer. And so not only did you have a Dynamo table that was poorly configured, taking this data and taking longer to do it, you were just getting a hit on both sides. It happens.

Jesse: That hurts my soul. 

Pete: So, what’d you find, Jesse? What was some of the good wishlist items that you're hoping for in 2021?

Jesse: So, I come from a background of a lot of infrastructure as code I've worked a lot with Terraform, I know enough about Chef to be dangerous to your production environment. One thing that I saw a couple people tweet about that I would love to see is mock AWS API endpoints for, effectively, unit tests for a lot of infrastructure as code. Because if you think about when you're building infrastructure as code, the only way that you can really test it is by running it, by actually seeing, “Can I actually create the resources that I think I'm creating with this infrastructure as code content?” So, I would love to see maybe a feature flag for AWS services through the API where you can say, “Hey, don't actually create this RDS database or this EC2 instance, but just return the results as if I did create it. Maybe leave the Instance ID blank or something like that.” And then you, in writing your unit tests, can confirm all the details that you would expect to see in that response. 

Pete: I feel like there was a—Atlassian, maybe, had a project that was something like this, some sort of a way of unit testing these things. Again, it was something on GitHub, so even if it was associated with a large publicly traded enterprise, I'm sure it's fallen into disrepair at this stage.

Jesse: [laugh]. I will say I found an open-source tool looking into this one, called LocalStack that allows you to basically spin up an instance on your local machine that acts as the AWS API endpoint so that it actually creates this mock endpoint for you locally on your machine. But effectively, I'd love to see that as a first-class citizen in AWS.

Pete: I think the way they're solving this is just having people buy Outposts instead, and having them just—“Oh, just run it against your Outpost. It's right there. It's all local.” So, one thing that I saw, which I had a really interesting reaction to because it's related to a previous company I worked at, so this person said, “I’ve got a lot of items in my #awswishlist, but a serverless version of Elasticsearch on AWS—or a serverless service that is Elasticsearch compatible—would certainly be at the top.” I don't know if this is sponsored by ChaosSearch because—

Jesse: [laugh].

Pete: —sponsorships are handled separately than the recordings. And this is definitely not me, specifically, sponsoring them other than the fact that I used to work at ChaosSearch, and a tool that we had, the technology that they had created which was so cool was this kind of concept. So, it's just really interesting to see other people getting to that point. But I think what's interesting about it is that Amazon has a serverless Aurora, which we use within Duckbill. It's a really interesting technology. 

There's a V2 version that has come out in part of re:Invent. And so this is a technology that is growing inside of there, having these, kind of like, on-demand databases for when I don't need a thing running all the time. I think Athena is kind of that concept right of ad-hoc queries. I don't need to run in SQL, I just want to run some queries every once in a while. It would be really fascinating to see this type of technology become more prevalent. 

I may not want to run Elasticsearch all the time. Let's be honest, no one wants to run Elasticsearch any of the time, [laugh], but maybe just a serverless version, I’ll end up paying more for, in general. Like Amy has said in the past, you'll have the loops causing additional spend, I'm sure that will increase your Elasticsearch—serverless Elasticsearch spend, as well. So, what was another item that you found, Amy, in your wishlist bag?

Amy: There is another type of database that users would like to see go serverless, and that's graph databases laid on top of Dynamo, which does make sense because a lot of times you don't want to have to spin up your own things to experiment with graph. And graph is a type of thing that once it's fully implemented, you can drop it on top of whatever your data source is. And AWS does have a graph service, Neptune, but just like all native services that are offered by AWS, it is a little much if you're just trying to figure out if it's the thing that you want. No one wants to—they look at that wall of Neptune documentation, it's like, “Maybe I don't need it right now.” And then they put it—it becomes one of those backlog things you learn later.

Pete: Or someone spins it up and says, “Oh, I’ll play around with this later.” And then—

Amy: And they leave it on, forever.

Pete: Yeah. Until they call us, and we find it for you. [laugh]. Jesse, what's in your bag? What's in your wallet, Jesse?

Jesse: [laugh]. If we want to continue my theme of scaring AWS engineers, I would love to see a better user experience for the Personal Health Dashboard. So, between the recent kinesis outage that prevented the public status page from being updated, and some of my own experience, basically being the canary in the coal mine, when the status page said, “Oh, yeah, everything's fine. All systems are go. It's good. You're fine.” 

But I've been the first person to call out, “Hey, I think there's actually something there going on.” I struggled with that public status page. And I know, I should use the Personal Health Dashboard more, but I just feel like it doesn't provide as much value as I want. I feel like it tells me things that I already know from other official sources, or doesn't tell me anything at all. So, I would love to see something better with the Personal Health Dashboard, maybe more engagement, maybe more or different alerts in the dashboard. Something that makes it more beneficial for me as an AWS customer.

Corey: This episode is sponsored by our friends at New Relic. If you’re like most environments, you probably have an incredibly complicated architecture, which means that monitoring it is going to take a dozen different tools. And then we get into the advanced stuff. We all have been there and know that pain, or will learn it shortly, and New Relic wants to change that. They’ve designed everything you need in one platform with pricing that’s simple and straightforward, and that means no more counting hosts. You also can get one user and a hundred gigabytes a month, totally free. To learn more, visit newrelic.com. Observability made simple.

Pete: Not to speak for you, but I think what you want is the Amazon simple status page, which is a simpler status page than the current status page that goes down a lot. I want to see a thumbs up or a thumbs down emoji. Is Amazon? Working thumbs up. Is one part of Amazon down? Thumbs down, right? Just, that's all I want to know. Because if it's something's wonky at Amazon. Is the thumb up or the thumb down. Just give me that.

Jesse: Yeah. Think about it: the website, is it down for everyone else, or just me? I just need to know, just a real quick, hot take. “Is it just me?” “Yeah? Everything else? Okay, cool.” I'll keep troubleshooting my own thing internally. Is there something bigger going on? Okay, cool, then I'll just let AWS sort out whatever they need to.

Pete: So, one of the last things that I had on my list, which I really liked this one because it can hopefully simplify a lot of the more complex network scenarios that we see is a all Amazon services private link. So, you've got gateway endpoints, right? Those are Dynamo and S3 only. Those are free. It's great. They're free services. How often do you get that? 

But then you have these interface endpoints for other Amazon services, maybe like Kinesis, or SQS, or something. Can we simplify this, where I have my stuff in a VPC that can't go to the internet in any way and it wants to talk to something inside the Amazon ecosystem? Can I just have my all services one—sure, I'm sure you're going to charge me, like, $800 per gigabyte to use it, but it will at least allow me to have, like, one of these things versus one per every service per every VPC per every account that I have. So, I think that would be a great one. And that would definitely solve a lot of folks’ pain out there. 

Jesse: I can speak from a little bit of experience here working with a client who basically had a standardized template that every new VPC they spun up, they spun up some number of private link endpoints as well for data that they knew was kind of the standard traffic going in and out of each VPC. So, several months down the road, they now have multiple VPCs with different levels of data traffic; some of them are actively using all of these private links, some of them are using none of the private links, and they're just hanging out there. So, in a lot of cases, these private links are just costing an hourly spend and not doing anything, whereas in some cases they're actively being used, and maybe they need more. I would so love to just see all of that consolidated into a single private link endpoint that all data traffic goes across. Like sure, I get that you're probably going to charge me for this level of convenience, but I would rather that then the networking headache of how do I make sure that this data gets transferred between VPCs without going out through the public internet? 

Pete: Yeah, absolutely. Anything left in your grab bag of #awswishlist items, Amy?

Amy: There is one thing that I want, and it's mildly frustrating because this is a source of most of my monitoring problems, that there's no way to tell on AWS when a Lambda has failed. And not like, it threw a bunch of errors and then it closed itself because that’ll show up in CloudWatch real easy, but if it times out, the way you know it timed out is because the runtime will be one second short of whatever maximum you set it to, and you have to find out what that time is and just find anything that close within that range. It's infuriating and confusing.

Jesse: Absolutely.

Amy: Someone literally put, “Just give me every time that a Lambda has failed.” I'm like, “Yeah, that would be super nice.”

Pete: So, maybe we ventured into the personal grievances section of our #awswishlist because, Amy, that sounds like a personal grievance, and it's a great one—

Amy: That wasn’t me.

Pete: [laugh].

Jesse: [laugh].

Amy: Mine is completely different. [laugh].

Pete: [laugh]. Well, Jesse, do you have any personal grievances, things that you're just like—in your usage of Amazon if this one thing was built in 2021, it would make my year.

Jesse: You know what? I'm going to go really easy because I think I've been really, really hard on AWS through most of this list. I'm going to make this one really easy. All that I want is the most important feature that every application should have, by this time: dark mode in the console. 

Pete: [laugh].

Jesse: That's all I want. I just want to be able to log in and see that beautiful, beautiful, dark-shaded background against all of that bright text. Now, I get that the color scheme might need to change a little bit, so that might be something that we have to involve product for, but it's so close. We’re so close.

Pete: We'll get there 2021. Maybe they’ll announce it in preview for re:Invent 2021. So, mine was a little bit more specific, and maybe on the nose, considering we spend so much of our time with the Cost and Usage Report and so much of our time in Cost Explorer. But one thing that I would really love to see—I mean, if there's a disparity between the CUR, between the Cost Explorer. And it really comes down to resource IDs. 

When you turn on the CUR, you have the option of turning on a resource ID, and let's be honest, you want to turn that on because when you do that, you can start, basically, getting around a lack of tagging. If, for example, you wanted to create a report of your bucket usage and break out the usage of that bucket via spend—like, what percentage of this bucket spend was in requests and what was in storage—the only way that I'm really aware of being able to find that easily without per-bucket tagging and that tag enabled as a billing thing is inside the Cost and Usage Report [00:17:24 unintelligible] with the resource ID. Now, maybe this is—I don't want to say as simple as because the billing team handles trillions upon trillions of records that they have to provide us insight into with Cost Explorer, but maybe adding resource ID functionality. 

Again, would I pay for this? I actually might because getting up to a resource ID level for S3 bucket resource IDs in Cost Explorer would allow me to not have to set up the CUR, find a place to put it, load it into Athena, turn on the Glue crawler, fire up Tableau and point it at it, just to get this report. So, that would be huge for me is that one feature. Is almost like parity, right, between the CUR and Cost Explorer. And—now, I see I would pay for that, but I’d just make Corey pay for everything, right? [laugh].

Jesse: [laugh]. Well, until Corey publicly drags us on social media for making him pay for things, and then, you know, we have to have a conversation. 

Pete: Yes, that's true. [laugh]. Awesome, well, it's a amazing start to 2021 because we made it; we're here. We're in the future. It feels good. Hopefully, the future will continue progressing in this way. But as always, it's a lot of fun to chat about some of the things that would be cool to see in 2021 or future years because they'll announce some of these things in preview, and maybe we'll see them in 2022, who knows? But Amy, thank you again for joining us and pulling your list together. Jesse, it's always a pleasure. 

Thank you all for listening. If you enjoyed this podcast, please go to lastweekandaws.com/review, give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekandaws.com/review, give it a five-star rating on your podcast platform of choice and tell us what is your hope for the future with AWS. Thank you. 

Announcer: This has been a HumblePod production. Stay humble.