Inspecting Amazon Detective (Whiteboard Confessional)

Corey: This episode is sponsored in part by Catchpoint. Look, 80 percent of performance and availability issues don’t occur within your application code in your data center itself. It occurs well outside those boundaries, so it’s difficult to understand what’s actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course, validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because we’ll throw that one in, too. And it’s used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle—but don’t hold that against them—and many more. To learn more, visit www.catchpoint.com, and tell them Corey sent you; wait for the wince.

Pete: Hello, and welcome to the AWS Morning Brief: Whiteboard Confessional. You are not confused. This is definitely not Corey Quinn. This is Pete Cheslock. I was the recurring guest. I've pushed Corey away, and just taken over his entire podcast. But don't worry, he'll be back soon enough. Until then, I'm joined by a very special guest, Jesse DeRose. Jesse, want to say hi?

Jesse: Howdy everybody.

Pete: Jesse and I are two of the cloud economists that work with Corey here at The Duckbill Group, and I convinced Jesse to come and join me today to talk about a new Amazon service that we had the pleasure—mm, you be the judge of that—of testing out recently, a service called Amazon Detective. This is a new service that I want to say was announced a couple of weeks ago, actually longer than that because, as you'll learn, it took us a little while to actually get a fully up and running version of this going, so we could actually do a full test on it. But as you can imagine, we get a chance to try out a lot of new Amazon services. And when we saw this service come out, we were pretty excited. Jesse, maybe you can chat a little bit about what piqued your interest when we first heard of Amazon Detective.

Jesse: So, we here do a lot of analysis work with VPC Flow Logs. There's so much interesting data to be discovered in your VPC Flow Logs, and I really enjoy getting information out of those logs. But ultimately, digging into those logs via AWS’s existing services can be a bit frustrating; it can be a bit time-consuming in order to go through the administrative overhead to analyze those logs. So, for me, I was really excited about seeing how AWS Detective automatically allowed us to dig into some of that data, ideally more fluidly, or more organically, or naturally, to get at the same information with, ideally, less hassle.

Pete: Exactly. So, for those that have not heard of AWS Detective yet, I'm just going to read off a little bit about what we read on the Amazon documentation that actually got us so excited. They talked a lot about these different security services like Amazon GuardDuty Macie, Security Hub, and all these partner products. But finding this central source for all of this data was challenging. 

And one of the things they actually called out which got us really excited is these few sentences. They said, “Amazon Detective can analyze trillions of events from multiple data sources such as Virtual Private Cloud (VPC) Flow Logs, AWS CloudTrail, and Amazon GuardDuty, and automatically creates a unified, interactive view of your resources, users, and the interactions between them over time.” It was actually this sentence that got us really excited because, as Jesse mentioned, we spend a lot of time trying to understand our clients’ data transfer usage. What is talking to what? Why is there charge for data transfer between certain services? Why is it so high? Why is it growing? And we spend, unfortunately, a lot of time digging around in the VPC Flow Logs. So, when we saw this, we got really excited because—well, Jesse, how do we do this today? How do we actually glean insight from Flow Logs?

Jesse: It's a frustrating process. I feel like there has got to be a better way for us to get this information from a lot of our clients, and every single time we have to ask our clients to send over or share these VPC Flow Logs. There's that little wince of the implied. “I’m so sorry that we have to ask you to do it this way,” because it's doable, but it requires sinking data between S3 buckets, creating and running Athena queries, there's lots of little pieces that are required to build up to the actual analysis itself. There's no first-class citizens when it comes to analyzing these logs.

Pete: It's really true. And Athena, the Data Factory—the Data Glue—what is it? Glue. You have to create a Glue Catalog. It's just a lot of work when we're really just trying to understand who and what are the top producers, consumers of data that is likely impacting spend for a client. 

So, we saw this and we thought to ourselves, “Wow, that one sentence it put in the list, it said, ‘The interactions between all of these resources and users over time.’” We got really excited for this. We also got excited because, of course, we love understanding how much things cost, but the pricing for Detective, it didn't seem that crazy. I mean, it's not great, but it's all based on ingested logs, which they don't really describe. So, our assumption is that if you send it your VPC Flow Logs, or CloudTrail logs, or whatever, you're going to pay for those on top of probably already paying for them today. So, that could be a deal-breaker for some clients out there.

Jesse: That's the thing that was super frustrating for me, or super interesting for me is that AWS Detective, in terms of pricing and in terms of technology and capability, doesn't replace any of these other components. It is additive, which, generally speaking, I think is great, but when you start looking at it from a price perspective, that means that you're going to pay for CloudTrail logs, and VPC Flow Logs, and GuardDuty, and Macie, and all of these other services, and now you're going to pay for AWS Detective on top of that. So, it feels like you're paying twice for a lot of these services, when you could do a lot of the same analysis work yourself. And it's probably not going to be as clean to do it yourself in terms of building out the Glue Catalogs that we talked about building out, Athena tables and queries. But ultimately, it may be less expensive because it's not ultimately paying for all these additive services on top of each other.

Pete: Exactly. I think we're definitely not being fair to the Amazon Detective product teams because we're trying to use this service, or we're hoping this service solves a really specific painful use case for us. And really, it's just based on what we found in their public-facing marketing.

So, how does this actually work? Well, we found some really great information online via Amazon. They did a great job documenting how this all works. Essentially, you enable Amazon Detective, and you enable CloudTrail, and VPC, and GuardDuty, you have to enable it in multiple accounts, and Jesse can talk a little bit more about some of the caveats we ran into just setting it up within our own services. What it does, though, it will distill that data down. 

So, it's going to consume all of these different data sources in, it will then give you this—ugh, it sounds terrible to say it—a single pane of glass for these different log types. So, if you have, for example, an IAM user that is associated with a large amount of network data transfer, could that be an exfiltration data attempt or something like that? So, essentially, what they're trying to solve here is, it's like a SIEM/SIM for Amazon created logs. That's really what it felt like to me after we had gone through this. What did you think, Jesse?

Jesse: I agree. I definitely felt like this is Amazon building their own SIM solution within AWS to effectively make all of these logs and alerts first-class citizens such that you don't have to send all of this data from your CloudTrail logs, from your GuardDuty findings, from your VPC Flow Logs into a third-party solution. You can send all of it directly to Amazon Detective, and that allows you to ultimately click through a lot of the findings in a way that creates deep links. So, ultimately, if you look at that single pane of glass—it hurts me to say it, too—then you can ultimately click through a finding to the GuardDuty page where GuardDuty is looking at the finding, or to the CloudTrail logs page, where CloudTrail can dig in deeper. There's a lot of opportunities for this deep linkage to allow you to better dig into the data that you would not ultimately get from a third-party solution; there would be a lot of back and forth with a third party solution between tabs and accounts, and it's a lot easier, a lot smoother with Amazon Detective to get all this data, or to click through a finding and find more information, find the information you need, and find the potential solution or potential remedy to the problem immediately.

Pete: Yeah, it does a really good job, from our testing, pulling these different disparate data sources together, and giving security engineers the ability to act on it. And where I think this could be actually a huge benefit is that there are a lot of companies that just don't have dedicated security teams. They still need to make it through SOC 2 or PCI audits, HIPAA compliance reasons, they need to show to auditors that they're analyzing these security threats, that they have this type of technology, and this could be a really easy way to get up and running. So, we took it on ourselves to go and turn on Amazon Detective because again, we wanted this to solve our VPC Flow Log, kind of, discovery issues. 

And while diving into it, as you can imagine, with a new Amazon service—or well, most Amazon services—there are some caveats. There are some rough edges that you have to be careful about. One of the things that we found, and why we turned it on in the first place, was you get a 30-day free trial. So, you can go and turn this on for your accounts; it is absolutely free for 30 days. But there was a very interesting caveat around this 30-day free trial when we turned it on, Jessie, what was this wonderful caveat?

Jesse: You get a 30-day free trial, but when you first turn Amazon Detective on in any account, it takes a minimum of 14 days to baseline. And what they refer to as baselining is effectively ingesting all of the data from this particular account, or multiple accounts if you are using AWS Organizations and pulling in data sources from multiple accounts—which we'll get to in a minute—and it brings all that data together in one single pane of glass, and runs some machine learning or AI analysis on top of this data, but it takes two weeks to set up. You have to wait a minimum of 14 days in order to get any data.

Corey: This episode is sponsored in part by ChaosSearch. Now their name isn’t in all caps, so they’re definitely worth talking to. What is ChaosSearch? A scalable log analysis service that lets you add new workloads in minutes, not days or weeks. Click. Boom. Done. ChaosSearch is for you if you’re trying to get a handle on processing multiple terabytes, or more, of log and event data per day, at a disruptive price. One more thing, for those of you that have been down this path of disappointment before, ChaosSearch is a fully managed solution that isn’t playing marketing games when they say “fully managed.” The data lives within your S3 buckets, and that’s really all you have to care about. No managing of servers, but also no data movement. Check them out at chaossearch.io and tell them Corey sent you. Watch for the wince when you say my name. That’s chaossearch.io.

Pete: That sounds like nearly half of my trial period that I'm just waiting. And it is; you would be correct that you would be waiting in about half the trial period. Now again, is that a deal-breaker for a lot of people? Probably not. As we found the—what is it—remaining 16 days of our trial was more than enough time to get a feel for what AWS Detective can do. But still, it felt a little… it felt a little Amazon of them.

Jesse: Absolutely. It was this great moment of, “Okay, we're here, we're ready, we're going to kick the tires.” We turned everything on, we invited other accounts, and then immediately it says, “Fantastic. Go take a coffee break, go back to your daily life for 14 days, and then come back, and then we might have some information for you.”

Pete: What Amazon doesn't realize is that in these current times, I don't actually have a life to go to. So, I just sat there hitting refresh for the next 14 days. It was a long, long wait.

Jesse: I can vouch; I did the same thing. It was part of my morning routine. Just, I have always enjoyed watching paint dry, and this was equally enjoyable and equally fun to just wait on that one main dashboard screen, not knowing if data was being adjusted, not knowing if there was any progress being made, just seeing the one single status banner that read, “Your data is currently baselining. Please wait.”

Pete: So, that wasn't the only thing we found. One of the other really interesting caveats—edge cases—is that for this to work, just, at all, you need to actually enable additional data sources. So, I mean, I don't know who you are if you don't have CloudTrail enabled. You should have CloudTrail enabled. But if you don't have CloudTrail enabled, you'll have to go and do that. You'll have to make sure that it's enabled for any accounts you want to integrate within the service. You need to turn on GuardDuty, you may not be using GuardDuty, but you need to go enable GuardDuty. And additionally, you need to enable the VPC Flow Logs for whatever VPC you want to include on this one.

These, in some cases, can cause additional charges to your account. One thing that I will say—because I do want to say something nice about Amazon—is that with this new service, they make it really clear during this trial period, how much this is going to cost you. So, there is a section in Amazon Detective that will essentially tell you, per account, which accounts that you enabled, whether it's 5 or 500, how much data it has ingested from those accounts, and essentially what your estimated bill is going to be so that when the trial is over, you actually do get informed; you get real information on what you want to do. And honestly, I love that feature. I think all new Amazon features should include that, especially including that ability to let you try it first, but also just say, “Hey, this is what it's going to cost you.” And then it's really for you to say, “Yeah, okay. This is worth it for me.” I think that's something that was great, and so kudos to the Amazon product team for including that in.

Jesse: One thing that we always discuss with our clients, always highlight with our clients is the importance of thinking about cost in every aspect of cloud cost optimization and management. And so if you are able to think about how much money you are going to invest, if you are able to predict how much money you are going to invest in a new architecture feature, or in this case, enabling Amazon Detective, it really helps you understand how much more money am I investing? How much more money am I spending on this service? Is that ultimately worthwhile for the business? And then you can make an informed business decision based on that information, rather than going in blindly saying, “We need a SIM solution, or we need some kind of additional security.” And then suddenly get the bill later and balk at it.

Pete: Exactly. So, we kicked the tires, we did spend the last of our trial period diving into the dashboard. We added some real data from some of our internal Duckbill accounts so we could see things going on. And that was great. It would, obviously, be probably a lot more useful if we had a lot more volume going on. 

But, you know what happened as we dove into this one? Well, when you go in blind to a new service, especially a new Amazon service, there isn't always a lot of great prompts to help guide you along. And this was no exception. When we landed onto the console after the baselining period was clear, you essentially just land on a search page, and it's just blank, and there's a search bar with a couple of suggestions. So, since there are no suggestions on things to search for, you have to at least start by picking the thing you want to search upon, whether it's IP address or account number, or something like that, only then will it then say, “Hey, here is some recommendations of things that you might want to investigate.” 

So, you really would use this because you know what you're looking for, or potentially you got an alert from another service. And I think that's where it's supposed to tie these together, is that it's the place you go after you got the alert from GuardDuty, or after you saw something strange. It didn't really feel like a done solution where, like, this is the place that you come to start. It's almost like this is where you go when something's happened. That was kind of my feeling on it. What were some of your thoughts, Jesse?

Jesse: I agree. I think that there is a ton of power in this service, but it's not intuitive. And that may be partially us diving in without having more data, that may be partially us diving in without poking around other services that flow into the service, like GuardDuty, and Macie, and VPC Flow Logs. This service has so much potential, and there is so much opportunity here, but it is very, very overwhelming to load the main dashboard and see just a single search pane. Like you said, I felt like I needed to know what I was looking for going in, immediately. 

This is not something that I had easy browse capability. But again to AWS’s credit, once we did start poking around, there is tons of amazing information. It's deep knowledge. As we mentioned before, deep links to other services, lots of really thorough, intricate details for things like VPC Flow Logs, and for findings that really allowed us to get a really clear picture of what was going on in our AWS accounts. So, I was really impressed with the amount of detail in all of these findings. But again, I would not have known that that amount of information, that amount of detail was available to me simply from the main search screen.

Pete: Yeah, only when you start searching upon different, maybe, IAM users or IP addresses does the full power of this application really become apparent, where you can see different applications by accessing IPs, by ports used, by bandwidth used, even to see API calls by IAM user, which I thought was super interesting as well. All of those little things buried within a search interface that, maybe if you're a security engineer, this is the solution that you were looking for because you have the questions, you just don't have the interface to go search upon it. So, I think what's great is that this is just the first version of this; this is just what was released. I'm actually very excited to see where it goes from here because one thing that we do know about Amazon is, actually, they do listen to their customers. And if you are a user of this service, you should definitely give it a try. If you're not, it's a 30-day free trial. You really don't have anything to lose. 

And before you decide to continue on with it and pay for it, they'll estimate about what it will cost you per month, then you can make that decision. So, high level, Amazon Detective, pretty cool service seems to be a bit of a missing link between a lot of these other services that would generate a lot of this data. So, probably has been in planning for a long time because there's a lot of these other services that didn't have any sort of centralized way of reporting on it. So, I think that's really cool. It's a really interesting service, and it's something we're going to keep an eye on at Duckbill Group, and you should definitely check it out as well. Well, Jesse, thank you so much for joining me today and being a proxy for Corey. Two of us don't seem to quite equal a Corey, but we'll keep working.

Jesse: Thank you so much for inviting me to this. I look forward to future sessions as we kick the tires on other services.

Pete: Fantastic. Well, if you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell me how much you miss Corey. Jesse, thank you again. Thank you very much. This is AWS Morning Brief: Whiteboard Confessional.

This has been a HumblePod production. Stay humble.