Blinded by QuickSight

Corey: This episode is sponsored in part by Catchpoint. Look, 80 percent of performance and availability issues don’t occur within your application code in your data center itself. It occurs well outside those boundaries, so it’s difficult to understand what’s actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course, validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because we’ll throw that one in, too. And it’s used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle—but don’t hold that against them—and many more. To learn more, visit www.catchpoint.com, and tell them Corey sent you; wait for the wince.

Pete: Hello, and welcome to the AWS Morning Brief. I am Pete Cheslock. I'm still here. I'm going to be here for a while I guess, but not alone. I'm here with Jesse. Jesse, thank you again for coming on board and keeping me company.

Jesse: Always a pleasure.

Pete: It's honestly just nice to talk to someone else that's outside of my little family unit or my pandemic crew.

Jesse: I would say it's nice to get paid to just talk about my feelings. But I mean, I'm not technically getting paid for this.

Pete: Yeah, I feel like I'm just trying to balance the conversations with coworkers, podcasting this, my kids at this point, have more Zooms than I do.

Jesse: [laugh]. I think that probably says something about our social lives and about ourselves. And I feel like I need to go rethink everything.

Pete: Well, my son who is six years old, he does a better job of managing his mute button than most full-grown adults I know.

Jesse: I feel like that's the fun thing. I really want to see how the next generation is going to grow up with technology, better understanding the mute button, and all of this video content than we do.

Pete: It is hilarious to hear my daughter yelling at her friends, “You're on mute.” [laugh]. Oh, well, what is not on mute today is both of us. We are talking about the most loved Amazon service, Amazon QuickSight.

Jesse: I think it's technically going to be on blast today rather than on mutes.

Pete: Yeah, I think we're going to struggle to keep this one on time. So, if we go long, I apologize in advance. But we're talking about QuickSight, which for those that maybe have never heard of QuickSight before, it's Amazon's business intelligence tool. The question you're probably asking yourself, to be perfectly honest, is why? Why did you even try QuickSight? 

Like what point, what thing were you solving that made you think of QuickSight? So, we're going to tell that story. But first, let's just pivot into BI tools, business intelligence tools. That's the category that QuickSight is technically in. So, we'll talk a little about that, and also how we actually use BI tools within Duckbill because that'll give you, hopefully, the context into answering that question of, “Why did you even try QuickSight, Pete? Why?”

Jesse: I mean, I feel like there's probably still going to be people asking us why after this podcast, and I'm sorry for those listeners. We don't have an answer for you. Maybe we're just masochists. We don't know.

Pete: It's just because it's there, I think is what the final answer is. [laugh].

Jesse: Absolutely. So, business intelligence tools solve a whole variety of problems and we could probably do an entire episode on them in general. They help you gain insights from your data, which is fantastic. I absolutely love that this is even a category of service out there. But today specifically, to keep it on track, we want to specifically talk about gaining insights from your spend data, your AWS spend data. And to do that, we really need to start by talking about the AWS Cost and Usage Report.

Pete: Yeah, the Cost and Usage Report—you might hear it referred to as the CUR. I heard it referred to as the CUR often and it took me quite a while to actually figure out what anyone was talking about. So, if you hear someone say the CUR, they probably mean the Cost and Usage Report. But this is the v2, we'll call it, version of the Amazon billing data. 

It's incredibly high fidelity, I think is the term. It's very granular; there's a lot of data in there. And it's not enabled by default; you need to actually go turn it on. But what's awesome about this tool is it can provide you some really deep insight into where your money is going, and the only cost for it is the cost to store the data. And the Cost and Usage Report itself, when you turn this report on and have it dumped into your S3 bucket location of choice, you can actually have it store into a couple different file formats. 

One of them is Excel CSV format. And the other one is a Parquet format, which is a columnar data store and is a lot more efficient for this type of data. And it's the Parquet version of this that we use, we tell our clients—clients of Duckbill—to turn this on and turn it on with Parquet because then you can use tools like Athena to query your data and just leave it in S3 and run those ad hoc queries. So, Athena, though, which we're not talking about Athena, is challenging to use, in some cases—

Jesse: Yeah.

Pete: —you have to know SQL, which if you don't know SQL you're kind of in a bad spot. So, we use a BI tool, a very popular one called Tableau to query our data on Athena. So, Athena is kind of the engine, you could also obviously put your CUR data into an actual database. But largely, the queries we're doing, these are all human-generated. We're fine if they take seconds; they don't need to happen in milliseconds.

Jesse: Yeah, I mean, there's lots of solutions out there. There's third party commercial apps like Tableau and Looker—RIP—there's open-source options like Metabase. But of course then, in true AWS fashion, there's also a hastily integrated acquisition called QuickSight.

Pete: So, I have this memory in my head—and hopefully someone will correct me if they're listening to it, and I'm wrong here—but I feel like QuickSight was actually an acquisition. Like Amazon, which really doesn't usually acquire a lot of teams or businesses into Amazon Web Services, with like a couple of pretty rare exceptions, I'm almost positive, that QuickSight was actually some other product that Amazon acquired into it. But the history of QuickSight from at least the Amazon umbrella started around 2015 is when they announced it at re:Invent, and I was there for that announcement. I remember that announcement clearly, and I still actually kind of laugh at it when it came out. Now, first off, that was 2015 is when it was announced, and not for nothing, it does not look like it has gotten much better in the five years that it's been operating since launch. 

But I do want to tell one story about that announcement at re:Invent that year because there was this common, I guess, trope of re:Invent. Amazon would announce a service, and people would play this drinking game, essentially—because that's what re:Invent mostly is. It's just one week-long drinking game—but they would walk the floor, you know, the expo hall of all the vendors there. And every time Amazon would announce some new service, you know, people would point out, “Oh, that company is going out of business. And that company is going out of business.” And I so clearly remember this, I remember when they announced QuickSight and you could hear people pointing to the Tableau booth—because Tableau obviously was there—pointing to the Looker booth and saying, “Ah, those companies? Gone.”

Jesse: Gone.

Pete: Vaporized, right? Amazon's going to stomp all over them. And then I'm also reminded of last summer's Tableau acquisition by Salesforce that was $15.7 billion. I'm also reminded of the Google acquisition of Looker this year $2.6 billion. So, meanwhile, QuickSight is still QuickSight. It's still there.

Jesse: Tableau and Looker are clearly doing well for themselves. And QuickSight is still QuickSight.

Pete: QuickSight’s still QuickSight. Now, this all goes back to, again, why did we even look at QuickSight?

Jesse: Yeah, we started with Tableau for our business intelligence use case, because that's what we've collectively used in previous companies. But there's arguably a case for each of these solutions, depending on your use case. So, we just started with Tableau because that's what we were familiar with. But we wanted to try QuickSight to see if it would solve our use case, the same as Tableau. We wanted to really get that apples to apples comparison, so to speak, to see if using QuickSight might get rid of some of the negative things that we experienced with Tableau, while also maybe doing some other positive things, to remove QuickSight and move all of our business intelligence work for your spend data into AWS itself and keep it all kind of collectively in the same ecosystem. So, we wanted to try QuickSight to see if it would solve our use case. And I will let Pete talk more about that once his eye stops twitching.

Pete: Yeah, it was quite an experience, that's for sure. Now, if you've never used Tableau, Tableau is… oh, how can I describe it? It's like a box of Legos. But not a box of Legos that’s, like, follow this really clear booklet and you'll end up with the Star Destroyer from Star Wars.. Or Star Trek. I don't know, one of those star movies. I'm angering so many people right now. But—is it even called the Star Destroyer. Yeah, it's a star destroyer, right?

Jesse: Yeah.

Pete: [laugh]. So, it's not like that. It's just a box of random Legos that you have to piece together yourself. Now, it's unbelievably powerful because you can build anything you want with that box of Legos, just like Tableau. You can build any sort of visualization. 

It is so insanely powerful, it can connect to a ridiculous amount of different back end interfaces to JSON files, to geospatial data. I mean, it's got everything; it's really full-featured. And that's great. We can do a lot with it. It's also extremely well documented and that's super helpful as a Tableau beginner trying to dive into some of the Cost and Usage Report data. 

But it's heavy. There's just a lot to it. And so we really wanted to see, is there something simpler? I know a lot of people use Looker, and they love it. And honestly, the main reason we passed on Looker was, honestly the Google acquisition did not help. Something about using a Google-owned product to help our clients analyze their Amazon billing data just felt not great to us. And also, we don't trust Google and pretty sure that sometime in the next year, they're just going to, like, quit Looker and just shut it down. Because that's the thing that happens.

Jesse: Yeah.

Pete: So, we went to QuickSight because we thought, “Well, hey, if there's something that's integrated, that's native to Amazon, then that could potentially be a really helpful way for us to maybe build some standard dashboards and templates that we can easily share with our clients and make available.” Like, that could be really cool. So, that's where we started. So, I went, and I actually signed up for QuickSight, and that's where I kind of ran into the initial thing. It's not like a normal Amazon service. 

It's not like—there's a little bit of pay as you go, but it’s, like, pay per user. That's how it's billed out. And so you kind of end up on a landing page, and it's like, “Do you want to sign up for this?” But you do get a free trial, I think it's 90 days, if I'm remembering right, so there's not really a lot of risk here. And also, it's not super expensive, I think it's 10s of dollars per user, right, so when comparing to the cost of, like, a Tableau, which is much, much more expensive. 

So, all those reasons are kind of like this seems like it should be good, and this should work out. Like, we should definitely dive into this one. So, I am a classic engineer, and I don't read directions before doing it. I really shouldn't have to; it should be that simple to set up. And I went through and clicked through on the QuickSight and setup Standard Edition because I didn't need all the enterprise-y features, and was trying to start building the dashboard. 

And honestly, other than some of the very complex getting access setup working, which was mostly I think, related to the fact that it's like IAM, it actually was pretty easy to get up and running. And granted, my Athena database was already set up with my CUR data, so like all the complexity of that was taken care of, it really was just, go to QuickSight, add a data source, point it to Athena, and I was able to run some queries. And you know what? Within a really short amount of time, I was able to build some dashboards and using some of the documentation that exists to do that. So, that's, I think, the one good thing we ran into with QuickSight. It was quick… question mark? [laugh].

Corey: This episode is sponsored in part by our good friends over a ChaosSearch, which is a fully managed log analytics platform that leverages your S3 buckets as a data store with no further data movement required. If you're looking to either process multiple terabytes in a petabyte-scale of data a day or a few hundred gigabytes, this is still economical and worth looking into. You don't have to manage Elasticsearch yourself. If your ELK stack is falling over, take a look at using ChaosSearch for log analytics. Now, if you do a direct cost comparison, you're going to say, “Yeah, 70 to 80 percent on the infrastructure costs,” which does not include the actual expense of paying infrastructure people to mess around with running Elasticsearch themselves. You can take it from me or you can take it from many of their happy customers, but visit chaossearch.io today to learn more.

Jesse: [laugh]. Yeah, it was great to initially set that up and see that, out of the box there was a minimal learning curve to get from zero to dashboard. And I really appreciate that. There was definitely that opportunity that made me have false hope that this was going to be this amazing product that would solve all of our problems, which, joke's on me, because—

Pete: Lies.

Jesse: —yeah, there’s—

Pete: Lies.

Jesse: —absolutely. And so I started playing around with it, and I wanted to see how easily we could recreate some of the dashboards that we use in Tableau right now. So, effectively, for our listeners’ sake, we create dashboards in Tableau that we ultimately want multiple different end-users to be able to view. So, effectively, we want to be able to copy and paste a template of dashboards that different users will be able to view with different data sources, depending on who's looking at it, and really just make this as templated as possible, as cookie-cutter as possible. That's not the QuickSight way. That's not how they want you to do things. I built a—

Pete: It’s not quick. That is not quick, that part. It's like the concept of I create a series of I think analytics or dashboards, and I might want to change the data source from A to B. The same data, but I want to point to a different client, right?

Jesse: Yep.

Pete: That is hard in a way that I don't understand why.

Jesse: Yeah, it's this whole process to open the existing dashboard that you've already built, duplicate it as a new analytics item, and then re-export or re-save it as a new dashboard. Why? Why? Why can't you just change the underlying data source for the existing dashboard?

Pete: Yeah, exactly. And that's something that we do quite a bit with our Tableau. I mean, we can have dozens of Tableau reports and Tableau dashboards, make one change to the underlying data source, and everything will update. Again, the data is the same. So, the queries are the same. 

It's the same SQL query, it’s the same type of queries we're running, we just want to point it at a different data set. And yeah, that was super hard, and honestly, I didn't really get a satisfying answer that how to do that without having to rebuild my dashboards. Now, I did find something that was pretty interesting looking to hopefully answer that problem. So, there is a website that is Amazon ran, but it looks like it exists on GitHub called wellarchitectedlabs.com.

And I hadn't seen the site before. I found it when I was doing some research into QuickSight and trying to answer some of these questions about how to, and I found actually this really great lab for QuickSight and creating some cost intelligence dashboards. And what I thought was really fascinating, and I wanted to try what this was like is, they had a concept of, “Hey, you can go and get access to these templates and load them into your QuickSight.” And I'm just thinking to myself, “That's what we want. That's what we want to build. Like, let me go and test out how that works.” So, this is where I ran into a new problem with QuickSight is I went through this lab, and one of the early points was like, “You need to make sure you have a QuickSight Enterprise Edition for this to work.” So, I just said, “Okay, great. Like, I'll just go and upgrade my QuickSight account from—

Jesse: What could go wrong?

Pete: —Standard to Enterprise, because I'm still—what could go wrong, right? And I looked at a couple of docs, and it was basically like, “This is how you upgrade. And oh, by the way, when you upgrade to enterprise, you can't downgrade.” And I think the docs even say something like, “You have to cancel your QuickSight, and then maybe start a new Standard Edition.” Which I thought was kind of hilarious. [laugh].

Jesse: That's odd.

Pete: It just feels very, I don't know, it's like, very wrong, very non-AWS is what it feels like.

Jesse: Yeah.

Pete: Anyway, I updated to enterprise, and every one of my dashboards and queries immediately fails. Just, nothing worked anymore. It was very sad.

Jesse: Very sad, and very, very frustrating.

Pete: So, I opened a support ticket, because that's what we do. Right? I actually got a response back pretty quick. And we hopped on a screen share, so I could kind of go through it. Again, the only change I made: upgrade to enterprise. 

This feels like something that should work well. But after, I don't know, maybe like a week or so, we finally got a response back what the issue appears to be, although I'm not sure I totally trust it, is that it has to do with something IAM related, where I am accessing this account as a federated user-role-based access, where that's just how we access all of our accounts is we have users that exist in a separate account, then the account that I'm accessing QuickSight feels like a pretty standard use case to me, but the support ticket is basically saying that that may not work. So, I don't totally know, and we're still working with Amazon support to kind of solve this problem, but if that is not on the roadmap, we would love to see that on the QuickSight roadmap.

Jesse: Yes, please. Absolutely. And it also feels very AWS that they said that we need to use a normal IAM User Federation instead of a Federated User. Like, wait, what? Those things are the same in my mind, so I mean, at this point, we just seem to be arguing semantics.

Pete: Yeah, I don't talk totally understand it, but I'm going to try to be the good person, and I'm going to follow the steps given to me by support, mostly just to see, like, does it work that way? If it does work that way, I will probably laugh. I will then probably cry a little bit.

Jesse: [laugh].

Pete: But at least then I'll be able to finish this lab so I can just see what I came to see. I want to see like, there clearly is a way to do this templating thing and point things at different data sources. What does that look like? That's all I really… that's all I really want in this world.

Jesse: And I will say two QuickSight’s defense, we did get really fast response times for all of our queries, which is a definite plus a huge gain over Tableau and other resources that we were testing before. So, there are a couple things that QuickSight does well, but that list is kind of shorter than I'd like it to be.

Pete: Yeah, I agree. I also did not test one of the additional features of QuickSight, which probably would change my feeling on it, maybe a little bit. They have an in-memory query engine, they call it SPICE, which is short for Super-fast, Parallel, In-memory Computation Engine. There's an additional charge for usage there. 

I haven't turned that on. Cost and Usage Report data, even if you're a heavy user, even if you've turned on, like, resource IDs, it's not that big and I don't need my queries that fast. If I make even some of the gnarliest dashboards showing me usage down to the specific resource ID, which that'll be an extremely high cardinality field. There’ll be a lot of things about every single resource, you might use every EBS volume, every EC2, every Lambda. I mean, there's just tons and tons of resource IDs, those queries still worked. Maybe it would take 30 seconds or something? It’s fine, right? I don't need a millisecond query. So, it's possible that maybe that's the game-changer technology. But we weren't even able to get that far. [laugh].

Jesse: Then it would really be true that the SPICE must flow in order for the rest of the universe to work as we want it to.

Pete: I hope that there was a Dune fan who came up with that, like, they came up with the term. Because, like, this is probably what happened is they came up with the term ‘SPICE’ and there's some internal Dune-related thing. And they had to like back into, well, what could SPICE stand for because I think it's the super-fast part of this that makes it feel like they backed into that analogy there. [laugh].

Jesse: Yeah, and saying that this SFPICE, will flow just doesn't have the same ring to it.

Pete: [laugh]. Oh, wow, that was dad joke level bad there, so I think—

Jesse: You’re welcome.

Pete: —I think on that note, is where we will put an end to QuickSight for today, at least. Maybe we will come back to this. But I'm really curious. Does anyone out there use QuickSight? Do you like it? Is it a necessary evil for you? Was it just there and you're stuck with it now? I'm super curious. 

These BI tools, there's a lot of them out there. I feel like they're getting more popular. More and more people are using these tools to overlay business metrics with usage metrics and cost metrics and things like that, so I'd love to hear from people who actually use this. And I hope someone out there can just tell me that like, “No, no, no. You're just doing it wrong. You're just using it incorrectly, and if you use it in this other way, it's actually really great.” So, I'm going to wait to see if anyone actually does that.

Jesse: To go back to our previous point, like, I do feel like this is therapy for me because we get to talk about things that are broken, and then eventually somebody on the internet will correct me and say, “No, no. Why aren't you doing it this way instead?” And in some cases, they're right, and I do it the other way and all of a sudden everything works the way that I want it to.

Pete: Exactly. So, if you're out there and you're using QuickSight, please give us your feedback. I would be very curious to learn more, and hopefully, I can make it through now that I have a fix to my support ticket. I'm going to continue on at the wellarchitectedlabs.com site. I'm going to go through the rest of this lab here and give it a swing. And then maybe I'll come back next time and say I was wrong and here's the reason why. But thus far I think QuickSight is eh, eh, it’s fine. Like, I give it a solid meh.

All right, and with that, if you enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast—like your hatred for QuickSight—please go to lastweekinaws.com/review, give it a five-star review on your podcast platform of choice, and just tell us all the terrible ways that you're using QuickSight to get an answer to your data. Thanks again.

Announcer: This has been a HumblePod production. Stay humble.