Tagging Isn’t Just About Cost

Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com. 

Jesse: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.

Amy: I’m Amy Negrette.

Tim: And I’m Tim Banks.

Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we’re actually going to talk about a very specific listener question that we didn’t get to last week, but really, we had so many thoughts on this topic that we wanted to break it out into its own episode. So, today we’re going to be talking about tagging, and the importance of tagging, and how tagging can be used. And when I say tagging, specifically we’re talking about user-defined cost allocation tags. The original question that I’ll read off was from [Aaron 00:00:58].

Aaron asks, “Is tagging over-recommended as a cost reporting mechanism? I recently took on managing my company’s AWS bill and when talking to AWS and reading third-party blog posts about cost management, a solid tagging strategy is often extolled this step zero for understanding AWS costs. Based on what I know about AWS so far, this approach seems like it may work for some aspects of cost management, but does not seem to be a sound strategy for more formal cost reporting, like budgeting or calculating total spend for a given product or cost center. To me, these activities require complete or near-complete accuracy the tags just don’t seem to be able to provide since there are some costs like data transfer that aren’t tagged, and the fact that the tags are not retroactive—” that’s a big one that I can say is super frustrating for me. “Is there something I’m missing here? Is there in fact, a way to use these tags to ensure that 100% of an AWS account’s costs are in fact attributed back to a specific cost center accurately? It seems drastically simpler to embrace a multi-account strategy where each account is simply billed to whatever cost center makes sense to the organization.” So, Amy and Tim, again, the main question here is, is tagging over-recommended as a cost reporting mechanism?

Tim: The simple answer is no, it is not over-recommended. And the question makes a lot of good points around some of the heartaches and some the problems that come with tagging, specifically about tags not being retroactive, but, if you’re going to make changes to reflect changes in the past, I mean, you know, I don’t really have a good answer for that, if we’re being honest. But if we’re talking about going forward, tracking costs from this point forward, tagging is going to be a much more concise solution than using multi-account strategy. That said, there are a lot of reasons you should use multi-account strategy and tagging together. Multi-account strategy and tagging strategies should definitely be an ‘and’ situation, not an ‘or’ situation. That’s like pizza or steak. No. It’s both pizza and steak.

And I feel like because there are a number of non-cost reasons to use multiple accounts, especially in AWS, the biggest concern of which are service limits, right? Service limits, as you know, are done by account by region, so, if I have a service limit of S3 buckets that I can create—and I think that the hard limit is, like, one thousand—once I need that one thousandth and one S3 bucket, I have to create another account. That account can still be production, it can still be for all the same things that I’ve used for anything else, but I had to add another account so I can spin up S3 buckets. So, how do I track those, what those buckets are for, what those costs are going to be? I’m going to track those with tags.

And I’m going to track those tags from the payer account, or from up in the organization. So, as you set up multiple accounts, you can have—even if they’re all production, they still need to be tagged. Even if they’re all dev, they still need to be tagged. If you’re using the account vending machine style stuff from Control Tower where you spin up a sandbox account, you run some stuff, and then you throw it away, tagging is going to be the best way to track those costs, not just the fact that this account is named a certain thing. Names are arbitrary; they don’t really reflect necessarily what they’re going to be for, accounts can come and go.

So, I don’t necessarily like the use of name. Plus, sometimes it’s hard to do that if you’re doing, like, [unintelligible 00:04:21] various countries and things like that, various languages. Different things can impart different meanings. Tags also still probably use language 
problems, but they are arbitrary values. You know you’re going to try and lump these all together; that’s all that matters.

So, I definitely think that, if we’re using tagging, tagging is going to let you be more concise with your costs, it’s going to let you apply costs across different accounts more readily, it’s going to let you apply costs across different cloud providers, especially if you use one of the CMP tools like CloudHealth, or Cloudcheckr, or something like that and you run production workloads from a single cost center across multiple clouds, you’re going to want to tag those in those tools, so, that way, you can keep a consistent track and more concise tracking of costs, versus just using account names. Account names after a while is going to just become unmanageable when it comes to tracking costs.

Amy: I totally agree. And one of the big things that I harp on, especially on this podcast, is that if you’re worried that it’s not going to be as explicit as other billing methods, you will still at least have that data. You will still know per resource—if it’s properly tagged—who it’s supposed to be charged to and who owns it. You would make that decision on an architectural level, you should also make it for your bill, just to make sure that if you ever need that information in the future, you can go get it. You’re not going to get it—since they don’t happen retroactively, then you may as well do it as early as possible.

Jesse: Yeah. It’s super frustrating that a lot of this information is not available retroactively. And while I understand the technical limitations to that, I can’t harp enough why starting to tag resources early is super, super critical to understanding that spend, and using that tagging setup, that tagging policy, to better understand your spend in a number of different ways. But again, I also want to call out that, like, I've been saying everything about tagging related to spend, there are other ways that tags can be beneficial to your 
organization. I’ve seen organizations where security needs to know, are all of the containers that were running patched to a certain level?

Are all of the AMIs that we’re running patched to a certain level? Tags can do that; tags can help you understand what resources are using a certain AMI version, or a certain container version, or other security pieces that are important for security to know and be able to understand that all of these resources are patched to the latest available version of whatever we’re looking at. One of the things that we talk about a lot in this podcast is having conversations with other teams because I feel like cloud cost management is not just an engineering responsibility. It’s a responsibility of finance, and product, and security, and IT because there’s all sorts of different groups that may ultimately be using the cloud. And that’s kind of important for everybody to be on the same page in terms of how you’re using the cloud. And so it’s not just about tagging so, you can know the cost of something, but tagging so that you can know all these other important things like security, like product details, like maybe IT details, all these other different use cases for different departments that are also involved in cloud usage.

Corey: This episode is sponsored in part by ChaosSearch. You could run Elastic Search or Elastic Cloud or Open Search, as they’re calling it now, or a self hosted out stack. But why? ChaosSearch gives you the same API you’ve come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for App performance monitoring, cyber security. If you’re using ElasticSearch consider not running ElasticSearch. They’re also available now on the AWS market place, if you prefer not to go direct and have half of whatever you pay them count toward your EDP commitment. Discover what companies like Hubspot, Clarna, Equifax, Armor Security and Blackboard already have. To learn more visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again.

Tim: Yeah. I think there’s this idea that comes, I think, from very legacy data center operations where you’re going to use an account name to, kind of, specify what it does and where it comes from in the same way that you would use, like, a host naming scheme to define what a computer is and what it does and things like that. And I think that can be practical, but sometimes it’s often short-sighted, especially as an organization grows, and you create more accounts, and you bubble up other accounts [unintelligible 00:08:21] accounts. It comes time to sign the EDP and you need to have a master payer account, you acquire some other accounts and things like that, and then all of a sudden, whatever naming schemes they used is now integrated into what your naming scheme is. And that becomes, maybe, unmanageable.

So, I’ve always preferred to have account names. I mean, if you need to have it specified, understand it’s going to just be for humans to, really quick, find it, but I’m just as content to have an account name be a UUID and then have some other kind of method for looking at what it does or assigning billing to it. Because in the end, like I said, I prefer to use tagged resources to define what they are and where they go. They are obviously going to be exceptions made for things that are, like, dev, test, UAT, or something like that, where [unintelligible 00:09:06] are different, but we’re still talking about changes on an account, and then you make the changes on the account as you need. And then if it’s for production, then obviously those accounts can be tagged as production. They don’t have to necessarily be named production.

Amy: Right, and I think, security boundaries and resource permissions aside, if you’re just looking at trying to track costs to a resource, an account ID is really just one piece of information as opposed to tags, where you can just overload it with as much information as you need.

Jesse: Absolutely. Now, one other thing that I do want to talk about is we’re talking about a lot of good use cases for tags. We should also talk about some of the not-so-good use cases for tags, or some of the not-so-great best practices for tags that we have seen. Amy, I know specifically you had some examples that you want to talk about.

Amy: Yes. [laugh]. So, this comes from having to do data normalization, back in the day. First thing you never want to do when developing your tag strategy is you want it to just determine things like casing, or whether or not you’re allowed to use spaces because I’ve seen in different places, not just on resource tagging, but also the way information is meta-keyed, where they have their key name identical to a completely different key name, like you have ‘product owner’ except ‘product owner’ is capitalized in one instance and not capitalized in another instance, and these are considered to be different things within the system. Whether or not that’s your intention, they will show up as different things on some visualizations. On other visualizations, they will get normalized and turned into the same thing. So, it really depends on what it is that you want your reports to look like and what you want these resources to be able to tell you.

Jesse: Yeah, that’s a really great point that one of the things that we haven’t potentially touched on for this episode, and is covered in a number of other podcast episodes and blog posts in general is a good tagging strategy is equally as important as tagging coverage. Knowing that the tags should all be uppercase or all lowercase, or use these types of characters and not those types of characters is equally as important as making sure that those tags are applied accurately across all of your resources. So, as you are talking about tagging, as you are thinking about tagging, even in the multi-account situation, it is important to think about, what are the best practices? What are the standards that you want for your tagging? And again, this may not be a conversation that you have in a silo by yourself; this may be a conversation that you have with a number of other teams because there may be a number of other teams that need certain information from tags and need to use certain letters or special characters. And you need to incorporate all of that; you need to include all of that in the tagging policy that you create.

Tim: I think it’s also important, though too, that with most analytics tools, even if it’s just, you know, Cost Explorer within AWS Console, you can still aggregate those tags together, especially if you’re doing costs, you can absolutely aggregate multiple cases and things like that. CloudHealth, I know you can select multiples or anything that matches a pattern regardless of case and do it that way. So, it is possible to work around those mistakes. It’s not a, “Oh, we didn’t have our tagging schema set up correctly, so, throw your hands up and give up.” It’s just something else you have to consider, and hopefully, you can normalize going forward.

Jesse: Yeah, absolutely.

Amy: And really, the other thing is to make sure that the tags that you choose makes sense for what you’re doing. So, if you are tagging the environment and that is the only tag that you put on a resource, then just know that when you start pulling things up in Cost Explorer or Cost and Usage Report, that’s the only thing you’re going to see. So, you’re only going to see things split up between your production account and your dev account; you’re not going to be able to see what service is actually costing you more money, or what storage, as associated to a team, has suddenly decided to grow beyond the usual predictive usage patterns.

Jesse: Yeah, we have some recommendations we can make if you are just getting started on your tagging journey, and I will make sure that information is shared in the [show notes 00:13:53]. But ultimately, again, it becomes a strategy conversation. It becomes a question of what are you trying to accomplish? What are the goals that you’re trying to accomplish? What is the information that you want out of tagging? Because that’s ultimately going to drive what you tag and why you tag.

All right, that’ll do it for us this week, folks. If you’ve got questions you’d like us to answer, please go to lastweekinaws.com/QA, fill out the form and we’d be happy to answer your question on a future episode. If you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star rating on your podcast platform of choice and tell us what are the most important things that you focus on in your tagging strategies? What are the things that you tag for your company?

Announcer: This has been a HumblePod production. Stay humble.