We’re currently hiring a content writer to write additional things here at Last Week in AWS. Our writing prompt for the role asks the applicant to write a small piece about what the most under-appreciated AWS service is.
Here’s my submission.
The AWS billing system
Everyone appreciates (or at least fears) S3, DynamoDB, EC2, and (as of last week!) Kinesis. But almost nobody appreciates the breathtaking complexity and disturbing accuracy of the AWS billing system.
In fact, you could easily be forgiven if you forget that it’s a service at all.
Want to bypass the console? Use Terraform or CloudFormation.
Want to bypass IAM? Do everything with your root account credentials. (HOLY GOD DO NOT DO THIS!)
But you aren’t going to bypass the AWS billing system, even if you never leave the “always free” tier.
The billing system takes a boatload of raw data tied to every API call that you make within AWS then distills it down into the Cost and Usage Reports, the (legacy) Detailed Billing Reports, whatever godforsaken format Cost Explorer uses, and—as your accounting group is painfully well aware—an invoice that shows up like clockwork every month.
Think about what it takes to achieve that.
Now think about what it takes to be able to achieve that—but also go back to raw data in order to prove a given bill segment’s accuracy years later.
You can’t cheat and take rounding “shortcuts” here. If you go back and regenerate customer-visible billing data from the source data, it absolutely has to match to the penny.
If someone finds a pricing bug (ahem) in a single customer’s favor? Someone at AWS has to be given the tools to find every customer who was impacted and issue a credit.
Think about the sheer scale of this system. Imagine the levels of complexity that must exist to make this work. And then let’s bolt on an uncomfortable truth that’s likely near and dear to every engineer who touches this system:
Every single customer uses it, but not a single customer would object if the service broke and stopped charging them.
Who thanks engineers who work on billing systems?
Engineering is generally a pretty thankless job, but that goes double for the billing team. Why would someone choose to work on this rather than S3 or DynamoDB, given the option? “It’s huge quantities of data!” Sure, but so is Kinesis—and people don’t shout at you when they find out that you worked on that thing in quite the same way.
Stepping aside from AWS for a minute, it’s rumored that the folks who work on core payments code at Stripe are incentivized to work on it—because core payments stuff is boring, but incredibly important.
It’ll never be appreciated. But it absolutely has to be done—and done well. (Despite whatever Agile process you adopt or claim to adopt, everything that touches payments is Waterfall. Fight me!)
While I don’t have any inside knowledge about AWS billing system staff turnover, almost everyone I got to know at re:Invent (AWS’s own version of Cloud Next) two years ago who works on that team isn’t there anymore.
There’s my entry, where’s yours?
If you’re interested in writing content for Last Week in AWS that showcases your personality and technical expertise, apply for the role today. We’d love to have you on board and look forward to seeing which AWS service you think deserves more recognition and why—and continuing the conversation from there.
