Being Upfront on CloudFront with Katrina Bakas

Katrina Bakas is a Senior Technical Product Manager at Amazon, working on CloudFront within AWS. She brings a lifetime of relentless curiosity to her role and a desire to simplify complex technologies to make them accessible for more folks. Previously, she brought the same inquisitiveness and desire to simplify to observability at Pivotal and VMware (upon acquisition), having spent time at start ups and in megacorporate Financial Services before that. She strongly believes the best tech is found at the intersection of psychological safety, Design, Engineering, and Product Management. 

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored in part by our friends at Fairwinds. Whether you’re new to Kubernetes or have some experience under your belt, and then definitely don’t want to deal with Kubernetes, there are some things you should simply never, ever do in Kubernetes. I would say, “run it at all.” They would argue with me, and that’s okay because we’re going to argue about that. Kendall Miller, president of Fairwinds, was one of the first hires at the company and has spent the last six years the dream of disrupting infrastructure a reality while keeping his finger on the pulse of changing demands in the market, and valuable partnership opportunities. He joins senior site reliability engineer Stevie Caldwell, who supports a growing platform of microservices running on Kubernetes in AWS. I’m joining them as we all discuss what Dev and Ops teams should not do in Kubernetes if they want to get the most out of the leading container orchestrator by volume and complexity. We’re going to speak anecdotally of some Kubernetes failures and how to avoid them, and they’re going to verbally punch me in the face. Sign up now at fairwinds.com/never. That’s fairwinds.com/never.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Katrina Bakas, who is a senior-product-manager-dash-technical-dash-that's-a-corporate-title-if-ever-I-heard-one at Amazon, specifically AWS’s CloudFront group. Katrina, welcome to the show.

Katrina: Thanks for having me, 

Corey. Glad to be here.

Corey: So, first, I understand that you're soon to be transitioning to a different team within Amazon, but fortunately, we're able to do this recording before that happens. What is a product manager dash technical? And of course, we will not forget the word ‘senior’ that goes in front of that either.

Katrina: Yeah, a product manager dash technical at Amazon is pretty much the same thing as a technical product manager anywhere else, which is to say, no one really knows what a product manager is and it varies wildly depending on your role, sometimes even within the same company.

Corey: Well, I have a friend who went down that path for a while, and one of the things that he kept encountering was at some companies that I won't name because you can tell a Google story from a mile away. It was great. “Oh, you're a TPM. Wonderful. Please solve this algorithm on the whiteboard first.” 

And it's a little ridiculous if you look at the role there where it's, you have to be able to prove that you can code so you can get a job wherein you'll never have to code again. But okay, we'll roll with that. Other companies that he spoke to, were able to have the conversation without ever getting into anything that could even remotely be considered engineering stuff. So, between those two endpoints across a broad spectrum, where does that land for AWS, first, and secondly, for you, personally?

Katrina: Amazon and within AWS, I get the sense that it varies a bit per team. You can sort of choose your own adventure at Amazon with as involved as you want to be. It really is the kind of place where you can find a team that really needs your skillset, and you can dive in really deeply there. I personally do not have a software engineering background, so I stay away from needing to make those implementation decisions. On my team in particular, I'm very much responsible for representing the customer in the room, and balancing what engineering tells me is possible, and know enough technical knowledge to understand if I'm getting sandbagged or not, but not prescribing what the implementation details should be. 

Corey: I always try and steer away from saying, “Oh, well, do you write code, or aren't you technical?” I think that's one of the biggest tropes out there that is provably false. Anyone who thinks that there's not technical detail and minutiae in that whole universe around any aspect of business is, frankly, naive. There is so much depth and insight that is valued across the entire board, that, oh, you don't know how to write a particular programming language, therefore, you're not technical is just a terrible take.

Katrina: I totally agree with you. I think one of the things that's even more frustrating in this conversation is folks who say, “Well, great. Now more people can be technical because Jamstack exists, so you can make an app.” I would go one step further to say that to be technical, you don't have to be able to develop an app, whether it's code, no-code, or anything else. 

It's about understanding what the thing is that you're working on and knowing the trade-offs of any particular feature, or chore, or technical debt that you're introducing. To me, it's just about understanding systems at least at a level where you're able to make trade-off decisions.

Corey: Yeah, the idea of Jamstack is just fascinating to me and something I plan on getting into a bit down the road if this year lets me. I love the concept; I love how approachable it's becoming. My static website template generator thing that I launched with a few years ago has been replaced by things that actually, you know, work. And it's nice to see that unfold. So, getting, I guess, away from the high level and into a bit more of the weeds of what you do, CloudFront is odd to me in that it feels on some level like it was, give or take, feature-complete—I know, I know. 

Don't email me—a while back because it's a CDN that has various points of presence across the world. Above my desk, I have a map of the AWS global infrastructure because I am really sad. And every time I wind up hearing about a new release, I put a new pin in there. And I look at this thing, and it's a sea of CloudFront locations around most of the world at this point. So, turning up new edge locations is valuable, obviously, and necessary. But I don't see a rapid pace of innovation around the product capabilities. And to be very clear, I'm not necessarily sure I want to either. Am I wrong? What am I missing?

Katrina: I think you hit the nail on the head here. A CDN exists as, ideally, something in as much of the background as it can. A CDN will accelerate your website and protect it from DDoS attacks from around the world or various other types of attacks. People come to CloudFront for security, and for reliability, and for acceleration. We do that remarkably well. 

Anything on top of that can be nice to have, but we don't focus so much on what are the additional bells and whistles that we can tack onto the core product that does things, as I mentioned, that we do really well today. 

Corey: So, I feel like I should also clarify slightly here where, if I look over the past year, there was no big announcement about it, but CloudFront distribution updates used to take—how do I put this directly—an age. And it has consistently dropped to sub-five minutes, which is awesome as a customer. But there was no big announcement about that. It was a quality of life improvement that I thought was incredibly appreciated from my side, but there are also aspects for certain use cases where that feels like an eternity as well. There are, of course, enhancements and stories about making it better to work with across the board. 

But it's also something that is relatively arcane. If you talk to the typical person just getting in technology, a CDN is not necessarily intuitive, and it has a bunch of sharp edges, if you're not careful, that can cut folks to ribbons that has nothing to do with CloudFront specifically, and everything to do with the fact that networking on a global stage is super hard.

Katrina: Networking on a global stage is super hard, and as someone that works in it every single day, I still find nuances just about every week that I didn't realize that I needed to know. I mean, even just taking a big step back and thinking about how the internet works at large and then replicating that in a private system so that your data can be more protected, and managing that without necessarily all of the same… let's say, benefits and headaches of liaising with all sorts of government entities as the public internet does. It's quite, quite complex. And the thing that I most look for as a PM for CloudFront is how can I pass the least amount of burden of managing all of that to my users while still delivering the value of that acceleration and protection at the same time.

Corey: It's also a bit of a challenge because I would argue that people's first experience as they learn how all of the various AWS services work with CloudFront isn't particularly friendly. Because if I'm looking at this from the Jamstack perspective—and one of my first outings with CloudFront was exactly this way: “Oh, I'm going to build a static website, and I'm going to have it live in an S3 bucket. Awesome. That's great. Now, I want it to, A) have its own domain name, which is not unreasonable, and I want it to work over SSL/TLS because this is no longer the 1990s.” 

Okay, to do that, now I have to bring CloudFront into play and it has an interface with ACM— the certificate management service that AWS offers—it has to integrate with S3, it has to integrate with Route 53—ostensibly a DNS provider, but here in reality, the world's greatest database—you have to get it to hook up to all of these different things, and as an initial starting out process, it's kind of formidable, at least from where I sit. And part of the problem is well, on the CloudFront specific piece is that the things that make it awesome—which is its flexibility— also make it terrifying of, “Wow. Those certainly are an awful lot of options of which I understand approximately none of them. Which ones can I safely ignore/take the default on/which one is going to turn my secure database backups into a new trending topic on Twitter?”

Katrina: Gosh, I'm so glad you brought up this topic, Corey, I came to Amazon, to CloudFront specifically, about a year ago, in particular, to focus on our developer experience. And while we've had some really big wins this year, like you mentioned, bringing our distribution update times down to under five minutes consistently, one of the things that is absolutely on our mind as a team, and on my mind in particular as a PM, is how to soften those sharp edges and just in general, reduce the burden of knowledge from the majority of the users who are coming to CloudFront for CDS—or CDN usage, rather—to something that's much more approachable. What I've found is that most of the users coming to CloudFront—and by most of I mean, on the order of between 80 and 90 percent of the users that come to CloudFront—do not ever change their default values, while still getting all of the perks of using a CDN, that acceleration and protection that we keep talking about. And if that's the case, how can I deliver a user experience to developers so that they know that they can safely ignore all of these other configurations, and when might I bring their attention back to the service when I think there's an optimization that they could make instead, intelligently. And more of those changes will be rolling out over the course of the next year. 

Corey: One of the challenges, too, is even right now, the values that you articulate of having a CDN— the security aspect and the performance aspect— when someone is setting out down this path at the first time, they don't actually care about either of those things, to be direct. And there's a strong argument that maybe they shouldn't because I'm building a ‘Hello World’ website, and I want to put that in a S3 equivalent, and then I want to put that out a website to the world. Yeah, if that's not secure, I couldn't possibly care less as long as it's not exposing, you know, access keys for my AWS account because there's nothing in a security sense to worry about other than, “Please don't make the browser pop up a giant warning instead of the website.” And from a performance perspective, when you're building out, oh, this is a simple HTML page that says, “Hello World.” “How impatient are you?” is sort of the default response here of, I want to make sure that this is an easy to hit website, but if you're on the other side of the world, it might take half-a-second round trip to pull that up. But a CDN will make that faster. Well, who can't wait half a second after hitting a link on a website for it to load? The real-world answer is that down the road, things get much more complicated and there are security concerns up the wazoo. And, well turns out most of us write websites like crap. 

And there are 30 sequential requests for the same thing, and that half-second latency for each one of those winds up multiplying and now your website takes two and a half minutes to load, so at that point, you may as well just store it in Glacier and wait for the retrieval times.

Katrina: I'm less concerned about that particular user of the Hello World app builder who's then adding on new and new tiers and then making their way to production by adding all of these different things onto their website. Maybe you start with Hello World and end up with a really cool site to sell Billy the Platypus merchandise on, where people can buy it from.

Corey: We are working on that.

Katrina: [laugh]. And that's a great use case. But at the same time, typically, as you're building up those building blocks, something tells you that you need this protection. Maybe when you're implementing your checkout process, you realize that that encrypted field could be better powered by using signed cookies or something like that. And then you get down this rabbit hole and discover that's a feature available of a CDN, or that you don't have to write it yourself, or something like that. 

What I'm possibly more interested in or maybe just more top of mind, for me, is the opposite use case where someone comes to AWS and says, “I have the world's coolest merch store. I need to put it up online. I know either not very much, or very little, or I just can't be bothered with knowing the intricacies of what a routing system is, but in figuring out how to set up my first eCommerce site, it tells me that I need something called an SSL/TLS cert, something called domain name registration, something called a CDN because that DDoS thing sounds scary.” How do we make that experience as easy as possible from that entry point, which is that you already have a predefined goal in mind. You already know that you need to serve this content that is really hot for your users in a way that's not going to negatively impact neither them nor you.

Corey: Yeah. There's a lot of opportunity there. I feel like those are the cloud stories of the future. I mean, this idea right now feels almost like a temporary aberration where you have a developer who wants to get into doing these things. It seems like the much more common path, down the road if not today, is going to look a lot more like, I'm just new to the workforce, or I'm changing careers, or I just graduated school, and I have an idea for a business. 

That business involves computers in some form, as most things do. Today, the answer is, you have to go to basically software engineering school, or at the very least cloud school to start understanding how a lot of these things fit together. Down the road, I'm more and more convinced all the time that that is not going to be a permanent state of affairs.

Katrina: Oh, I totally agree. And I would challenge you on that even one step further, which is that I'm more curious about the folks who set out to solve any particular problem, and that solution tends to have some sort of technical impact to it usually involving a website in some way, but more so than saying, I'm going to set out to create a business that's going to do this particular thing, sometimes it's just, “I’m really tired of having to keep track of what's in my fridge, so I'm going to start in an Excel spreadsheet of all of the ingredients, and then maybe I want some sort of logic on top of it that removes them as I use them. And oh, that gets really cumbersome in Excel, so I'm actually going to throw that into this simple webpage. And I know how to do that because there are a billion tutorials and blog posts out there.” And now you've just backed into creating your first app when you never set out to create an app. [laugh].

Corey: Yeah, it's almost on some level, too, where we start to see the parallels between that and career aspects and career paths, where I will never be a software engineer professionally, according to some people because I didn't go to Stanford and get a degree. Awesome. That is certainly a take people can have. But you're deeply involved in this and have a level of insight into the running of a CDN that I'm never going to have because I'm not getting the same hands-on experience that you are. But your degree is in I believe international studies and biology.

Katrina: That's right. My undergrad degrees are in international studies and biology. I was pretty convinced I was going to be a neurosurgeon for quite a long time.

Corey: And now you wound up going through a variety of different career moves. You were at Pivotal and then VMware when they did that whole shell game, acquisition, divesting. I don't even know who owns what anymore nor do I particularly care. You were in a bunch of other areas: loyalty marketing, you were in large financial services, and now you're a senior TPM at Amazon. What about your background makes you—I don't know how to put this politely, so I'm just not even going to try—relevant to that?

Katrina: Yeah. The thing that makes me relevant to being a really good product manager is caring, at the core, about what users’ experiences are, whether it was in loyalty marketing, or working in startups on real estate CRMs, or any other job that I've had so far. At the end of the day, I just want to solve interesting problems with simple solutions that actually impact the lives of users in some way. Sometimes, that's a really clever problem, and sometimes it's a problem of how can I make you care less day-to-day about the thing that I work on by the end of the time that I'm working on it. So, for me, that variation in my background, having a lot of different roles that have exposed me to all sorts of different industries and therefore different users, has allowed me to think about or to be better at understanding and seeking out what my users’ needs are, and then bringing those into a business and, frankly, just advocating really well internally, by being able to describe what impacts they have on users and therefore what impact that has on our business, and making good cases for developing better experiences.

Corey: One of the most valuable things that I see time and again, is folks who have come to this industry—by which I will round to Cloud as a whole— through, I guess we can call it non-traditional paths, but that always feels weird because I know more people who have come to this industry through those paths than folks who went and got the degree at Stanford. So, I really do object to that framing on some level. But I find that the folks who have that wild, varied background lead to teams that generally tend to be stronger, which is something that the studies have borne out as well. Now, that is the abstract. In a much more practical sense on some level, it's a CDN. 

This is not the sort of thing where I would—going to put up a bit of a straw man for you to knock down, so please don't write me emails, listeners—it seems like, on some level, the idea of having diverse and inclusive teams, including a pretty wide diversity of background isn't the most important thing when you are running a CDN. It's not something that is going to, for example, suffer from algorithmic bias issues where we're seeing a lot about that in the news these days. It's not going to, for the most part, wind up being something that is a differentiation along axes that are problematic to various constituencies. It's basically a, we make the network faster. The end. What is the value of having a more humanistic approach to these things?

Katrina: I think the value of having a more humanistic approach to providing a really robust and usable CDN is understanding that if CloudFront, as an org, only hired die-hard DevOps folks who have only ever worked on DevOps, only have these extremely coding-rich technical backgrounds who just love building extremely beefy apps, then the way that the product is going to be made is going to reflect that. The problem is that users of CDNs vary wildly away from that. There are people, like we mentioned, who want to tinker around with every configuration or who have to because they run really heavy and robust workloads, like maybe video streaming platforms, or other applications with highly concurrent needs, like big banking applications, and things like that, all the way down to the Billy the Platypus commerce site that you're building right now, and not having that same level of expertise. And what I think is the value of having, and what I've seen is the value of having folks come in with a variety of different backgrounds is just a greater ability to empathize because they can relate to different people's experiences. And when you have a thousand of all of the same exact person working on a thing, then you're going to end up with a thing that reflects all of those thousand people. 

But if you have a thousand different perspectives working on a thing, you're going to end up with something that works well for, well, more than just that one persona type.

Corey: No, you raise an excellent point. There are ways now to get things up and running effectively with CloudFront for those folks among us whose first language is not JSON. So, you're right in what you're saying. The user story is becoming much more simplified; the ability to have things start being removed from the console. I don't know if RTMP— if I'm getting an acronym right or not— is still an option that is actively promoted on the CloudFront side, the fact that I could do things with WebSockets now if I know what those are, but it doesn't nag the heck out of me about them and make me feel like I'm a fool for not knowing. 

It's becoming much more approachable to the point where, do I have to use this, or is there a click by numbers option I can use instead? There is something to be said for folks who are representing different customers getting a voice.

Katrina: Absolutely. And I would also challenge the notion that there is not algorithmic bias necessarily in CDNs or anything like that. I mean, there may not be yet, but as we continue to invest in different compute at the edge and logic at the edge functions where you can write all sorts of really interesting applications that instigate really interesting network impacts, like logic that may, let's say, have downstream impacts on the rest of your infrastructure, but only doing so when a customer requests it, the way that we continue to invest in compute at the edge and as we continue to push different logic to the edge, what we have to keep in mind is that the edge is simply just a proxy to the rest of your system if not configured right, or sometimes if configured exactly right. So, whatever logic is being built and introduced to the system can be really harmful to end-users, and we take that responsibility very seriously that we're giving people, as we continue to build CloudFront, continue to build different at-the-edge functionality, that we're giving people more and more power over these systems that are, I mean, just deeply ingrained in all sorts of networking infrastructure. And the more that you're able to fine-tune and customize there, the more possibilities of [laugh] different attack vectors there are at the same time. That sounds really scary, but all of that to say that the more flexibility that we offer, and the more that that investment in the edge logic continues, I see more and more possibility for just more creative usage.

Corey: I will say that I had the privilege of talking to a sarcastically senior engineer on the CloudFront team a year or two ago after I was doing my typical way of filing bugs against AWS services, which is complaining loudly on Twitter. And they had the grace to always respond with that typical Amazonian response of, “Interesting. Can you tell me more about your use case?” As opposed to the much more direct slash honest, “What the hell's your problem?” Which is basically what I deserve. [laugh]. 

And the answer that I had at the time was—I was complaining about the 45-minute distribution updates. And the problem that they said—and they're right, by the way— “Do you want us to just lie to you and come back when we wind up having the update not done, and you'll be serving different data than you think to customers? Or do you want it to be technically correct? Because we strive for correctness here.” Because that's a great question. 

It takes time to update stragglers and the rest. And my answer was also a very legitimate, “I could not possibly care less. The reason I care about this at all is that in order to update a Lambda-at-edge function, I have to wait for a distribution update to finish.” There's no option for me to just deploy it to a single POP, basically, instantaneously in the region I'm developing in; I have to wait, at the time, 45 minutes to discover, yep, I forgot to put a comma in there and there are syntax errors because see previous notes about terrible coding practices that I do. 

And that was a use case that, it turned out, was echoed by more than a few people. We’re all secretly terrible developers, as it turns out. And that understanding was taken and obviously acted on because it's now way faster. But it's still challenging, in some ways to have a four-minute wait when I want to see if I forgot a comma again. But it's more tenable. 

It means that writing a Lambda-at-edge function isn't going to be something that's going to take two days for me to spit out before I finally get it right. Things improve all the time, and it's easy to complain about the things we don't have, rather than the things that we've been given.

Katrina: It is often easy to complain about the things that we don't have versus the things that we’re given. I, as a PM, absolutely understand, though. Sort of like, as users, we’re conditioned to always want more to always want better, we're conditioned to want the perfect thing that works for our exact use case. And to understand [laugh]—and, as you said, complain loudly about that when it doesn't work out. I take that responsibility quite seriously. 

If there are ways that we can improve our users' lives, like continuing to work on the change propagation time, so that it continues to be lower, lower and you have more of that instantaneous feedback. There are certainly other ways of solving that problem that we've heard lots of requests for, like different sandbox environments, for example, or a blue-green deploy system or, like you said, the ability to specify a specific edge locations to where you can deploy code. For some people, that's a persistent request. They want to always be able to deploy certain logic by edge location at a very granular level because that suits their business needs. Taking all of those and weighing them in not only for what will work for a subset of our user population, but what will work for all of our consumers at large as part of a major in-demand CDN with customers of all different sizes, all over the world, from single dev shops to massive multinational corporations. 

We have to consider—what I try to consider as a product manager is what are the features and the, sort of, bells and whistles and the customizations that I can offer that impact the most of our users in the most applicable way, rather than how do I design for just one customer's need?

Corey: Some people need dozens of tools to visualize their stack. But not you. You’ve got New Relic Navigator. All your hosts, services, containers, and anything else you can monitor are in one dense hex view, so you can check system-wide health at a glance.And you can sort by platform, service, app, or any other tag, which makes it easy to navigate yourentire system and see what needs your attention. Go to https://newrelic.com and get started for free.

Corey: And this is the thing that I've had to learn again, and again, and again. Every time I think I've seen it all in the context of a given AWS service, or even AWS as a whole. All I have to do is talk to one more customer and I'll discover a use case that I hadn't considered or hadn't envisioned as being something that real people did. No judgment intended on that one, by the way. And you really do go about learning from customers as you go. 

But it's also hard, particularly on the CloudFront level, where of all the environments I've been in, there have been remarkably few requests I've had of a CDN that were not a modified form of either, make it easier, or make it faster, or get out of my way more. Those are the only three requests that everything winds up in a CDN piece that I ever saw going down that path. And I thought that that was it. Okay, CDNs are boring, why does everyone care? And then I started talking to customers. So, from that perspective, what do you think, in what you've seen, is the most misunderstood aspect of CDNs and/or the most curious use cases you've seen?

Katrina: The most misunderstood aspects of CDN that I have experienced in my work has been this notion of—back to the beginning of our conversation in this podcast— the notion of my CDN should be everything to me. This kind of conflation of CDN and application, or Jamstack, or application builder tooling that people tend to sometimes come to CloudFront with a very particular need, or they come to CloudFront thinking, “This is where I'm going to build my entire website, and by the time that I'm done with the console, I'm going to start with nothing and have a working website that will allow my users to purchase the thing and check out.” And that's been a really interesting space to, kind of, step into and figure out what that balance is, for me as a PM, in terms of where I want to continue to advocate for taking the product. Do we want to be everything to everyone, or do we want to do our core competencies really, really well? 

In terms of the most interesting use cases that I've seen in the CDN space, gosh, it’s sort of limitless. Some of the more fascinating ones to me are understanding how really large players out there you see CNS in incredibly complex ways. Talking customers who have hundreds, if not thousands of CloudFront distributions, all of that do something a little bit differently and serve a unique purpose for their tech stack, typically always going towards one final application. So, the interconnectedness of it all completely blows my mind, especially when I use the end-user-facing product in realizing that when I navigate, say, to a website, and let's say, watch a movie or something like that— say that I'm watching something on Prime Video, let's say, it blows my mind that complexity that can be underlying for this in a way that customers somehow can keep track of all at the same time and continue to iterate and improve on, and it just kind of magically works for me as an end-user.

Corey: That's the biggest thing that continues to amaze me almost in equal measure. First, the fact that I can wind up having some crappy line of code and push a button and it gets deployed at world scale for fractions of a penny, followed secondly by the fact that I am pissed off about minutia of that experience and want to take to Twitter and drag AWS for it. Those two senses of wonder are always competing in my mind. And on Twitter, I do bias for the ladder, I admit, but it's magic; you have built what is fundamentally magic.

Katrina: We have. We built fundamentally what is magic through a lot of conversations with users and less back-and-forth debates about what is doing the right thing and then, of course, fast iteration and learning from it. There's so much that goes on, as with any technology, but behind the scenes, even in what we'll call a slow feature time, if you'd like to call it that, just all of these underlying improvements and underpinnings that continue to make the experience better and easier to use and at the same time, enable people who didn't know anything about a CDN before to deploy their code to over 220 edge locations around the world. The idea that someone who's never written a line of code before can, within a day for sure, learn all there is about it in tutorials, and online, and whatnot, and then come to a service like CloudFront and deploy that brand new code, the first line they've ever written, absolutely mind-boggling to me. 

Corey: One thing that never occurred to me to ask before, but since I've got you on a recording and it's impossible to say no when I ask these things, I'm going to ask my question now about this. AWS is somewhat famous for not having a global control plane; it's avoided things like global outages of every region at the same time. Those boundaries don't really exist with CloudFront, I have never yet seen an announcement of this new feature has come to CloudFront—and there have been a lot of those— but only to the following three global regions with more to come in the future. Features are rolled out either globally, or not at all, and I can't remember, although I'm sure someone will have some document somewhere that’ll yell at me for this, a single global CloudFront outage. How does that happen without the blast radius being either monstrous, which it clearly is or the pace of innovation slowing to a crawl which, snark aside, it isn't? 

Katrina: Through a lot of diligence, Corey. That's the [laugh] that's the best answer I can possibly give, having this extremely globalized service, that is still though, by nature, are at the core of being a network of interconnected physical locations deployed all over the world. Those physical locations in each particular place are separated, so we are, to some extent, naturally protected from large natural disasters by having edge locations in each place, much like different data centers or availability zones. There does exist that aspect of that fragmentation. The flip side of that is that we also are extremely conscious of the fact that CloudFront operates at a global scale— with a small caveat of, except for in China because of the Great Firewall of China and having to operate as a separate legal and network entity within China— that globally we have this responsibility of making sure that the software that sits on top of all of those physical edge locations can reroute and does reroute traffic consistently based on what's available, and not only what's available, but what's most performant. 

So, we continue to keep it at absolute top of mind to be able to redundantly support workloads through any sort of localized or even regional outages that may happen so that we can continue to serve the information that we need to.

Corey: It's never ceased to amaze me just how much of these various concerns AWS has to hold in tension with each other. Because customers want new features, and they want them quickly. They also don't want you to drop their entire banking presence onto the floor while doing it. And every time I had a feature request, the answer has been, there are extremely complicated reasons why it doesn't do X. I have never yet come to an AWS team with a feature request and had them drop a cup of coffee and, “Holy crap, we never thought of that before.” 

It's always the, “We'd like to, but it's really hard.” And on the one hand, I'm very sympathetic about that, and these things do take time. On the other, I'm not paying my fractions of a penny so that AWS can only solve the easy problems. So, again, it's a study of contrast, a study of tension. And I feel like just through this conversation, I now have a better idea of what product management is than I did when I started half an hour ago.

Katrina: That's lovely. That's a great meeting of my goal then, of spending this time with you. [laugh]. 

Corey: Exactly. So, one question I have before we call this a show is, what is the best and worst you've ever felt about your job as a product manager?

Katrina: Oh, gosh, that's a great question. The best—or the worst, I'll start with, which is more fun.

Corey: The worst is always the good stories. It’s the, “What do you regret the most?” It is always a great question to ask someone. It's super weird when you ask it in a job interview, but that's a separate story.

Katrina: [laugh]. I would love to hear Screaming in the Cloud of just, Corey’s regrets. That would be lovely.

Corey: Oh, my stars. I don't know if anyone else would. That's the problem. Once I start complaining, I don't stop. But you're not getting out of the question that easily.

Katrina: The worst I've ever felt my job as a product manager was sitting down, I had worked on this very large feature release with my team for about six months at another company, had had tons of user interviews felt really good about the research that we did at the design phases; had these iterations with real live customers in hand, testing things out, did all of these recordings, like, couldn't have done anything better by the book. Sat down when it was in beta with the customer, had them test it out. And the customer just struggled. They could not get through the very first thing they needed to do to even log into the application, much less use the new feature, which was this really tricky authentication maneuver. So, they sat down, they struggled, they struggled, they struggled, they couldn't get any further. 

And finally, this customer, who I knew really well, had built a good relationship with, turned around to me and said, “I’ve been doing my job for 20 years and I've never felt worse at it than I do today. And it's because this experience has been so rough.” And as a product manager, that's about the worst thing in the world that you can hear from a user is that you're making them feel bad at their job. And the lesson that that very brutal experience taught me was that while being so focused on the new and exciting thing, I was completely ignoring these really fundamental steps that it takes to actually access that new and exciting thing, which in this case was something as simple as a back end authentication.

Corey: That one absolutely resonates across the board. I believe wholeheartedly, that if a user is working with an AWS service and feels dumb, that is a failing of the AWS service team. And that's a very hard line that I draw, in that none of the services that AWS offers—mostly. I'm going to carve out exceptions for some of the advanced ML stuff, and certainly the quantum computing folks. Don't get me started on that. 

But for most of these things, the fundamental concept of what the various services do is not that complicated. Someone who has gotten far enough to have spun up an AWS account specifically should be able to grasp on some level what these things are. And every time they're struggling trying to get something done, in an idealized version of the world, that's a service team failure and is an area in which a customer can reasonably expect the cloud provider service team to do better than they have. It's the divinely dissatisfied in perpetuity approach.

Katrina: Absolutely. Anytime that I can free up my users’ busy time to get to the core of whatever it is that they have to do. With the CDN, for example, I want to give my end-users time back to do whatever they want to do. That is not spending hours figuring out which CloudFront configurations apply to them. You would much rather be doing something else, be it coding an application, be it spending time with your kids, be it literally anything else in the world than trying to figure out what this very, very particular setting might be. 

And for me, that's a big part of that, is making sure that things that users have to do routinely are easy and fast and get them to the core of the thing that they have to do or that they want to do. So, basically, removing the obstacles from the things that you have to do to get you to the things that you want to do, faster.

Corey: Yes. I do want to caveat my statement as well because I know that there is some junior product manager out there somewhere at some startup, who's going to hear that and think, oh, well, our predictive algorithms are really good, so we're just going to go ahead and send things to our customers that we know that they would like and charge them for it. No. Stop. That is called credit card fraud. No. Stop it, you still need to get consent from users to sell them things. The fact that I'm not entirely sure whether that disclaimer was a joke or not, should tell you a lot about the state of our industry.

Katrina: [laugh]. We want things to be easier to use. I think that's a great fine line to call out, that I don't want to be making predictions about what I think you might want and get that wrong. That feels worse than just giving you the option to do what you want super easily.

Corey: Right. There's a reason that none of the cloud providers have implemented the web console version of Clippy of, “It looks like you're building a CloudFront distribution. Would you like help?” No. Because the first time it gets it wrong, it annoys the hell out of everyone.

Katrina: Damn. Cat’s out of the bag. Roll back the feature release, everyone.

Corey: Exactly. Yeah, but I assume it's also going to be Amazon-centric, so it wouldn't be a paperclip because that's too derivative. Instead of, I don’t know, a binder clip.

Katrina: [laugh].

Corey: Or a nail gun. There we go. The jokes would write themselves at some point. So, that was the worst experience that you had as a product manager. But I don't like ending on down notes. What's the best?

Katrina: The best experience that I've had working as a product manager was—different company again, a couple of years back. I took the time to sit down and, in laypeople's terms write out what a platform as a service meant. It ended up, I guess, setting me up well, for my career at Amazon because it was about six pages of a narrative of just what a platform as a service was, what the Cloud was, how it integrates, how servers play together. Wrote an analogy about how these things interconnect and why it matters. And just sort of understanding that there were probably a lot of people in my same shoes, maybe even working at my same company who, despite working on a platform as a service, didn't quite understand what it was that it did and why that layer of abstraction was important, much less the underlying infrastructure as a service that powered that thing. 

And taking the time to write that out in really simplistic terms then created the experience where when I shared it out, I had several folks come up to me and just say, “Hey, I've never really understood that thing. Thanks for making it approachable, and thanks for making me understand this piece of technology in a way that I otherwise wouldn't have.” And lowering that burden of information that you've had to go seek out yourself for other people that I went through the experience of finding myself in, and trying to understand and reading through incredibly dense technical blogs to understand something that could have been simplified much better has probably been the pinnacle of my product management experience so far. Just simply being able to take something incredibly complex, describe it very simply to people, and removing the burden from them having to go self-serve that information through the same pain that I had to understand it was a really great experience for me.

Corey: There's a lot of power in being able to, I guess, have the position in the market—which I do, and I basically took it because I figured, what's the worst-case someone's going to do? Take away my birthday—and say, “I don't get it.” And that tends to be valuable because every single time I've done that, people have come forward and said, “Oh, thank God, it wasn't just me. I just didn't want to say it out loud.” 

But there's much more power in being able to take that feedback and fix it; make it understandable in a way that is extraordinarily approachable. There's a reason that people don't just ask for a list of API options as documentation. They want to understand what business problem do I have that this solves for me. They want to see code samples, but they also want to see something in between the very high-level and the very low-level that explains considerations of using this, the trade-offs, the design reason why you would use this, and very often, why you wouldn't use a particular feature, product, or service. Anytime someone says, oh my product’s awesome and you should use it for everything, they're doing a sales job and not a particularly good one at that.

Katrina: Absolutely. I think the really apt analogy here is, you should disseminate whatever information you have and whatever knowledge you have, in a way that is consumable to your users. And the analogy to me that comes up is a water fountain. If someone is thirsty and—thirsty for knowledge, in this case— and you are trying to supply them with water, you might think that the very best thing that you can do is open up a fire hose on them because it's more water than they could ever use, and it's fantastic, and you're giving them all of these things. But you're also going to drown them. If you give them something that's more approachable and let them opt into consuming more as they're ready, it’s a much better experience for users, and you're enabling them to have their needs met, to opt into more information, to self-serve that at the same time, and then continued to grow with them.

Corey: I think that's a terrific sentiment to end the episode on. If people want to hear more about what you have to say, where can they find you?

Katrina: Folks can find me @katrinabakas on Twitter. Feel free to reach out anytime.

Corey: Excellent. Thank you so much for taking the time to speak with me. I appreciate it.

Katrina: Thanks, Corey. It was my pleasure.

Corey: Katrina Bakas, senior technical product manager at AWS’s CloudFront. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, and in the comments. Tell me what year you graduated from Stanford.

Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey atscreaminginthecloud.com, or wherever fine snark is sold.

This has been a HumblePod production. Stay humble.