Walking the Arcane Halls of AWS with Rachel Kelly

Episode Summary

Often on “Screaming” we get folks whose careers in tech don’t follow the typical path. This is certainly the case for Rachel Kelly, Senior Infrastructure Engineer at Fastly, who has pulled a kind of “Benjamin Buttoning” move, and has done a career repatriation. Rachel has gone from managing just the SaaS web app to managing vast groups of servers and data centers worldwide. Rachel talks about her repatriation and gives the why, what, where, and how of her current position. Rachel wanted to tap into some of the more arcane aspects of how the internet works, and at Fastly she gets to be a part of that group. Rachel and Corey bat some AWS considerations across the net and have an enlightening conversation about the multitudes of AWS. From AWS offerings, to nuances of hiring, to some practices that need to be left in the rear view—Rachel and Corey cover the spread!

Episode Show Notes & Transcript

About Rachel
Rachel Kelly is a Senior Engineer at Fastly in Infrastructure, and is a proud career-switcher over to tech as of about eight years ago. She lives in the Pacific Northwest and spends her time thinking about crafts, cycling, leadership, and ditching Google. Previously, she worked at Bright.md wrestling Ansible and Terraform into shape, and before then, a couple years at Puppet.  You can reach Rachel on twitter @wholemilk, or at [email protected].

Links:

Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.


Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.


Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key or a shared admin account isn’t going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport’s unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And no, that is not me telling you to go away, it is: goteleport.com.


Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. A periodic subject that comes up from folks desperate to sell people things is this idea of cloud repatriation, where people have put their entire business in the cloud decided, “Mmm, not so much. I’ll build some data centers and move it there.” It’s an inspiring story if you’re selling things for data centers, but it’s not something we’re seeing widespread evidence of, and I maintain that.


Today, we’re going to talk about that, only completely different. My guest today is Rachel Kelly, senior infrastructure engineer at Fastly. And no, Fastly has not done a cloud repatriation of which I am aware. But Rachel, you’ve done a career repatriation. You went from working with 
AWS in your previous company to working in bare metal. First, welcome to the show, and thank you for joining me.


Rachel: Thanks, Corey. Super happy to be here.


Corey: Now, let’s talk about why you would do such a thing. It feels almost like you’re Benjamin Button-ing here.


Rachel: Yeah, a bit. The normal flow has been to go from sort of a sysadmin level, where you’re managing servers fairly directly, to an operational level, where you are managing entire swathes of servers to entire data centers and so forth. But I went from managing just the SaaS web app to managing enormous groups of servers in data centers all over the world. And I did that because the provisioning of the web app, even on AWS, was absolutely my favorite part. What I’ve always wanted to get better with is the Linux and networking side of how our internet runs, and at Fastly, we are responsible for such a huge percentage of traffic all over the world. We have enormous customers who rely on us to deliver that data. And I get to be part of the group of people that puts those enormous groups of servers into production.


Corey: I started my career in the more traditional way of starting out in data centers, building things out, and then finally scampering off into a world of cloud. And you learn things going through the data center side of the world that don’t necessarily command the same levels of attention in the cloud environment because you don’t have to think about these things. Networking is a great example. During the Great Recession, there was a salary freeze. I was not super thrilled in my job, but I couldn’t find another one, so I spent the year learning how networks worked, and it made me a better systems administrator as a direct result of this. Same story with file systems, not necessarily because I did extensive amounts of work with their innards, but because every sysadmin interview under the sun asked the same questions about how inodes work, how journaling works, et cetera, and you have to be able to pass the trivia-based hazing process in order to get a job 
when you’ve just been fired from your last one.


So, that became where I was focusing on these things. And now looking at a world of cloud, feels like we don’t really need that in any meaningful sense. I mean, a couple people need to know it, but by and large no one has to think about it. So, is that just a bunch of useless knowledge that is taking up valuable space in your brain that could be used for other stuff or do you think that there’s a valid story for folks who are working in purely cloud environments to still learn how the things underlying these concepts work?


Rachel: First of all, I think that there is so much that we can do with less particular networking knowledge than we’ve ever needed in the past, thanks so much in part to AWS and all of their hangers-on. But yes, there are still people who need this networking knowledge. And once you have that kind of knowledge, once you’re able to see how the routes talk to each other, and how your firewalls actually work, and how to abstract out these larger networks and determining your subnetting and everything, you can utilize that really beautifully, even in something like VPC on AWS. Without that kind of knowledge, like, you can still get quite a bit done—which I think is a testament to the power of abstraction in AWS—but I mean, boy oh boy, what you can do once you have some of that knowledge.


Corey: I’m not allowed in the AWS data centers because I’m very bad at dodging bullets, but I find the knowledge is still useful because it helps me reason about things. When I know what—at least in a traditional environment—it’s doing, I know what AWS is emulating, and I can safely assume that I haven’t discovered some bug in their network stack for almost anything reasonable that I’d be working on other than maybe their documentation explaining it. So, when I start reasoning about it from that perspective, things make a lot more sense. And that’s always been helpful. The argument historically has been when you’re hiring—at least in the earlier days of cloud—well, I’m trying to hire, but it’s hard to find cloud talent, so the story was always, “Oh, don’t worry. If you’ve worked in a data center, we’ll teach you the cloudy pieces because it’s the natural evolution of things.” And there’s a whole cottage industry of people training for exactly that use case. Because you are who you are, and doing what you do, how do you find hiring works when you’re going the exact opposite direction?


Rachel: Oh, my gosh, it’s so interesting. In my area, we are trying to build these huge groups of servers based on bare metal. Do we hire sysadmins? Maybe. Do we hire ops folks? Maybe. Do we hire network engineers? Also, maybe.


There are so many angles that we need to be aware of when pulling new talent into our area. And I think it’s fascinating what all of these different, largely, like, non-programmer types have to contribute to the provisioning process. We need someone with expertise in security, and quality, and networking, and file systems, and everything else between those items. And it’s really exciting seeing what people can add to our process.


Corey: There’s so much in there that I love, but at the part I’m going to focus on is you’re talking about new hires as being additive. And that is valuable. It can lead to some pretty toxic and shitty behaviors, where it’s, “We want to make sure everyone we hire is schmucks we’ve hired now.” Like, no, that is not what we’re talking about. But culture is something you get whether you want it or not, and I firmly believe teams are atomic, when you bring someone new in or let someone go, you haven’t changed the team, you have a new team, in many respects, and that dynamic becomes incredibly important.


The idea of hiring people for strength has always been what I look for, as opposed to absence of weakness, where it’s okay, I’m going to ask you a whole bunch of questions around all the different aspects of computing; I’m going to find the area you’re bad at, and we just beat the snot out of you on that. It’s, yeah, if I want to join a fraternity, I would.


Rachel: [laugh]. Yeah, when I was job seeking, I wound up in interviews at places where their method of interviewing was very much hazing. “Well, let’s see, I haven’t read your resume. It says that you’ve set up a few things with Nginx. Do you know about this particular command in Nginx?” It’s like, “Well, geez, I could look it up and figure it out, but that’s not the point of this job.”


I mean, we work together collaboratively every day, and if that doesn’t sound familiar to you, I’m going to leave this interview. But yes, I mean, everybody’s additive. There was another gal who joined at the same time that I did at Fastly, and we both have a very operational background. And we were additive to the very strong networking and data center engineers who were already on the team. And as far as I can tell, the team changed overnight when we joined.


It is now our role—both this other gal’s and mine—to work so much on the automation piece of our build process, which has been focused on lightly in some areas, but that we can bring that with—even just shell scripting, we are able to enhance that process by so much. And I just fantasize about the day that we can get someone in who is directly on our team and focused on security, or directly on our team and focused on testing. The heights we could soar to with that kind of in-department knowledge, where we’re still focused on creating these builds, it’s just so exciting to think about.


Corey: It is and it’s easy to look at data centers as the way things used to be but not the future at all, but CDNs are increasingly becoming something very different than they used to be. And I admit I’m a little stodgy; I tend to fight the tide. There’s value in having something that is serving static assets close to your customer. There’s value to the CDN, in following the telco story, of aspiring to be more than just the quote-unquote, “Dumb pipe,” because that’s a commodity; you want to add differentiated value. But I’m also leery to wind up putting things that look like business logic into the edge at this stage.


And I’m starting to feel like I might be wrong as far as the way that the world views these things. But I like the idea that if a CDN takes an outage—which is not common, but it does happen—that I should be able to seamlessly—well, “Seamlessly”—failover to a different CDN within an hour or so. But if there’s significant business logic in your CDN, you’ve got to either have that replicated in near real-time between the two providers, or your migration is now measured with a calendar instead of a stopwatch.


Rachel: Yeah, absolutely. I mean, that’s an incredibly hard problem. We want to be able to really provide that uptime. And we don’t really have outages. Everybody remembers—well, listeners of this show will probably remember, the Fastly outage, but—


Corey: The Fastly outage, and that’s the—


Rachel: The—


Corey: —best part is the fact that I’m talking about ‘the’ and everyone knows the one I’m talking about, that says something.


Rachel: Yeah. In June of this year, we had an outage for 45 minutes, and it was just an incredible and beautiful effort on the engineering side to get us back up as quick as possible. There were a handful of naysayers, certainly, in the outage, but we fixed it real fast. One thing that I loved was your tweet about it in June, when our outage happened. “The fact that Fastly was able to detect, identify, and remediate this clearly complex problem as quickly as they did may be one of the most technically impressive things I’ve seen in years.” I appreciated that so much. So, many folks internal to Fastly appreciated that point of view so much because the answer to should I have a backup CDN? Like, yeah, maybe, and it is complicated because you have so much logic on the edge right there, but really, the answer is, we really do a good job of staying up. And that cannot be the full picture for any company that needs just a ton of HA, but that is what we’d really like to present, we really want you to be able to trust us. And I feel like we have demonstrated that.


Corey: I would argue from where I sit you absolutely have. If this were a three times a week situation, it wouldn’t matter, no one would care because no one’s going to trust the CDN that breaks like that.


Rachel: Right.


Corey: It gets to the idea of utility computing. And that means different things to different people, but to me, what that says is that when I use an actual utility, like water or electricity, when I turn the faucet or flip a switch, I don’t wonder if it’s going to work or not. Of course, now I have IoT light switches, so I absolutely wonder if it’s going to work or not, but going to the water story, yeah, I turn on the faucet, if something doesn’t happen, or the water comes out a different color than expecting, I have immediate concerns. And that is extraordinarily atypical and I can talk about that one time it happened. It’s not that every third time I go and wash my hands, the water catches fire because there’s fracking nearby, or something. Or it’s poisonous because I live in Flint. It is just a thing that works.


No one is going to sit here and have a business problem and say, “You know what I really need? I really need a local point of presence close to my users so that the static asset can be served more quickly and efficiently to this.” No, the business problem is, “Our website is slow, so people aren’t using it.” It’s how do you speak to things like that? And how do you make working with it either programmatically or through a console—because surprise, business users generally don’t interact with things via APIs—how do you make that straightforward? How do you make that accessible, and Fastly does—


Rachel: Oh gosh.


Corey: —a bang-up job on this.


Rachel: I think that Fastly has done a good job on it. How that has happened, I simply cannot tell you whatsoever. I am so far from support and marketing. I know that those folks work their tails off and really are focused on selling the story of you need your assets to be more easily delivered to the people who want to consume it. No, and you would never use that as a soundbite for Fastly because it [laugh] it sounds like a robot said it.


Corey: It’s always—I was gonna interesting, but I’m also going to go with strange—the ability to, for whatever reason, build out a large scaling infrastructure business like this—CDNs are one of those businesses where you’re not going to come up with this in your garage and a cloud provider tonight and be ready to deploy in a couple of weeks. It takes time to get these facilities out there. It takes tremendous capital investment. But I want to switch a little bit because I know that you’re a believer in this in the same way that I am. As much fun as it is to talk smack about cloud providers, I think it’s impossible to effectively understate just how transformative the idea of being able to prototype things via a cloud provider is.


Yeah, it’s not going to be all businesses, I’m not going to build a manufacturing company on a cloud provider overnight in my spare time, but I can build the bones of a SaaS app and see if it works or not without having to buy infrastructure or entering into long-term contracts. I just need a credit card and then I’ll use a free tier that’s going to lie to me and then hit me with a surprise $60,000 bill. But yeah, you know, the thought is there.


Rachel: The thought is there. I think that if you know a little bit what you’re doing with a not even terribly clever operations engineer to get into AWS with you, you can prototype that for pretty cheaply. If you’re not spending all this money on transfer fees and whatever else. If you really just want this small mock up of hey, does this work? Can it be reached from the network? Again, getting your networking knowledge in will only serve you, even in this setting, even though we’re in the modern era.


I mean, I think it’s incredible, and I think it’s responsible for the total democratization of the modern internet as we know it. Yes, there are other cloud providers, but AWS is who brought this to everybody. Their support for when you run into a jam is some of the most technical and capable of any support organization I’ve ever interfaced with. And at my previous role we did all the time because, you know, the internet gets complicated, if you can imagine that. And I just think that’s phenomenal.


On AWS, I want something where I’m hooking up some VPC to this Redis Database over here to a few EC2 instances with backups going over here, and some extremely restricted amount of dummy data flowing from all of those objects. And there’s nothing like that. [laugh].


Corey: Oh, yeah. And part of the reason behind this, as it turns out, is architectural. The billing system aspires to an eight-hour consistency model, in which case, I spin up something and it shows up in the bill eight hours later. In practice, this can take multiple days. But it’s never going to get fixed until the business decides, all right, you can set up a free tier account with the following limits on it, and to get past these, you have to affirmatively upgrade your account so we can start charging you and we automatically going turn things off or let you stop adding storage to it or whatnot, whenever you cross these limits.


Well today, you can do whatever you want for the first eight hours. And the way to fix this is, cool, Amazon eats it. Whenever their billing system doesn’t catch something, they eat the free tier. And given how much they love money, and trimming margins, and the rest, suddenly you have an incentive because if someone screws up royally and gets that $60,000 bill before the billing system can clamp down on it, okay, great. I would rather the $1.6 trillion company eat that bill than the poor schmoo sitting in their dorm room halfway around the world.


Rachel: That’s such a good point. Some schmo in their dorm room. How many kids have been bitten by this that we don’t hear about because people become ashamed of “Stupid mistakes” like that—that was big air quotes, for those of you at home. It’s not a stupid mistake.


Corey: People think I’m kidding when I say this, but Robinhood had a tragic story, right? A 19-year-old was day-trading, saw on the app that he had lost $900,000—which turned out not to be true once things settled—and killed himself. And that is tragic. It is not a question of if, it’s a question of when someone sees this, reads that you’re on the hook for it, support takes a few days to respond, they see their life flashing before their eyes because in many cases, that is more money than people in some of these places will expect to earn in a year, and does something horribly tragic. And at that point, there’s a bell that has been rung that cannot be unrung.


Of all the things I want to fix, yeah, I complain and I whine about an awful lot of stuff, but this is the one that has the most tragic consequences. No story for a human is going to end in tragedy because of the usurious pricing for Managed NAT Gateway data transfer, but a surprise bill that we know support is going to wipe over something like that, that is going to break people. And that’s not okay.


Rachel: No, it’s not okay. I think that you write very well about that topic in particular, and I really would love to see some changes take place. I 
know that Amazon knows their business better than to need to rely on some Adore Me-style subscription model that you can’t figure out how to get out of. Like, have some faith in your products or don’t sell it.


Corey: I really, really wish that more companies saw it that way. And the hell of it is the best shining example is a recurring sponsor of this show: Oracle Cloud. Oracle is, let’s be honest, they’re Oracle; that’s less a brand than a warning label in many cases, but I’ve often said the Oracle Cloud biggest challenge is the word Oracle at the front of it—


Rachel: Absolutely.


Corey: —because their service offering is legitimate, their free tier is actually free—I’ve been running some fairly beefy stuff there for over a year, and have never been charged a dime for it. And it’s not because I’m special; it’s because I haven’t taken the affirmative upgrade-my-account step. And their data transfer pricing is great. Within the confines of those things, yeah, it’s terrific. I can’t speak to what it looks like a super large-scale for a cloud-native app, yet, but that’s going to change; people are starting to take them a lot more seriously.


And I’ve got to say, in previous years in the re:Invent keynotes, they’ve made fun and kicked at Oracle a fair bit, which no one has any sympathy for. Now, I don’t think that would lend the same way, just among people who have decided to suspend disbelief long enough and kick the tires in the Oracle free tier. It’s like, well, yeah, you can say a lot of negative things about Oracle—and I have a list of them—but you know, what I never got with Oracle: A surprise bill. And its Oracle we’re talking about, where surprise billing is the entire reason that they—


Rachel: It’s the model.


Corey: —are a company.


Rachel: Yeah. [laugh].


Corey: That is the model. And in this case, they are nailing it. And I’ve often said that you can buy my attention, but not my opinion. Long before they sponsored this show, I was talking, like, this about this particular offering. “Oh, so you’re saying we should migrate everything to Oracle databases?” “Good, Lord, no. Not without talking with someone who’s been down that path.” And almost everyone who has will scream at you about it. It’s a separate model. It’s a separate division. It’s a separate way of thinking about things. And I’m a big fan of that.


Rachel: Oh, that’s great. There have been ruinous results of Oracle’s decisions and acquisitions in our industry, and yet, this does appear to be a slice of the market that they have given autonomy to the people running it. And I feel like that’s really the key. I know just a hair about the product process—the new product introduction process at Amazon in general, And therefore, I actually do have a bit of faith that they will fix this. It’s just a huge problem, and when Oracle is eating your lunch, I mean, I just—you really have some things to reconsider.


Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn’t heard of before, but they’re doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they’re using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they’re able to wind up taking what you’re running as it is, in AWS, with no changes, and run it inside of their data centers that span multiple regions. I’m somewhat skeptical, but their customers seem to really like them, so that’s one of those areas where I really have a hard time being too snarky about it because when you solve a customer’s problem, and they get out there in public and say, “We’re solving a problem,” it’s very hard to snark about that. Multus Medical, Construx.ai, and Stax have seen significant results by using them, and it’s worth exploring. So, if you’re looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That’s risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.


Corey: I am an Amazon fan. I think that given the talent, and the insight, and the drive that they have there—not to mention the fact that they’re a $1.6 trillion company—if they want to do something, it will get done. And there are very few bounds I would put on it. Which means that everything that Amazon does, is, on some level, a choice. There are very few things they could not achieve with concerted effort if they cared enough.


Corey: I want to also tell a story about you for a change, because why not? Back in 2018, I was just really getting to have an audience, and the rest, and I found myself at the replay party at re:Invent. And it was a weird moment for me because I’d finished most of my speaking stuff, I had hung out with my meetups and my friends and the rest, and I’m wandering around the party—


Rachel: Your DevOps stand-up, as I recall.


Corey: That’s what it w—that’s what it was. Yeah, my DevOps stand-up, cloud comedy, whatever you want to call it. And I’m walking around, and it’s isolating and weird after something like that—back in the before times, at least—and when people know me as a character, more or less, but not as a person, and it’s isolating, and it’s lonely, and it’s—again, you don’t feel great after four days in Las Vegas, and it’s dark, and it’s hard to tell who’s who we ran into each other and just started walking around and having a conversation outside because apparently 4000 decibels as a little much for volume for both of us. And it was just great finding someone who I can talk to as a human being. There’s not enough of that in different ways. Because remember, back then, I was an independent consultant I didn’t have colleagues to hang out with. It was—


Rachel: Oh, that was pre-Duckbill.


Corey: That was when I was still the Quinn Advisory Group.


Rachel: Oh, very good. Okay. Yes, I do remember that.


Corey: The Duckbill Group was formed about a month-and-a-half after that as memory serves.


Rachel: Oh, okay. Cool.


Corey: But yeah, same problem. It’s, how do I build this? How do I turn this into something was a separate problem that hadn’t quite—hadn’t come up with an answer yet. So, I’m an independent consultant, wandering around, feeling lonely. My clients are all off doing their own things because it turns out that I’m great at representing clients in meetings with Amazon execs, but lousy at representing them on the dance floor.


So, it was just the empathy that exuded from you was just phenomenal. And I don’t know ever thank you for just how refreshing it was to be able to just step back from the show for a minute and be a person. So thanks.


Rachel: Oh, likewise. I remember I had gotten in touch with you beforehand as well to say, like, “I’m going to be at re:Invent. I don’t know any women who will be there. Can you please introduce me to some?” And you introduce me to some lovely people who, along with you, really helped me navigate my first re:Invent in a huge way, which was—you think it’s going to be overwhelming, multiply that by ten or a hundred. That is how much information is coming at you all the time when you are at re:Invent.


So, to go to this funny party where there was like some EDM DJ, who I think was, like, well-known or something in 2018, be like, [laugh] that’s really not my thing. But I want to bum around this party, I do want to see what’s going on, and if I can touch base with anybody else that I have met during this conference. And I remember we, kind of like, stuck close to each other. And that was so—that was, it was so human. And I appreciated that so much from you as well.


I was sent by my company—as anybody who goes to [OSCON 00:31:03] or re:Invent are, if they pay full freight [laugh]—it was so lovely to just have a buddy to bum around with and make fun of things, and talk shop, and everything in between.


Corey: I do want to give one small tip, something buried in there that I think is just something I’ve been doing extensively for a while, but I haven’t really ever called it out, or at least not recently—and I’ll do a tweet thread about this after we’re done recording—the counterpoint that I want to that I want to point out is that introductions are great, but every person I introduced you to, I had your permission to give their email address to them, and I reached out to them independently in every case and said, “Hey, someone would like”—once I was had your permission to reference you—“She would like to talk to other folks who don’t look like me who are going to re:Invent. May I introduce you?” The idea of a double opt-in introduction goes so far. And I’m talking about this for folks who aren’t me. In my case, fine. If some rando wants to introduce me to some other rando, knock yourself out. There is very little showing up in my inbox that I am not going to have some way of handling. But not everyone thinks about things that way, and it just shows a baseline level of human respect.


Rachel: Yeah, absolutely. I actually just did that this morning. I’m sure all of us get these calls a few times a year: “I’m thinking about switching to tech because the money’s there, the stability is there, the job market is there, and I have been underpaid and treated poorly for a long time,” or whatever variation on that story that I know we all are aware of. And I talked with him for a while last night, and then I put him in touch with the dual opt-in emails with someone in the field that he’s looking at, exactly, and a recruiter friend of mine to help give more perspective on the industry as a whole. And with both of those people, I asked permission to introduce them to the friend of mine who had reached out to me, and both of them responded right away because when you are fielding questions like these all day, you become familiar with the kindest way to do that.


And I really love being able to use my network in that way. Yes, I know a person at X, and yes, I would love to introduce you to Y. And I will make sure that everybody agrees and knows that this is coming, and I’m not just taken by surprise. Where I do get those emails and I understand that etiquette is something to learn, it isn’t directly common-sense sometimes. And then you sit down and you think about it, or someone says to you like, “I really need you to give me a heads up before giving my contact information to someone that I don’t know.”


Corey: It happens. It’s about being accessible. It’s about making the industry better than it is. And on that topic, I have one more area I want to delve into before we call it a show, and that is you are on the program committee for SeaGL, the Seattle GNU/Linux conference.


Rachel: That’s right.


Corey: I have fond memories of that conference, once upon a time. I gave a keynote a few years ago back when I was, you know, able to go places without it being a deadly risk, and much more involved in the community side of the world when it comes to conferences. I’ve unfortunately pulled back from a lot of it, just due to demands on my time. But great conference. Enjoyed a lot of the conversations once you, sort of, steered around the true believers around some areas of things, to the point where it subverts, you know, being civil to people. But it was a good conference. There was a lot to recommend it.


Rachel: SeaGL is a beautiful little conference. It is community-focused. We don’t let sponsors get on stage. We really restrict how much the people giving us money are able to dictate what we do. What we do is create a platform for people to discuss open-source in a human way, I would say.


I think in our earlier days, we had a lot of focus on software freedom at all costs, and that has softened in the name of humans and social justice in a way that I feel very proud of. I have been the program chair for three years now, and it’s just wonderful seeing the trends that come up every year. Our conference is Friday and Saturday, November 5th and 6th, so I hope that by the time you hear this, you will still have an opportunity to go to that; I’m not sure. Some of the themes this year have just been so interesting. It’s all about—and this will be very interesting to a particular subset of people, and maybe not to everybody—but about open-source governance, and how do we maintain the soul and the purpose of an open-source project, while keeping people housed and fed who are working on these things, and to not sign over all the rights of a given project to our corporate overlords and such.


So, there’s a number of talks that are going to be talking about that. A few years ago, the trend that I was really excited about that I personally gave a talk about as well, is how to start owning and managing your own data entirely. I gave a talk on trying to get off Google, which is Herculean and close to impossible. And I understand that, and that’s frustrating. But you know, we see these trends where we’re trying to help our community protect itself and remain open at the same time in a technical and open-source context. And it’s just an exciting and lovely organization and event each year. This is our second year being virtual. I was shocked by how good our virtual experience was last year. And I have high hopes for this year, too. So, I hope you can come check it out.


Corey: I would highly recommend it though I believe this will be airing after the show goes out.


Rachel: Ah darn.


Corey: But there’s always next year.


Rachel: That’s right. And they’re all recorded as well, all the talks will be recorded. The publication date on those might be a little bit after but 
yes, they will all be up.


Corey: But we will of course include links to that in the [show notes 00:37:13] because there’s always next year.


Rachel: That’s right.


Corey: I want to thank you so much for taking the time to speak with me. If people want to learn more, where can they find you?


Rachel: I think probably the best place is on Twitter. That is @wholemilk on Twitter. Like, the dairy product by the gallon that’s me.


Corey: And that link to that will go in the [show notes 00:37:33] as well. Thank you so much for taking the time to speak with me today. I really appreciate it.


Rachel: Thank you, Corey. This has been great.


Corey: It really has. Rachel Kelly, senior infrastructure engineer at Fastly. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that you should absolutely shove your business logic fully into the CDN, then wind up not being able to edit the comment because it’s locked to a single CDN.


Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.


Announcer: This has been a HumblePod production. Stay humble.

Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key or a shared admin account isn’t going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport’s unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And no, that is not me telling you to go away, it is: goteleport.com.

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. A periodic subject that comes up from folks desperate to sell people things is this idea of cloud repatriation, where people have put their entire business in the cloud decided, “Mmm, not so much. I’ll build some data centers and move it there.” It’s an inspiring story if you’re selling things for data centers, but it’s not something we’re seeing widespread evidence of, and I maintain that.

Today, we’re going to talk about that, only completely different. My guest today is Rachel Kelly, senior infrastructure engineer at Fastly. And no, Fastly has not done a cloud repatriation of which I am aware. But Rachel, you’ve done a career repatriation. You went from working with AWS in your previous company to working in bare metal. First, welcome to the show, and thank you for joining me.

Rachel: Thanks, Corey. Super happy to be here.

Corey: Now, let’s talk about why you would do such a thing. It feels almost like you’re Benjamin Button-ing here.

Rachel: Yeah, a bit. The normal flow has been to go from sort of a sysadmin level, where you’re managing servers fairly directly, to an operational level, where you are managing entire swathes of servers to entire data centers and so forth. But I went from managing just the SaaS web app to managing enormous groups of servers in data centers all over the world. And I did that because the provisioning of the web app, even on AWS, was absolutely my favorite part. What I’ve always wanted to get better with is the Linux and networking side of how our internet runs, and at Fastly, we are responsible for such a huge percentage of traffic all over the world. We have enormous customers who rely on us to deliver that data. And I get to be part of the group of people that puts those enormous groups of servers into production.

Corey: I started my career in the more traditional way of starting out in data centers, building things out, and then finally scampering off into a world of cloud. And you learn things going through the data center side of the world that don’t necessarily command the same levels of attention in the cloud environment because you don’t have to think about these things. Networking is a great example. During the Great Recession, there was a salary freeze. I was not super thrilled in my job, but I couldn’t find another one, so I spent the year learning how networks worked, and it made me a better systems administrator as a direct result of this. Same story with file systems, not necessarily because I did extensive amounts of work with their innards, but because every sysadmin interview under the sun asked the same questions about how inodes work, how journaling works, et cetera, and you have to be able to pass the trivia-based hazing process in order to get a job when you’ve just been fired from your last one.

So, that became where I was focusing on these things. And now looking at a world of cloud, feels like we don’t really need that in any meaningful sense. I mean, a couple people need to know it, but by and large no one has to think about it. So, is that just a bunch of useless knowledge that is taking up valuable space in your brain that could be used for other stuff or do you think that there’s a valid story for folks who are working in purely cloud environments to still learn how the things underlying these concepts work?

Rachel: First of all, I think that there is so much that we can do with less particular networking knowledge than we’ve ever needed in the past, thanks so much in part to AWS and all of their hangers-on. But yes, there are still people who need this networking knowledge. And once you have that kind of knowledge, once you’re able to see how the routes talk to each other, and how your firewalls actually work, and how to abstract out these larger networks and determining your subnetting and everything, you can utilize that really beautifully, even in something like VPC on AWS. Without that kind of knowledge, like, you can still get quite a bit done—which I think is a testament to the power of abstraction in AWS—but I mean, boy oh boy, what you can do once you have some of that knowledge.

Corey: I’m not allowed in the AWS data centers because I’m very bad at dodging bullets, but I find the knowledge is still useful because it helps me reason about things. When I know what—at least in a traditional environment—it’s doing, I know what AWS is emulating, and I can safely assume that I haven’t discovered some bug in their network stack for almost anything reasonable that I’d be working on other than maybe their documentation explaining it. So, when I start reasoning about it from that perspective, things make a lot more sense. And that’s always been helpful. The argument historically has been when you’re hiring—at least in the earlier days of cloud—well, I’m trying to hire, but it’s hard to find cloud talent, so the story was always, “Oh, don’t worry. If you’ve worked in a data center, we’ll teach you the cloudy pieces because it’s the natural evolution of things.” And there’s a whole cottage industry of people training for exactly that use case. Because you are who you are, and doing what you do, how do you find hiring works when you’re going the exact opposite direction?

Rachel: Oh, my gosh, it’s so interesting. In my area, we are trying to build these huge groups of servers based on bare metal. Do we hire sysadmins? Maybe. Do we hire ops folks? Maybe. Do we hire network engineers? Also, maybe.

There are so many angles that we need to be aware of when pulling new talent into our area. And I think it’s fascinating what all of these different, largely, like, non-programmer types have to contribute to the provisioning process. We need someone with expertise in security, and quality, and networking, and file systems, and everything else between those items. And it’s really exciting seeing what people can add to our process.

Corey: There’s so much in there that I love, but at the part I’m going to focus on is you’re talking about new hires as being additive. And that is valuable. It can lead to some pretty toxic and shitty behaviors, where it’s, “We want to make sure everyone we hire is schmucks we’ve hired now.” Like, no, that is not what we’re talking about. But culture is something you get whether you want it or not, and I firmly believe teams are atomic, when you bring someone new in or let someone go, you haven’t changed the team, you have a new team, in many respects, and that dynamic becomes incredibly important.

The idea of hiring people for strength has always been what I look for, as opposed to absence of weakness, where it’s okay, I’m going to ask you a whole bunch of questions around all the different aspects of computing; I’m going to find the area you’re bad at, and we just beat the snot out of you on that. It’s, yeah, if I want to join a fraternity, I would.

Rachel: [laugh]. Yeah, when I was job seeking, I wound up in interviews at places where their method of interviewing was very much hazing. “Well, let’s see, I haven’t read your resume. It says that you’ve set up a few things with Nginx. Do you know about this particular command in Nginx?” It’s like, “Well, geez, I could look it up and figure it out, but that’s not the point of this job.”

I mean, we work together collaboratively every day, and if that doesn’t sound familiar to you, I’m going to leave this interview. But yes, I mean, everybody’s additive. There was another gal who joined at the same time that I did at Fastly, and we both have a very operational background. And we were additive to the very strong networking and data center engineers who were already on the team. And as far as I can tell, the team changed overnight when we joined.

It is now our role—both this other gal’s and mine—to work so much on the automation piece of our build process, which has been focused on lightly in some areas, but that we can bring that with—even just shell scripting, we are able to enhance that process by so much. And I just fantasize about the day that we can get someone in who is directly on our team and focused on security, or directly on our team and focused on testing. The heights we could soar to with that kind of in-department knowledge, where we’re still focused on creating these builds, it’s just so exciting to think about.

Corey: It is and it’s easy to look at data centers as the way things used to be but not the future at all, but CDNs are increasingly becoming something very different than they used to be. And I admit I’m a little stodgy; I tend to fight the tide. There’s value in having something that is serving static assets close to your customer. There’s value to the CDN, in following the telco story, of aspiring to be more than just the quote-unquote, “Dumb pipe,” because that’s a commodity; you want to add differentiated value. But I’m also leery to wind up putting things that look like business logic into the edge at this stage.

And I’m starting to feel like I might be wrong as far as the way that the world views these things. But I like the idea that if a CDN takes an outage—which is not common, but it does happen—that I should be able to seamlessly—well, “Seamlessly”—failover to a different CDN within an hour or so. But if there’s significant business logic in your CDN, you’ve got to either have that replicated in near real-time between the two providers, or your migration is now measured with a calendar instead of a stopwatch.

Rachel: Yeah, absolutely. I mean, that’s an incredibly hard problem. We want to be able to really provide that uptime. And we don’t really have outages. Everybody remembers—well, listeners of this show will probably remember, the Fastly outage, but—

Corey: The Fastly outage, and that’s the—

Rachel: The—

Corey: —best part is the fact that I’m talking about ‘the’ and everyone knows the one I’m talking about, that says something.

Rachel: Yeah. In June of this year, we had an outage for 45 minutes, and it was just an incredible and beautiful effort on the engineering side to get us back up as quick as possible. There were a handful of naysayers, certainly, in the outage, but we fixed it real fast. One thing that I loved was your tweet about it in June, when our outage happened. “The fact that Fastly was able to detect, identify, and remediate this clearly complex problem as quickly as they did may be one of the most technically impressive things I’ve seen in years.” I appreciated that so much. So, many folks internal to Fastly appreciated that point of view so much because the answer to should I have a backup CDN? Like, yeah, maybe, and it is complicated because you have so much logic on the edge right there, but really, the answer is, we really do a good job of staying up. And that cannot be the full picture for any company that needs just a ton of HA, but that is what we’d really like to present, we really want you to be able to trust us. And I feel like we have demonstrated that.

Corey: I would argue from where I sit you absolutely have. If this were a three times a week situation, it wouldn’t matter, no one would care because no one’s going to trust the CDN that breaks like that.

Rachel: Right.

Corey: It gets to the idea of utility computing. And that means different things to different people, but to me, what that says is that when I use an actual utility, like water or electricity, when I turn the faucet or flip a switch, I don’t wonder if it’s going to work or not. Of course, now I have IoT light switches, so I absolutely wonder if it’s going to work or not, but going to the water story, yeah, I turn on the faucet, if something doesn’t happen, or the water comes out a different color than expecting, I have immediate concerns. And that is extraordinarily atypical and I can talk about that one time it happened. It’s not that every third time I go and wash my hands, the water catches fire because there’s fracking nearby, or something. Or it’s poisonous because I live in Flint. It is just a thing that works.

No one is going to sit here and have a business problem and say, “You know what I really need? I really need a local point of presence close to my users so that the static asset can be served more quickly and efficiently to this.” No, the business problem is, “Our website is slow, so people aren’t using it.” It’s how do you speak to things like that? And how do you make working with it either programmatically or through a console—because surprise, business users generally don’t interact with things via APIs—how do you make that straightforward? How do you make that accessible, and Fastly does—

Rachel: Oh gosh.

Corey: —a bang-up job on this.

Rachel: I think that Fastly has done a good job on it. How that has happened, I simply cannot tell you whatsoever. I am so far from support and marketing. I know that those folks work their tails off and really are focused on selling the story of you need your assets to be more easily delivered to the people who want to consume it. No, and you would never use that as a soundbite for Fastly because it [laugh] it sounds like a robot said it.

Corey: It’s always—I was gonna interesting, but I’m also going to go with strange—the ability to, for whatever reason, build out a large scaling infrastructure business like this—CDNs are one of those businesses where you’re not going to come up with this in your garage and a cloud provider tonight and be ready to deploy in a couple of weeks. It takes time to get these facilities out there. It takes tremendous capital investment. But I want to switch a little bit because I know that you’re a believer in this in the same way that I am. As much fun as it is to talk smack about cloud providers, I think it’s impossible to effectively understate just how transformative the idea of being able to prototype things via a cloud provider is.

Yeah, it’s not going to be all businesses, I’m not going to build a manufacturing company on a cloud provider overnight in my spare time, but I can build the bones of a SaaS app and see if it works or not without having to buy infrastructure or entering into long-term contracts. I just need a credit card and then I’ll use a free tier that’s going to lie to me and then hit me with a surprise $60,000 bill. But yeah, you know, the thought is there.

Rachel: The thought is there. I think that if you know a little bit what you’re doing with a not even terribly clever operations engineer to get into AWS with you, you can prototype that for pretty cheaply. If you’re not spending all this money on transfer fees and whatever else. If you really just want this small mock up of hey, does this work? Can it be reached from the network? Again, getting your networking knowledge in will only serve you, even in this setting, even though we’re in the modern era.

I mean, I think it’s incredible, and I think it’s responsible for the total democratization of the modern internet as we know it. Yes, there are other cloud providers, but AWS is who brought this to everybody. Their support for when you run into a jam is some of the most technical and capable of any support organization I’ve ever interfaced with. And at my previous role we did all the time because, you know, the internet gets complicated, if you can imagine that. And I just think that’s phenomenal.

On AWS, I want something where I’m hooking up some VPC to this Redis Database over here to a few EC2 instances with backups going over here, and some extremely restricted amount of dummy data flowing from all of those objects. And there’s nothing like that. [laugh].

Corey: Oh, yeah. And part of the reason behind this, as it turns out, is architectural. The billing system aspires to an eight-hour consistency model, in which case, I spin up something and it shows up in the bill eight hours later. In practice, this can take multiple days. But it’s never going to get fixed until the business decides, all right, you can set up a free tier account with the following limits on it, and to get past these, you have to affirmatively upgrade your account so we can start charging you and we automatically going turn things off or let you stop adding storage to it or whatnot, whenever you cross these limits.

Well today, you can do whatever you want for the first eight hours. And the way to fix this is, cool, Amazon eats it. Whenever their billing system doesn’t catch something, they eat the free tier. And given how much they love money, and trimming margins, and the rest, suddenly you have an incentive because if someone screws up royally and gets that $60,000 bill before the billing system can clamp down on it, okay, great. I would rather the $1.6 trillion company eat that bill than the poor schmoo sitting in their dorm room halfway around the world.

Rachel: That’s such a good point. Some schmo in their dorm room. How many kids have been bitten by this that we don’t hear about because people become ashamed of “Stupid mistakes” like that—that was big air quotes, for those of you at home. It’s not a stupid mistake.

Corey: People think I’m kidding when I say this, but Robinhood had a tragic story, right? A 19-year-old was day-trading, saw on the app that he had lost $900,000—which turned out not to be true once things settled—and killed himself. And that is tragic. It is not a question of if, it’s a question of when someone sees this, reads that you’re on the hook for it, support takes a few days to respond, they see their life flashing before their eyes because in many cases, that is more money than people in some of these places will expect to earn in a year, and does something horribly tragic. And at that point, there’s a bell that has been rung that cannot be unrung.

Of all the things I want to fix, yeah, I complain and I whine about an awful lot of stuff, but this is the one that has the most tragic consequences. No story for a human is going to end in tragedy because of the usurious pricing for Managed NAT Gateway data transfer, but a surprise bill that we know support is going to wipe over something like that, that is going to break people. And that’s not okay.

Rachel: No, it’s not okay. I think that you write very well about that topic in particular, and I really would love to see some changes take place. I know that Amazon knows their business better than to need to rely on some Adore Me-style subscription model that you can’t figure out how to get out of. Like, have some faith in your products or don’t sell it.

Corey: I really, really wish that more companies saw it that way. And the hell of it is the best shining example is a recurring sponsor of this show: Oracle Cloud. Oracle is, let’s be honest, they’re Oracle; that’s less a brand than a warning label in many cases, but I’ve often said the Oracle Cloud biggest challenge is the word Oracle at the front of it—

Rachel: Absolutely.

Corey: —because their service offering is legitimate, their free tier is actually free—I’ve been running some fairly beefy stuff there for over a year, and have never been charged a dime for it. And it’s not because I’m special; it’s because I haven’t taken the affirmative upgrade-my-account step. And their data transfer pricing is great. Within the confines of those things, yeah, it’s terrific. I can’t speak to what it looks like a super large-scale for a cloud-native app, yet, but that’s going to change; people are starting to take them a lot more seriously.

And I’ve got to say, in previous years in the re:Invent keynotes, they’ve made fun and kicked at Oracle a fair bit, which no one has any sympathy for. Now, I don’t think that would lend the same way, just among people who have decided to suspend disbelief long enough and kick the tires in the Oracle free tier. It’s like, well, yeah, you can say a lot of negative things about Oracle—and I have a list of them—but you know, what I never got with Oracle: A surprise bill. And its Oracle we’re talking about, where surprise billing is the entire reason that they—

Rachel: It’s the model.

Corey: —are a company.

Rachel: Yeah. [laugh].

Corey: That is the model. And in this case, they are nailing it. And I’ve often said that you can buy my attention, but not my opinion. Long before they sponsored this show, I was talking, like, this about this particular offering. “Oh, so you’re saying we should migrate everything to Oracle databases?” “Good, Lord, no. Not without talking with someone who’s been down that path.” And almost everyone who has will scream at you about it. It’s a separate model. It’s a separate division. It’s a separate way of thinking about things. And I’m a big fan of that.

Rachel: Oh, that’s great. There have been ruinous results of Oracle’s decisions and acquisitions in our industry, and yet, this does appear to be a slice of the market that they have given autonomy to the people running it. And I feel like that’s really the key. I know just a hair about the product process—the new product introduction process at Amazon in general, And therefore, I actually do have a bit of faith that they will fix this. It’s just a huge problem, and when Oracle is eating your lunch, I mean, I just—you really have some things to reconsider.

Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn’t heard of before, but they’re doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they’re using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they’re able to wind up taking what you’re running as it is, in AWS, with no changes, and run it inside of their data centers that span multiple regions. I’m somewhat skeptical, but their customers seem to really like them, so that’s one of those areas where I really have a hard time being too snarky about it because when you solve a customer’s problem, and they get out there in public and say, “We’re solving a problem,” it’s very hard to snark about that. Multus Medical, Construx.ai, and Stax have seen significant results by using them, and it’s worth exploring. So, if you’re looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That’s risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.

Corey: I am an Amazon fan. I think that given the talent, and the insight, and the drive that they have there—not to mention the fact that they’re a $1.6 trillion company—if they want to do something, it will get done. And there are very few bounds I would put on it. Which means that everything that Amazon does, is, on some level, a choice. There are very few things they could not achieve with concerted effort if they cared enough.

Corey: I want to also tell a story about you for a change, because why not? Back in 2018, I was just really getting to have an audience, and the rest, and I found myself at the replay party at re:Invent. And it was a weird moment for me because I’d finished most of my speaking stuff, I had hung out with my meetups and my friends and the rest, and I’m wandering around the party—

Rachel: Your DevOps stand-up, as I recall.

Corey: That’s what it w—that’s what it was. Yeah, my DevOps stand-up, cloud comedy, whatever you want to call it. And I’m walking around, and it’s isolating and weird after something like that—back in the before times, at least—and when people know me as a character, more or less, but not as a person, and it’s isolating, and it’s lonely, and it’s—again, you don’t feel great after four days in Las Vegas, and it’s dark, and it’s hard to tell who’s who we ran into each other and just started walking around and having a conversation outside because apparently 4000 decibels as a little much for volume for both of us. And it was just great finding someone who I can talk to as a human being. There’s not enough of that in different ways. Because remember, back then, I was an independent consultant I didn’t have colleagues to hang out with. It was—

Rachel: Oh, that was pre-Duckbill.

Corey: That was when I was still the Quinn Advisory Group.

Rachel: Oh, very good. Okay. Yes, I do remember that.

Corey: The Duckbill Group was formed about a month-and-a-half after that as memory serves.

Rachel: Oh, okay. Cool.

Corey: But yeah, same problem. It’s, how do I build this? How do I turn this into something was a separate problem that hadn’t quite—hadn’t come up with an answer yet. So, I’m an independent consultant, wandering around, feeling lonely. My clients are all off doing their own things because it turns out that I’m great at representing clients in meetings with Amazon execs, but lousy at representing them on the dance floor.

So, it was just the empathy that exuded from you was just phenomenal. And I don’t know ever thank you for just how refreshing it was to be able to just step back from the show for a minute and be a person. So thanks.

Rachel: Oh, likewise. I remember I had gotten in touch with you beforehand as well to say, like, “I’m going to be at re:Invent. I don’t know any women who will be there. Can you please introduce me to some?” And you introduce me to some lovely people who, along with you, really helped me navigate my first re:Invent in a huge way, which was—you think it’s going to be overwhelming, multiply that by ten or a hundred. That is how much information is coming at you all the time when you are at re:Invent.

So, to go to this funny party where there was like some EDM DJ, who I think was, like, well-known or something in 2018, be like, [laugh] that’s really not my thing. But I want to bum around this party, I do want to see what’s going on, and if I can touch base with anybody else that I have met during this conference. And I remember we, kind of like, stuck close to each other. And that was so—that was, it was so human. And I appreciated that so much from you as well.

I was sent by my company—as anybody who goes to [OSCON 00:31:03] or re:Invent are, if they pay full freight [laugh]—it was so lovely to just have a buddy to bum around with and make fun of things, and talk shop, and everything in between.

Corey: I do want to give one small tip, something buried in there that I think is just something I’ve been doing extensively for a while, but I haven’t really ever called it out, or at least not recently—and I’ll do a tweet thread about this after we’re done recording—the counterpoint that I want to that I want to point out is that introductions are great, but every person I introduced you to, I had your permission to give their email address to them, and I reached out to them independently in every case and said, “Hey, someone would like”—once I was had your permission to reference you—“She would like to talk to other folks who don’t look like me who are going to re:Invent. May I introduce you?” The idea of a double opt-in introduction goes so far. And I’m talking about this for folks who aren’t me. In my case, fine. If some rando wants to introduce me to some other rando, knock yourself out. There is very little showing up in my inbox that I am not going to have some way of handling. But not everyone thinks about things that way, and it just shows a baseline level of human respect.

Rachel: Yeah, absolutely. I actually just did that this morning. I’m sure all of us get these calls a few times a year: “I’m thinking about switching to tech because the money’s there, the stability is there, the job market is there, and I have been underpaid and treated poorly for a long time,” or whatever variation on that story that I know we all are aware of. And I talked with him for a while last night, and then I put him in touch with the dual opt-in emails with someone in the field that he’s looking at, exactly, and a recruiter friend of mine to help give more perspective on the industry as a whole. And with both of those people, I asked permission to introduce them to the friend of mine who had reached out to me, and both of them responded right away because when you are fielding questions like these all day, you become familiar with the kindest way to do that.

And I really love being able to use my network in that way. Yes, I know a person at X, and yes, I would love to introduce you to Y. And I will make sure that everybody agrees and knows that this is coming, and I’m not just taken by surprise. Where I do get those emails and I understand that etiquette is something to learn, it isn’t directly common-sense sometimes. And then you sit down and you think about it, or someone says to you like, “I really need you to give me a heads up before giving my contact information to someone that I don’t know.”

Corey: It happens. It’s about being accessible. It’s about making the industry better than it is. And on that topic, I have one more area I want to delve into before we call it a show, and that is you are on the program committee for SeaGL, the Seattle GNU/Linux conference.

Rachel: That’s right.

Corey: I have fond memories of that conference, once upon a time. I gave a keynote a few years ago back when I was, you know, able to go places without it being a deadly risk, and much more involved in the community side of the world when it comes to conferences. I’ve unfortunately pulled back from a lot of it, just due to demands on my time. But great conference. Enjoyed a lot of the conversations once you, sort of, steered around the true believers around some areas of things, to the point where it subverts, you know, being civil to people. But it was a good conference. There was a lot to recommend it.

Rachel: SeaGL is a beautiful little conference. It is community-focused. We don’t let sponsors get on stage. We really restrict how much the people giving us money are able to dictate what we do. What we do is create a platform for people to discuss open-source in a human way, I would say.

I think in our earlier days, we had a lot of focus on software freedom at all costs, and that has softened in the name of humans and social justice in a way that I feel very proud of. I have been the program chair for three years now, and it’s just wonderful seeing the trends that come up every year. Our conference is Friday and Saturday, November 5th and 6th, so I hope that by the time you hear this, you will still have an opportunity to go to that; I’m not sure. Some of the themes this year have just been so interesting. It’s all about—and this will be very interesting to a particular subset of people, and maybe not to everybody—but about open-source governance, and how do we maintain the soul and the purpose of an open-source project, while keeping people housed and fed who are working on these things, and to not sign over all the rights of a given project to our corporate overlords and such.

So, there’s a number of talks that are going to be talking about that. A few years ago, the trend that I was really excited about that I personally gave a talk about as well, is how to start owning and managing your own data entirely. I gave a talk on trying to get off Google, which is Herculean and close to impossible. And I understand that, and that’s frustrating. But you know, we see these trends where we’re trying to help our community protect itself and remain open at the same time in a technical and open-source context. And it’s just an exciting and lovely organization and event each year. This is our second year being virtual. I was shocked by how good our virtual experience was last year. And I have high hopes for this year, too. So, I hope you can come check it out.

Corey: I would highly recommend it though I believe this will be airing after the show goes out.

Rachel: Ah darn.

Corey: But there’s always next year.

Rachel: That’s right. And they’re all recorded as well, all the talks will be recorded. The publication date on those might be a little bit after but yes, they will all be up.

Corey: But we will of course include links to that in the [show notes 00:37:13] because there’s always next year.

Rachel: That’s right.

Corey: I want to thank you so much for taking the time to speak with me. If people want to learn more, where can they find you?

Rachel: I think probably the best place is on Twitter. That is @wholemilk on Twitter. Like, the dairy product by the gallon that’s me.

Corey: And that link to that will go in the [show notes 00:37:33] as well. Thank you so much for taking the time to speak with me today. I really appreciate it.

Rachel: Thank you, Corey. This has been great.

Corey: It really has. Rachel Kelly, senior infrastructure engineer at Fastly. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that you should absolutely shove your business logic fully into the CDN, then wind up not being able to edit the comment because it’s locked to a single CDN.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Announcer: This has been a HumblePod production. Stay humble.

Newsletter Footer

Get the Newsletter

Reach over 30,000 discerning engineers, managers, enthusiasts who actually care about the state of Amazon’s cloud ecosystems.

"*" indicates required fields

This field is for validation purposes and should be left unchanged.
Sponsor Icon Footer

Sponsor an Episode

Get your message in front of people who care enough to keep current about the cloud phenomenon and its business impacts.