A Non-Traditional Path into the SRE Folds with Serena Tiede

Episode Summary

This week Serena Tiede, an SRE at Optum, joins Corey to talk about the world of SREs. Serena discusses their mix of traditional and non-traditional background and making the jump from electrical engineering to tech. Serena tells us about their beginnings at Optum and the different, and welcome, challenges of moving from system to system as an SRE. They talk about what Serena carrys forward from their background, starting in security and moving over to becoming an SRE and learning Docker. Corey and Serena also discuss the interminable nature of the cloud and the vast differences between when we’re footing the AWS bill—and when its the company’s problem! It turns out Serena learned some tricks on keeping costs down by tuning in to the various “Last Week in AWS” podcasts. You can do the same!

Episode Show Notes & Transcript

About Serena 
Serena Tiede is a SRE at Optum, a healthcare technology company that manages everything from the delivery of care to the management of patient data. Prior to becoming an SRE they were a Kafka operator for real time security logging and ingestion. In their off time, they moonlight as the proud admin of an incredibly over engineered Minecraft server.  


Links:

Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.


Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!


Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.


Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. A recurring theme of this show has been for a while, where does the next generation of cloud engineer come from because the path I walked of being a grumpy Unix admin isn’t really as commonly available as it once was, and honestly, I wouldn’t wish my path on anyone in good conscience. My guest today is Serena Tiede, who’s a site reliability engineer at Optim and didn’t start their career as a grumpy systems administrator. Serena, welcome to the show.


Serena: Hey, thanks for having me. I’m so pumped to be here.


Corey: Don’t worry, that will soon pass. What I’m wondering is, you didn’t come to be an SRE through a giant ops background of clawing your way up by dealing with hardware and data centers and driving at unsafe speeds in the middle of the night because someone tripped over a patch cable in the data center. You have a combination of traditional/non-traditional background. Tell me about that.


Serena: Yeah. So, it’s funny you mentioned hardware. So, I went to school for electrical engineering, went to University of Minnesota because you want to do engineering, you pretty much going to one of the big state schools in the Midwest. So, I grew up and was like, “I want to be a 
hardware designer.” I’m terrible at it. So terrible. [laugh].


Corey: Wait, I didn’t realize that you could want to be things you were bad at. If somebody told me that early on my career, it’s, “Huh. This might have taken a very different turn, and far more productive one.” I just assumed if I wasn’t good at something I should give up and never try it again.


Serena: Oh, I took the courses and was like, “Whoa, this is circuit design? Not for me.” Then I ended up just taking a bunch of engineering math courses. So, I took communications, the digital signal processing, controls, and started programming. I was like, all right, let’s do embedded systems. No one was hiring and then come internship time, there’s this little company that I’ve never heard of called Optim. And they’re like, “We want software engineers.” Well, I can write C. Does that count?


Corey: Oh, question, of course, to really ask is, “Oh, can you really write C having gone through it?” The more I talk to people who’ve been writing C for their entire career, and you ask them, “Can you write C?” The answer is, “Not really slash reliably. I can basically type and sometimes it works.” And, “Oh, thank God they’re mortal, too.” Was my response.


Serena: Oh, my opinion: no one should learn C unless there are specific reasons why. And those reasons are: you’re doing embedded systems where I had to learn how to write in assembly, for three weeks, and then my professor at the end said, “Hey, we’re writing C. Be thankful; it’s a high-level language.”


Corey: That is terrifying. But let’s get back to this idea of you going to school for electrical engineering, and you didn’t just dabble in it; you graduated with a degree in electrical engineering, didn’t you?


Serena: Oh, yes, I did. I graduated. It was fun even though, unfortunately, it still had my dead name on the diploma. So, I refer to that as my… matrix, Mr. Smith moment. [laugh].


Corey: They won’t go back and edit and reissue it under your actual name?


Serena: I haven’t bothered to look, but I almost consider it just kind of hilarious and just keeping it that way.


Corey: No. Again, I am not one to ever advise people how to deal with names. When I changed my name back in 2010 or so I wound up getting a whole lot of strange looks over it. And honestly, it is no one’s business, except how you interact with a name. Not the direction that we need to go in on this. I’m more interested in understanding, on some level, how you got a degree as an electrical engineer and then immediately landed a job writing software. That one feels a little strange. Can you talk me through it?


Serena: Oh, yeah. So, pretty much I took a bunch of operating systems classes and was like, “Wow, this computer science thing is cool.” But I was too far in the electrical engineering track to change degrees. So, I got the degree, ended up working at Optim. I originally started off in security, oddly enough, for my internship, then came back, did a—you know, we have a rotational program so I did security for six months and then… I wound up on this team for my second rotation where their literal job description, “Write RESTful APIs in streaming applications.”


Corey: So, it wasn’t even a software job that focused on the close-to-the-hardware stuff where you’re doing embedded systems. Like, that would at least make a bit more intuitive sense to the way I see the world. No, this was full-on up-the-stack REST API stuff.


Serena: Oh, yeah. I tried embedded, but in my market, it was all medical devices, and between all of us listening here, I don’t do well with medicine. Get very squeaked out, very faint. So decided, all right, let’s go up the stack, and turns out, it’s, like, okay, Kafka Streams. And then we were trying to figure out, “Okay, why our services—like, how do we know if it’s saturated?”


I’m like, “Oh, well, we have this Prometheus thing. This sounds cool.” And it was deployed on, like, you know, a rudimentary Kubernetes cluster. “Oh, hey, there’s this cool service discovery thing. Let’s do that.” And then one thing led to another. Thanos was coming out, and before it had a release candidate, I decided my claim to fame at the company was like, “All right, let’s do this Thanos thing because it seems really cool. I read about it on Reddit.” And the distinguished engineer in the room was like, “Oh, yeah, I heard about it on Hacker News. Do it.” I did; it was rough, but it was so cool. And then I come back, like, a year later because I went back to security for a wee bit, and the same monitoring stack is still there. And they were like, “Hey, can you do more monitoring things and pivot to observability?”


Corey: Yeah, let’s skip past the obvious joke that I could make about someone at a healthcare company saying, “Let’s do it because I read about it last night on Hacker News,” because it’s just too easy at some point. It’s odd, though, because I always held the belief, somewhat publicly, that an SRE role was not going to be a junior role. It was something that required quasi-significant experience to wind up moving into it, it’s always felt like a transition from traditional ops roles or folks who are deep in the weeds that have been doing software engineering at scale to a point where they see how these systems fail over time in production scenarios. It doesn’t sound like that was your path at all. Not to delegitimize your path by any stretch of the imagination. This is more to do with me reevaluating how I view SRE, as a field that people get into and how they approach it.


Serena: I just fell into it. And the reason why I bring up my digital signal processing background is a lot of the SRE stuff I look at all of our time-series metrics, and it’s like, “Oh. Well, this is just a real-time stream of data that we scrape periodically.” And it’s like, “Oh, cool. So, we can look at our averages, percentiles, I can eventually do some really cool fancy digital filtering.” And kind of was like, “Oh, wow. I, kind of, know the math behind a lot of this stuff and just have to just brute force apply it in places.”


Corey: Tell me a little bit more about that because with my approach to SRE—which let’s be clear, was fairly crap—the math that I tended to do was mostly addition and subtraction, and for the really deep stuff, I used the most common tool to manage anything at scale, Microsoft Excel, and that mostly handled even the addition and subtraction for me what math?


Serena: So, for me, a lot of it comes down to—I actually have my signals book in the other room—the big concept behind all these systems is the concept of sampling. You’re not going to, real-time, get memory and CPU data every second. Processors are running at gigahertz of speed, you would need double that to recreate your signal with full fidelity. That’s the Nyquist sampling theorem. But you kind of can fudge the numbers a little bit and just say, “Ehh, do we need that granular detail?”


We’re not trying to reproduce what happens in the past, we’re just trying to see what’s going on now. So, I say okay, 15-second scrape interval, things are looking good and then rolling into what I’m doing later of applying, like, “All right, let’s do some fun control loops,” because people wanted service-level objectives. People want service-level objectives; everyone loves them some SLOs and SLAs. No one wants to figure out, by hand, what their baseline is. But again, some fancy—this is more controls math—figure out what your baseline is just automatically and do some little magic in the frequency domain, courtesy of Laplace transforms, and that’s it. I can just automate that for you and remove the human from the equation.


Corey: I’m still somewhat astounded by the fact that people calculate these things out mathematically instead of, you know, dead reckoning and confident-sounding estimation.


Serena: It’s really just bringing that electr—like, controls background to software. Honestly, I’m kind of baffled that no one else is found this hack because I’m just thinking, “Oh, well, I can’t be that unique. Someone else has to have done that.” And then I talk to the people in the room and it’s like, “Oh, wait, no, I am the only person here.” [laugh].


So, that’s my whole thing. Everything is just applied math. And all of our human dead reckoning, it’s great, but it doesn’t scale well. You know, my boss wanted me to figure out how to do our SLOs for the entire team, and turns out realist—and when it came time to hire, realistically, cloning myself was not an option. [laugh]. So—


Corey: For better or worse, it seems like it isn’t. So, what was your first exposure to the SRE-style space? You started off in security, but looking at the timelines on this, it wasn’t that long ago. It feels like you were probably not exposed, in many cases, to physical data centers as much as you would be cloud, or at least not having to image bare-metal systems. Were you up at the AMI level, or was it beyond that in having virtual machines that moved around into full-on containers, or serverless?


Serena: So, I started my internship in 2016, and got my full-time offer in 2017. And we started having our—container platforms started becoming this up-and-coming thing. You know, my lead engineers were like, “All right, you've got to learn this thing called ‘Docker.’” And I have never heard of it, but I was just amazed that, “Wow, I can just run these little, little itty bitty pods anywhere on this hardware.” And later on, I did do some, like, virtual machine stuff, but I’ve had the luxury of all of these years of pain and toil, to be able to say, “Oh, yeah. I can just manage things with Ansible, create my Docker files, and do everything from a code deploy pipeline style. And it was awesome.” And I just can’t fathom what it’s like to work without those tools, but knowing… the past, it’s kind of like, “Wow, we have gotten a lot farther. Things are abstracted. This is actually kind of nice.”


Corey: It kind of is, on some level. I feel like my initial reticence towards containers—I gave a talk: “Heresy in the Church of Docker,” which sort of put me on the speaker map once upon a time—and it was about all the things that Docker as a technology didn’t really have good answers for. Honestly, the reason that I gave the talk was I assumed that it did have answers and I was just unaware of them, and I just gave the talk so I could publicly become the idiot who didn’t know what they were talking about and then get “well actually’d” to death by [ducks 00:12:40] slash Googlers. And it turns out that no, no, at that point in time, these things were not well understood or solved for. The observability stories, the metrics, the logging, the orchestration, the security story, the how you handle things like state, et cetera, et cetera, et cetera.


And Kubernetes these days has largely solved a lot of those problems, but I don’t dabble in those spaces just because of outright ornery. Back then it was a weird problem, but these problems have largely gotten solved in some ways. But I sort of just skipped over the whole Kubernetes slash container renaissance, and personally, I went directly into the serverless world. What’s your take on that?


Serena: Oh, so as someone who loved Kubernetes, I was a serverless skeptic, initially. I was like, “Well, I can just build my Docker file and write the deployment manifest. No big deal.” And then I started working on my side project. For, I think, better purposes, my iCloud account is tied to my credit card and I have to actually be on the hook for cloud bills. And I use GCP for my home lab and lo and behold, 1 million requests a month for free. And I love the sound of free when it’s my money on the line.


Corey: Oh, yeah, company money versus enterprise money, radically different scales. I mean, if you try and sell me personally a $50 hamburger, I’m going to tell you to go to hell. If you try to sell me, as representative of my company, a $50 hamburger, I’m going to need a receipt.


Serena: Exactly. And then also, like I’m just running through, I was redoing one of my serverless functions and watching the deploy steps. And then one of my coworkers introduced me, he’s like, “Hey, Serena, you hear this thing called ‘Buildpack?’” and I’m like, “No. What on earth is that thing?” And he’s like, “Oh, well, you take your code, and then it just magically turns into a container.” I’m like, “Well, crap. Show me.” And lo and behold, code goes in one end, nice little container comes out the other. And that crap was magic.


Corey: It really does change the world if you let it. I think. I know it sounds like a ridiculous, I guess, hype-driven thing to say, but for the right use case, it’s great because it removes the entire administrative burden from running services. Now, critics are going to say that well, that means you’re just putting all of your reliability in the hands of your cloud provider. Yeah, we’re kind of all doing that already; serverless just, sort of, forces us to be a little bit more honest with ourselves about that.


Serena: Oh, yeah. I mean, even if you self-host things, you’re relying on your data center ops people to, like, make sure, oh, I don’t know, your machines don’t literally catch fire. We literally had a bug one time where it’s like, “Why is this one node bad?” “Oh, actually—hey, did you increase the fan speed?” Someone had to literally go increase the fan speed for whatever servers, which, again, in the serverless and cloud provider world, I don’t think about that. The cloud is just infinite to me. It’s just computers and APIs as far as the eye can see. It’s wonderful.


Corey: It really is. It’s amazing, and it’s high level, and on some level, you went from getting a degree that required you to write assembly and super low-level stuff and figure it out hardware works into, let’s be honest, writing in your primary language, which for all of us in SRE-land is, of course, YAML.


Serena: Oh, I am a very spicy YAML engineer. YAML and a little bit of Go for what I need to make things go.


Corey: You ever notice there’s never a language called ‘Stay,’ or ‘Stop,’ or anything like that? It’s always about moving to the next thing. And we in engineering always have sprint after sprint after sprint. Never a, “It’s a marathon, not a sprint. Relax. Walk. Enjoy the journey.” Nope, nope, nope. Faster, further, sooner.


Serena: Yeah, it is honestly weird because my relatively short career span, you know, it’s 2021 and I graduated in 2017. The company is like, “Hey, you’re a senior software engineer now.” Here’s a program, here’s a budget. Go forth.


Corey: Oh, that’s lucky. It must have been amazing to have an actual budget. When I started out, I was in one of those shops where it’s, “Yeah, Palo Alto wants $4,000 for that appliance. That’s okay. We have some crappy instances and pfSense, and you know, we could wind up spending eight weeks of your time to build something not as good. Get on it.”


Serena: While the hilarious part is I’m stressing out about every single dollar I’m spending and then my boss is like, “Oh, you know, your budget is super small potatoes, right, compared to like our other stuff? Don’t sweat it. It’s fine.”


Corey: I keep making this point to the cloud providers where they’re somewhat parsimonious free tiers are damaging longer-term adoption because I look at building something myself, in my spare time in my dorm room or whatnot, and I’m spinning up some instances that talk to each other and I want to use a load balancer and I want to use a managed NAT gateway—God forbid—and at the end of the month, I get a bill for $300. And it’s, what the hell is this? I thought I was on the free tier and it scares the living hell out of us. So, we learn not to use those services that are higher level and differentiated. And then when we start working in environments that have budgeting and are corporate, we still remember that, and, “Oh, don’t use that thing. It’s expensive.” And you’ll inadvertently spend 80 times as much in what your employer is paying for your time, rather than using the high-level thing because they could not care less about a $500 a month charge. And it’s this weird thing that really serves as a drag on adoption.


Serena: It’s super wei—I actually literally had this conversation with one of my engineers who wanted to, “Hey, we’re trying to expose a GRPC thing.” And I had issues getting it to work with an ingress. And he’s like, “Do you want me to take a crack at that?” And I’m like, “Look at the price of the load balancer.” And I’m like, “Unless you can figure it out in half an hour… it is literally more expensive for you to continue tilting at that windmill than for us to just leave it be.” [laugh]. And it’s also weird. I have my personal stuff where I’m trying to keep my cloud bill to, you know, maybe a humble $100 a month max, versus, “Oh, the enterprise? Oh, yeah. That’s just logging that you’re paying for.” Which is baffling to me.


Corey: I feel like as engineers, we always, always, always fall into this trap. And maybe I fall into it worse than others because my entire business is actually lowering the bill. But when I started as an independent consultant, my bill was something like seven bucks a month, which yeah, I’m pretty content with that. And I started looking at ways to golf it lower, which in most cases is never worth the time, but in my case, I should really understand every penny of the AWS bill or I’m going to have a problem someday. And now I look at it recently because we have a number of engineers building things here, and our bill was over $2,000 a month.


And true story, by the way, it turns out that your AWS bill is not so much a function of how many customers you have; it’s how many engineers you have. And I look at this and, “Oh, my God, we need to fix that immediately.” And I spent a little bit of time on it and knocked 500 bucks off, and, “Whew, that’s better.” And it still bugs me to see a $1500 bill; it feels like it’s an awful lot of money. I mean, think of what you can buy for 1500 bucks a month.


And then in the context of the larger business picture, compared to payroll, compared to all the other nonsense we use, like Tableau, for example, it’s nothing. It is a rounding error that gets lost in the weeds. I never understood that before having access to company budgets. When I was an employee, this was never explained to me, so I was always optimizing for absolutely the wrong thing in hindsight. It feels like this is part of the problem that we run into as a culture when we don’t give our staff context to make the right decisions.


Serena: Yeah, I actually do appreciate the way my company does things because I am, like—not personally, my bank account, but I am, like, responsible if someone should ask, “Hey, what’s this charge for?” I have to say, “Oh, well, it’s for all of these things, and we need that.” But for the most part, it’s been really weird to, kind of, learn, like, one of the ways I, kind of, sped up my, like, “Okay, I need to learn how business works. What do I do?” Well, quite honestly, a lot of my cloud cost tips I have learned from your various podcasts. [laugh].


Corey: Uh-oh, that’s a problem.


Serena: No, but like, all of a sudden, all this stuff and just hanging out on tech Twitter and hearing all the advice of people and then… it was, kind of a weird way of, like, yeah, years-wise, yeah, some people might look me askance and be, like, “You’re really a senior engineer?” But then they hear me speak and it’s all about like, “Oh, well, I”—again—“I stand on the shoulders of giants,” which is awesome, and I’m honestly just hoping that one day I will write something that is very cool and then someone will say, “Oh, well, they were right on these things, but not right on this. Let’s edit this to make it a little bit better.” And the standing on the shoulders of giants trend continues.


Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.io


Corey: I’m a little taken aback by the fact that you’ve learned a lot of this stuff from the podcast because I tend to envision when I’m telling stories about this, companies that show ads, or my mythical Twitter for Pets startup. I have to remember that there are banks, like, is one of the examples of serious businesses that I use all the time. But you’re in healthcare. I’m sorry, that’s more serious than finance, just because—I hate to say this because it sounds incredibly privileged and I don’t even care—it’s only money. What is money compared to the worth of someone’s life?


I don’t think that you can ever draw an equivalent and I feel dirty every time I try. When you’re working with things that impact people’s ability to access healthcare, that is more important than showing banner ads. And a lot of the stories I tell about, “Maybe it’s okay to have downtime.” Because yeah, if AWS takes a region down issue for an afternoon and you can’t show ads to people or your website isn’t working, yeah, that’s kind of sad and it’s obviously not great for your business, but at the same time, the stories in the news are always about Amazon’s issue, not about your specific issue. If you’re in an environment where there’s a possibility that people will die if what you have built is not available, we’re having a radically different conversation.


Serena: Exactly. Fortunately, for me, I personally, not working in the, like, kind of, care delivery space, but the stuff I’m working on right now is supporting, you know, that lovely end-of-the-year where it’s open enrollment, all the employers are saying, “Hey, time to re-up your benefits.” Yeah, it’s kind of a big deal that our site doesn’t go down. Because—


Corey: Yeah. And open enrollment, to my understanding, changes based upon what plan you’re on. I’ve known companies that have open enrollment in the summertime. I believe ours winds up coinciding pretty closely with the calendar year, but I’ve certainly worked in environments where that wasn’t true. So, being able to say, “Oh, it’s fine. It’s April; no one’s doing open enrollment now.” Is it actually true?


Serena: So, it totally depends on which part of your business. If you’re going through the healthcare exchanges, that’s usually more in the fall. I think the Medicare plans, those are a little bit before the individual enrollments. And there’s a ton of these things that even though I just work tangentially, that I’m just not even in the know for. And then, of course, we talk about open enrollment, but the thing that a lot of people don’t really talk about is, so what happens when your plan goes live on January first of the next year? Yep. Our site’s still got to be up. And it’s a responsibility I take really seriously because it impacts so many people.


Corey: It really does. And it shouldn’t, to be clear. I try to avoid getting overly political on this podcast, but the state of healthcare in the United States as of the time of this recording is barbaric. And I really, really, really hope there comes a day where someone’s listening to this and laughing because it’s such an antiquated and outmoded story that isn’t true anymore. But I’m terrified that it won’t be.


And yeah, having access to a website lets you sign up for healthcare during a limited period of availability, if you miss that window, you don’t have healthcare, in many cases, until the following year when open enrollment opens again, or honestly, you wind up changing jobs because that is a qualifying event to change healthcare. “Well, I missed the open enrollment window, so I have to quit and take a job somewhere else,” is a terrifying thing. It’s bad for the business for a variety of reasons, but that pales in comparison to the fact that people have to make life-altering career decisions based upon a benefit that is routed through an employer when it should not be. Okay, I’ll climb off my soapbox.


Serena: Oh, it’s bizarre to me. Honestly, for better or worse—I argue worse—but I’m honestly optimistic. One of the weirdest things I saw that stuck out from the most recent stimulus bill was, “Oh, hey. We’re having a special enrollment period during a pandemic.” And I’m like, “You know, it’s not a hundred percent.


Maybe we should just extend it to the whole year.” But it’s better than what was the previous state, where it’s like I can’t make—I mean, even in my work life, I can’t make everything perfect. I can’t make outages go away, but I can make things just a touch better. And that’s all I can do.


Corey: Sometimes all we can do, and I wish there were better ways to handle that. I don’t know what the future is going to hold, but I also think that there are bright areas. There are aspects that are promising as far as the future being brighter than today. The overall trend—I hope—is for humanity to uplift itself.


Serena: Totally.


Corey: Again, I do want to highlight that you went in a very strange direction where you went from software engineering—a generally pleasant job—to SRE, which is horrible and would not be recommended to anyone. What guidance would you have for people who are, for some godforsaken reason, trying to figure out what their career trajectory is going to be like, and thinking that they might want to become an SRE—even if they’re not in tech yet—because for some reason they hear the stories and think there’s some nobility in suffering or whatnot?


Serena: Well, for starters, for me, it kind of came down to get real good with this great math. It’s boring, but that’s kind of the bread and butter of the concepts I’ve learned. Also for junior people, if you’re also just curious—say you’ve written an app, go over to OpenTelemetry. Go, like, instrument your stuff and see how many requests you get in a day. Start getting your hands dirty with instrumentation.


Look at how cool it is, and then maybe you want to start structuring your logs; maybe you start end up doing tracing. But at the end of the day, it’s all, for me, I think best learning is just experiential, and you know, one of the things where how do you learn from production outages? Go to happy hour with some of the senior people and listen to the stories that they tell. With enough time they become funny, but they’re also valuable learning things.


Corey: The aspect I would push back on is the hard requirement around discrete math. I don’t deny that it has been helpful for what you’ve done and how you do it. I don’t know how any of that stuff works on paper; I have an eighth-grade education. That was never my path and never my strong suit. I would agree that knowing it would have made aspects of what I do easier, but the bulk of it I don’t necessarily know that I would agree. I guess, my counterpoint slash pushback would be that if you thought you’d like this, but you don’t want to deal with the math, it’s not a hard requirement, and I don’t think that I would frame it as one.


Serena: Actually, that is a very good catch. It is not a hard requirement. I am not sitting here in my notebook, scribbling away at equations. But the concepts that I’ve learned from a while back, it’s the concepts are way more important than the actual computation itself. Because 
computers do that, and a computer will absolutely run circles around me.


Corey: Most of us do, unless, you know, the computer is an overheating processor from Intel. But that’s a little bit of a low blow. Not that it 
stopped me. But it was a low blow.


Serena: Well, I mean, your local science supply shop might have some liquid nitrogen. Maybe.


Corey: So, what’s next for you? You started off in security slash software engineering, transitioned on over to SRE work. What’s the next step? 
What’s the beyond for you?


Serena: Ohh, great question. So, I don’t really know. I’m enjoying the SRE thing. At some point, might write a book trying to make all the concepts I have learned from my electrical engineering degree, maybe a bit more accessible, be it a series of blog posts, maybe a book. I would love to get a book published. And honestly, just writing more because knowledge should be shared, and if someone learns something from my nonsense experiments on my home lab, then cool; it’s all worth it.


Corey: I’d agree with that. I’m a big fan of learning in public. One of the, I guess, magical things that I do, for lack of a better term, is that I will stumble my way through learning a new concept that I have no idea what I’m doing, and when I get lost, I call it out because invariably, I’m not the only person who runs into that problem. But for folks who don’t have—I don’t know if it’s the platform, the seniority, the perceived gravitas, the very intentional misdirection where I fooled the entire world into thinking I know what the hell I’m doing, whatever that is, most people have a problem with admitting they don’t know something and learning in public, so anytime I can take up that mantle or that burden, I love doing it, just because I don’t have any technical credibility to lose from my point of view. I wish that were more accepted and more common. That’s why I’m so intentional about being able to talk, on some level, about the things I don’t understand or the things that I don’t get.


Serena: I love that. I used to read a bunch of philosophy books, way back when, and my big thing, this great quote—I always get it confused, Plato or Socrates, but it’s, “I know that I know nothing,” and I just run with that because I mean, even though fortunately, for me, my corner of the internet, as a non-binary person, no one’s really mean to me when I say, “Okay, I broke my DNS,” because, honestly, I knew DNS conceptually when I was setting up my Minecraft server for friends, but I never really got it until I, well, kind of, broke it, [laugh] and eventually fixed it. But I hope that over time, it becomes more acceptable to say, “I don’t know things.” Within my team, I tell anyone that’s working with me when they’re asking me a question, say, “I don’t know, but I have a feeling this rabbit hole, this trail of crumbs might lead us to an answer.” And then it’s a fun little adventure.


Corey: I miss the days when I could describe what I do is a fun little adventure. It’s now, “Oh, dear Lord, it’s this bullshit again.” [sigh]. That was my sign that I was burned out, in time, find other things to do than keeping sites up.


Now, I have no on-call responsibilities because there’s no real site to keep up. Thank you, serverless, I get to sleep at night again. But there are times I miss aspects of working in the trenches, of being able to dive deep into a problem on a very large scale architecture. The grass is always greener, somehow.


Serena: The grass is always greener. In a weird way, I actually, I complain about my on-call weeks, but I actually kind of love them. There’s a weird camaraderie about all of us dealing with a shared thing. And on my team, it’s really cool because we do this whole thing where, you know, I have these junior people asking, “Oh, am I going to go on call?” And we’re like, “Well, unfortunately, you’re not quite fully baked yet. Not quite ready. Once you’re here longer with us, then yeah, we’ll go walk you through a game day and make sure you can do all the things. But being on-call, it should not be a punishment for people.” Honestly, it’s just the greatest feedback mechanism that guides me because I say, “Wow, this stinks. This could be better.” And then try to make it better.


Corey: If people want to learn more about what you’re up to, how you think about these things, or potentially even reach out for advice, where can they find you?


Serena: So, I am on Twitter at @Serena—S-E-R-E-N-A—Tiede—T-I-E-D-E. DMs are open; come bug me. I got my lovely blog. It’s just blog.serenacodes.com. It’s pretty bare-bones, but I’ll have some new content up there hopefully pretty soon, once I get around to writing it. And say hi. I like meeting new people and learning new things. Adventures await.


Corey: And we will, of course, put a link to that in the [show notes 00:34:30]. Thank you so much for taking the time to speak with me. I really appreciate it, Serena.


Serena: Hey, thank you. I am so happy to be here. This was one of my life goals, and now I don’t know what to do now that I’ve gone up here.


Corey: That’s the problem with achieving these bucket list items. It’s, “Oh, well, I wake up the following day. Now, what do I do?” And when life 
eventually returns to normal, on some level. [laugh]. Thanks so much for your time. I really appreciate it.


Serena: Thank you. Have a great day.


Corey: Serena Tiede, site reliability engineer at Optim. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment saying that if you think that C is a high-level language, oh, just wait until you explore the beauty and majesty of Rust.


Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.


Announcer: This has been a HumblePod production. Stay humble.
Newsletter Footer

Get the Newsletter

Reach over 30,000 discerning engineers, managers, enthusiasts who actually care about the state of Amazon’s cloud ecosystems.

"*" indicates required fields

This field is for validation purposes and should be left unchanged.
Sponsor Icon Footer

Sponsor an Episode

Get your message in front of people who care enough to keep current about the cloud phenomenon and its business impacts.