Talking Shop with a Unix Historian with Tabitha Sable

Episode Summary

Tabitha Sable is a systems security engineer at Datadog who moonlights as a Unix historian. Join Corey and Tabitha as they discuss what it was like to join Datadog right when the pandemic shut everything down, how Tabitha got experience with Unix workstations, what was going on in Murray Hill, New Jersey, in the early 1970s, whether operating systems matter any more or not, what happens when you type www.google.com into your browser and press enter, why job interviews are awful, how Tabitha conducts interviews, the power of referring people for jobs, why you should hire for strengths instead of absence of weakness, and more.

Episode Show Notes & Transcript


About the Guest

Tabitha Sable has been a hacker and sysadmin since the turn of the century. She serves Kubernetes as co-chair of SIG Security and an associate member of the Product Security Committee. She loves to build tools and make friends, and puts those skills to work coordinating the efforts of the infrastructure, security, and product teams at Datadog. Outside of work, she can often be found organizing or participating in Capture the Flag contests and loves "pretty much anything with wheels."

Links Referenced: 

Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.


Corey: This episode is sponsored by a personal favorite: Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front end frameworks to figure out or access controls to manage, just ship the tools that will move your business forward fast. Okay, let’s talk about what this really is. It’s Visual Basic for interfaces. Say I needed a tool to, I don’t know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas: buttons, checkboxes, tables, et cetera. Then I can wire all of those things up to queries with all kinds of different parameters: post, get, put, delete, et cetera. It all connects to virtually every database natively, or you can do what I did, and build a whole crap ton of Lambda functions, shove them behind some APIs gateway and use that instead. It speaks MySQL, Postgres, Dynamo—not Route 53 in a notable oversight, but nothing’s perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces. And I don’t know front end. That’s the most important part here: Retool is transformational for those of us who aren’t front end types. It unlocks a capability I didn’t have until I found this product. I honestly haven’t been this enthusiastic about a tool for a long time. Sure they’re sponsoring this, but I’m also a customer, and a super happy one at that. Learn more and try it for free at retool.com/lastweekinaws. That’s retool.com/lastweekinaws, and tell them Corey sent you because they are about to be hearing way more from me.


Corey: When you think about feature flags—and you should—you should also be thinking of LaunchDarkly. LaunchDarkly is a feature management platform that lets all your teams safely deliver and control software through feature flags. By separating code deployments from feature releases at massive scale—and small scale, too—LaunchDarkly enables you to innovate faster, increase developer happiness—which is more important than you’d think—and drive transformation throughout your organization. LaunchDarkly enables teams to modernize faster. Awesome companies have used them, large, small, and everything in between. Take a look at launchdarkly.com, and tell them that I sent you. My thanks again for their sponsorship of this episode.


Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Tabitha Sable who, among many other things, is a system security engineer at Datadog. Tabitha, thank you for joining me on the show today.


Tabitha: Thank you very much, Corey. I'm delighted to be here.


Corey: So, you're relatively new at Datadog, which for those who are relatively new to the sector is apparently a monitoring company, not as it turns out when you mispronounce it, Tinder for Pets. I learned that the fun, difficult way when I got yelled at on Twitter for that a year or so ago.


Tabitha: I'm sure that was hilarious.


Corey: I said, “Who wants to date a dog?” Well, ideally, compared to the trash fire that is some people, lots of folks. But I digress. So, you've been there since April, sort of the very beginning of the pandemic era, for lack of a better term. What was that like joining a company where, “You’re here just in time to never meet your coworkers in person.”


Tabitha: It's been really wild. I work remotely from the Midwest. The two biggest Datadog offices are in New York and Paris, and the teams that I'm associated with are, for the most part, split pretty evenly between New York and Paris. But also within each country, a fair number of people are remote, and I'm one of those remote people. So, I had been planning all along to be working remotely, but not in this way. 


So, it's interesting that I've been in the Datadog office, but never as an employee. For people like me, there's a few unexpected advantages. Like, for example, when you're meeting someone over Zoom, it's much more difficult to not be able to remember their name at a time of high pressure where you would cause yourself embarrassment because you can just look at the little words that are underneath their picture.


Corey: Well, I've seen people who are still capable of messing that up perfectly effectively. Messing up Zoom etiquette is always a terrific way to do things. Especially if you just kill video, and then we can go back to the old call center days of putting inordinate amounts of faith into the effectiveness and accuracy of the mute button.


Tabitha: [laugh]. Oh my goodness, yeah.


Corey: It’s like, “Well, you've worked with electronics. How far do you trust them?” “Well, not super far. Certainly not bet my entire career on it.” Yeah, there's a reason that more people should exercise caution with those things.


Tabitha: You know, I’ve been accused of being wholesome, and honestly, I think I'm just a little too simple to get into those kinds of situations. I know that I would absolutely be the worst at remembering whether the mute button is on or not. And so I just try to keep it on the up and up, and for the most part that saves me from myself.


Corey: You want to talk about the most destructive virus in the world: something that has Zoom tell you you're muted when you're not. I feel like that would be one of those old-school prank viruses that—


Tabitha: Oh my gosh.


Corey: —you can talk about things that would transform society? That's right there with it.


Tabitha: Yeah, yeah. You would learn so many things that a lot of people would be unhappy with, but in a lot of cases, when somebody shows you their ass, and then that you need to treat that person with care. Like, that hurts in the short term, but it saves you problems in the long term.


Corey: Yeah, it's painful to discover that people that you look up to and admire are trash goblins. But I prefer that to not knowing, and unknowingly introducing them as someone, “Oh, they're great. You should talk to them.” People remember when you introduce them to horrifying nightmare people.


Tabitha: And bad news is unfortunate to get, but it generally beats not knowing. Yeah. A hundred percent.


Corey: Yeah. So, I know that we're talking about things in the past rather than forward-looking. And I see no reason whatsoever to change that. I don't believe this is a core function of your day job, or at least I hope it's not but you moonlight apparently, as a Unix historian.


Tabitha: [laugh]. Yeah, it's fun. Unsurprisingly, there's history there. So, when I was just a tiny little person—and by that, I mean a teenager—I had the experience of being involved in the computing club at school. And this was around the turn of the century. 


And so we had what were at that time a trash heap of outdated Unix workstations. We had an R 3000 indigo, like Iris Indigo, like the adorable purple dorm fridge, little SGI box; there was a Sun-3 machine with a VT100, plugged into a 25-pin serial port on the back of it. And there's a funny story there, which I'll share with you later if you're interested in. But those sorts of things; we had these Unix workstations that were 15 years old at the time. And so a lot of my early exposure to Unix and to system administration, was from, kind of, herding these wheezy old pieces of crap into providing some useful services for students. The first server that I ever ran myself was a 386 that I put OpenBSD on so that people can play Rogue.


Corey: Oh, that is a blast from the past. I mean, I started out in the university setting as well; my first professional job in the space was 2006. So, I was a bit after this to some extent, but I was a grumpy Unix admin because, let's be honest, have you ever met a different kind of Unix admin? I haven't. And BSD was the way that we went—full stop—for a year because I had opinions, and if no one really has any better idea what direction to go in, it's not the worst direction you could pick.


Tabitha: Yeah. So, yeah, we had all of these things. And that was where I got a lot of my really early experience with the concepts of system administration, with providing a service that people would actually use, and that would help them achieve some goal or help them have a good time. And so I got interested in those sorts of machines and in the software from that era. And then that intersects really well with this sort of general thing about the way that I understand the world, where I don't really feel comfortable with my understanding of something as it is unless I can connect it to a little bit of history, a little bit of how did it get to be this way. 


And so sometimes I'll ask, “What does that word mean?” But I don't mean, “What is the definition of that word in the dictionary?” What I really mean is, “What is the etymology of that word?” So, this kind of point of view sort of permeates my experience of life, and that in common with my early history with Unix, has made me prone to researching what was going on in Murray Hill, New Jersey in the early ’70s.


Corey: Tell me more. So, in the ’80s, there was something worse going on in New Jersey, namely, I lived there. And my favorite part of New Jersey is not living there anymore. But the ’70s are before my time; tell me more.


Tabitha: So, this story starts a little bit earlier than that, in the ’60s. So, computers were expensive, but they were starting to get to be really good. And so—


Corey: Well, Apple is working on reversing that trend, too. They're trying to make computers more expensive, bless their hearts.


Tabitha: So, yeah. In the ’60s, there was a group project, sort of the group project made in heaven I guess, between MIT, Bell Labs, and Honeywell—if I remember correctly—to produce a computer that could be used as a service, like the computer utility. And so they were making Multics, which is this huge operating system that was very highly specified, and some people still believe was the best system that never finished getting built. And so that project was kind of slowly grinding to a halt under the weight of its own design specifications. For example, one of the design specifications of Multics was it had to be written in PL/I, but it turned out that at the time, there was no PL/I compiler. 


And so then they had to define the implementable subset of PL/I and then write the operating system in that because even the language was too big to compile at the time. But the project was kind of grinding to a halt, and eventually, Bell Labs pulled out of it. And some of the folks who were working on that were disappointed because they had a huge Honeywell mainframe all to themselves, and it was running great with three or four users, and it was a really collaborative experience that was unlike what you got on other systems at that time. So, they wanted to start doing that same kind of computing, but also management at Bell Labs was very serious, they were not going to get into another operating system project.


Corey: So, many good things came out of Bell Labs and its history could have been so different. 


Tabitha: Right?


Corey: We take a look at the modern technologies we take for granted, a majority, it feels like, and trace their lineage back to Bell.


Tabitha: Oh, my goodness, yes. Learning about what it was like to be at Bell Labs in that time is so interesting because it seems like very much a product of its time, but also one of the best products of its time. So, yeah, a bunch of the people that had worked on Multics wished that they had a computer. And they couldn't buy one, so they looked around, and they found a PDP-7 that was, like, sitting in a corner of a lab room because it had been bought for a certain project and the project was finished. And it wasn't big enough to run anything on. So, they were like, “Sweet, can we just use this?” And the answer was, “Yes.” And so that was how Ken Thompson, and Dennis Ritchie, and friends started writing Unix.


Corey: It's amazing just to think back to those days when getting even time on a system was this incredibly valuable thing. How do you get access to this? I mean, the reason that I went in a software direction instead of hardware, personally was, let's not kid ourselves; people who have seen me on Twitter have a pretty good idea of exactly how accident-prone and clumsy I am in social interactions, let alone in the physical world. It maps. When you screw up hardware, that's super expensive, especially when you're me as a kid and have basically no money to use on these things. 


And whereas with software, “Well, I can just make a backup copy.” And you always remember to make a backup right after you really could have benefited from having made a backup. But you can unwind software and start over in different ways, whereas with the physical world, generally speaking—though there are exceptions—computers don't ever work the same way after you let the magic smoke out the back.


Tabitha: Yeah, yeah. There's the whole range of horrible things that you can do to your hardware, some of which are recoverable and some of which are not. And where that border is depends very much on what skill level you have, how steady your hands are, and what kind of tools you have in your house. 


Corey: So, that does, of course, lead to the question, as someone from the perspective of a Unix historian, here in this year of our lord 2020—as of the time of this recording, but it’ll be 2021 by the time it hits the world—is the operating system relevant anymore? Does it matter, other than to a few folks who are going super deep into the stack for tuning or performance reasons?


Tabitha: I really love that question. That's a hard one. I feel like the answer is yes and no. It kind of doesn't, to the extent that you don't use it. And so a lot of the things that made Unix feel comfortable—for a certain sort of person anyway—are getting to the point where they don't matter anymore because they’re UX concerns, and the UX is replaced. 


So, if you're talking about deploying containers onto Kubernetes, or on to some kind of cloud-hosted container service, there's Unix in there, but you don't ever touch it. You write the Docker file and you build the thing, and the rest of it just goes. And so to that extent, it kind of doesn't. But there's a different way of looking at it, I think, which is, what does the developer experience look like? And that gets more and more relevant as the user experience gets less relevant because of the fact that you're shifting your relationship to the operating system. 


The operating system never goes away, but you're not banging at a keyboard typing in commands anymore. And so the sorts of concerns—like, how memorable are the command words? How lengthy are they? Are they annoying to type—become much less of a thing, but is the API of high quality? Is it easy to write to the API? Does the API help you to not write bugs? All of those sorts of things become more and more important because that becomes your dominant way of interacting with the operating system. And so, yes and no.


Corey: It feels on some level like it's akin to the network where, in a modern cloud era, you don't have to think about the network because it's all abstracted away. And I've checked with AWS extensively, they will not let you go hands-on hardware with their switching equipment because they don't believe in having fun. But I found that when I got deeper into learning networking—during the Great Recession when suddenly there's a salary freeze, so I want to leave my job, but no one's hiring, so I can't leave my job. What am I going to do to pass the time? I'll get my CCNA. 


And I learned a lot about how networks work. And not that I wanted to become a network engineer, but suddenly, I understood a lot more about how the systems that wrote on top of the networks were built and why they did the things that they did, instead of hand waving my way past subnet masking, for example. I understood what the hell it meant. It wasn't the magic spell that you would just repeat because it's what you heard the elder say; now it's something you understand. And that opened up a lot of doors. 


Tabitha: Absolutely.


Corey: I can't shake the feeling that understanding operating system fundamentals is heading in that same direction. Sure, most of the time, I'm just writing a Lambda function; I just want it to invoke the code, and I want it to do the thing until suddenly I see an emergent behavior that I don't fully understand. What's it doing under the hood? I don’t know. I used to work in shops where there was no one to really escalate to beyond me, which is, A) a giant problem for those companies, but, B) it teaches a certain level of self-reliance where, how do I start finding out what this looks like? 


Why do I build a reproducible skeleton case for a mailing list? How do I frame a question? I mean, far and away, the best way I ever found to solve a problem myself was to start writing an email, “When I wind up doing this, just like it says in the documentation, except the documentation has a comma there, and I don’t—oh.” And then we close that draft and never speak of it again. There's something to be said by framing the question in such a way—like, rubber duck style—that opens up a bunch of understanding. But figuring out what the underlying tools and services are that power this, seems like it's a critical step. It's not super appreciated these days.


Tabitha: I think that this is a really interesting tension that we're having to deal with right now. I feel like I would get voted off the internet if I didn't call out the paper “The Ironies of Automation” right now. The author escapes me right now, but it's easy to search for on the internet. And it's great. And I feel like that applies here, too, where the whole advantage of making these abstractions is that it lets the people who are riding on the upper layers be able to achieve their goals without having to actually understand everything that's going on in the lower layers. 


But there has never been an abstraction that didn't leak. There's never been an abstraction that was actually perfect. And so there are always emergent behaviors that come out because of the way that the abstraction is implemented. And the real substrate that's underneath of it affects the layers above in ways that the abstraction says that it's not supposed to. And I think it's beautiful, and also frustrating. 


So, that balance there of, I'm so glad that we have all these layers of abstraction because it demonstrably lets people get so much more done, but it also makes room in the world for people like me, who just cannot stop themselves from opening something up to see what's inside of it. Especially if you tell me that I don't need to worry about what's inside of it, or I don't need to think about how that works. Nothing makes me want to dig into it more than that.


Corey: If people are asking for, “Well, I have a couple of weeks off. What should I be doing with that time in my spare time? Because I enjoy this stuff as a hobby, where do I go next?” As if the people have a curriculum here. But my answer is yeah.


Tabitha: We're actually super, super lucky. We kind of do have a curriculum, if your flavor matches my flavor anyway, I cannot recommend anything more highly than the International Journal of Proof of Concept, or Get Out The [BLEEP] Out


Corey: Yes.


Tabitha: —which is a hackerzine. It's been running for several years now, and they do short, conversational articles on an astounding variety of really deep, delightful, and fun, reverse engineering projects.


Corey: And we're throwing a link to that puppy in the [00:21:26 show notes].


Tabitha: Yeah, yeah. One of my favorite past articles there was achieving “Reliable Code Execution on a Tamagotchi”. That's the kind of stuff that you get in POC or GTFO.


Corey: This is amazing. I like that better than my curriculum which was, always look at the thing in your stack of whatever it is you're running, and then magic happens and then that kicks off this other piece. It winds up on covering areas in which you're generally weak. One of my favorite interview questions just from a showing how candidates think perspective, but terrible from a pass or fail perspective is, “I type www.google.com into a browser and I press enter. Assuming that they have not yet deprecated google.com, what happens next to make that work?” 


And you can go anywhere with that question: the network, the browser, the DOM discussion, the debouncers in the keyboard, electricity. You can talk about TCP handshakes, you can talk about Google's deprecation policy; it's hard to get me not to. And there's so many different ways to take that, that you can spend an entire lifetime on that. There's a GitHub repository—or GifHub repository—where someone documented everything that happens to make that work. And it's a collaborative effort, and it is enormous at this point. I’ll have to see if I can dig that out and put that in the [00:22:43 show notes] as well.


Tabitha: Yeah, that's one of those kinds of questions that you can use for good or evil. And as someone who would like to use it for good, it breaks my heart that it is so frequently used for evil because I would absolutely never ask someone that question in an interview, not because I am afraid of what I would do to them during that conversation, but because I am afraid of how it would make them feel and what the baggage a question like that will pull in from all of their previous bad experiences being abused by interviewers who just wanted to show off that they were smarter than somebody.


Corey: Yeah, I've just pulled it up, and I'll throw it into the [00:23:28 show notes]. Yeah, it has 291 commits, 69 contributors, 48 pending pull requests, 81 issues. Yeah, it's incredibly deep. This is why I don't like this in the form of a pass or fail question. Job interviews are awful; I will die on that particular hill.


Tabitha: Oh, man, I love them so much, and I absolutely agree with you. They're awful.


Corey: Yes, I love going through them—on either side of the table—because you have conversations; you get to see what other people are into, and that's great. But you're also evaluating people on a skill set that for almost every job, they're not going to need again until they interview for a different job somewhere else. I don't know about you, but I don't tend to write a whole hell of a lot of code on the whiteboard on purpose.


Tabitha: [laugh].


Corey: I don't need to implement quicksort. I don't need to invert binary trees or avoid link lists to wind up finding a cycle loop, or whatever it is that people are asking about these days.


Tabitha: Yeah.


Corey: It’s not interesting to me. It's fun to play the games of who knows more, but they’re games you play with coworkers and friends, not, “I am going to dictate the course of your career by how well you answer this question. But no pressure.”


Tabitha: Yeah. Yeah, yeah. And like, that's why I have taken so strongly to interviewing and why I enjoy the opportunity of interviewing so much because I've had the good fortune in my career, not to have very many of those awful sort of, “If you can jump through these 17 hoops in exactly the way that I prescribed, then you pass,” sort of interviews. I have had a far lower than usual number of them. And I'm grateful for that. 


And so I'm also grateful then, for the opportunity to pay that forward by trying to have a good and interesting conversation with every single candidate that I interview. Sometimes we have a great conversation because, actually, they are an astoundingly good fit for the role and, hooray; other times, I might know two minutes in that this is very likely not going to be a yes from me, but for the next 58 minutes, or whatever, my time is that person’s, and I want to make sure that they have a chance to talk about the way they think about technology, the way they approach problem-solving, in whatever way is going to show what they're good at—as well as it can—and hopefully give them things to think about to improve themselves in the future, to improve their interviewing performance in the future, but also just to improve the way that they approach things in the future.


This episode is sponsored by our friends at New Relic. If you’re like most environments, you probably have an incredibly complicated architecture, which means that monitoring it is going to take a dozen different tools. And then we get into the advanced stuff. We all have been there and know that pain, or will learn it shortly, and New Relic wants to change that. They’ve designed everything you need in one platform with pricing that’s simple and straightforward, and that means no more counting hosts. You also can get one user and a hundred gigabytes a month, totally free. To learn more, visit newrelic.com. Observability made simple.


Corey: I want to jump in. I mean, before you get letters, and oh, will you get letters if we're not careful to clarify here. Like, “Well, if you know you’re not going to hire someone, why would you waste their time?” Well, first, if I could reliably say this person is ‘no hire’ in the first two minutes of meeting them, that would be incredible in so many different aspects of life. But I have my initial gut impression, and then I have the actual studied impression so I can back up the decision either way.


Tabitha: Mm-hm. Got to have reasons. Like, just what you feel isn't enough.


Corey: Yeah. And as other benefits. One, it gives this person exposure to the interview process, if that's one of the areas they're weak on; it also lets you see how this person thinks. And again, remember, it's not a, “No,” it's a, “Not right now, for this role.” And it's also a marketing opportunity. 


The goal of every interview, in my experience has been that even if you turn down the candidate, or you make an offer and they reject the offer, they should walk away from the experience thinking, “That was such a great experience. I would love to try again for a better fitting role and/or recommend it to other people I know.” Because people remember this stuff. People talk about these things. I still have nightmare stories about awful interview processes. I went through the Google interview process twice, and after the second time, I swore that it was such a degrading experience I would not go through it a third time. And here we are.


Tabitha: [laugh]. Yeah, yeah, yeah. Or to put it the other way, there are organizations that I've had really great interviewing experiences with, and for whatever reason—my reasons or their reasons—did not end up working there, but I met people through the process of interviewing there, and I learned about the organization and I developed a lot of respect for both the people and the organization. And I keep in touch with some of those people, and I send them referrals. I try to get people to work at Datadog because I'm happy here and I want to have more people around that are the sort that I want to work with. But it's not just purely mercenary. I don't actually refer everyone that I know to Datadog, but I do refer people to the places where I think that they'll be happy. 


Corey: I will refer people like crazy. It doesn't take much to do it. And introductions always help. But I also am always clear to draw a line between referring someone and recommending someone. That's a much shorter list. And there's a difference between, “You should talk to such and such.” Versus, “If you don't hire this person, I would very much like to know why because one of us is very wrong about something and I don't think it's me.”


Tabitha: [laugh]. That's a great point. And like—


Corey: Yeah, ‘the shut up and hire them’ list versus the ‘some yahoo found their way into my inbox and asked if I could introduce them to someone at your company.’ Yeah, I'm thrilled to do that. And again, if you're listening to this show and there's someone that I know at a company that you would love to work at and want an introduction, please reach out; that's what I do. I enjoy doing that and it helps short circuit a lot of the HR application process that screens on things that, frankly, are not germane to your ability to do the job. I am thrilled to introduce people to one another. But there is a difference between that and, “I have worked with this person in the past; I cannot say enough good things about their work product, and they're a joy and treasure to work with.” Those are two different things. And I have no problem giving one of those recommendations to anyone, and I love when I find someone I can give the other one to.


Tabitha: I am super grateful for you bringing that up. Because, A) I think it's a good thing to be saying here on air, but also, B) that's a concept that I really needed in my life because I did not have that concept before. And so when I have done referrals, I have only done them in the, “I highly recommend this individual based on my personal experience and opinion,” sense. But you make a really good point about how introducing people to each other is not expensive, and can help people in ways that you don't understand.


Corey: And that's part of it, too is, again, don't view life in a transactional manner. My question I always like to ask people after I have a cup of coffee or lunch with him is what are you up to and how can I help? Who can I introduce you to? It takes basically no effort from me, but it has the potential to change the trajectory of someone's life. Our lives are built on relationships. 


And that's why I love sitting down and doing the interview conversations. I mean, here at The Duckbill Group, we go to extensive lengths to make sure that our interview process is not, “All right, we're going to find the thing you're weakest at and beat you up on that.” Like, what the hell is that? Are you trying to hire for strengths, or absence of any discernible weakness? The latter leads to a really weird culture.


Tabitha: Yeah, a culture where everybody's afraid they're going to get stabbed in the back by somebody else. Like, that's not actually a high-performing culture.


Corey: Oh, yeah. I mean, if you're talking to me for a technical interview process, and you ask how good I am with database. I'm like, “Oh, I'm great with DNS, thank you for asking.” That's probably a warning sign that I'm not a great fit to bring in when your DBA has a question. However, there are a lot of other things that I am particularly skilled at. It comes down to finding where someone is strong, where someone is weak, and looking holistically at the team and start to build out a highly functional team in the areas you need to have this. We could do a whole show on this sort of thing.


Tabitha: Oh my god, yes.


Corey: I'm just annoyed to the point of ranting now, the culture of hazing that is technical interviewing in an awful lot of shops. Because here's the hell of it: they've done studies on this; they’ve found what works and what doesn't at large scale for hiring effectively, and there's this entire Silicon Valley culture of, “We're good at programming, which probably means we're good at everything else, too, so we're going to discount what the quote-unquote ‘experts’ say and reinvent the job interview from first principles.” Do you hear yourselves? Stop it.


Tabitha: Oh, my goodness, a hundred percent. I always have to be on the lookout for that kind of stuff because my educational background is math and physics, and so I have to learn things from first principles. And if I don't, it never really sits well with me; I never really feel like I get it. But also, that comes with it a need to be quite clear on the distinction between, “I have experience in this area,” versus, “I am speculating from first principles. You should take this for what it's worth,” which might be something but it might be nothing at all.


Corey: Yeah. It's just this entire, weird, horrifying culture that we live in. The thing is, is I find you can tell an awful lot about companies by how they buy their people. And that is something that I think companies lose sight of is: candidates talk, and if you think otherwise, you're about to be sadly disabused of that notion.


Tabitha: Oh my gosh, especially as you get into more specialized areas. Sometimes it feels like there are ten Kubernetes security people and we all know each other. It's not really like that, but it can feel that way sometimes.


Corey: Yeah. I feel like we could do a whole ‘nother episode one of these days, about the joys, trials, and tribulations of technical interviewing. The problem is, I'm not the best person to have that conversation anymore because, at some point, I went through a transition from, “I have to get the job to put food on the table.” Which, respect. I get it. I have been there. I have nothing but respect for people in that position. 


And it switched to, “Okay, now that I have options in my career, what's the right company and what's the right fit?” And when I stopped pretending to be something I wasn't and being much more myself, I found that many job interviews were far shorter as a result, but because I was sussing out places I didn't want to work. It's one of those things where the old joke of, “What's your biggest weakness?” “Honesty.” “Well, I don't think honesty is a weakness.” “Well, I don't give a [BLEEP] what you think.” 


It comes down to not being a great thing for getting yourself hired, but it's filtering out the places you don't want to work. So, you've just completed the interview process yourself; you're working at Datadog. The fact that you will appear on this show and admit to working at Datadog implies that it's not a terrible place to be. Having gone through that process more recently than I have, any tips you have for anyone who's listening as far as what things to look for in an employer, red flags, things that really make you take a step back on the other hand, and say, “This is a place I want to be.” And, again, if you're listening to this and hiring, pay attention, this is also for you.


Tabitha: Oh, that's also a good one. For me, I was getting a lot from reading between the lines of the interview process. We were just talking about interview process, and trying to have a good and interesting conversation for both parties even if it didn't seem that the interview was necessarily leading towards a hire, and the more that I felt like my time was being well-used in the interview process, the more that I felt like I was enjoying meeting someone, maybe learning something, having an interesting conversation that seemed also to correlate with places where people liked their jobs, where people had positive relationships with their coworkers, where people helped each other out instead of backstabbing each other. And to me, that was a really important selector for where I wanted to work. I want to solve fun tech problems, I want to solve fun people problems with others who have that same goal. I don't want to waste my time on watching my back so it doesn't get stabbed.


Corey: Exactly. People have so much energy, and they can either focus on moving things forward, or they can focus on covering their own ass and doing everything the most defensible way possible. That's a gross oversimplification. I mean, depending on the company, sometimes downside risk management is more important than speeding time to market or getting code out. It turns out most banks don't have a culture of, “I wrote this code last night. Let's YOLO it into production, it's probably fine.” But in most cases, people will optimize for what you reward, culturally.


Tabitha: One hundred percent. For better and for worse.


Corey: I have been in multiple job interviews in my life as a candidate where I wound up successfully recruiting the interviewer because you start talking about work-life balance, and how the rest works, and at some point, they become too honest. And… “Yeah, this isn't the kind of place where you'd be happy,” “Yeah. That's kind of why I am where I am now.” “Are you folks hiring?” “Well, we could be.” And suddenly, we're having a different conversation.


Tabitha: You know, that is an absolute delight. And it reminds me of an interviewing story that was one of the best interviewing experiences that I never had. A friend of mine was a manager, managing a pen testing team, and at that time I was looking for a job, and I was potentially interested in it. So, I didn’t, like, apply, but I just asked her, “Hey, tell me a little bit about this.” And we had the conversation. And eventually, she told me, “I think that you could do this; I think you could be really successful at this, but honestly, I am afraid that you would get bored. You probably don't want to do this.” And that was the truest expression of friendship. And I'm still super grateful to her.


Corey: There's something to be said for if you're going through the process of interviewing and you don't know how to handle things. For God's sake, reach out to people. You're not alone in this, I promise. And find people who've done well in their career and ask them what their tricks are, ask what their secrets are, ask what they wish they'd known. People like the ambition in most cases, and people love giving advice. Just remember, everyone has their own opinion, and not all of them are great.


Tabitha: Yeah, yeah. You got to try to understand what is the context that has led to this opinion because sometimes you can learn from good advice, and in good circumstances, you can also learn from bad advice, as long as you recognize that it's bad advice and read between the lines.


Corey: Absolutely. So, thank you so much for taking the time to speak with me today. If people want to learn more, where can they find you?


Tabitha: Easiest place to find me is on Twitter. My Twitter handle is @tabbysable. Otherwise, you know, I'm around on the internet. I'm on a lot of the same big industry Slacks that you are, and those sorts of things. But I love to get messages on Twitter.


Corey: Oh, it's my favorite thing because then I feel so good about reading them and then forgetting to respond.


Tabitha: Oh—


Corey: Don’t take it personally.


Tabitha: —my gosh, the management of the DM inbox is such a trash fire, and I'm just lucky that I get a manageable number of them.


Corey: Exactly. Thank you once again. I appreciate your taking as much time with me as you have.


Tabitha: It's been great. It's been so much fun. Thank you so much for having me.


Corey: Tabitha Sable, systems security engineer at Datadog. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me what happens when I type google.com into my browser and press enter.


Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com, or wherever fine snark is sold.


This has been a HumblePod production. Stay humble.



Newsletter Footer

Get the Newsletter

Reach over 30,000 discerning engineers, managers, enthusiasts who actually care about the state of Amazon’s cloud ecosystems.

"*" indicates required fields

This field is for validation purposes and should be left unchanged.
Sponsor Icon Footer

Sponsor an Episode

Get your message in front of people who care enough to keep current about the cloud phenomenon and its business impacts.