Hear the dulcet tones of Corey Quinn

Transcript

Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.

Jesse: Welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.

Amy: I’m Amy Negrette.


Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure because I mean, who doesn’t love to complain about AWS? I feel like that’s always a good thing that we can talk about, no matter the topic. Today, we’re going to be talking about the ‘cloud cost management starter kit.’ So, the starter kit seems to be a big fad that’s going around. If you’re listening to this episode, you’re probably thinking, “It’s already done. It’s over.”


But I still want to talk about it. I think that this is a really relevant topic because I think a lot of companies are trying to get started, get their hands started in cloud cost management. So, I think this would be a great thing for us to talk about: what’s in our cloud cost management starter kit?


Amy: And it really will help answer that question that I get asked a lot on: what is even a cloud economist, and what do you do?

Jesse: Yeah, I mean, given the current timeframe, I haven’t gone to any parties recently to talk about what I do, but I do feel like anytime I try to explain to somebody what I do, there’s always that moment of, “Okay. Yes, I work with computers, and we’ll just leave it at that.”


Amy: It’s easier to just think about it as we look at receipts, and we kind of figure things out. But when you try to get into the nuts and bolts of it, it’s a very esoteric idea that we’re trying to explain. And no, I don’t know why this is a real job. And yet it is.

Jesse: This is one of the things that always fascinates me. I absolutely love the work that I do, and I definitely think that it is important work that needs to be done for any organization, to work on their cloud cost management best practices, but it also boggles my mind that AWS, Azure, GCP, haven’t figured out how to bake this in more clearly and easily to all of their workflows and all their services. It still boggles my mind that this is something that exists as—

Amy: As a thing we have to do.

Jesse: As a thing we have to do. Yeah, absolutely.


Amy: Well, the good news is, they’re going to change their practices once every six weeks, and we’ll have a new thing to figure it out. [laugh].

Jesse: [laugh]. So, let’s get started with the first item on our cloud cost management starter kit. This one is something that Amy is definitely passionate about; I am definitely passionate about, as well. Amy, what is it?

Amy: Turn on your CUR. Turn on your CUr. If you don’t know what it is, just Google AWS CUR. Turn it on. It will save you a headache, and it will save anyone you bring in to help you [laugh] [unintelligible 00:02:59] a huge headache. And it keeps us from having to yell at people, even though that’s the thing that if you pay us to do it, we will totally do it for you.

Jesse: If you take nothing away from this episode, go check out the AWS Cost and Usage Report—otherwise known as CUR—turn it on for your accounts, ideally enable it in Parquet format because that’s going to allow you to get all that sweet, sweet data in an optimized manner, living in your S3 bucket. It is a godsend. It gives you all the data from Cost Explorer, and then some. It allows you to do all sorts of really interesting business intelligence analytics on your billing data. It’s absolutely fantastic.


Amy: It’s like getting all of those juicy infrastructure metrics, except getting that with a dollar sign attached to it so you know what you actually doing with that money.


Jesse: Yeah, this definitely is, like, the first step towards doing any kind of showback models, or chargeback models, or even unit economics to figuring out where your spend is going. The Cost and Usage Report is going to be a huge first step in that direction.


Amy: Now, the reason why we yell at people about this—or at least I do—is because AWS will only show you the data from the time that it is turned on. They do have it for historical periods, but if you enable it at a specific point, all of your reports are going to start there. So, if you’re looking to do forecasting, or you want to be able to know what your usage is going to be looking like from this point on, turn it on as early as possible.

Jesse: Absolutely. If you are listening to this now and you don’t have the CUR enabled, definitely go pause this episode, enable it now, and come back and listen to the rest of the episode because the sooner you have the CUR enabled, the sooner you’ll be able to get those sweet, sweet metrics for all of your—

Amy: And it’s free.

Jesse: [laugh]. Yeah, that’s even the more important part. It’s free. There’s going to be a little bit of data storage costs if you send this data to S3, but overall, the amount of money that you spend on that storage is going to be optimized because you’re saving that CUR data in Parquet format. It’s absolutely worthwhile.

All right, so number two; the second item on our cloud cost management starter kit, is getting to know your AWS account manager and account team. This one, I feel like a lot of people don’t actually know that they have an AWS account manager. But let me tell you now: if you have an AWS account, you have an AWS account manager. Even if they haven’t reached out to you before they do exist, you have access to them, and you should absolutely start building a rapport with them.

Amy: Anytime you are paying for a support plan, you also have an account manager. This isn’t just true for AWS; I would be very surprised for any service that charged you for support but did not give you an account manager.

Jesse: So, for those of you who aren’t familiar with your account manager, they are generally somebody who will be able to help you navigate some of the more complex parts of AWS, especially when you have any kind of questions about your bill or about technical things using AWS. They will help you navigate those resources and make sure that your questions are getting to the teams that can actually answer them, and then make sure that those questions are actually getting answered. They are the best champion for you within AWS.

If you have more than a certain threshold of spend on AWS, if you’re paying for enterprise support, you likely also have a dedicated technical account manager as well, who will be basically your point person for any technical questions. They are a great resource for any technical questions, making sure that your technical questions are answered, making sure that any concerns that you have are addressed, and that they get to the right teams. They can give you some guidance on possibly how to set up new features, new architecture within AWS. They can give you some great, great guidance about the best ways to use AWS to accomplish whatever your use case is. So, in the cases where you’ve got a dedicated technical account manager as well, get to know them because again, they are going to be your champion. They are here to help you. Both your account manager and your technical account manager want to make sure that you are happy with AWS and continue to use AWS.

Amy: And the thing to know about the account manager is, like, if you ever run into that situation where, oh, something was left on erroneously and we ended up with a spike, or this is how I was understanding the service to work and it didn’t work that way, and now I have some weird spend, but I turned it off immediately, if you ever want to get a refund or a credit or anything, these are the people to talk to; they’re the ones who are going to help you out.

Jesse: Yeah, that’s a great point. It’s like, whenever you call into any kind of customer support center, if you treat the person who answers the phone with kindness, they are generally more likely to help you solve your problem, or generally more likely to go out of their way to help you solve your problem. Whereas if you just call in and yell at them, they have no interest in helping you. So—

Amy: You’ll never see that refund.

Jesse: Exactly. So, the more that you can create that rapport with your account manager—and your technical account manager if you have one—the better chances that they will fight for you internally to go above and beyond to make sure that you can get a refund if you accidentally left something running, or make sure that any billing issues are taken care of extremely fast because they ultimately have already built that rapport with you. They care about you and the way that you care about them and the way that you care about continuing to use AWS.

Amy: There’s another note about the technical managers where if you are very open with them on what your architecture plans are—“We’re going to move into this type of EKS deployment. This is the kind of traffic we think we’re going to run, and we think it’s going to be shaped this way”—they’ll help you out and build that in most efficient way possible because they also don’t want the resources out there either being overutilized or just being run poorly. They’ll help you out in trying to figure out the best way of building that. They’ll also—if AWS launches a new program and you spent a lot of money on AWS, maybe there’s a preview program that they think will help you solve a very edge case kind of issue that you didn’t think you had before.

Jesse: Absolutely.

Amy: Yeah. So, it’s a great way to get these paths and get these relationships because it helps both parties out.

Corey: This episode is sponsored in part by VM Ware. Because lets face it, the past year hasn’t been kind to our AWS bills or, honestly, any cloud bills. The pandemic had a bunch of impacts. It forced us to move workloads to the cloud sooner than we would otherwise. We saw strange patterns such as user traffic drops off but infrastructure spend doesn't. What do you do about it? Well, the CloudLive 2021 Virtual Conference is your chance to connect with people wrestling with the same type of thing. Be they practitioners, vendors in the space, leaders of thought—ahem, ahem. And get some behind the scenes look into the various ways different companies are handling this. Hosted by Cloudhealth by VM Ware on May 20th the CloudLive 2021 Conference will be 100% virtual and 100% free to attend. So you really have no excuses for missing out on this opportunity to deal with people who care about cloud bills. Vist cloudlive.com/corey to learn more and save your virtual seat today. Thats cloud l-i-v-e.com/corey c-o-r-e-y. Drop the “e,” we’re all in trouble. My thanks for VM Ware for sponsoring this ridiculous episode.

Jesse: So, the third item on our cloud cost management starter kit is identifying all of your contracts. Now, I know you’re probably thinking, “Well, wait. I’ve just got my AWS bill, what else should I be thinking about?” There’s other contracts that you might have with AWS. Now, you as the engineer may not know this, but there may be other agreements that your company has entered into with AWS: you might have an enterprise discount program agreement, you might have a private pricing addendum agreement, you might have an acceleration program—migration program—agreement. There’s multiple different contracts that your company might have with AWS, and you definitely want to make sure that you know about all of them.

Amy: If you’re ever in charge of an architecture, you’re going to want to know not just what your costs are at the end of the day, but also what they are before all your discounts because those discounts can maybe camouflage a heavy usage if you’re also getting that usage covered by refunds and discounts.

Jesse: Absolutely, totally agreed. Yeah, it’s really, really important to understand, not just your net spend at the end of the day, but your actual usage spend. And that’s a big one that I think a lot of people don’t think about regularly and is definitely important to think about when you’re looking at cloud cost management best practices and understanding how much your architecture is actually costing you on a team-by-team or product-by-product basis.

Amy: Also, make sure if you’re doing reservations that you know when those reservations and savings plans ent—

Jesse: Yes.

Amy: —because you don’t want to have to answer the question, “Why did all of your costs go up when you actually have made no changes in your infrastructure?”

Jesse: Yeah. Half the battle here is knowing that these contracts and reservations exist; the other half of the battle is knowing when they expire so that you can start having proactive conversations with teams about their usage patterns to make sure that they’re actually fully utilizing the reservations, and fully utilizing these discounts, and that they’re going to continue utilizing those discounts, continue utilizing those reservations so that you could ultimately end up purchasing the right reservations going forward, or ultimately end up renegotiating at the correct discount amount or commitment amount so that you are getting the best discount for how much money you’re actually spending.

So, the last item on our cloud cost management starter kit is thinking about the non-technical parts of projects. Amy, when you think about the non-technical parts of projects, what do you think about?

Amy: Non-technical always makes you think of people and process. So, this would be the leadership making the decisions on what those cost initiatives are. Maybe they want to push this down to the team lead level: it would include that. Or maybe they want to push it down to the engineering level, or the individual contributor level. There are some companies that are small enough that an engineer can be completely cognizant and responsible for the spend that they make.

Jesse: Yeah. I think that this is a really, really critical item to include in our starter kit because leadership needs to be bought into and back whatever work is being done, whatever cloud cost management work is being done. But also teams need to be empowered to make the changes that they want to make, make the changes that will ultimately provide those cloud cost management optimization opportunities and better cost visibility across teams. So, does everybody know what their teams are empowered to do, what their teams are capable of? Does everybody know what their teams are responsible for on the flip side? Do they ultimately know that they are responsible for managing their own spend, or do they think that the spend belongs to somebody else? Also, do they understand which resources are part of their budget or part of their spend?

Amy: It’s the idea that ownership of—whether it’s a bill, whether it’s a resource—comes down to communication, and level setting. Do we know who owns this? Do we know who’s paying for it? Do they know the information in the same way? Is there someone who’s outside who can figure out this information for themselves? Just making sure that it’s done in a clear enough way that everyone knows what’s going on.

Jesse: Absolutely. Well, that will do it for us this week. Those are our four main items for our cloud cost management starter kits. If you’ve got questions you’d like us to answer, please go to lastweekinaws.com/QA, fill out the fields and submit your questions.

If you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us, what would you put in your ideal starter kit?

Announcer: This has been a HumblePod production. Stay humble.

Links:

• Github main site: https://github.com/
• The Open Guide to Amazon Web Services:https://github.com/QuinnyPig/og-aws
• Slackhatesthe.cloud: slackhatesthe.cloud
• Global Diversity CFP Day: https://www.globaldiversitycfpday.com/
• Twitter: https://twitter.com/deniseyu21
• Personal site: deniseyu.io

Transcript


Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.

Corey: This episode is sponsored in part by VM Ware. Because lets face it, the past year hasn’t been kind to our AWS bills or, honestly, any cloud bills. The pandemic had a bunch of impacts. It forced us to move workloads to the cloud sooner than we would otherwise. We saw strange patterns such as user traffic drops off but infrastructure spend doesn't. What do you do about it? Well, the CloudLive 2021 Virtual Conference is your chance to connect with people wrestling with the same type of thing. Be they practitioners, vendors in the space, leaders of thought—ahem, ahem. And get some behind the scenes look into the various ways different companies are handling this. Hosted by Cloudhealth by VM Ware on May 20th the CloudLive 2021 Conference will be 100% virtual and 100% free to attend. So you really have no excuses for missing out on this opportunity to deal with people who care about cloud bills. Vist cloudlive.com/corey to learn more and save your virtual seat today. Thats cloud l-i-v-e.com/corey c-o-r-e-y. Drop the “e,” we’re all in trouble. My thanks for VM Ware for sponsoring this ridiculous episode.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Denise Yu, who's a senior software engineer at a company that mispronounces itself as GitHub, but it's actually pronounced Jif-ub. Denise, welcome to the show.


Denise: Thanks so much for having me, Corey. I am excited to scream in some clouds with you.

Corey: Yes, and I do want to point out that Jif-ub is not accepted by the Jif-ub marketing team, and they're upset. But that's the nice thing about this being my show; they don't get to dictate the nonsense that I say. But this is not you endorsing my correct pronunciation, as an employee, in any way. Because that's not your role. You're a senior software engineer. What do you do?


Denise: So, I work on a bunch of different things. I actually originally joined GitHub back in March at the beginning of this global pandemic that we're still living through, somehow, in the year 2021. But I joined a year ago to work on a team called ‘Community and Safety.’ That team has been rolled into other teams by this point, partly because the original team did too good of a job at creating abuse prevention tools for the platform. But these days, I still work on community-facing tools, which is very exciting. 


I'm actually in a department called Communities. And our whole charter is to build tools that make life better for open-source maintainers. So that's a mission that I'm actually really excited about. I think [laugh] of all the products that I've worked on, this is one of the few that I can say I actually really want to make life better for the end-users. [laugh].

Corey: Oh, absolutely. For better or worse, GitHub—or Jif-up—has taken a central role in the open-source community. And on some level, I find it—how do I frame this—kind of weird in that we've taken this protocol that is widely decentralized, and that's its entire point. Well, what do we do about this? Oh, immediately, we're going to re-centralize it. That's the right answer. 

And it just never ceases to amuse me that that's what we have taken from it as a society. But it works. I can't argue otherwise. And I maintain—and this is a bet that I am definitely going to the wall with—that in five years or so we are going to look back at Microsoft acquiring GitHub as a transitional moment.


Denise: Yeah, I think so. I think even though the Git technology is built for resilience and distributed work, I think what GitHub as a platform has shown that the process of building software is more than just pushing lines of code. The code is the artifact of collaboration, but we still need a place to do that collaboration, which is why—I’ve been using GitHub for five or six years now, and I only realized a few years ago that things like pull requests and issues don't exist outside of GitHub. That's an abstraction that's so, so deeply baked into my idea of what it means to work collaboratively. That didn't exist until GitHub invented it, which is pretty wild.


Corey: Yeah, it's bizarre in a number of different levels. But so much of what we think is part of Git is not. It is the GitHub abstraction, but it's also something that is widely copied by all of GitHub’s competitors, in many respects. So, the line has gotten very, very blurry. And how people come to Git is also fascinating. 


I used to go to a Git training, I think in 2009, conducted by a GitHub employee—I may be misremembering the year by a few years in either direction—and it was a multi-day process, and it was complex, and I left it feeling in many ways that I had more questions than I answered. Now, I point people to a GitHub page that talks about how to use Git, and they're mostly there. So, it isn't that Git necessarily has improved as a product, it's that GitHub has made it far more accessible. And let's face it, after a few million times practicing, you get very good at explaining complicated things in simple ways.

Denise: Oh, yeah. For sure. It's not a huge surprise that internally, we use GitHub for everything. So, if you want to, I don't know, collaborate on writing a—even a new blog post, new marketing copy, or new documentation, all of that happens through GitHub. And I think that people with different levels of proficiency with command line—actually these days, you don't even need command line. You do everything through the UI, which I think is pretty neat.

Corey: Oh, it's phenomenal. And I do want to call out that I am the maintainer of an open-source project on GitHub. We know it's good because it has [28.3000 00:06:18] stars; at the time of this recording, almost 3000 forks. And what it is, is the “Open Guide to AWS” it's a big markdown document that more or less explains what all of the AWS services do. 

There are over 200 contributors, we have an online Slack community at slackhatesthe.cloud because they don't like open-source community stuff very well, and we make fun of them for it. But my point being is that even without knowing how to write a single line of code, this is more or less just a big readme that explains different aspects of what AWS is because it's incredibly hard to adjust to that. And that is every bit as much an open-source project as anything that's included in NPM, or any command line tool that you'll use in any aspect of your job.

Denise: Yeah, exactly. And I think the nice thing about editing a document like an AWS Guide, or anything else, really. A couple years ago, I would have said, “Okay, that sounds like it should be a wiki page.” Wiki pages support revisions, collaboration—or maybe a Google Doc or something. But the nice thing about putting it somewhere like GitHub, is like, well, you're already—probably all your other tools already integrate into GitHub. Like, you maybe get Slack notifications, when you have a PR requires review, or whatever it is. It's just so much easier to have everything in one place. And you also get the cool green squares for contributing. [laugh].


Corey: Oh, of course, the most important thing is fill that thing up. It’s—talking about gamification causing weird behavior patterns. Yeah, we have a GitHub workflow on this, that fires off on every pull request that looks at the entire site, and validates the thousand and some odd links are still resolving and valid because when you link to this many different sites across the internet, link rot’s a real thing, and it's depressing and sad to be able to look at this and, “Oh, that didn't work.” Now, we do have to fix some of this because it winds up, in some cases, flagging people's submitted pull requests as not being valid, but it has nothing to do with what they're submitting because I'm bad at this. But the fact that that's built in and is available for use is incredible. Every time I look at GitHub’s site, it's gotten better than the last time I looked at it.

Denise: Oh, that's so nice.


Corey: Well, can't shake the feeling that at some point, there's going to be at the proper moment, a deploy to Azure button that as soon as someone clicks, whatever they've written is suddenly running in some environment in Azure. Now, if it comes too soon, it's going to be terribly implemented, and no one's going to trust it; if it comes too late, then people will already have solved this problem in different ways, so timing is going to be critical. But that is going to be—well-executed at least—a potential sea change in how people approach Cloud and what the default environment to run something on it.

Denise: Wait, have you looked at Codespaces? Or seen the demo around that?

Corey: I was hoping you would bring those up. Back in the before times, I would travel kind of a lot. And the only computer I brought with me was my iPad.


Denise: Mm-hm. Yeah, a lot of people do this.


Corey: Oh, yeah. Trying to get a environment on an iPad, it turns out, it's not terrific. The solution that I fell back to was, well, I've been using VI for 20 years, so I'll just SSH into an instance and call it good. But there were interesting projects of varying degrees of success that would run VS Code in a instance somewhere and then just present its interface as a web page. GitHub has actually integrated this as a core offering: all right, you click a button, it spins it up, and it's there. 

Denise: Yep.

Corey: And oh, my God, is that better.

Denise: Yeah. I think it's still in limited beta. So, I think you still have to request access to it—I don't know if this will still be the case in March, but yeah, I've tried it out a couple of times, and it works really, really well. My only gripe is that I don't like to use VS Code, so I'm going to have to learn VS Code in order to use it.

Corey: I've got to say that is my single biggest barrier to adoption. I already know how to use VI, but a lot of the VS Code stuff, of you having a full featured IDE that does all of these things, is extraordinarily heavy of a lift for me. It more or less is breaking 20 years of muscle memory.

Denise: Yeah. Tons of my co-workers love VS Code and use it every day. I only know how to use RubyMine for, [laugh] like, large monolithic Ruby development. And if I'm not doing work on the monolith, I don't want to use an IDE otherwise; I want to just use VI. I'm working in a small enough codebase where I know the name of every file, then I don't want to use the IDE for it. 

The VS Code to me sits in this middle ground that I know I need to learn how to exist in that middle ground, but it requires me going off to the internet and hunting down every single plugin that I need to put onto VS Code to make it feel like RubyMine again. And that's the thing that I don't want to do.

Corey: Well, I'll take it a step further, and most of the tutorials I see about how to get up with VS Code as an IDE are JavaScript based—or YavaScript. [unintelligible 00:11:01] pronunciation once again—and my lingua franca is crappy Python. So, whenever I look for, oh, I want a VS Code tutorial for Python, the first eight pages of it are here the different ways you can do virtual environments and dependency management because that is Python. It focuses on that, and it winds up getting hung up on those implementation details before ever getting to, “Here's a reasonable Python project that someone who's not very good at Python can understand, and here's how VS Code can add value to it.” I mean, those posts exist, but they're hard to find. And it honestly makes me feel like an imposter for not knowing JavaScript.

Denise: Yeah, the feature that I use the most in a full IDE like RubyMine is command-click to go to definition. And with Ruby, you get the correct definition about 70 percent of the time, but it's still better than nothing. Because if I'm not doing that, I'm doing a full text search for the method definition, and that's like—my brain can only work with one of those two things. With VS Code, after 15 minutes of trying to find the right plug-in that does click to go to definition and failing, I was just like, this is just not worth it. Nobody should have to live through this much pain. [laugh]. I don't want to do this anymore. [laugh].


Corey: Exactly. It becomes this situation where at least when you're starting to learn something, it's breaking everything you're used to, and it feels like instead of helping you, you're fighting with it.

Denise: Yeah, this reminds me of how people talk about switching their keyboard layout to Dvorak, or whatever—

Corey: Oh, my God. I want to do that on one hand because I'm deep into the mechanical keyboard space. But on the other is, I don't really have six months to teach myself to type all over again.

Denise: Yeah, that's the thing. It's like, maybe it'll speed up my typing by, like, two percent. But I already type really fast. I think I would have extremely diminishing returns on that. But also, I don't want my brain to just not work for weeks or months. 

Corey: Oh, it is worse than that because as soon as you leave that and go on to someone else's computer—which admittedly is not nearly as frequent as it used to be when I was in help desk and desktop support—“Oh, can you type in your password on this thing?” “Absolutely not. Thank you for asking.” It's effectively having to go back and restart it over. I understand that it's more efficient, but you need a societal shift to it, not individuals doing it piecemeal.

Denise: Yeah, that's true.


Corey: So you're relatively recent, as far as hires over a Jif-ub. You joined mid-pandemic, or early pandemic, depending on how you want to slice that, and you've never been to their headquarters. What is that like?

Denise: Everyone keeps telling me how awesome HQ is like, and I’m like, “Cool.” I actually have a brick wall in my office that looks like some of the spaces in HQ, I guess, so people get super confused when I tell them that I'm in Toronto. 

Corey: Oh, yeah, I've been to HQ a bunch of times for various things. It's great. You’ve really—

Denise: Yeah.

Corey: —got to see it. I'm not helping, am I?


Denise: [laugh]. You're really not helping. I really want to go to HQ. I was meant to go to HQ in March. So, my start date was March 13th, or 14th—I can't remember—in 2020. And I was part of the first onboarding class to not get to go out to HQ because that was when the travel restrictions around COVID were finally being implemented. So I'm very sad about that.


Corey: I will say that GitHub has the advantage of a lot of other companies in that—to my understanding—they had a remarkably resilient remote culture before this all hit. Is that correct, or am I misremembering my history?

Denise: No, you're right. I think the figure was 75 percent of employees don't live within commuting distance to HQ when I first joined.

Corey: Well, not with that attitude.


Denise: [laugh]. Yeah, that's true. We just got to try harder. But I think that number is a lot higher now because we have been hiring internationally. And we've grown a lot in the last year or so that I've been here. 

Yeah, so I think we were better equipped than a lot of other companies to go full remote because it basically just was telling that 25 percent of employees, “Don't come into HQ anymore.” And giving people a little bit more office budget, I guess, to get set up fully for full time remote work. There are a lot of challenges though, especially when you join as a remote employee and you've never worked in a fully remote environment before. It took me a really, really long time to get used to the fact that Slack was my portal to everything. 


Corey: Oh, yeah. Here at the Duckbill Group, we were full remote from the beginning, just because I decided to merge my consultancy with my business partner’s ten days before he moved from across the street to Portland, Oregon. So, we at that point decided, all right, we're full remote. And we've never had a critical mass of staff in one city to the point where they're became a de facto headquarters because as soon as that happens, it changes your culture. Trying to retrofit something like full remote to a company that historically has always been face-to-face, it shatters a lot of the dynamics. So, on some level it was easy for us. On the other, it feels like we're at a bit of a disadvantage in some respects, once things return to normal. But that's a future problem, unfortunately.


Denise: Yeah, I think the biggest thing is learning to work asynchronously, which GitHub was already pretty good at. So I'm pretty used to opening a PR on Wednesday evening, for example, and not really expecting to get feedback on and until the next day, Thursday, or even Friday, especially if I needed a feedback from a different team. Actually, that's something that I've learned since joining GitHub; that wasn't something I was used to before because I've only ever worked in co-located environments before. Someone didn't review my PR in time, I would take a Nerf gun and go over to them [laugh] and gently ask them to [laugh] review my PR. But it's been very interesting. 

So, I also used to work at Pivotal, which I'm sure you've heard rumors about our kind of cultish pairing [laugh] culture. So, my default state of working was to constantly be talking to another human about what I was doing all the time. And so, suddenly joining a fully remote company in a time when we have to work a little bit more asynchronously than before because people's lives are just completely upended by this pandemic, that was a big adjustment.


Corey: So, in the before time, something else that you were effectively renowned for was not only giving conference talks yourself, but helping other people give conference talks, too. Let's talk a little bit about that because speaking at conferences was always one of my passion projects because I love the sound of my own voice, simply love it. And helping other people do that, too, was also something I was very interested in doing, but it was always hard for me to get traction around getting people to let me help them for reasons that are probably blindingly obvious to everyone except me. So, tell me about that. What's the story?

Denise: Yeah, so I have always been pretty comfortable speaking in public. I think it's something that I almost took for granted. And the reason for that is because I spent upwards of 10 years of my teenage and adult years doing competitive debate. It was traveling to debate tournaments pretty much every single weekend in college and spent a lot of time doing both structured and off-the-cuff speaking in front of large groups of people. And when I got into tech, it took me a long time to realize that not everyone is like me. I kind of had this special strand, I guess of imposter syndrome, where I think that if I know how to do a thing, then everybody else knows how to do that thing also. Which is aggressively not a correct assumption to make, ever. 


Corey: Oh, same here across the board. It's one of those I thought for a while I fit in tech because the stereotypes seem to fit me super well. It's like, yeah, “We're bad with people.” “Absolutely.” “We prefer dealing with computers to humans.” “Absolutely.” “I hate my boss.” “Absolutely.” “I'm good at programming…” “Oh… okay, never mind.” But yeah, there's a lot that I think people miss about what it takes to give a good talk. It's not technical mastery. It is an ability to tell a story.

Denise: Exactly. I think you hit the nail on the head there. And so after I realized that, okay, I am, at least at the median. I actually think I'm quite good at public speaking and telling stories about technical subject matter, and breaking things down in a way that someone without as much technical context as me can understand it. The thing that got me into doing more and more public speaking, specifically at tech conferences was, after I gave my first talk, people just kept coming up to me, basically, and being like, “Hey, I saw you talk to this conference. Do you want to come talk at this other conference that I'm running?” or whatever. I think like, once you've got a couple of larger stage conference talks under your belt, you almost don't really have to look for speaking opportunities anymore.

Corey: No, they fall into your lap. And then you have different problems such as, oh, I've been invited to speak at this conference. Let me do some checking to figure out are they a trash fire, or are they a reasonable place?

Denise: Yeah. Am I going to be the only not white guy speaking at this conference?

Corey: Am I going to be on a manel? I won't do it. Is there not a code of conduct that has actual teeth and doesn't read as a joke? I won't do it. And that stuff is way more important to me than the technical content of the talk.

Denise: For sure. For sure.

Corey: But tying it back to even the stuff that GitHub does, one of the talks that was transformative for how I approach this was a Git talk, “Terrible Ideas in Git.” And I like to teach people things by counterexample, and the first iteration of that talk would have been great if the Git maintainers themselves had been the target audience for this, but two people would have loved that and the rest of the audience would have been looking at this with what the hell are you talking about? So, I had to refactor it, and it made it a far stronger talk all the way to the point where there's jokes for everyone in there. But my favorite time giving the talk, I said that, “I now need to be able to explain Git in such a way that my mother would understand it. And that is a problematic way of framing on ageism and sexism basis in most cases, but not this time because she's sitting in the front row. Hi, mom.” That was one of my favorite speaker moments that was just, it's hard to get that one pulled off, mostly because getting my mother to fly to various parts of the world and watch me speak is a bit of a heavier lift than one expected. But here we are.

Denise: Yeah, I use a similar litmus test for my stepdad. My stepdad is not a technologist. He is a biologist by training, actually. But he asks me a lot of questions about what I do. And back when I worked at Pivotal, I had to explain cloud virtualization to him every time I went home. [laugh]. 

I don't know if it's gotten easier. With GitHub, I feel like GitHub is a little bit easier to explain to the average person than Cloud Foundry. But my bar kind of is, every time I put together a talk, do I think that my stepfather could understand the content.

Corey: This episode is sponsored in part by our friends at New Relic. If you’re like most environments, you probably have an incredibly complicated architecture, which means that monitoring it is going to take a dozen different tools. And then we get into the advanced stuff. We all have been there and know that pain, or will learn it shortly, and New Relic wants to change that. They’ve designed everything you need in one platform with pricing that’s simple and straightforward, and that means no more counting hosts. You also can get one user and a hundred gigabytes a month, totally free. To learn more, visit newrelic.com. Observability made simple.

Corey: Increasingly, I found the best way to explain GitHub to folks who aren't familiar with it and are non-technical but work in the business space has been, “You know how Microsoft Word has track changes? Yeah, imagine that only without the copy-of-copy-of-file-final-use-this-one-date.bak.PDF.doc? Yeah, imagine that. Instead, this makes this streamlined and easy for multiple people to work on at the same time,” at which point they stare at you hungrily and say, “Oh, my God, how can I use this for my work?” And I feel like there's a breakout moment waiting for GitHub if they ever decide to focus on that part of the market because my God, is there an appetite for it?

Denise: Yeah, I think like that kind of revision driven editing, I think that's applicable to so many different domains. I've heard people experimenting with the idea for—I think, Figma does something similar for designs for visuals, but I've also heard the idea floated of this being done for, like, sound mixing, sound engineering, which I think would be super, super cool. I don't know how you could visualize that, but someone who has more knowledge of that space, I'm sure could come up with a way.


Corey: Oh, yeah, the challenge I find with stuff like that, for podcasts, for example, is GitHub gets very, very angry because Git gets very, very angry when you start doing things like checking in large uncompressed media files, and then making small changes to them, but you can't diff that so every revision for video work is, “Oh, here's another 150 gig file to put up,” and that becomes a problem.

Denise: Yeah.


Corey: Yeah, there's a lot that's neat out there. I also want to credit GitHub with fixing the state of working with Git because let's face it, Git makes everyone feel stupid at some point. The question is, is just how far along are you before that hits? And invariably, the approach was always, when you get into that territory, you wind up asking the question on some forum somewhere, and then you wind up getting a few minutes later, this giant essay where it starts out, “Well, Git is not really a version tracker, it's instead of distributed graph that”—and then you just skip down eight paragraph to the bottom where they give you the one-liner that fixes the problem. 

GitHub has fixed the result of queries for, “How do I X in Git?” And just gives you the answer outright. It's transformative, and it's one of the things that everyone takes for granted because no one really stops to think this documentation is awesome, and accessible, and answers my problem, but the result of it is, is that you just don't see people complaining about how hard Git is anymore.


Denise: Yeah, yeah. Oh, my God, Git is so hard. I think the first-time I correctly did an interactive rebase—like, I typed all the arguments correctly. I think I just got up and did a lap around the office. [laugh]. There's just all this random stuff that you have to remember when you're using—especially some people use the desktop GUIs or whatever. I always just use command line. But it took forever and I still make a ton of mistakes with it.

Corey: Oh, we all do. That's the beautiful part of it. The nice thing is, “Oh, well the nice thing about Git is that nothing ever really gets truly lost.” And you say, “Well, how do you figure?” “Oh, I copy the entire directory to a backup before I do anything.” Which is on the one hand, awful version control and, two, has saved my bacon multiple times. So, if it's stupid and it works, it's not really stupid, now is it?


Denise: It's not stupid if it works. [laugh].

Corey: So, a passion project of yours that I want to talk about has been teaching people to submit to conferences, to give talks at conferences, to convince them that yes, you can do this and you're good at it, you have a story that people want to tell, the things you find easy are not easy for everyone. ‘Give that story voice’ has been a recurring theme that you do. How has that manifested during this year of isolation?


Denise: Ohh. This year’s definitely been tough. So, usually when I do that pitch, I run workshops specifically around a certain CFP. Like I help out conference organizers in the Toronto area. Like, I did a workshop for devopsdays last year, and another one for GoCon, also meant to be last year. 


And usually it's like, okay, here's how you write a strong proposal—and I'll have the committee in the room—and be like, here's what the committee is looking for; ask these people questions about what they would choose to program. And this is how you get your talk submitted. This is how you get accepted; you need to understand who your audience is. It's a product question. It's exactly the same mindset that you need to get into when you're playing product manager, or if you're actually a product manager. 

You need to understand who your target customer or audience is, what they're looking for, and how do you give that to them on a plate. This year, it's slowed down a lot, obviously because there have been fewer conferences, so I've just had fewer opportunities to do this kind of pitch without—it feels weird telling people to write talks without there being a place to submit a talk to. But I think—I found that there still is room for content creation, even if that isn't a conference talk. So, within the company, within my networks, I do try to encourage people to write blog posts. Err on the side of writing what you've learned; if you spent a couple days doing a technical investigation and you found it interesting, write it down so other people can learn about it. 


On the conference talk front, I think the unique benefit of doing conference talks and getting up on stage is you do build a personal brand. After a little while that kind of transcends the company that you're at, granted that you're not talking about your company's products in every single talk. 

Corey: Absolutely. And a lot of companies get profoundly insecure about that. And I'm sorry; they're wrong. That is not a bad thing. Anyone who gives a talk for your company will eventually become better known than your brand for some aspects and in some areas—harder at places like GitHub than it is Twitter for Pets, but that's okay—and you have to be okay with that and have a plan for how you're going to transition them out when they outgrow the role and move somewhere else. But that's okay. And companies that are insecure about that drive me up a wall.

Denise: Yeah, there have been a few times when I went to very cloud specific conferences and talked about things that my company is working for, but other times, I've gone to just very general conferences where it was just like, let's talk about Agile, or let's talk about software, like super, super generally. And the ones who have gone to the really general conferences and talk about things that matter to me—like cross-functional collaboration, or test driven development, or whatever it is—those are the ones that I've actually managed to get people to apply to my company without even saying that we're hiring. It's a little mind blowing.

Corey: It is. People want to work with interesting people, and when you find someone who's able to tell a story that touches you on some level, which is the point of telling stories, then at some point, the idea of I'd like to work with that person on ongoing basis becomes actively compelling. People ask so well, it's not like you could hire a whole lot of people who do what you do, so hiring and recruiting is going to be a problem. It's like, well, it is but it's because it's finding the right people with the right alignment, but we have an awful lot of candidates to choose from for basically any role, just because for better or worse, I make a splash.


Denise: Yeah, exactly.

Corey: So, as people are looking to get into the space and they're new to speaking, how do you get them to a point where there's at least a certain level of confidence? I would say, “How do you get them to not be terrified?” But I’ve been speaking almost full time for years, and I'm still scared to death every time I give a talk when I start out.

Denise: Me too. I'm still terrified. I almost switch into a different persona when I go on stage, and I don't remember anything that happens. I don't know what—maybe there's psychology [laugh] around this, but I literally blank out the 40 minutes that I'm on stage. So, the most common objection that I hear about why someone doesn't want to give a talk is they'll say, “But everybody already knows about this. The thing that I want to talk about has already been talked about. There's already too much content out there. I wouldn't add anything new.” 

And for a long time, my response to that was just, I don't care. I still want to hear what you have to say, you're going to offer a fresh perspective, you're going to offer a new experience, you're going to explain something in a way that clicks for someone. Your explanation might be the one that works for them that they haven't been able to figure out by listening to anybody else. So, I historically said those things, and I still one hundred percent those things, and I still say those things to people but I actually came across something interesting on Twitter today. Someone was actually talking about fanfiction. I don't know if you’ve written or read fanfiction in the past. When I was a kid I used to—

Corey: I can neither confirm nor deny…

Denise: [laugh]. There’s varying levels of fanfiction out there. But fanfiction communities are actually incredibly collaborative places. I remember being, I don't know, like a 10 or 11-year-old and writing a lot of anime fanfiction, basically just writing myself into the Pokemon universe and things like that. And we would post these on forums. 

It'd be forums of strangers working together, and we’d post drafts. And then we would give each other feedback on those drafts. And in the end, there'd be 60 different pieces of fanfiction, roughly about the same universe, every story is a little bit different, and nobody was ever self-conscious that this had already been done. Because the whole point of fanfiction is that there's a lot of it, the value of the community is in the fact that the content is abundant. Not that every single piece of content is unique. 


So, I think a similar thing is true for content that's technical storytelling. Just because someone else has said this thing before, it doesn't matter. The fact that there's more content out there about this thing is good for the community, building a community around this thing. So, I don't know, that was something that just resonated with me a lot, and I just really liked that framing.


Corey: We have this weird perspective that the things that we know are easy, the things we don't know are hard, so once you've learned how to do something, everyone else knows it. I mean, one of my most popular Twitter threads that exploded that I didn't see coming was just me talking about random things that live in my shell config file, and what they do. It's, “Well, this is stuff everyone knows, and everyone has something like this, right?” And it got into a dissection of what these things do, and how it works, and of course, it's snarky because that's my excuse for a personality, but it also shined a light on something that I forgot, once again: not everyone knows this. A lot of the viral tweets you see going around are people noticing something that we've all seen and been ignoring because that's just the way it is, and someone makes an observation about it and, oh, yeah, that's right. That is something that everyone can relate to.

Denise: Yeah. 


Corey: People want stories.


Denise: Yeah, exactly. I think about every six months, I try to retweet—every review cycle around mid-year, end-of-year. It’s like, “Hey, just a reminder, if someone on your team has done something for you write this down. Catalog what they did, write it down, send it to them, send it to their manager, put it into their file so they can use it at review time.” And every time I talk about this, I always get a ton of engagement. 


And I have to remind myself that this is not obvious. Even if someone was told to do this at some point in the past, reminding people to be deliberate and be active about giving each other peer feedback, writing stuff down. You just can't remind people to do that enough, I think.


Corey: Yeah, it's one of those things where, “Well, I've already given that talk, so I have to give a different one.” No, you don't, I promise you, you have not given your talk to everyone in the industry, and the reason I know that is because by that point, you would see the view count on YouTube or whatnot—or the video winds up—and be flabbergasted because the industry is bigger than anyone thinks. I feel like I tell the same joke from time to time, but they always get laughs from the audience because it's new to them. My business partner and my romantic partner find it incredibly challenging, both—those are two people, incidentally—they find it incredibly irritating because, “Yes, yes, we've heard that one.” And every once a while they sit up, “Oh, Corey got a new joke.” But the rest of the world doesn't respond that way.

Denise: Yeah, exactly. So, I don't know if we've talked about this already, but that closes the loop on the whole imposter syndrome piece where—just because I think one strain of imposter syndrome is telling yourself, just because I know this thing, means everybody else must also know it. I must be the last person in the world to learn this piece of information.

Corey: There is no piece of technology, or anything even tangentially related to technology, that you couldn't do a tweet thread on and people would learn things about it. How an if-then-else loop works, for example. That is—sorry—ahh—do you see what I'm talking about? That's exactly what I mean. See, I meant to say ‘Conditional,’ and instead I said ‘Loop’ and we all trip over these things; they're all challenging, and the fact that we can still learn new things about that of, how does a flow control conditional actually work? 


What does that mean? Sure, anyone who spends their days writing code all the time is going to have some topical understanding of it, but not everyone's going to, first, be that person, or secondly, understand why that's valuable and relevant. You could do an entire conference talk on nothing more than that concept alone and it would be a dynamite talk.

Denise: Yeah, exactly. I did go through a wave of imposter syndrome around last year, or maybe—uh, I don't even know what year is it. [laugh]. A little while back, I was feeling uncertain about this talk about distributed systems that I've done a few times. And then I got a random message on Twitter from a student who told me that she had just gotten her first engineering role out of university—ever, ever—and she said, I watched your talk on distributed systems; you gave me the vocabulary to talk about this stuff in an intelligent way, and I've landed my first role as an entry level—I forget exactly what kind of engineer but—I mean software engineering—but I forget if it was like infrastructure, or app dev, I don't know. But it was like, that kind of feedback, just—that's really cool to hear.


Corey: I'm going to take it a step further and request anyone listening to this who’s seen the talk that has potentially changed the way they view things or helped them advance in their career, track down whoever gave that talk and send them a message. One of the hardest parts about being a speaker is that you very rarely get feedback. I mean, even on this podcast, I almost never get direct feedback online. People talk to me about it in person, when they run into me at a conference, in the before times. But it's not a thing where people wind up drafting an email, “Dear sir, I found your podcast to be compelling, and also you are a jackass.” I would love letters like that.


Denise: Well, I don't love letters like that, so please don't write me letters like that. [laugh].

Corey: Well, in my case, they're well-deserved. In your case, it would just be offensive.


Denise: [laugh]. But yeah, reach out to those strangers whose content you've learned from, I think it can actually make someone's day.


Corey: Absolutely. So, on the other side of that, if someone's listening to this, and they want to start giving a talk, but they don't know where to start, how to build a CFP, where to begin, where should they start? What is the next step for those people?


Denise: So there's an initiative that I, in the before times, mentored and helped organize local chapters for. It's called Global Diversity CFP—it's either Global CFP Diversity or Global Diversity CFP, I always forget—day. And even if you're not a person who has a marginalized identity, they've done a really, really great job of consolidating a bunch of resources to get first-time speakers off the ground.

Corey: And we will put a link to that in the [show notes 00:36:34], of course.


Denise: Cool. So, I recommend reading through what they have. So it's actually an event that take place on a full day; it's a little bit different this year, I think they might be running it remotely this year. But all of their content is geared for first-time speakers, and they have a lot of material that's pepping people up to take the first step. But I would say that beyond that, the best way to get started, honestly, is to just pick a topic that you care about, and just try to write a talk. 

And if you don't have a place to deliver that talk—if there's no meetup, or conference or anything coming up, you should just record yourself giving the talk to your computer, actually. This is the thing that my debate coach used to have us do in college: the best way to improve as a speaker is to record yourself speaking and watch it. It's going to be terrible the first few times because you're going to be like, “Oh, God. I look so awkward.”


Corey: Power through it. I still can't watch myself speak.

Denise: [laugh]. Like, what are my hands doing? But you can only catch those things and improve at them once you're aware that you're doing them. So, that is the fastest way to improve your presence and your speaking ability.

Corey: Yeah. I want to thank you for taking the time to speak with me and go through all these various topics today. If people want to learn more about you, where can they find you?

Denise: So, you can find me on the internet on the cursed blue website at @deniseyu21 is my handle on Twitter. My personal website is deniseyu.io because—I used to have .com but some kids stole it and squatted on it after I forgot to renew it. So I'm not deniseyu.com anymore.

Corey: Yeah, the joy of changing usernames on different platforms and the rest. I hear you.


Denise: Yeah. Yeah, probably those two places. If you want to talk about anything that I talked about on the podcast, feel free to DM me on Twitter. Please don't send me mean feedback that starts with “Dear sir,” in my Twitter DMs, but other than that, I—

Corey: Generally never a good plan for anyone.


Denise: [laugh]. No. 


Corey: Thanks so much for taking the time to speak with me. I really appreciate it.

Denise: Yeah. Thanks so much, Corey for having me on. It was a lot of fun. I feel like we covered a lot of ground. Probably too much. [laugh].


Corey: It really was. And we did. Denise Yu, senior software engineer at Jif-ub. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment containing a correctly formatted Git rebase command.


Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey atscreaminginthecloud.com, or wherever fine snark is sold.


This has been a HumblePod production. Stay humble.