Screaming in the Cloud
aws-section-divider
Audio Icon
The Rise of the Agile Data Center with Tim Banks
Episode Summary

About the Tim Banks
Tim Banks is currently with Packet, an Equinix Company, where he is a Principal Solutions Architect. His tech career spans over 20 years through various sectors. Tim’s initial journey into tech started as a US Marine, having originally joined the Marine Corps to be a musician. He was later reassigned into an avionics specialty based on the results of standardized testing. Upon leaving the Marine Corps, he went on to work for hardware manufacturers and defense contractors as a civilian. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for large Unix-based datastores.


Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with engineering groups in his current role. Tim is also a husband and a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and Pan American Brazilian Jiu-Jitsu champion in his division.


Links Referenced: 

Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.


Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined once again by Tim Banks, by popular demand. We spoke last year and it was fantastically well-received to the point where everyone demanded more of him. And you're back, Tim, in a new year with a new job. You were at Mission Cloud, and now you've joined Equinix Metal as a principal solutions architect. First, congrats. Secondly, what gives?


Tim: Well, thank you, Corey. I'm glad to be here, and I appreciate it. So, I spent some time in the TAM world, both at AWS and at Mission, and I made the jump into solution architect—actually principle solution architect at Equinix Metal. And I think there's a couple reasons why I made that jump. 


First and foremost is that I still wanted to be in a place where I'm talking to the customer because that's where I think I can excel in both helping the customer and helping my company because I have an engineering background, I have engineering chops, but I also know how to talk to people and more importantly, listen to people and figure out what they need and figure out what they want, and then try and find a way to get those two things to happen. So, sticking within that role versus going to, like, a people management role, or versus going to some other kind of role, I think it was very important to me. The other thing that was very important to me were dollars.


Corey: Oh, yes. Hopes, dreams, and mission are all very important, but I can pay exactly none of my rent with those things.


Tim: No, I wish I could, with best intentions, but they don't translate well to keeping the lights on. You know titles matter, so to get the principal title was very important to me because I've done the work, so to be able to have the title and then the financial backing that goes with that title was extremely important to me. People have asked, “Well, you were only there for a year,” or a little more for a year, and, “What gives?” And the fact of the matter is that for many roles, for many people, and in many companies, you're going to get a far better raise by changing companies and changing jobs than you will by just trying to go through the normal internal promotion track. There's a lot of reasons for that, they vary from org to org, but the numbers don't lie. 


There are a lot of people that have a career that looks like that, where you are staying at a place one to two years, and then you change jobs or change companies and there was a significant pay raise that went along with that. And if you look at what the pay raises that come with the internal promotions and review periods and stuff like that, they never match what you'll get by going to another company. I wish to say they would. I wish that companies would do more investment in keeping their people around, but that's just not the case.


Corey: Oh, I'm right there with you. It's one of those I'm lobbying for a three percent versus four percent raise after I've been somewhere for a year versus, well I can get twenty percent if I change jobs. And partially that's because my skills have changed and improved, presumably. But at some point, this obviously does cap out, but if I'm talking to someone else, and I'm able to get a few tens of thousands of dollars in raise, then why not? It feels like it's absolutely one of those things that lends itself to just pure self-interest if nothing else.


Tim: Absolutely. And so I have a strategy behind it. And so every year, on my year anniversary date or so, being with a company, I will look at other jobs and I will take some interviews. And I'll do that for two reasons: the first reason is because searching for a job and interviewing is a skill. It is absolutely a skill, and if you don’t—


Corey: Oh God, yes. And it evaluates you on a list of things that you really only bring into practice when you're looking for a job. So, if you've been somewhere for eight years, that skill has atrophied massively, and you're being judged based on that ability.


Tim: And it's atrophied massively, and the problem is too many people are exercising that skill when they absolutely need to instead of exercising it when they don't have to. It's like do you want to wait to run a mile when you're being chased by wolves, or do you maybe want to run a mile every so often to make sure you can, and then if you do get chased by wolves, you're very comfortable running that mile. If you are not interviewing; if you don't know how to interview; if you don't know what interviews look like now; if you don't know what questions they’re asking; if you don't know what skills they’re looking for; if you don't know how your resume supposed to be [00:06:09 unintelligible]; if any of these things, and then heaven forbid you get laid off, or for some reason like that, and then now you have to interview under duress to make your mortgage payment and to feed your children, that's a lot more stress, combined with being unfamiliar with what you've got to do. So, just as a means of practice and as honing my professional skills, I will go out, and look for, and interview for jobs every year.


Corey: I strongly endorse the entire approach.


Tim: And now the second reason I do that is because when I do go in for my annual review, for my performance review, or whatever, I want to know what the value of my skill set is to outside companies. And that way, when it comes time for review, and the company I'm working for now says, “Hey, well, this is what you are worth to us.” And that's what they're doing when they give you your raise or evaluate for whatever annual raise, or performance rate you’re due for, they're saying, “This is what your job is worth to us. This is what the value of your work to this company.” And if my work is more valuable to another company, I will bring that up to my current employer, and if they don't see that the value is the same, then I think we've reached that point, an amicable parting of the ways.


Corey: I agree wholeheartedly with everything you just said. I've gone on quote-unquote, “Practice interviews” past in my career, and either wound up hiring the interviewer, or that practice interview suddenly became very real, and I took a job there for a variety of different reasons. I don't have many regrets about my current situation in life, but one of them is that given that I own the company, it's very challenging for me to reliably and in good faith do that because it first sends a message about the company that isn't great, and two I can't imagine what something that would lure me out of here—that wasn't an acquisition—would look like. And I don't want to waste people's time and cast those doubts into the ecosystem. But I really enjoyed aspects of the job interview process.


Tim: Yeah, I think the one thing that I take away the most from the job interview process is figuring out how to present my skill set to different groups of people. So, I can say that, “Yes, I know containers, I know best practices for engineering, I know DevOps, I know SRE, I know all these languages, I know all these automation methods.” But how do I convey that to someone in the manner that they're asking the questions? If I interview for a startup folks from Silicon Valley, those questions are going to look a lot different than if I'm interviewing with someone like Amazon, or Google, or Facebook, or something like that. They ask questions differently, they look for different answers to find the same underlying set of skills. And so you have to learn how you can convey your skillset and how you can convey what other qualities have to different people who are asking the same question different ways.


Corey: Absolutely. It's also strongly valuable to understand this is a form of corporate hazing, for lack of a better term. In practice, you are never going to need to invert a binary tree or implement quicksort yourself. That's what lunatics would do. 


But people love asking those questions, and it's more or less a, do the CS students’ secret handshake to pass this and show that you belong here. Which is deeply problematic on a number of different levels. But that's where it seems like it goes, and I've never been a fan of the trivia questions. It still feels on some level like these companies are interviewing for a lack of weakness rather than hiring for strengths.


Tim: Yeah. It is, in its very core, just gatekeeping. And not gatekeeping to say, “Oh, we have a standard and this is what's done,” because as you mentioned, most of these have nothing to do with the actual job you’re going to perform.


Corey: Yeah, as opposed to, what, all those companies that have no standards? Please. 


Tim: [laugh].


Corey: Except maybe Facebook, but that's a moral issue.


Tim: Oh. No lies detected, Corey. But I think what it ends up doing is it ends up just satisfying the interviewer, which is a whole other problem in and of itself. I will say, though, that I will ask very upfront now, what does the interview process look like? How long is it? 


Are there going to be, you know, a coding test, whiteboarding, like that? Now, and at this point in my career, if someone's going to have me up there whiteboarding for an hour, I'm going to pass because I don't have to at this point. And I've worked very hard. I've done a lot of whiteboarding to get to this point, but I don't have to anymore. It is an antiquated practice. People that suffer from various forms of anxieties, ADHD, or other kinds of neurodivergentcy at all, whiteboarding can be crippling and is not a good test of their ability to do a job, unless your job is to actually code on a whiteboard in front of strangers.


Corey: And it's awful. The last time I was subjected to that I successfully pivoted the interview with a, “Look. What is it you're actually trying to uncover?” “Well, whether you have technical competence.” It was for a DevRel style role. “Cool, why are you talking to me?” “Oh, everyone loves your work, and we're highly aware of your technical competence and your authenticity with developers.” “Cool. So, what is the purpose of this again?” 


And it turned into a let me show you some code I wrote, and then I can make fun of it for you and see if you find that more engaging than me implementing some API I've never seen before. And sure enough, it worked out super well. But it's this entire idea of what are you actually trying to uncover. Becomes incredibly annoying.


Tim: Exactly. One of the interview styles that I like the most is probably the narrative style interviews where you want me to tell you a story, and from that story, you're going to glean certain things about my personality, certain things about my ability to troubleshoot, certain things about my ability to own a task, certain things about my ability to break something down or explain something, but you're going to want me to tell you a story, versus answering questions. And when I find, if I let people talk and tell a story in their own voice, I can figure out what it is they're trying to communicate to me fairly well. And maybe I'll ask some questions to dig in a little bit if I want to get a little more meat on something. 


But I still essentially want them to do most of the talking: I want them to tell me about themselves; I want them to tell me about their experience in whatever way makes them feel the most comfortable. Because in the end, that's what you're going to be doing. If someone feels comfortable when they're working, they're going to be able to get across what they're thinking, what they want to do, how they can help. And that's what people should be looking for interview, versus Jeopardy.


Corey: I wholeheartedly agree with your position on this. But let's move on a little bit. I'm curious as to why you would go to work for Equinix after spending as much time as you have in, shall we say, the hyperscaler cloud world. It feels on some level like that's a very strange direction. Which based upon my understanding of you, is a near certainty that there's something I’m very clearly missing.


Tim: So, I will say that I first started off—because I've been in it for so long—in the days before public clouds, in data centers, and I've still, for the first few years of using public clouds had to break myself of the kind of mentality that most people have when they're working in data centers. I will say though that Equinix Metal—and Equinix as a whole—are moving in a direction that you would not typically expect from a data center provider. They are moving in a direction that is much more what you'd expect from a new up-and-coming cloud provider. Talking about trying to be more elastic and more agile, quicker deployments, using API's versus sending in a ticket to a human to do things, but also moving in a direction where they understand that you're going to have presence in a public cloud, or several public clouds, and how can I tie all these assets together? And that's what's fascinating about the whole thing because I know, having worked in public held long enough, that there are a lot of workloads and a lot of use cases that maybe you wouldn't want to run in a public cloud.


Corey: Well, tell me a little bit more about that because I would agree with you, but the answer I would give to that question in 2015 versus the answer I would give to that question in 2021 are radically different, and honestly, much smaller now than they used to be. It turns out, the Cloud got better. Anything that required deterministic performance was a no-go back then, well, that kind of changed. A lot of HPC workloads weren't effective. Well, now they kind of are. What am I missing?


Tim: So, I think the biggest thing that I see and something that will probably appeal to you, knowing you to be the person that you are, is that when you are running in a data center or when you own your equipment, your costs are very predictable. And that's something that you will typically don't see in a public cloud. If I need to spin up something real quick and in a hurry—I mean, I can cost-cut with Spot; I can cost-cut with Reserved Instances if I've made reservations, or I have an idea of what it's going to be in on-demand. But if everything I'm going to spend is unknown until the end of the month, that can be a little harrowing.


Corey: And let's not kid ourselves either. The things we spin up in a hurry, tend to be load-bearing forever. 


Tim: Yeah. [laugh].


Corey: One of my production tables’s in DynamoDB, and my serverless application is called ‘clicktracker-test.’ Yeah, funny how that works out. One of the other tables has the word ‘dev’ buried in it because I make terrible decisions, and is it working? Great. Am I going to deploy it clean? Of course not. Should I? Yes. Will I? No because I'm bad at things.


Tim: Well, there's so many long-running production workloads that are still running off of some developer’s credit card right now. So, there are decisions that we make on the spur of the moment, without a lot of, necessarily, planning, that we end up sticking with—like you said—for that very reason. How many POCs or MVPs are still basically what people are running now? But what you end up with as an unpredictable cost. You can forecast it pretty well at this stage, but it's not predictable. 


It's going to change month-by-month, year-by-year in ways that are not definitive. Versus making CapEx, where you buy an allocation of hardware and that's what it's going to cost you. The other thing I think, too, that really gets people on using public cloud is your bandwidth transfer. I know you've seen that before. It’s like—


Corey: Oh my god, yes. It's the Achilles heel across the board. There are still some workloads that are heavily based on that, that are not an appropriate fit for the Cloud. Look at Netflix, for example. They're very public about the fact that they don't stream anything from AWS. 


They have their Open Connect custom CDN that they built out themselves. Given who they are, that makes perfect sense. But even imagining the heavily discounted pricing—and yes, anyone who thinks that something the scale of Netflix is paying retail pricing for cloud for anything is missing something very key here—it winds up being a complete difference. It's just a non-starter. Even extortionate levels of discount still make it untenable.


Tim: And it's so wildly unpredictable. Because you can have a workload that, “Well, we know what our compute’s going to look like, and we can predict what infrastructure we're going to need for traffic,” but if your traffic spikes and your bandwidth usage goes way up, that's going to be a bill you weren't expecting. Every time. And I've seen that sticker shock from people; like, “I don't understand how our AWS bill went up $45,000 this month.” And it was all in data transfer. 


So, there's a certain amount of predictability around being in a data center on those things, especially within Equinix, that appeals to a lot of people. I do think, too, if you need special custom hardware, you can get similar configurations in public cloud providers, but it's still going to be an approximation. If you really want to be able to turn all the little dials that you need for your specific workload, that's still going to be something that you're going to need to have something on-metal to do.


Corey: True. Now, let's be very clear here; I do want to call things out. I am objective, and that is perceived as authenticity when in reality, I'm just a jerk. But if I pull up and click around in the Equinix website, the bandwidth pricing there—at default egress—starts at five cents a gigabyte, which, sure, it's better than the nine cents that the big providers are charging, but that still is non-trivial. Oracle, for example, starts at below a penny a gigabyte, which is, okay, that feels on some baseline level to be directionally correct. All the big hyperscalers all feel like they're still charging 1998 prices for bandwidth, and it drives me nuts.


Tim: Mm-hm. Well, because they can. There's no incentive for them to charge lower, even though AWS will say, “Well, we lower our prices on this all the time.” And yeah, they do; they're very good at lowering prices on some things, but they're still going to keep that bandwidth pricing, like you said, at that 1998 price.


Corey: “We have lowered the pricing on this type of instance that you don't use in a region you didn't know existed. We're counting that as another price cut.” At some point, it’s come on. It's this one of those things that actually matters to people or not?


Tim: No. No, it doesn't. But it looks good. It makes for good marketing material, so that's how it happens. When folks say, “Oh, we're going to go on AWS and we're going to save money in the long run because we're going to get a EDP and we're going to get a private pricing agreement, and they cut prices on their stuff all the time.” 


But you signed an EDP and like—or the private pricing agreement—and you're like, “Oh, we still have to grow twenty percent year-over-year.” Well, can you imagine signing an EDP in 2019, and you're a business that makes its money off of a service industry or people going out places?


Corey: I don't have to imagine it. This is actually what I do as a consultant. I have customers who were hurting and how do they wind up restructuring those EDPs and PPAs? Contract negotiation with AWS is one of the things that we find ourselves doing a lot more than I thought we would when we started doing it. And it leads to great outcomes on both sides of it, but yeah, it turns out that there are hiccups in the road, that the growth is not always hockey stick upward and to the right. The real world is messy, and it happens. And we talk about architecture aspirationally, and then we talk about the architecture that we have here in reality, and one of those things is not nearly so shiny as the others.


Tim: It's funny, when I would have architecture reviews with customers, the first thing I would say is, “Tell me what your architecture looks like now. Let's talk about what you could make your perfect architecture look like and let's find out where in the middle we can land, actually.”


Corey: Oh, absolutely. I've been saying for a long time that you can design a working architecture to solve a customer's problem on a whiteboard pretty easily. That's something that anyone with a baseline level of certification or experience can wind up doing pretty effectively, and it'll work. But the challenge is, all right, where are you now? How do you get to that future state? 


And what is the benefit of doing it? Because “It'll save us money in the longer term,” is surprisingly, uncompelling as a reason to undertake a project like that. There has to be a value proposition that goes beyond pure cost savings or it's unlikely to get done.


Tim: It's true. You're going to run up against either, the bean counter—there's always going to be a bean counter of some kind, whether it's engineering hours, or whether it's going to be actual money—as to say whether we can do this or not. And that's where you have to make those kind of proposals lucrative. You have to find out what motivates them to say, okay, well, this is now worth it, addressing whatever concerns I had. And that's hard. 


But that's almost never a technical problem. That's almost never a technical problem. It's almost always resolving somebody's fears or resolving someone's insecurity, or sometimes stroking somebody's ego; more times the ego-stroking than I care to admit. But once you get there, then you can finally say, okay, now, we have enough buy-in to be able to make this digital transformation, or whatever it is they want to call it.


Corey: The big challenge that I'm seeing across the board is, we just had re:Invent happen, and I want to talk to you about that in a minute or two, but the challenge that you see is these companies get on stage, and they talk about how amazing they are and how their transformation has worked. And you look at that on stage, and then you look at your own environment, and you feel bad as a direct result. And it's almost like it's conference-ware on some level. In practice, no one is ever completely honest when they're on stage at a conference talking about their architecture. They're talking about a particular workgroup, or they're talking about a proof of concept. It is very rare that you'll see someone get on stage and say, “This is how it works across our organization,” and be accurate about it. Because most things are inherently brownfield or there's legacy that has to be accounted for. Legacy, of course, being engineering-speak for, “It makes money.”


Tim: I think it's interesting when we talk about what it takes to get—especially these large enterprises—to make these kinds of changes, and the thing that they don't tell you on the stage is, “Also, AWS threw us a million dollars worth of credits.”


Corey: Oh, I love it when people wind up making economic decision based upon credits. “Oh, they'll pay for the migration with MAP credits.” Or, “They'll give us a whole bunch of things through Activate.” And that's great and all, I appreciate the value, but things get bigger. I mean, I was astounded, relatively recently, to discover that with the DuckTools launch of our SaaS offering here, as well as the experiments we're running and the stuff we're doing for our customers, we're now a $25,000 a year AWS bill. 


And that took me aback on some level. Now in practice, it doesn't really matter. That's two grand a month and change. And compared to the cost of running this place, no one cares. But it keeps going up. It only ever goes in one direction. 


Eventually, we're going to care, and we're very well suited for when that day comes, but by the same token, it's also the cobbler’s kids have no shoes style of story. Yeah, optimizing our bill is always going to be a distant second place to optimizing our clients’ bills.


Tim: Yeah. And as business goes, almost everyone does it the same way. What I do think is important to note is that when you talk about being on AWS and that bill is always going to go up, you don’t, sometimes, even have to do anything. You just have to exist. Your bill will go up over time just because you're storing more objects in S3, right? 


Because you're storing more EBS volumes. And if you don't do those optimization tips that we both talked about the past, then you will never save money. You can save money even if you are growing, but you have to be very purposeful about it. You have to be very intentional about it. You have to do—you actually have to build cost savings into your architecture; it can't be an afterthought because you're never really going to make anything happen. 


If you have an architecture—and then I've worked with folks before where—one of my old customers runs advertisements and monetization for game ads. And a year ago, a year and a half ago, they pushed to change their workload into a very stateless workload that can be run on Kubernetes and in Spotinst. And so this allowed them to have very, very well-orchestrated Spot instances that was easily 95 percent of their infrastructure. So, that way, when the pandemic hit and a bunch of more people were home, and they were playing games—like, kids were playing games, adults were playing games like that—and their usage tripled. Their AWS bill only took a little bit of a hit up because they had built cost optimization into their architecture before they needed to.


Corey: Absolutely. Strongly agree. I mean, the sad part for us was during the pandemic when we saw folks have their user activity drop off a cliff, and their infrastructure spend remains stable. People talk about elasticity, but whether they're living it or not is really a subject of some debate. Many applications still require, if you add or remove a node from a cluster, all the other cluster members need to be made aware of that. That's not a great candidate for auto-scaling as it turns out.


Tim: It is very painful to watch people try to turn things down and realize that they cannot. 


Corey: Oh, yes. Because historically—and I understand why. It made sense. If you scale up rapidly, great, you're going to be able to solve for that problem because if you don't, you're dropping customers on the floor. If you fail to scale down, well, you're just spending a little extra money, and that is never as big of a deal as it could be. So, it's always a trailing function; it's always something people think about after the fact. And the consistent lie we always tell ourselves is after this sprint ends, we're going to start doing everything the right way and clear on technical debt. 


Tim: [laugh].


Corey: Yeah. Sure, you will.


Tim: Oh. It's like when people clean up the garage, like, “Oh, yeah, we're going to get in there, we're going to throw all this stuff out, and then the garage will be clean and we’ll be able to put both cars in here.” And either you've always been that way, or it's never going to happen. I've never seen anybody who had a dirty garage that's cleaned it out and then from that point forward can always put both cars in the garage. It just doesn't happen.


Corey: The only way it happens is if you wind up, effectively paying someone to do it for you—


Tim: Exactly.


Corey: —which wasn't the metaphor I was intending to go with, but that seems to be the way that I've experienced it. My home office was always a mess until I started paying someone to clean it up behind me. And that's helpful.


Tim: Do you mean paying someone, like, maybe The Duckbill Group to come in straighten out your—


Corey: It wasn't the direction I was going in, but… and again, it's one of those stories of, what is the actual impact a customer is looking to have, it's never the, “Oh, just pay us and all your problems go away.” It very rarely is that simple. But it's a start, and it highlights what you could do, what you should do, what we would do in your shoes. But it's varied; there's a reason that we do this as a consulting service and we're not just selling software in isolation. We're selling some power tools that we do use to solve very specific problems, but we're not doing this as a ‘sign up for SaaS and never worry about your bill again,’ because there's no API for business insight. 


Tim: That’s true.


Corey: There's no context that you can glean programmatically to avoid recommending terrible things. And after a few terrible recommendations, people stop caring what you have to say.


Tim: I think that’s something that I talked about when I was working at Mission and something that I like to carry forward in any role: when you talk to customers—what I like to tell people who are new to customer-facing roles is that people can get AI recommendations for basically anything now. People can get machine learning and some kind of algorithmic recommendation for basically anything at this point. But what a machine cannot really effectively give you, right now at least, is insight based on your experience. And if you have a wealth of experience, and you've been able to go back and examine it, especially when the experience is making your own mistakes, right, of which I am very, very versed in, if you don't have that experience, you cannot say to others, “I know what the numbers say, but there's more to the story and here's why.” When you can do that, people will pay you for that because you are going to keep them from making mistakes. 


You are going to keep them either from spending more money because they haven't optimized or from spent more money because something bad happened and now they have to pay out a bunch of customers. So, when you talk about a consultancy, you're talking about giving customers—you know, people paying you for you telling them things to do, right? That's important. The ability to say that, “Hey, I have been there, and this is what I have seen, and this is what I see for you coming forward if you don't handle this.” I mean, that's a big thing. 


I think the garage cleaning analogy works, especially for us because oftentimes it's way easier for us to throw out somebody else's stuff than it is for us throw out our own. If I'm going into the house and I'm going to clean up the garage, if you don't tell me something absolutely has to stay, I'm going to throw it away. But if I'm going there, you're going into my house—or if I'm going to my house, I will find it, “Oh, I can't throw this away. My girlfriend from my sophomore year of high school, you know, I was wearing this when she looked at me the first time, so I got to keep it.” You know, like, all these little reasons why that you cannot really downsize things when it's your own.


Corey: Right. This is the big problem I see, [00:29:08 unintelligible] conference-ware where it's this idea of, here's what you could do if you wound up approaching it—the same sort with reference architectures, same story with the ‘hello world’ style stuff. The real world is messy, and well, there's context that means I can't safely do that for a variety of reasons. Sure, you can sit there and say that's because my process and culture are shitty, but that process or/and our culture also generates a sizable pile of revenue. So, how do I fix this? And meeting people where they are, rather than shaming them into meeting you where you are—as a vendor—seems to be something that all of the major cloud providers have been focusing on for a while now.


Tim: Yeah, I think it's interesting. We talked about it in the context of—you mentioned re:Invent before, we're talking about cloud providers. One of the things that cloud providers spent a long time doing was telling you that you shouldn't be on-premise and you should go to public clouds and here's various thousands of reasons why. They would do everything they could to get you off of being in a data center, being on metal, into public cloud. And if you notice at this year's re:Invent, they talked about putting a bigger emphasis on AWS Outposts, they talked about EKS Anywhere, ECS anywhere, they talked about the open-source distro for EKS you can run anywhere. 


And they mentioned a couple things about migrating databases back and forth, and things like that. And what I think it was is an acknowledgment that people are still in the data center, will continue to be in the data center or will be expanding into the data centers for various reasons. And so instead of trying to fight against that, they are now trying to make sure they can still make their money while people are doing that. Which is brilliant, number one, but number two, it is actual customer obsession because the customers are telling them, “Hey, this is what we want to do.” And AWS is saying, “Okay, that's fine, but we still want your money.”


Corey: Yeah, again, there's a cynical perspective, and there are uplifting perspectives to take on it. But I don't work for AWS. I’ll let them articulate what exactly this [laugh] they mean on this. I am curious as to what you saw coming out of re:Invent. What did you take away from the conference? And by extension, what did we learn about ourselves from all of last year, both as individuals and as a society?


Tim: The thing I took away from re:Invent was they literally put a big emphasis on large customers—enterprises—reinventing themselves technologically, [00:31:25 crosstalk].


Corey: Because that's where the money is.


Tim: Oh, sure. Sure. And they had a couple of startups, but the biggest thing was, they were talking about how large companies changed to meet the demands of 2020. And there were a lot of ways that large companies had to do some innovation. They talked about, was it—I think the bank—I think was a Capital One, they cut that four-hour wait time for folks to talk to an agent to, like, ten minutes based on doing some serverless functions into Lambda stuff, which I thought was brilliant. 


And that's something that everyone can understand. If you got laid off from your job, you got a pay cut, and you need to talk to the bank about something, can you imagine having to wait four hours on a weekday to talk to somebody about something super important, like, you know, mortgage payment, or bills, or something like that? That would be nightmarish. That is a real effect. That is a real thing that affects real people. 


And I think is important. A lot of things that we do in tech, people don't really know, people don't really feel. You know, like, if you say, “Hey, I reduce response time on this API.” Well, that's all well and good, but most people are never going to know that. But if you said, “Hey, we did a lot of work so that you don't have to wait four hours on the phone to talk to the bank.” 


That's something people were going to notice. And I think that's important. I do think that the overall theme of people changing and doing things differently through AWS technologies to meet the demands of 2020 was timely. I think it was inspiring in a lot of ways, but I think a lot of it, they're also kind of telling on themselves a little bit that maybe—the unwritten thing is, maybe the things that they were doing in the past aren't going to serve them going from 2020 on. The answer, the other questions like what do we learn as a society, and as individuals in this year? I think is—


Corey: Yeah, versus what should we have learned as a society, which we're going to stay away from because that is way too depressing.


Tim: Yeah. Here's what I think we learned: I think we learned, number one, that our cultures, our work cultures, and work organizations are nearly not as smoothly designed as maybe we thought they were and that they rely too much on people being in proximity, one with another, to exchange information, or to find information more importantly. I think that we realized that we don't document enough things. I think that we realized that things that we do you have documented are too hard to find. I think that we realized that we don't really know how to talk to, or check in on, or check in with people if we are not physically in front of them. 


And I think that a lot of companies that were remote-first companies had a huge advantage over companies that were trying to pivot to being remote-based companies. And what you saw were some companies did it really well, and some companies kind of didn’t. And so those companies, you'll see where they're bringing people back as soon as they can. You know, so whether they should or shouldn’t, that's a different discussion. 


But I think that's one thing we discovered about ourselves; we really learned, how is our culture? How is our company culture? How is our organizational culture? How are our practices in dealing with people who have real concerns? Without getting on the soapbox too much, but something I mentioned in the past is like, how do we address our employees’ real-life needs as a company? 


Whether it's our female engineers, are we making sure that they're supported, if their mothers, so they don't have to leave the workforce? Are we making sure that people have time for mental health breaks instead of just scheduling them from meeting to back-to-back meeting for 10 hours every day for four weeks straight? Hey, are we checking in on our people just in general? Are we making sure our people—hey, do you have the power on? Do you have high-speed internet at home?


Corey: Even if you are full remote, as we were when we started this whole thing, it's not the same. And if wow, it turns out our company can't ever do remote work. Look how terrible that year was. Yeah, not really a fair test, if I'm being very direct. This year has been remarkably different. 


Tim: It has been extremely different. And I think even for us, people who were full remote beforehand, we were able to still have the advantage of, every couple of weeks, couple months, being able to go and fly somewhere and see all the people that we talk to on Twitter, and have fun, and exchange pleasantries, and actually have real physical contact with these people. And you don't have that this year. And that's changed a lot of jobs. Like people who do DevRel, their jobs changed, I think, dramatically in 2020 because they're not doing conferences like they were. 


Conferences have changed. The value-add for conferences has changed quite a bit where people are realizing that, do I really want to pay $1500 to fly to a hotel to hear people just give a talk, or would I rather hear people discuss panels or talk to people in the hallway tracks? A lot of people have missed the hallway tracks. A lot of people have missed interacting one with another, exchanging ideas completely off the cuff about something, and just watching where it goes. Just sitting there and having a person just talk on the stage doesn't seem as valuable, I think, anymore because people are sitting there watching videos of people doing the same thing. And it's maybe not as fun. And I'm not saying that's universal across the board. Some people are very compelling speakers—


Corey: The entire world is defined by exceptions. That's fine. No argument here.


Tim: Yeah. But I do think, by and large, people now are realizing the value of sitting around, in a group of people, in a room talking about this thing that comes up and just letting that idea take root, and electrify people, and write stuff down. And I've been at conferences that have started a conversation with someone in the hallway that ended up being a product in two months. And that's the kind of stuff that people pay for. That's the kind of enrichment that you have. 


And yeah, the vendor booths and swag are one thing, having been both a participant and sponsor at a conference, the vendor experience is quite different, and I think that that needs to be rethought how that gets done going forward. And I do think another thing that needs to be rethought is, who are we bringing to the conferences? If you noticed, a lot of conferences were free, and people that could otherwise not attend because they couldn't afford, either personally, or for the companies to fly them to someplace and put them in a hotel—you know, it's always going to be fly into some very expensive place and put them in a very expensive hotel, to listen to—


Corey: And it’s people who can basically justify being not at work for a week for the duration of that conference. 


Tim: Oh, absolutely. So, all these combinations mean that only a very few people in an interest group would be able to actually participate. But now when you say, “Well, you can do it from your workday, you can pop in the session, then work some. It doesn't cost you anything, and you can literally get to it from anywhere.” Now I saw more people—different types of people—actually being able to participate in some of the activities. 


Mind you, they were different than a conference was before, but I do think what you're seeing is that maybe we need to find a different way of doing this so that we can get more people into these things; you get more people into these things, you get more thoughts, you get more products, you get more interactions. That's going to be a net benefit for tech as a whole, for whatever user group, or special interest group, or specialty that you're talking about. The more people you can get in there discussing more ideas, the better that's going to be.


Corey: I absolutely agree with basically everything you've just said. And for better or worse, there's an awful lot of things we need to learn as we move forward as a society. And hopefully, we'd look back on this with some silver lining we can take out of it, otherwise, it just becomes this entire year of nonsense. 


Tim: Yeah.


Corey: And that's too depressing to really contemplate.


Tim: Yeah. If we go into 2021, when it becomes quote-unquote, “Normal” again and we're all going back to work, whatever that looks like. And we looked exactly the same that we did in 2019, then all the shit that happened this year was for nothing. And that would be the greatest tragedy of it all, if we didn't learn anything, we take anything away. I would really like to see the overall health of people—physical health, mental health, you know, what's your burnout level, what's your home stress level—become a more important focus for these businesses. 


Because if you have a thriving individual there, that person is thriving, and that person is healthy, and that person is happy or as happy as they can be on whatever their circumstances are, they're going to be a better and more productive employee than they would be if they were [BLEEP] miserable.


Corey: Yep. Wholeheartedly agree.


Tim: So, what does this have to do with tech, Tim? Because I've had that question, too. It's like, well, when you're in your stand-up meeting, and you're talking to somebody, and you notice that hey, maybe they sound down, maybe they're a little off, maybe something's not right with them. Take a second and like, you know, “Hey, person, let me get five minutes after work.” And just check in on them. 


See how they’re doing, and see if there's anything you can do. Because the real benefit of having people managers—versus tech and engineering leads—is that they're going to address things that aren't necessarily technical but that are good for org, good for the business, good for the employees. Like, if you handle somebody’s pay problems, if you handle somebody’s insurance problems, and certainly you can maybe handle—it’s like, “Hey, you need a day off and I noticed you haven't taken any PTO in three months. Why don't you take a couple days. Something. Get away from the thing. Let me switch around some on-call so you can have a break.” These are things that you should be doing as people managers. People management is a skill. It may be it's not a technical skill, but it is a skill that is very important to technical work.


Corey: It absolutely is. And I think that people lose sight of that at their own peril. And once again, we've had a terrific conversation on this show. If people want to learn more about you and who you are, what you do, what you believe, what you're working on, et cetera, et cetera, where can they find you?


Tim: They can find me on Twitter @elchefe: E-L-C-H-E-F-E.


Corey: Fantastic, and we will of course throw links to that in the [00:41:05 show notes]. Thank you so much for taking the time to speak with me. Every time we have one of these conversations. I always wish we could go longer.


Tim: I appreciate it. Corey, I'm always glad to come back anytime.


Corey: Don't worry, I'll take you up on that. 


Tim: Awesome.


Corey: Tim Banks, currently a principal Solutions Architect at Equinix. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts, whereas if you hated this podcast, please leave a five-star review on Apple Podcasts along with an ignorant comment telling me why I should get over the high cost of data transfer egress pricing from cloud providers.


Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com, or wherever fine snark is sold.


This has been a HumblePod production. Stay humble.








Episode Show Notes and Transcript
About the Tim Banks
Tim Banks is currently with Packet, an Equinix Company, where he is a Principal Solutions Architect. His tech career spans over 20 years through various sectors. Tim’s initial journey into tech started as a US Marine, having originally joined the Marine Corps to be a musician. He was later reassigned into an avionics specialty based on the results of standardized testing. Upon leaving the Marine Corps, he went on to work for hardware manufacturers and defense contractors as a civilian. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for large Unix-based datastores.


Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with engineering groups in his current role. Tim is also a husband and a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and Pan American Brazilian Jiu-Jitsu champion in his division.


Links Referenced: 

Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.


Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined once again by Tim Banks, by popular demand. We spoke last year and it was fantastically well-received to the point where everyone demanded more of him. And you're back, Tim, in a new year with a new job. You were at Mission Cloud, and now you've joined Equinix Metal as a principal solutions architect. First, congrats. Secondly, what gives?


Tim: Well, thank you, Corey. I'm glad to be here, and I appreciate it. So, I spent some time in the TAM world, both at AWS and at Mission, and I made the jump into solution architect—actually principle solution architect at Equinix Metal. And I think there's a couple reasons why I made that jump. 


First and foremost is that I still wanted to be in a place where I'm talking to the customer because that's where I think I can excel in both helping the customer and helping my company because I have an engineering background, I have engineering chops, but I also know how to talk to people and more importantly, listen to people and figure out what they need and figure out what they want, and then try and find a way to get those two things to happen. So, sticking within that role versus going to, like, a people management role, or versus going to some other kind of role, I think it was very important to me. The other thing that was very important to me were dollars.


Corey: Oh, yes. Hopes, dreams, and mission are all very important, but I can pay exactly none of my rent with those things.


Tim: No, I wish I could, with best intentions, but they don't translate well to keeping the lights on. You know titles matter, so to get the principal title was very important to me because I've done the work, so to be able to have the title and then the financial backing that goes with that title was extremely important to me. People have asked, “Well, you were only there for a year,” or a little more for a year, and, “What gives?” And the fact of the matter is that for many roles, for many people, and in many companies, you're going to get a far better raise by changing companies and changing jobs than you will by just trying to go through the normal internal promotion track. There's a lot of reasons for that, they vary from org to org, but the numbers don't lie. 


There are a lot of people that have a career that looks like that, where you are staying at a place one to two years, and then you change jobs or change companies and there was a significant pay raise that went along with that. And if you look at what the pay raises that come with the internal promotions and review periods and stuff like that, they never match what you'll get by going to another company. I wish to say they would. I wish that companies would do more investment in keeping their people around, but that's just not the case.


Corey: Oh, I'm right there with you. It's one of those I'm lobbying for a three percent versus four percent raise after I've been somewhere for a year versus, well I can get twenty percent if I change jobs. And partially that's because my skills have changed and improved, presumably. But at some point, this obviously does cap out, but if I'm talking to someone else, and I'm able to get a few tens of thousands of dollars in raise, then why not? It feels like it's absolutely one of those things that lends itself to just pure self-interest if nothing else.


Tim: Absolutely. And so I have a strategy behind it. And so every year, on my year anniversary date or so, being with a company, I will look at other jobs and I will take some interviews. And I'll do that for two reasons: the first reason is because searching for a job and interviewing is a skill. It is absolutely a skill, and if you don’t—


Corey: Oh God, yes. And it evaluates you on a list of things that you really only bring into practice when you're looking for a job. So, if you've been somewhere for eight years, that skill has atrophied massively, and you're being judged based on that ability.


Tim: And it's atrophied massively, and the problem is too many people are exercising that skill when they absolutely need to instead of exercising it when they don't have to. It's like do you want to wait to run a mile when you're being chased by wolves, or do you maybe want to run a mile every so often to make sure you can, and then if you do get chased by wolves, you're very comfortable running that mile. If you are not interviewing; if you don't know how to interview; if you don't know what interviews look like now; if you don't know what questions they’re asking; if you don't know what skills they’re looking for; if you don't know how your resume supposed to be [00:06:09 unintelligible]; if any of these things, and then heaven forbid you get laid off, or for some reason like that, and then now you have to interview under duress to make your mortgage payment and to feed your children, that's a lot more stress, combined with being unfamiliar with what you've got to do. So, just as a means of practice and as honing my professional skills, I will go out, and look for, and interview for jobs every year.


Corey: I strongly endorse the entire approach.


Tim: And now the second reason I do that is because when I do go in for my annual review, for my performance review, or whatever, I want to know what the value of my skill set is to outside companies. And that way, when it comes time for review, and the company I'm working for now says, “Hey, well, this is what you are worth to us.” And that's what they're doing when they give you your raise or evaluate for whatever annual raise, or performance rate you’re due for, they're saying, “This is what your job is worth to us. This is what the value of your work to this company.” And if my work is more valuable to another company, I will bring that up to my current employer, and if they don't see that the value is the same, then I think we've reached that point, an amicable parting of the ways.


Corey: I agree wholeheartedly with everything you just said. I've gone on quote-unquote, “Practice interviews” past in my career, and either wound up hiring the interviewer, or that practice interview suddenly became very real, and I took a job there for a variety of different reasons. I don't have many regrets about my current situation in life, but one of them is that given that I own the company, it's very challenging for me to reliably and in good faith do that because it first sends a message about the company that isn't great, and two I can't imagine what something that would lure me out of here—that wasn't an acquisition—would look like. And I don't want to waste people's time and cast those doubts into the ecosystem. But I really enjoyed aspects of the job interview process.


Tim: Yeah, I think the one thing that I take away the most from the job interview process is figuring out how to present my skill set to different groups of people. So, I can say that, “Yes, I know containers, I know best practices for engineering, I know DevOps, I know SRE, I know all these languages, I know all these automation methods.” But how do I convey that to someone in the manner that they're asking the questions? If I interview for a startup folks from Silicon Valley, those questions are going to look a lot different than if I'm interviewing with someone like Amazon, or Google, or Facebook, or something like that. They ask questions differently, they look for different answers to find the same underlying set of skills. And so you have to learn how you can convey your skillset and how you can convey what other qualities have to different people who are asking the same question different ways.


Corey: Absolutely. It's also strongly valuable to understand this is a form of corporate hazing, for lack of a better term. In practice, you are never going to need to invert a binary tree or implement quicksort yourself. That's what lunatics would do. 


But people love asking those questions, and it's more or less a, do the CS students’ secret handshake to pass this and show that you belong here. Which is deeply problematic on a number of different levels. But that's where it seems like it goes, and I've never been a fan of the trivia questions. It still feels on some level like these companies are interviewing for a lack of weakness rather than hiring for strengths.


Tim: Yeah. It is, in its very core, just gatekeeping. And not gatekeeping to say, “Oh, we have a standard and this is what's done,” because as you mentioned, most of these have nothing to do with the actual job you’re going to perform.


Corey: Yeah, as opposed to, what, all those companies that have no standards? Please. 


Tim: [laugh].


Corey: Except maybe Facebook, but that's a moral issue.


Tim: Oh. No lies detected, Corey. But I think what it ends up doing is it ends up just satisfying the interviewer, which is a whole other problem in and of itself. I will say, though, that I will ask very upfront now, what does the interview process look like? How long is it? 


Are there going to be, you know, a coding test, whiteboarding, like that? Now, and at this point in my career, if someone's going to have me up there whiteboarding for an hour, I'm going to pass because I don't have to at this point. And I've worked very hard. I've done a lot of whiteboarding to get to this point, but I don't have to anymore. It is an antiquated practice. People that suffer from various forms of anxieties, ADHD, or other kinds of neurodivergentcy at all, whiteboarding can be crippling and is not a good test of their ability to do a job, unless your job is to actually code on a whiteboard in front of strangers.


Corey: And it's awful. The last time I was subjected to that I successfully pivoted the interview with a, “Look. What is it you're actually trying to uncover?” “Well, whether you have technical competence.” It was for a DevRel style role. “Cool, why are you talking to me?” “Oh, everyone loves your work, and we're highly aware of your technical competence and your authenticity with developers.” “Cool. So, what is the purpose of this again?” 


And it turned into a let me show you some code I wrote, and then I can make fun of it for you and see if you find that more engaging than me implementing some API I've never seen before. And sure enough, it worked out super well. But it's this entire idea of what are you actually trying to uncover. Becomes incredibly annoying.


Tim: Exactly. One of the interview styles that I like the most is probably the narrative style interviews where you want me to tell you a story, and from that story, you're going to glean certain things about my personality, certain things about my ability to troubleshoot, certain things about my ability to own a task, certain things about my ability to break something down or explain something, but you're going to want me to tell you a story, versus answering questions. And when I find, if I let people talk and tell a story in their own voice, I can figure out what it is they're trying to communicate to me fairly well. And maybe I'll ask some questions to dig in a little bit if I want to get a little more meat on something. 


But I still essentially want them to do most of the talking: I want them to tell me about themselves; I want them to tell me about their experience in whatever way makes them feel the most comfortable. Because in the end, that's what you're going to be doing. If someone feels comfortable when they're working, they're going to be able to get across what they're thinking, what they want to do, how they can help. And that's what people should be looking for interview, versus Jeopardy.


Corey: I wholeheartedly agree with your position on this. But let's move on a little bit. I'm curious as to why you would go to work for Equinix after spending as much time as you have in, shall we say, the hyperscaler cloud world. It feels on some level like that's a very strange direction. Which based upon my understanding of you, is a near certainty that there's something I’m very clearly missing.


Tim: So, I will say that I first started off—because I've been in it for so long—in the days before public clouds, in data centers, and I've still, for the first few years of using public clouds had to break myself of the kind of mentality that most people have when they're working in data centers. I will say though that Equinix Metal—and Equinix as a whole—are moving in a direction that you would not typically expect from a data center provider. They are moving in a direction that is much more what you'd expect from a new up-and-coming cloud provider. Talking about trying to be more elastic and more agile, quicker deployments, using API's versus sending in a ticket to a human to do things, but also moving in a direction where they understand that you're going to have presence in a public cloud, or several public clouds, and how can I tie all these assets together? And that's what's fascinating about the whole thing because I know, having worked in public held long enough, that there are a lot of workloads and a lot of use cases that maybe you wouldn't want to run in a public cloud.


Corey: Well, tell me a little bit more about that because I would agree with you, but the answer I would give to that question in 2015 versus the answer I would give to that question in 2021 are radically different, and honestly, much smaller now than they used to be. It turns out, the Cloud got better. Anything that required deterministic performance was a no-go back then, well, that kind of changed. A lot of HPC workloads weren't effective. Well, now they kind of are. What am I missing?


Tim: So, I think the biggest thing that I see and something that will probably appeal to you, knowing you to be the person that you are, is that when you are running in a data center or when you own your equipment, your costs are very predictable. And that's something that you will typically don't see in a public cloud. If I need to spin up something real quick and in a hurry—I mean, I can cost-cut with Spot; I can cost-cut with Reserved Instances if I've made reservations, or I have an idea of what it's going to be in on-demand. But if everything I'm going to spend is unknown until the end of the month, that can be a little harrowing.


Corey: And let's not kid ourselves either. The things we spin up in a hurry, tend to be load-bearing forever. 


Tim: Yeah. [laugh].


Corey: One of my production tables’s in DynamoDB, and my serverless application is called ‘clicktracker-test.’ Yeah, funny how that works out. One of the other tables has the word ‘dev’ buried in it because I make terrible decisions, and is it working? Great. Am I going to deploy it clean? Of course not. Should I? Yes. Will I? No because I'm bad at things.


Tim: Well, there's so many long-running production workloads that are still running off of some developer’s credit card right now. So, there are decisions that we make on the spur of the moment, without a lot of, necessarily, planning, that we end up sticking with—like you said—for that very reason. How many POCs or MVPs are still basically what people are running now? But what you end up with as an unpredictable cost. You can forecast it pretty well at this stage, but it's not predictable. 


It's going to change month-by-month, year-by-year in ways that are not definitive. Versus making CapEx, where you buy an allocation of hardware and that's what it's going to cost you. The other thing I think, too, that really gets people on using public cloud is your bandwidth transfer. I know you've seen that before. It’s like—


Corey: Oh my god, yes. It's the Achilles heel across the board. There are still some workloads that are heavily based on that, that are not an appropriate fit for the Cloud. Look at Netflix, for example. They're very public about the fact that they don't stream anything from AWS. 


They have their Open Connect custom CDN that they built out themselves. Given who they are, that makes perfect sense. But even imagining the heavily discounted pricing—and yes, anyone who thinks that something the scale of Netflix is paying retail pricing for cloud for anything is missing something very key here—it winds up being a complete difference. It's just a non-starter. Even extortionate levels of discount still make it untenable.


Tim: And it's so wildly unpredictable. Because you can have a workload that, “Well, we know what our compute’s going to look like, and we can predict what infrastructure we're going to need for traffic,” but if your traffic spikes and your bandwidth usage goes way up, that's going to be a bill you weren't expecting. Every time. And I've seen that sticker shock from people; like, “I don't understand how our AWS bill went up $45,000 this month.” And it was all in data transfer. 


So, there's a certain amount of predictability around being in a data center on those things, especially within Equinix, that appeals to a lot of people. I do think, too, if you need special custom hardware, you can get similar configurations in public cloud providers, but it's still going to be an approximation. If you really want to be able to turn all the little dials that you need for your specific workload, that's still going to be something that you're going to need to have something on-metal to do.


Corey: True. Now, let's be very clear here; I do want to call things out. I am objective, and that is perceived as authenticity when in reality, I'm just a jerk. But if I pull up and click around in the Equinix website, the bandwidth pricing there—at default egress—starts at five cents a gigabyte, which, sure, it's better than the nine cents that the big providers are charging, but that still is non-trivial. Oracle, for example, starts at below a penny a gigabyte, which is, okay, that feels on some baseline level to be directionally correct. All the big hyperscalers all feel like they're still charging 1998 prices for bandwidth, and it drives me nuts.


Tim: Mm-hm. Well, because they can. There's no incentive for them to charge lower, even though AWS will say, “Well, we lower our prices on this all the time.” And yeah, they do; they're very good at lowering prices on some things, but they're still going to keep that bandwidth pricing, like you said, at that 1998 price.


Corey: “We have lowered the pricing on this type of instance that you don't use in a region you didn't know existed. We're counting that as another price cut.” At some point, it’s come on. It's this one of those things that actually matters to people or not?


Tim: No. No, it doesn't. But it looks good. It makes for good marketing material, so that's how it happens. When folks say, “Oh, we're going to go on AWS and we're going to save money in the long run because we're going to get a EDP and we're going to get a private pricing agreement, and they cut prices on their stuff all the time.” 


But you signed an EDP and like—or the private pricing agreement—and you're like, “Oh, we still have to grow twenty percent year-over-year.” Well, can you imagine signing an EDP in 2019, and you're a business that makes its money off of a service industry or people going out places?


Corey: I don't have to imagine it. This is actually what I do as a consultant. I have customers who were hurting and how do they wind up restructuring those EDPs and PPAs? Contract negotiation with AWS is one of the things that we find ourselves doing a lot more than I thought we would when we started doing it. And it leads to great outcomes on both sides of it, but yeah, it turns out that there are hiccups in the road, that the growth is not always hockey stick upward and to the right. The real world is messy, and it happens. And we talk about architecture aspirationally, and then we talk about the architecture that we have here in reality, and one of those things is not nearly so shiny as the others.


Tim: It's funny, when I would have architecture reviews with customers, the first thing I would say is, “Tell me what your architecture looks like now. Let's talk about what you could make your perfect architecture look like and let's find out where in the middle we can land, actually.”


Corey: Oh, absolutely. I've been saying for a long time that you can design a working architecture to solve a customer's problem on a whiteboard pretty easily. That's something that anyone with a baseline level of certification or experience can wind up doing pretty effectively, and it'll work. But the challenge is, all right, where are you now? How do you get to that future state? 


And what is the benefit of doing it? Because “It'll save us money in the longer term,” is surprisingly, uncompelling as a reason to undertake a project like that. There has to be a value proposition that goes beyond pure cost savings or it's unlikely to get done.


Tim: It's true. You're going to run up against either, the bean counter—there's always going to be a bean counter of some kind, whether it's engineering hours, or whether it's going to be actual money—as to say whether we can do this or not. And that's where you have to make those kind of proposals lucrative. You have to find out what motivates them to say, okay, well, this is now worth it, addressing whatever concerns I had. And that's hard. 


But that's almost never a technical problem. That's almost never a technical problem. It's almost always resolving somebody's fears or resolving someone's insecurity, or sometimes stroking somebody's ego; more times the ego-stroking than I care to admit. But once you get there, then you can finally say, okay, now, we have enough buy-in to be able to make this digital transformation, or whatever it is they want to call it.


Corey: The big challenge that I'm seeing across the board is, we just had re:Invent happen, and I want to talk to you about that in a minute or two, but the challenge that you see is these companies get on stage, and they talk about how amazing they are and how their transformation has worked. And you look at that on stage, and then you look at your own environment, and you feel bad as a direct result. And it's almost like it's conference-ware on some level. In practice, no one is ever completely honest when they're on stage at a conference talking about their architecture. They're talking about a particular workgroup, or they're talking about a proof of concept. It is very rare that you'll see someone get on stage and say, “This is how it works across our organization,” and be accurate about it. Because most things are inherently brownfield or there's legacy that has to be accounted for. Legacy, of course, being engineering-speak for, “It makes money.”


Tim: I think it's interesting when we talk about what it takes to get—especially these large enterprises—to make these kinds of changes, and the thing that they don't tell you on the stage is, “Also, AWS threw us a million dollars worth of credits.”


Corey: Oh, I love it when people wind up making economic decision based upon credits. “Oh, they'll pay for the migration with MAP credits.” Or, “They'll give us a whole bunch of things through Activate.” And that's great and all, I appreciate the value, but things get bigger. I mean, I was astounded, relatively recently, to discover that with the DuckTools launch of our SaaS offering here, as well as the experiments we're running and the stuff we're doing for our customers, we're now a $25,000 a year AWS bill. 


And that took me aback on some level. Now in practice, it doesn't really matter. That's two grand a month and change. And compared to the cost of running this place, no one cares. But it keeps going up. It only ever goes in one direction. 


Eventually, we're going to care, and we're very well suited for when that day comes, but by the same token, it's also the cobbler’s kids have no shoes style of story. Yeah, optimizing our bill is always going to be a distant second place to optimizing our clients’ bills.


Tim: Yeah. And as business goes, almost everyone does it the same way. What I do think is important to note is that when you talk about being on AWS and that bill is always going to go up, you don’t, sometimes, even have to do anything. You just have to exist. Your bill will go up over time just because you're storing more objects in S3, right? 


Because you're storing more EBS volumes. And if you don't do those optimization tips that we both talked about the past, then you will never save money. You can save money even if you are growing, but you have to be very purposeful about it. You have to be very intentional about it. You have to do—you actually have to build cost savings into your architecture; it can't be an afterthought because you're never really going to make anything happen. 


If you have an architecture—and then I've worked with folks before where—one of my old customers runs advertisements and monetization for game ads. And a year ago, a year and a half ago, they pushed to change their workload into a very stateless workload that can be run on Kubernetes and in Spotinst. And so this allowed them to have very, very well-orchestrated Spot instances that was easily 95 percent of their infrastructure. So, that way, when the pandemic hit and a bunch of more people were home, and they were playing games—like, kids were playing games, adults were playing games like that—and their usage tripled. Their AWS bill only took a little bit of a hit up because they had built cost optimization into their architecture before they needed to.


Corey: Absolutely. Strongly agree. I mean, the sad part for us was during the pandemic when we saw folks have their user activity drop off a cliff, and their infrastructure spend remains stable. People talk about elasticity, but whether they're living it or not is really a subject of some debate. Many applications still require, if you add or remove a node from a cluster, all the other cluster members need to be made aware of that. That's not a great candidate for auto-scaling as it turns out.


Tim: It is very painful to watch people try to turn things down and realize that they cannot. 


Corey: Oh, yes. Because historically—and I understand why. It made sense. If you scale up rapidly, great, you're going to be able to solve for that problem because if you don't, you're dropping customers on the floor. If you fail to scale down, well, you're just spending a little extra money, and that is never as big of a deal as it could be. So, it's always a trailing function; it's always something people think about after the fact. And the consistent lie we always tell ourselves is after this sprint ends, we're going to start doing everything the right way and clear on technical debt. 


Tim: [laugh].


Corey: Yeah. Sure, you will.


Tim: Oh. It's like when people clean up the garage, like, “Oh, yeah, we're going to get in there, we're going to throw all this stuff out, and then the garage will be clean and we’ll be able to put both cars in here.” And either you've always been that way, or it's never going to happen. I've never seen anybody who had a dirty garage that's cleaned it out and then from that point forward can always put both cars in the garage. It just doesn't happen.


Corey: The only way it happens is if you wind up, effectively paying someone to do it for you—


Tim: Exactly.


Corey: —which wasn't the metaphor I was intending to go with, but that seems to be the way that I've experienced it. My home office was always a mess until I started paying someone to clean it up behind me. And that's helpful.


Tim: Do you mean paying someone, like, maybe The Duckbill Group to come in straighten out your—


Corey: It wasn't the direction I was going in, but… and again, it's one of those stories of, what is the actual impact a customer is looking to have, it's never the, “Oh, just pay us and all your problems go away.” It very rarely is that simple. But it's a start, and it highlights what you could do, what you should do, what we would do in your shoes. But it's varied; there's a reason that we do this as a consulting service and we're not just selling software in isolation. We're selling some power tools that we do use to solve very specific problems, but we're not doing this as a ‘sign up for SaaS and never worry about your bill again,’ because there's no API for business insight. 


Tim: That’s true.


Corey: There's no context that you can glean programmatically to avoid recommending terrible things. And after a few terrible recommendations, people stop caring what you have to say.


Tim: I think that’s something that I talked about when I was working at Mission and something that I like to carry forward in any role: when you talk to customers—what I like to tell people who are new to customer-facing roles is that people can get AI recommendations for basically anything now. People can get machine learning and some kind of algorithmic recommendation for basically anything at this point. But what a machine cannot really effectively give you, right now at least, is insight based on your experience. And if you have a wealth of experience, and you've been able to go back and examine it, especially when the experience is making your own mistakes, right, of which I am very, very versed in, if you don't have that experience, you cannot say to others, “I know what the numbers say, but there's more to the story and here's why.” When you can do that, people will pay you for that because you are going to keep them from making mistakes. 


You are going to keep them either from spending more money because they haven't optimized or from spent more money because something bad happened and now they have to pay out a bunch of customers. So, when you talk about a consultancy, you're talking about giving customers—you know, people paying you for you telling them things to do, right? That's important. The ability to say that, “Hey, I have been there, and this is what I have seen, and this is what I see for you coming forward if you don't handle this.” I mean, that's a big thing. 


I think the garage cleaning analogy works, especially for us because oftentimes it's way easier for us to throw out somebody else's stuff than it is for us throw out our own. If I'm going into the house and I'm going to clean up the garage, if you don't tell me something absolutely has to stay, I'm going to throw it away. But if I'm going there, you're going into my house—or if I'm going to my house, I will find it, “Oh, I can't throw this away. My girlfriend from my sophomore year of high school, you know, I was wearing this when she looked at me the first time, so I got to keep it.” You know, like, all these little reasons why that you cannot really downsize things when it's your own.


Corey: Right. This is the big problem I see, [00:29:08 unintelligible] conference-ware where it's this idea of, here's what you could do if you wound up approaching it—the same sort with reference architectures, same story with the ‘hello world’ style stuff. The real world is messy, and well, there's context that means I can't safely do that for a variety of reasons. Sure, you can sit there and say that's because my process and culture are shitty, but that process or/and our culture also generates a sizable pile of revenue. So, how do I fix this? And meeting people where they are, rather than shaming them into meeting you where you are—as a vendor—seems to be something that all of the major cloud providers have been focusing on for a while now.


Tim: Yeah, I think it's interesting. We talked about it in the context of—you mentioned re:Invent before, we're talking about cloud providers. One of the things that cloud providers spent a long time doing was telling you that you shouldn't be on-premise and you should go to public clouds and here's various thousands of reasons why. They would do everything they could to get you off of being in a data center, being on metal, into public cloud. And if you notice at this year's re:Invent, they talked about putting a bigger emphasis on AWS Outposts, they talked about EKS Anywhere, ECS anywhere, they talked about the open-source distro for EKS you can run anywhere. 


And they mentioned a couple things about migrating databases back and forth, and things like that. And what I think it was is an acknowledgment that people are still in the data center, will continue to be in the data center or will be expanding into the data centers for various reasons. And so instead of trying to fight against that, they are now trying to make sure they can still make their money while people are doing that. Which is brilliant, number one, but number two, it is actual customer obsession because the customers are telling them, “Hey, this is what we want to do.” And AWS is saying, “Okay, that's fine, but we still want your money.”


Corey: Yeah, again, there's a cynical perspective, and there are uplifting perspectives to take on it. But I don't work for AWS. I’ll let them articulate what exactly this [laugh] they mean on this. I am curious as to what you saw coming out of re:Invent. What did you take away from the conference? And by extension, what did we learn about ourselves from all of last year, both as individuals and as a society?


Tim: The thing I took away from re:Invent was they literally put a big emphasis on large customers—enterprises—reinventing themselves technologically, [00:31:25 crosstalk].


Corey: Because that's where the money is.


Tim: Oh, sure. Sure. And they had a couple of startups, but the biggest thing was, they were talking about how large companies changed to meet the demands of 2020. And there were a lot of ways that large companies had to do some innovation. They talked about, was it—I think the bank—I think was a Capital One, they cut that four-hour wait time for folks to talk to an agent to, like, ten minutes based on doing some serverless functions into Lambda stuff, which I thought was brilliant. 


And that's something that everyone can understand. If you got laid off from your job, you got a pay cut, and you need to talk to the bank about something, can you imagine having to wait four hours on a weekday to talk to somebody about something super important, like, you know, mortgage payment, or bills, or something like that? That would be nightmarish. That is a real effect. That is a real thing that affects real people. 


And I think is important. A lot of things that we do in tech, people don't really know, people don't really feel. You know, like, if you say, “Hey, I reduce response time on this API.” Well, that's all well and good, but most people are never going to know that. But if you said, “Hey, we did a lot of work so that you don't have to wait four hours on the phone to talk to the bank.” 


That's something people were going to notice. And I think that's important. I do think that the overall theme of people changing and doing things differently through AWS technologies to meet the demands of 2020 was timely. I think it was inspiring in a lot of ways, but I think a lot of it, they're also kind of telling on themselves a little bit that maybe—the unwritten thing is, maybe the things that they were doing in the past aren't going to serve them going from 2020 on. The answer, the other questions like what do we learn as a society, and as individuals in this year? I think is—


Corey: Yeah, versus what should we have learned as a society, which we're going to stay away from because that is way too depressing.


Tim: Yeah. Here's what I think we learned: I think we learned, number one, that our cultures, our work cultures, and work organizations are nearly not as smoothly designed as maybe we thought they were and that they rely too much on people being in proximity, one with another, to exchange information, or to find information more importantly. I think that we realized that we don't document enough things. I think that we realized that things that we do you have documented are too hard to find. I think that we realized that we don't really know how to talk to, or check in on, or check in with people if we are not physically in front of them. 


And I think that a lot of companies that were remote-first companies had a huge advantage over companies that were trying to pivot to being remote-based companies. And what you saw were some companies did it really well, and some companies kind of didn’t. And so those companies, you'll see where they're bringing people back as soon as they can. You know, so whether they should or shouldn’t, that's a different discussion. 


But I think that's one thing we discovered about ourselves; we really learned, how is our culture? How is our company culture? How is our organizational culture? How are our practices in dealing with people who have real concerns? Without getting on the soapbox too much, but something I mentioned in the past is like, how do we address our employees’ real-life needs as a company? 


Whether it's our female engineers, are we making sure that they're supported, if their mothers, so they don't have to leave the workforce? Are we making sure that people have time for mental health breaks instead of just scheduling them from meeting to back-to-back meeting for 10 hours every day for four weeks straight? Hey, are we checking in on our people just in general? Are we making sure our people—hey, do you have the power on? Do you have high-speed internet at home?


Corey: Even if you are full remote, as we were when we started this whole thing, it's not the same. And if wow, it turns out our company can't ever do remote work. Look how terrible that year was. Yeah, not really a fair test, if I'm being very direct. This year has been remarkably different. 


Tim: It has been extremely different. And I think even for us, people who were full remote beforehand, we were able to still have the advantage of, every couple of weeks, couple months, being able to go and fly somewhere and see all the people that we talk to on Twitter, and have fun, and exchange pleasantries, and actually have real physical contact with these people. And you don't have that this year. And that's changed a lot of jobs. Like people who do DevRel, their jobs changed, I think, dramatically in 2020 because they're not doing conferences like they were. 


Conferences have changed. The value-add for conferences has changed quite a bit where people are realizing that, do I really want to pay $1500 to fly to a hotel to hear people just give a talk, or would I rather hear people discuss panels or talk to people in the hallway tracks? A lot of people have missed the hallway tracks. A lot of people have missed interacting one with another, exchanging ideas completely off the cuff about something, and just watching where it goes. Just sitting there and having a person just talk on the stage doesn't seem as valuable, I think, anymore because people are sitting there watching videos of people doing the same thing. And it's maybe not as fun. And I'm not saying that's universal across the board. Some people are very compelling speakers—


Corey: The entire world is defined by exceptions. That's fine. No argument here.


Tim: Yeah. But I do think, by and large, people now are realizing the value of sitting around, in a group of people, in a room talking about this thing that comes up and just letting that idea take root, and electrify people, and write stuff down. And I've been at conferences that have started a conversation with someone in the hallway that ended up being a product in two months. And that's the kind of stuff that people pay for. That's the kind of enrichment that you have. 


And yeah, the vendor booths and swag are one thing, having been both a participant and sponsor at a conference, the vendor experience is quite different, and I think that that needs to be rethought how that gets done going forward. And I do think another thing that needs to be rethought is, who are we bringing to the conferences? If you noticed, a lot of conferences were free, and people that could otherwise not attend because they couldn't afford, either personally, or for the companies to fly them to someplace and put them in a hotel—you know, it's always going to be fly into some very expensive place and put them in a very expensive hotel, to listen to—


Corey: And it’s people who can basically justify being not at work for a week for the duration of that conference. 


Tim: Oh, absolutely. So, all these combinations mean that only a very few people in an interest group would be able to actually participate. But now when you say, “Well, you can do it from your workday, you can pop in the session, then work some. It doesn't cost you anything, and you can literally get to it from anywhere.” Now I saw more people—different types of people—actually being able to participate in some of the activities. 


Mind you, they were different than a conference was before, but I do think what you're seeing is that maybe we need to find a different way of doing this so that we can get more people into these things; you get more people into these things, you get more thoughts, you get more products, you get more interactions. That's going to be a net benefit for tech as a whole, for whatever user group, or special interest group, or specialty that you're talking about. The more people you can get in there discussing more ideas, the better that's going to be.


Corey: I absolutely agree with basically everything you've just said. And for better or worse, there's an awful lot of things we need to learn as we move forward as a society. And hopefully, we'd look back on this with some silver lining we can take out of it, otherwise, it just becomes this entire year of nonsense. 


Tim: Yeah.


Corey: And that's too depressing to really contemplate.


Tim: Yeah. If we go into 2021, when it becomes quote-unquote, “Normal” again and we're all going back to work, whatever that looks like. And we looked exactly the same that we did in 2019, then all the shit that happened this year was for nothing. And that would be the greatest tragedy of it all, if we didn't learn anything, we take anything away. I would really like to see the overall health of people—physical health, mental health, you know, what's your burnout level, what's your home stress level—become a more important focus for these businesses. 


Because if you have a thriving individual there, that person is thriving, and that person is healthy, and that person is happy or as happy as they can be on whatever their circumstances are, they're going to be a better and more productive employee than they would be if they were [BLEEP] miserable.


Corey: Yep. Wholeheartedly agree.


Tim: So, what does this have to do with tech, Tim? Because I've had that question, too. It's like, well, when you're in your stand-up meeting, and you're talking to somebody, and you notice that hey, maybe they sound down, maybe they're a little off, maybe something's not right with them. Take a second and like, you know, “Hey, person, let me get five minutes after work.” And just check in on them. 


See how they’re doing, and see if there's anything you can do. Because the real benefit of having people managers—versus tech and engineering leads—is that they're going to address things that aren't necessarily technical but that are good for org, good for the business, good for the employees. Like, if you handle somebody’s pay problems, if you handle somebody’s insurance problems, and certainly you can maybe handle—it’s like, “Hey, you need a day off and I noticed you haven't taken any PTO in three months. Why don't you take a couple days. Something. Get away from the thing. Let me switch around some on-call so you can have a break.” These are things that you should be doing as people managers. People management is a skill. It may be it's not a technical skill, but it is a skill that is very important to technical work.


Corey: It absolutely is. And I think that people lose sight of that at their own peril. And once again, we've had a terrific conversation on this show. If people want to learn more about you and who you are, what you do, what you believe, what you're working on, et cetera, et cetera, where can they find you?


Tim: They can find me on Twitter @elchefe: E-L-C-H-E-F-E.


Corey: Fantastic, and we will of course throw links to that in the [00:41:05 show notes]. Thank you so much for taking the time to speak with me. Every time we have one of these conversations. I always wish we could go longer.


Tim: I appreciate it. Corey, I'm always glad to come back anytime.


Corey: Don't worry, I'll take you up on that. 


Tim: Awesome.


Corey: Tim Banks, currently a principal Solutions Architect at Equinix. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts, whereas if you hated this podcast, please leave a five-star review on Apple Podcasts along with an ignorant comment telling me why I should get over the high cost of data transfer egress pricing from cloud providers.


Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com, or wherever fine snark is sold.


This has been a HumblePod production. Stay humble.







View Full TranscriptHide Full Transcript