Holiday Replay Edition – Continuous Integration and Continuous Delivery Made Easy with Rob Zuber

Rob Zuber is a 20-year veteran of software startups; a four-time founder, three-time CTO. Since joining CircleCI, Rob has seen the company through its Series B, Series C, and Series D funding and delivered on product innovation at scale. Rob leads a team of 150+ engineers who are distributed around the globe.

Prior to CircleCI, Rob was the CTO and Co-founder of Distiller, a continuous integration and deployment platform for mobile applications acquired by CircleCI in 2014. Before that, he cofounded Copious an online social marketplace. Rob was the CTO and Co-founder of Yoohoot, a technology company that enabled local businesses to connect with nearby consumers, which was acquired by Appconomy in 2011.

Announcer: Hello, and welcome to Screaming in the Cloud with your host cloud economist, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: If you asked me to rank which cloud provider has the best developer experience, I'd be hard-pressed to choose a platform that isn't Google Cloud. Their developer experience is unparalleled and, in the early stages of building something great, that translates directly into velocity. Try it yourself with the Google for Startups Cloud Program over at cloud.google.com/startup. It'll give you up to $100k a year for each of the first two years in Google Cloud credits for companies that range from bootstrapped all the way on up to Series A. Go build something, and then tell me about it. My thanks to Google Cloud for sponsoring this ridiculous podcast.

Corey: This episode is brought to us by our friends at Pinecone. They believe that all anyone really wants is to be understood, and that includes your users. AI models combined with the Pinecone vector database let your applications understand and act on what your users want… without making them spell it out. Make your search application find results by meaning instead of just keywords, your personalization system make picks based on relevance instead of just tags, and your security applications match threats by resemblance instead of just regular expressions. Pinecone provides the cloud infrastructure that makes this easy, fast, and scalable. Thanks to my friends at Pinecone for sponsoring this episode. Visit Pinecone.io to understand more.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Rob Zuber, CTO of CircleCI. Rob, welcome to the show.

Rob: Thanks. Thanks for having me. It's great to be here.

Corey: It really is, isn't it? So you've been doing the CTO dance, for lack of a better term, at CircleCI for about five, six years now at this point?

Rob: Yeah, that's right. I joined five and a half years ago. I actually came in through an acquisition. We were building a CI/CD platform for mobile, iOS specifically, and there were just a few of us. I came in an engineering role, but within, I think a year, had taken over the CTO role and have been doing that since.

Corey: For those of us who've been living under a rock and recording podcasts, CI/CD or Continuous Integration/Continuous Delivery has gone through a bit of, shall we say, evolution since the term first showed up. My first exposure to it many moons ago was back when Jenkins was still called Hudson, and it was the box that you ran that it would wait for some event to happen, whether it was the passing of time, a commit to a particular branch, someone clicked a button, and then it would run a series of scripts, which sort of lent itself to the idea of the hacker news anthem, "That doesn't look hard. I can build that in a weekend." Now, we've seen a bit of growth in that space of not just, I guess the systems you can run yourselves, but also a lot of the SaaS offerings around this. That's the, I guess, the morons journey from my perspective to path through CI/CD. That's almost certainly lacking nuance. What is it, I guess in the real world with adults talking about it?

Rob: Yeah, so I think it's a good perspective, or it's a good description of the perspective that many people have. Many people enter into this feeling that way. I think, specifically when you talk about cloud providers in CircleCI, we do have an on-prem offering behind the firewall. No one really runs anything on-prem anymore. But we have an offering for that market, but the real leverage is for folks that can use our stuff, multi-tenant SaaS cloud offering. Because, ultimately it's true. Many people have start with something simple from a code based perspective, right? I'm starting out, I've got a small team. We have a pretty simple project, maybe a little monolith Ruby on rails, something like that. Actually, I think in the time of the start of CircleCI. Probably not too many people kick off the rails monolith these days because if you're not using Kubernetes and Docker, then you're probably not doing it right.

Corey: So, the Kubernetes and Docker people tell us?

Rob: Yeah, exactly. They will proudly tell you that. We'll come back around to that point if we want to, but so you have simple project and you have simple CI, right? You may just have a simple script that you're putting in a Jenkins box or something like that, but what ultimately ends up happening is it gets complicated, and as it gets complicated, it becomes a bigger and bigger distraction from the thing that you're really trying to do, right? You're trying to build a business to ... I don't know, to do ride hailing, to do scooter sharing, what's big these days. You might be trying to do any of the ...

Corey: Oh, my project is Twitter for pets. We're revolutionizing the world of pet communication.

Rob: Right. And do you want to spend your time working on pet communication or on CI/CD, right? CI/CD is a thing that we understand very well, we spend our time on it every day, we think about some of the depths of it, which we can go into in a second. One of the things that gets complicated, amongst others, is just scale. So you build a big team, you have multiple projects and you have that one box under your desk where you said, "Oh, it's not that hard to build CI/CD. Now, everybody's waiting for their stuff to run because someone else got in there before them and you're thinking, okay, well how do I buy ... maybe you're not buying more boxes, you're building out something in a cloud provider and then you're worrying about auto scaling because it starts to cost you too much to run those boxes, and how do you respond to the amount of load that you have on any given day?

Because you're crunching for a deadline versus everybody's taken a week off. Then, you want to get your build done as quickly as possible. So you start figuring out how to paralyze the work and spread it across those machines. The list goes on and on. This is the reality that everyone runs into as they scale their work. We do that for you. While it seems simple and ... I said I came in through an acquisition, we were building CI/CD for iOS, and I was that person. I said, "This seems really simple. We should build it and put it in the market." It didn't take us very long to get that first version to build, and it had to be generic to support many different types of customers and their particular builds.

It was a small start but we started to run into the same problems, and then of course as a business, we ran into the problem of getting access to customers and all those things and that's why we joined CircleCI and that became what is now our iOS offering. But there is a lot of value that you can get quickly, to your point, but then you start focusing time and energy on that. I often refer to it, others in the industry refer to these sorts of things as undifferentiated heavy lifting. Something that becomes big and complex over time and is not the core of your business. Then as you start to invest in it, as we invest in it, then we build capabilities that most people wouldn't bother to build when they write that first bash script off a trigger or whenever, around helping you get your project set up, handling the connection into hooks, handling authentication so that different users only have access to the code they should have access to, maybe isolating access to production secrets, for example, if you're doing deploy.

The kinds of things that keep coming up over and over in CI/CD that people don't think about on that first pass but ended up hunting them down the road.

Corey: What do you think that people tend to misunderstand the most about CI/CD as you take a look at that throughout the ecosystem? From my perspective, when it was a box that you ran, behind the firewall as you say, the problem was is that everyone talked about, "Oh yes, we use cattle, not pets, except the box that does the builds. Of course, that box has a bunch of hand-built stuff on it that's impossible to replicate. It has extraordinary permissions into production environments and can do horrifying things, and it was always the star of various security finding reports. There are a number of us who came up from an operation side viewing CI/CD as, in some ways, a liability, which I understand is a very biased and one sided perspective. But going beyond that, what are people missing? What are they not seeing about the CI/CD landscape?

Rob: One thing that I think is really interesting there, well, one thing you call that was just resiliency, right? We think about that in the way that we operate that system. We have a world of cattle because we've managed to think about that as a true offering. So, as you scale and you start to think, "Oh, how do I make this resilient inside my operation?" That's going to become a challenge that you face. The other thing that I think about that I've noticed over the years is, I want to call it division of labor or division of responsibilities. Many of those single instance or even multi-instance self-managed CI/CD tools end up in a place where, past any size of team, honestly somebody needs to own it and manage it to make sure it's stable.

The changes that you want to make as a developer are often tied to basically being managed by that administrator. To be a little clear, if I have a group responsible for running CI/CD and I want to start building a different type of code or a different project, and it requires a plugin or an extension to the CI/CD platform or CI/CD tool, then I need to probably file a ticket and wait for another department who is generally not super motivated to get my code out into production, to go make a change that they are going to evaluate and review and decide ... or maybe creates conflict with something somebody else is doing on that system. And then you say, "Oh well actually we can't have these co-installed so now we need two systems." It's that division of responsibilities. Whereas, having built a multi-tenant cloud offering, we could never have that. There is no world in which our customers say to us, "Hey, we want this plugin installed. Can you go do that for us?"

Everything that is about how the development team thinks about their software and how they want their build to run, how they want their deploys to run, etc, needs to be in the hands of the developers, and everything that is about maintenance and operation and scale needs to be in our hands. It has created a very clear separation out of necessity, but one that even ... I mentioned that you can deploy CircleCI yourself and run it within a team, and in large organizations, that separation really helps them get leverage. Does that make sense?

Corey: It really does. I think we're also seeing a change in perspective around resiliency and how this works. I once worked at a company I will not name where they were. It was either CircleCI or TeamCity. This was years and years ago where I don't recall exactly what they were using, but it doesn't matter because at one point the service took an outage, and in typical knee jerk reaction, well, that can never happen again. So they wound up doing all of the CI/CD work for some godforsaken reason on a Raspberry PI that some developer brought in and left in the corner of the office. Surprise, it took an awfully long time for tests to run on basically an underpowered toy project. The answer there was to just use less tests because you generally don't need to run nearly as many.

I just stared at people for the longest time when it came to that. I think that one of the problems that we still see, I know when I write code myself, I'm as guilty of this as anyone, I am a terrible developer and don't believe in tests. So, the CI/CD pipeline that I tend to look at is more or less a glorified script runner. Whenever I make a commit to this branch, go ahead and run the following three lines script that does a serverless deployment and puts it where it needs to go, and then I'll test it manually, or it's a pre-production environment so it's not that big of a deal. That can work for some use cases, but it's also a great thing that no one actually depends on the stuff that I write for day-to-day business operations or anything critical. At what point does it stop being a script runner?

Rob: Well, to the point of the scale, I think there's a couple of things that you brought up in there that are interesting to me. One is the culture of testing. It feels like one of these areas of software development, because I was around in a time when no one really understood what it was to do automated testing. I won't even go into TDD, but just, in general, why would I do that? We have this QA team, it's cost effective to give it to a bunch of people. I'm thinking backwards or thinking back on that, it all seems a little bit well, wrong. But getting to the point where you've worked effectively with tests takes a little bit of effort. But once you have that, once you've sat and worked on something and had the feedback loop of, oh, this thing's not working. Oh, I'll just change this, now it's working.

Really having that locally, as a developer, is super rewarding, in my mind and enabling I guess I would say as well. Then you get to this place where you're excited about building tests, especially as you're working in a team, and then culturally you end up in a place where, I put up a PR and someone else looks at it and says, "I see you're making an assumption or I believe you're making an assumption here, but I don't see any way that that's being validated. So please add testing to ensure that is actually true." Both because I want to make sure it's true now, but when we both forget that you ever wrote this and someone else makes a change, your assumptions hold or someone can understand that you were making those assumptions and they can make appropriate changes to deal with it.

I think as you work in a team that's growing and scaling and beyond your pet project, once you've witnessed the value of that, you don't want to go back. So, people do end up writing more and more tests and that's what drives the scale at least on the testing and CI side in a way that you need to then manage that. Going the opposite direction of what you're describing, which is, hey, let's just write fewer tests and use cheaper machines, people are recognizing the value and saying, "Okay, we want that value, but we don't want to bottleneck everyone with an hour long build to run all these. So how do we get a system that's going to scale and support that?"

Corey: That's what's fascinating, is watching that start to percolate beyond the traditional web applications with particular blessed languages and into other things. For example, in my copious spare time, I'm the community lead for the open guide to AWS, which is a GitHub project that has 25,000 stars or so, so you know it's good, where it's just a giant markdown document that lists the 10,000 tips and tricks that we all wish we'd known when we'd gotten started with AWS, and in a format that's easily consumable. The CI/CD approach we have right now, which I believe is done through Travis, is it just winds up running a giant link checker in parallel across the thousands of links that are ... sorry, I wanted to say 1,200 links, that are included within that document.

There's really not a lot else we can do in that type of environment. I mean, a spellchecker with all of the terms of art involved would more or less a seg fault itself to death as soon as it took a look, but other than making sure we don't have dead links, and it feels like there's not a lot of automation or testing opportunity in something like that. Is that accurate? Am I completely wrong and missing something?

Rob: I've never built that particular site so it ... I mean, it sounds reasonable. I think that going the other way, we often think about, before we kick off a large complex set of testing for a more complex application, maybe then a markdown document, a lot of people now will use things similar to what you're using, like maybe part of my application is a bunch of links to outside docs or outside sites that I'm referencing or if I run into a problem, I link you to our help site or something and making sure all that stuff is validated. Doing linting on the structure and format of code itself. One of the things that comes up as you scale out of the individual script runner is doing that work in parallel. I can say, you know what? Do the linting over here, do the link checking over here. Only use very small boxes for those.

We don't happen to have Raspberry Pi's in our infrastructure, but we can give you a much smaller resource, which costs you less if you're not going to be pushing the limits of that. But then, if you have big integration tests or something which need more space than we can provide that as well, both in a single channel or pathway to give you the room to move faster and then to break that out and break up your work. At an extreme example, and of course, anyone who's done parallelization knows there's costs to splitting up work in like the management overhead. But if you have 1200 links, like you could check them all at the same time. I doubt that would be a good use of our platform, but you could check 600 in one and 600 in another, or 300s at a time or whatever, in find the optimal path if you really cared about getting that done more quickly.

Corey: Right. Usually, it's not that big of a concern and usually it winds up throwing errors on existing bad links, not something that has been included in the pull request in question. Again, there's nothing that is so awesome that I can't horribly misuse it for something ridiculous. It's my entire stock and trade. It's why I believe route 53 remains the best database option for everyone, but it's fun going through this space and just seeing how things have evolved. One question I do have since you come from a background, by way of acquisition, that was aimed squarely at this, historically, it seems that running a lot of testing on mobile devices, specifically iOS devices, was the stuff of nightmares because you couldn't really run that in any meaningful way in a virtualized environment. So, it generally required an awful lot of devices. Is that still the case? Has that environment changed radically since I last worked at a mobile shop?

Rob: I don't think so, but I think we've all started to think a little bit differently. We got started in that business because we were building iOS apps and thought, wow, the tooling here, it's really frustrating. To be clear, at CircleCI and at that business, we were solving the problem of managing the machines themselves, so the portion of the testing that you would run effectively in a simulator, not the problem of the device farm, if you will. But one of the things that I remember, and so this is late 2013, early 2014 as I was working on mobile apps was people shifting the MVC layers a little bit such that the thing that you needed to test on a device was getting smaller and smaller, meaning putting more logic in, I forget what the name was specifically, but it was like the ... I don't want to try to even guess.

But basically pulling logic out of the actual rendering and down into what we'll call state transitions I guess. If you think about that in modern day and look at maybe web frameworks like React, you're trying to just respond with rendering on top of a lot of state change that happens underneath that. In that model, if you thin out the user interface portion, you make a lot more of your code testable, if that makes sense. The reason we're all trying to test on all these different devices is often that we've baked a lot of business logic into the view layer. Does that make sense?

Corey: Yeah, it absolutely does. Please continue.

Rob: Instead of saying, well, all our logic's in the view layer, so let's get really good at testing the view layer, which means massive device farms and a bunch of people testing all these things, let's make that layer as thin as possible, and there's analogies for this in even how we do service design these days and structure the architecture of systems, basically make the boundaries as thin as possible and the interaction with the outside world as thin as possible. That gives you much more capability to effectively test the majority or much larger portions of your business logic. The device farm problem is still a problem. People still want to see how something specifically renders on a particular screen or whatever. But by minimizing that, the amount that you have to invest in that gets smaller.

Corey: This episode is sponsored in part by our friends at Uptycs, because they believe that many of you are looking to bolster your security posture with CNAPP and XDR solutions. They offer both cloud and endpoint security in a single UI and data model. Listeners can get Uptycs for up to 1,000 assets through the end of 2023 (that is next year) for $1. But this offer is only available for a limited time on UptycsSecretMenu.com. That’s U-P-T-Y-C-S Secret Menu dot com.

Corey: You mentioned device farm, which is an app choice, given that that is the name of an AWS service that has a crap ton of mobile devices that you can log into and it's one of my top candidates for the, did I make this service up to mess with you competitions? It does lead us to an interesting question. CI/CD has gotten an increased amount of attention lately from pretty much everyone. AWS, as is typical for Amazon, tends to lie awake at night worrying that someone somehow is making money that isn't them. So their product strategy distills down to, yes. So, they wound up releasing a whole bunch of CI/CD oriented products that at launch were, to be polite, terrible. Over time, they've gotten slightly better, but it's still a very confusing ecosystem there.

Then we see things like Azure dev ops who it seems is aimed at a very similar type of problem and they're also trying to challenge Amazon on the grounds of terrible names of services. But we're now seeing an increased focus from the first party providers themselves around the CI/CD space. What does that mean for existing entrenched players who have been making a specialty out of this for a lot longer than these folks have been playing with it?

Rob: It's a great question. I think about the approaches very differently, which is probably unsurprising. Speaking of lying awake at night or spending all day thinking about these things, this is what we do. You've the term script runner a few times in the conversation, the thing that I see when I see someone like AWS looking at this problem is basically, people are using, the way that I think about it, is maybe less the money, although it translates pretty quickly. People are using compute to do something, can we get them to do that with us? Oddly enough, a massive chunk of CircleCI runs on AWS so it doesn't really matter to them one way or another, but they're effectively looking to drive compute hours and looking to drive a pathway onto their platform.

One thing about that is it doesn't really matter to them in my perspective, whether people use that particular product or not. As a result, it gets the product investment that you put in when that's the case. So, it's a sort of a check the box approach like, hey we CI and we have CD like other people do. Whereas, when we look at CI and CD, we've been talking about some of the factors like scaling it effectively and making it really easy for you to understand what's going on. We think about very much the core use case, what is one of our customers or users doing when they show up? How do we do that in a way that maximizes their flow? Minimizes the overhead to them of using our system, whether it's getting set up and running really quickly, like talk about being in the center of how much of the world is developing software.

So we see patterns, we see mistakes that people are making and can use that to inform both how our product works and inform you directly as a user. "Hey, I see that you're trying to do this. It would go better if you did this." I think both from the, honestly, the years that we've been doing this and the amount that we've witnessed in terms of what works well for customers, what doesn't, what we see going through just from a data perspective, as we see hundreds of thousands of builds running, that rich perspective is unique to us. Because as you said, we're a player that's been doing this for a really long time and very focused on it. We treat the experience with, I guess I'm trying to figure out a way to say this that doesn't sound as bad as it might, but a lot of people have suffered a lot with CI/CD.

There's a lot that goes into getting CI/CD to work effectively and getting it to work reliably over time as your system is constantly changing. Honestly, there's a lot of frustration, and we come in to work every day thinking about minimizing that frustration so that our customers can go spend their time doing what matters to them. Again, when I think you sort of ... a lot of these big players present you with a runtime in which you can execute a script of your choosing. It's not thinking about the problem in that way and I don't see them changing their perspective. Honestly, I just don't worry about them.

Corey: Which is a very fair tack to take. It's interesting watching companies and as far as how much time and energy they spend worrying about competition versus how much they focus instead on customers. To turn it around slightly, what makes what you do challenging in some respects, I would imagine is that a lot of your target market is themselves, developers. Developers, in my experience, are challenging customers in that, first, they tend to devalue their own time to the point where, oh, that doesn't sound hard. I'll build that overnight. Secondly, once you finally win them over to the idea of paying for something, it's challenging to get them to have the necessary signing authority. At best, they become champions. But what you do has to start with developers in order to win widespread adoption and technical buy-in. How does that wind up manifesting as approach to, well, some people call it developer relations, developer advocacy. I refer to those folks as developers because I have problems, but how do you folks view that?

Rob: Yeah, it's a really insightful view actually because we do end up in most of our customers, or in the environments of our customers, however you want to describe it, as a result of the enthusiasm of individual developers, development teams, much more so than ... there are many products certainly in enterprise software and I don't really think purely in enterprise, but there are many products that can only be purchased by the CIO or the CTO or whatever. Right? To your question of developer relations, we spend a lot of time out in the market talking to individuals, talking at conferences, writing content about how we think about this space and things that people can do. But we're a very product driven company, meaning both, that's what we think about first, and then support it with these other things.

But second, we win on product, right? We don't win in the market because you thought the blog posts that we wrote was really cool. That might make you aware of us, but if you don't love the product, I mean, developers, to your point, they want to use things that they really enjoy using. When developers use the product and love the product and they champion it and they get access because they might work on a side project or an open source project or maybe they worked in another company that used CircleCI and then they go somewhere else and they say, "What are we doing? Life is so much better for you Circle CI, those sorts of things. But it very much comes from the bottom up. It's pretty difficult to go into an organization and say, "Hey, you should push this down to all of your developers."

There's a lot of rejection that comes from developers on mandated tooling. We have to provide knowledge, we have to provide capabilities in our product that appealed to those other folks. For example, administrators of our tooling, or when it gets to the point where someone owns how you use CircleCI versus just being a regular user of the product. We have capabilities to support them around understanding what's happening, around creating shared capabilities that multiple teams can use, those sorts of things. But ultimately, we have to lead with product, we have to get in into the sort of hearts and minds of the developers themselves and then grow from there and everything we do from a marketing, developer relations myself, I spend a lot of time talking to customers who are out in the market, is all about propping up or helping raise awareness effectively. But there's nothing that we can do if the product doesn't meet the needs of our customers.

Corey: That's what it seems like it comes down to a fair bit. It's always weird to consider that, at its heart, developer relations is marketing. The folks I talk to who argue against that, it seems that it comes from a misunderstanding of what marketing actually is. It's not buying ads in airports, it's not doing podcast advertisements. That's a subject near and dear to my heart. It's not about annoying people by showing up at their office with the sales team. It's about understanding what their challenges and problems are and then positioning a solution that ideally solves them in a place that and in a way that they can be receptive to. Instead, people tend to equate marketing to this whole ridiculous statistics driven nonsense that doesn't really resonate with anyone and I think that that's unfair to everyone involved.

That said, I will say that having spent a fair bit of time in this space, I've yet to see anything from CircleCI that has annoyed me to the point where I would have remembered it, which is awesome. I don't see it in flight magazines, generally. I don't see it on obnoxious people try to tackle me as I walk through an expo hall and want to scan my badge. It just seems very well executed and you have some very talented people working for you. To that end, you are largely a distributed company, which is fascinating. Did it start that way? Did it happen that way by a quirk of fate?

Rob: Yeah, I those two things probably come together. The company, from very early days, now I wasn't there but I think some of our earliest engineers were distributed and the company started out basically entirely as engineers. It's a team solving problems of other engineers, which is ... it's a fun challenge. There were early participants who were distributed. Mostly, when you start a company and no one has ever heard of you and no one knows if you're going to be successful, going and recruiting is generally a different game than when you're, certainly, when you're where we are now. There were some personal relations that just happened to connect with people around the globe who wanted to participate.

We started out pretty early with some distribution, and that led to structuring the org in a way, both from a tooling and process perspective. A lot of that sort of happens organically, but building a culture that really supported that. I personally am based in the Bay Area, so we have headquarters in San Francisco, but it doesn't really make a difference if I go in versus just stay and work from home on any given day because the company operates in such a way that that distribution is completely normal.

Corey: We accidentally did the same thing. My business partner and I used to live across the street from each other and we decided to merge a week before he moved out of state to Portland. So awesome. Great. We have wonderful timing on all of these things. It's fun to build it from that way, build that way from the ground up. The challenge I've always seen is when you start off with having a centralized office and everyone's there, except this one person who, no matter how you try to work around it, is never as involved. So it feels like the sort of thing you've absolutely got to be building from day one, or otherwise, you're going to have a massive cultural growing pain as you try to get there.

Rob: Yeah, I think that's true. So I've actually been that one person. I, at some point in my career prior to CircleCI, was helping out a company founded by some friends of mine based in Toronto. I grew up in Toronto. I kicked off a project and then the project grew and grew until I was the one person out of maybe 50 or 60 who wasn't in an office in Toronto. It got to the point where no one remembered who I was and I was like, "Cool, I think I'm done. I'm out." I was fine with that. It was always meant to be a temporary thing, but I really felt that transition for the organization. I would say in terms of growing, I mean, yes, if you start out, it goes both ways, if you start out distributed, you're going to remain distributed.

There are certain things that get more challenging at scale, right? If everybody is sort of just in their home all over the globe, then the communication overhead continues to increase and increase in just understanding who people are, who you should be talking to. You need to focus-

Corey: There's always the time zone hierarchy.

Rob: Ooh, the time zones are a delight, yes. I would say like we talk a lot about, in this industry, Dunbar's number and sizes of teams and the points at which things get more complex. I think there's probably a different scale for distributed teams. It takes fewer people to reach a point where communication gets challenging, and trust and all the other things that go with Dunbar's views. You kind of have that challenge and then you start to think, oh well, then you have some offices, because we actually have maybe six physical offices, partly because in our go to market org, we've started to expand globally and put people in regional offices.

There's this interesting disconnect. I don't know about disconnect, but there's a split in how we operate in different parts of the org. I think what I've seen people ... well, I don't know about succeed, but I've seen people try when you start out with one org, or sorry, one location is, let's not jump to that one person somewhere else and then one person somewhere else kind of thing, but build out a second office, build out another office, like pick another location where you think you ... it's often, certainly where we are, in the Bay Area, it's often driven by just this market. Finding talent, finding people who want to join you, hanging onto those people when there are so many other opportunities around tends to be much more challenging. When you offer people alternatives, like you can stay where you are but have access to a cool and interesting company or you can work from home, which a lot of people value, then there's different things that you bring to the table.

I see a lot of people trying to expand in that way, but when you are so office-centric, a second office I think is a smoother transition point than just suddenly distributing people because, especially the first and second one, unless you're hiring in a massive wave, are really going to struggle in that environment.

Corey: I think that's probably one of the more astute things that's been noticed on this show in the last couple of years. If people want to hear more about what you have to say and how you think about the world, where can they find you?

Rob: I would say, on our blog, I tend to write stuff there as do other people. You talked about having great people in the organization. We have a lot of great people talking about how we think about engineering, how we think about both engineering teams and culture and then some of the problems we're trying to solve. So, off our site, circleci.com, and go to our blog. Then, I attend to is to speak and hangout on podcasts and do guest writing. I think I'm pretty easy to find. You can find me on Twitter. My handle is z00b, Z-0-0-B. I know I'm not super prolific, but if someone wants to track me down and ask me something, I'd probably be more than happy to answer.

Corey: You can expect some engagement as soon as this goes out. Thank you so much for taking the time to speak with me today. I appreciate it.

Rob: Yeah, thanks for having me. This was a ton of fun.

Corey: Rob Zuber, CTO at CircleCI. I'm Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple podcasts. If you've hated this podcast, please leave a five-star review on Apple podcasts along with something amusing for me to read later while I'm crying.

Announcer: This has been this week's episode of Screaming in the Cloud. You can also find more [email protected] or wherever fine snark is sold.

Announcer: This has been a HumblePod production. Stay humble.