What an “Agilist” Brings to the Engineering Table with Cliff Moon

Cliff is an Agile Consultant and self proclaimed “computer botherer.”

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.io

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. My guest today has done an awful lot of things over the course of his career: startup engineering; software work; founded two startups; has been an engineering manager a bunch of places; has been the CTO at UpGuard, for example; and consulted at one point on HBO’s Silicon Valley. Also of note, he is now these days, a renowned Agile consultant. Cliff Moon, welcome to the show.

Cliff: [laugh]. Hi, Corey. Thanks for having me.

Corey: So, you and I have had energetic conversations about Agile, and based upon that context calling you an Agile consultant for enterprises is basically a deadly insult at this point. Let’s get some context on that. For those who have not heard of the term because they live wonderful, blessed lives. What is Agile? A lot of people talk about it, but always the presupposition that people listening know what it is.

Cliff: Yeah, that’s a great place to start. So, let’s go back into, sort of, prehistory. What we call Agile today is, I guess, several generations removed from the original thoughts of Agile. So, in case folks aren’t aware, to kind of lay the background, there was a group of software developers, I think it might have been in 2000, might even have been earlier than that, who came up with what they call the Agile Manifesto. And I’m not going to go through a point by point, one, because I don’t remember it; two, it’s not germane. But—

Corey: But it’s called a manifesto. I mean, if you take a look at things that have been written historically that are called manifestos, very few of them are good. Like, generally ‘manifesto’ sounds like something you wind up writing in a cabin somewhere, right before you wind up doing some sort of horrible crime that winds up living in infamy for 30 years.

Cliff: Yeah, yeah. Manifestos, they get a bad rap for good reason. Anyway, let’s not go down that [laugh] that rabbit hole. But yeah, so Agile Manifesto, right. Basically, it was this group of people, they said, “Hey, we don’t think we’re developing software the right way. This is unnecessarily painful. We’re doing things in kind of a silly fashion. Let’s refocus it around the customer. Let’s do this,” yadda, yadda, yadda.

And again, like you’re saying, it’s a manifesto; it’s not very prescriptive about what to do to solve the problem, it really just points out problems and then gives a bunch of vague statements about, “Here’s the things we should value,” or whatever. And so again, like we’re saying, as a manifesto it kind of mutates from there, and then everyone agrees; they say, “Yep, this is the wrong way. Let’s try a better way.” And down through the line, what ended up happening was a lot of people figured out that they can make money doing this, and make money being an Agile consultant, or an Agilist, if you’re, [laugh] if you like that title. So, it’s people who come in, and I guess, try to teach you how to do Agile the right way.

And the problem is that the right way usually ends up being Scrum. And so Scrum, if we can get into that, is this idea of having a two-week sprint, you plan out the work you’re going to do for that sprint, there’s a bunch of meetings you have to do that are kind of mandatory: you do a stand up every day, you do a retrospective, you do sprint planning, et cetera, et cetera. And so it’s this, like, cake-in-a-box, right? So, it’s like a ready-made thing. And, like, ta-da, now we have Agile, we’ve implemented Agile, we’ve done this Scrum thing.

The problem, of course, is that most people when they implement this—and it’s certainly not the Agile consultant in most cases because they’re basically there to just bake the cake, and then keep soaking hours until they’re forced to leave.

Corey: The problem that I had, whenever I wound up dealing with, I guess, Agile consultants in large enterprises, they always looked a lot like Agile trainers. And I don’t know what they were charging because it doesn’t matter what they were charging; they wound up gathering the entire engineering department in part of the building for two or three days to talk about how tickets worked, and how planning worked, and how to iterate forward, and how to wind up planning for spikes, and all the various terminology, and how to work with different tooling and the rest. And the reason didn’t matter what these people cost was because it was absolutely dwarfed by the sheer cost of every engineer in the company sitting through this nonsense for the better part of a week.

Cliff: Oh, absolutely. And then it’s an ongoing thing, right?

Corey: Well, it’s supposed to be an ongoing thing.

Cliff: Yeah, it’s an ongoing concern. You end up having all of these meetings that you have to do every two weeks or, God forbid, every one week, or whatever the iteration speed that they’ve laid out is. And the thing that gets lost in the sauce here is, why are you doing this? What is the point of all of this? And I think one of my favorite things to do if I’m at a company that they’re implementing Scrum, and they’ve got their sprints, and then we have our retrospective, one of the things I love to do is I love to touch the third wire during retrospective because that’s when you’re supposed to bring up, “Hey, what can we do better? What could have gone well?”

And that’s what I like to say, “Hey, can we just not do this?” [laugh]. And the response that I get is usually an indication of how hard of a job I’m going to have of trying to deprogram people. Because what it ends up being is that—and especially if you have an Agile consultant, and Agile teacher, whatever, they’re not going to like the sound of that at all, right? It’s like, “Why are we doing any of this? What is the point?”

And when you dig into it, especially when we talk about Scrum, it sort of confuses a bunch of different goals that a lot of companies don’t even necessarily have anymore. One of the tenets of Scrum is that every two weeks—or whatever your iteration speed is, whether it’s one week, two weeks, whatever—the whole system has to be shippable. So, that means that everything has to work together correctly, and then you can ship an entire, like, vertical, or monolith, or whatever. The problem with this is that, especially related to if people are deploying to the cloud or if they’re running some sort of SaaS service, this is a meaningless statement; the way that people develop software in that arena today is things get shipped immediately. The system is always shippable because the system is always up because prod is always up.

And so you ship your component, everything is backwards compatible, and then your features are behind a feature flag. So, the idea that, oh, everything has to be set in stone on a two-week cycle or whatever, it doesn’t mean anything anymore, unless of course, you have a physical artifact that you actually send to a customer, like a CD, or an image to download, or something. But if you’re doing cloud-based software—

Corey: Or a giant rocket.

Cliff: Yeah. Or giant rocket, yes. Oh, God. [laugh].

Corey: For some things, you always using waterfall. It’s like, giant rockets going to space or—more realistically for most of us or more prosaic at least—is billing systems; people don’t generally tend to iterate forward on things that charge customer credit cards. It’s a lot of planning, a lot of testing, and they roll it out, and everyone’s sweating bullets for a while.

Cliff: Right. And I would submit that, at least in my experience, most companies which have tried to implement a Scrum-type process, what they’re actually doing is they’re running a two-week waterfall. Because a lot of times they’ve got a lot of technical debt, so the idea that you can ship things immediately might be a little bit shaky, and so what you end up doing is you have this iteration speed of, like, two weeks, and then you have to plan everything out for that, and then you have to go through testing, QA, acceptance. The whole cycle has to run in a two-week sprint. And it truly is a sprint in that case because it’s too much work, it’s too much stuff, and everything just falls apart. And then they wonder why they can’t ship any software anymore. Well, it’s because you adopted this process. [laugh].

Corey: Oh, I’ve been in environments where we’ll sit down and do quarterly or, God forbid, annual planning about what we’re going to build this year, via Agile. It seems a little unlike what Agile professes to be. Now, other than the sheer aspect of hypocrisy surrounding all of this, you 
take it a step further and say that it in many ways causes harm.

Cliff: Yeah. It causes harm in a couple of different ways. One of the ways that I think it is most harmful is the effect that it has on junior engineers, so people who are just starting their career, folks who are just coming out of college, and, you know, in most colleges, they don’t really teach you software engineering processes, or software engineering practices beyond the nuts and bolts of the code, or the theory of the code, or whatever, but they don’t teach you how to work in a professional environment. And so then you get a lot of folks just entering their first job, and they learn the way to do things at that job. And then they go on, they move to another job.

And someone might have, you know… they might go through ten different companies in their career, maybe some more, but they learn a certain way of doing things, and then all of a sudden, it’s like, “Yeah. I know how to do Agile. We did it at company X, Y, and Z.” And then they cargo-cult it and take it to the next place if they don’t already have it. And so it’s this sort of inculcation of younger engineers into this way of doing things that is completely harmful because most places, they don’t sit you down and tell you why we’re doing this because they don’t necessarily know why we’re doing it either.

Like I said, a two-week sprint with Scrum, the system is shippable every two weeks, you have to go through testing, and yadda, yadda, yadda, this may actually make sense in some cases. And professionally, in my experience, I’ve designed certain processes that are similar to that. Longer timeframes, but they were designed towards both the product and the team, and, sort of, the interval that they had to ship on. But in most places I’ve been, no one’s thinking about it from that perspective; they’re not thinking, “We have to design our processes around the software, or customers, or whatever.” They just kind of do, either whatever the Agile consultant tells them or whatever they learned at the last place. And so it has this effect of replicating a cargo-cult mentality throughout the entire industry, which is sad.

Corey: I’ve talked to a number of relatively Junior folks who have not heard of Agile or Scrum or any of these higher-level concepts about software development methodologies. They just walked into the workplace one day, and everyone’s doing two-week planning sessions. I’ve had people ask me six months into their career, “Why is it called a sprint?” Or, “What is up with the swim-lane style things? It seems weird, but everyone I talk to is used to it. Is it this company thing, or is this an industry thing?” And, on some level, it’s, “No, it’s just a terrible thing [laugh] that’s sort of like a mind virus that wind up taking root in an awful lot of people’s minds.”

Cliff: Yeah, absolutely. And so when we talk about the damage being done, I think that’s the worst. When you think about new people getting into the industry, having a fresh perspective, and that perspective or having an opportunity to forge a new way to do things, that kind of gets ground out of them immediately and they have to do things this set way. And this especially goes for people who end up at large companies where it’s just like, you’re not going to change anything. You’re going to get in there and you’re going to do it their way and then that’s it.

In the rare case when someone comes out of college or comes out of a training program and then they go to a startup that doesn’t have as much structure, those are really the only sorts of areas where you even have an opportunity to innovate in terms of the process of how we develop software. Because otherwise, it’s just set in stone at this point.

Corey: So, you’re given a blank slate—or a blank whiteboard, as the case may be, or God forbid, a blank Jira board—how would you structure it instead? How would you advise companies to think about software development? Since I think it’s pretty clear that an awful lot of what they’re doing today either isn’t working or is some weird bastardized hybrid of different methodologies that doesn’t really have a name other than something cynical, like ScrumBut.

Cliff: [laugh]. Yeah, so that’s a great question. So, I think where I’ll take this is, I can talk a little bit about—I mean, I’ve done this before, right, so I’ve been hired into several different startups as either, like, an engineering manager, or a director, or basically, like, hired management, and typically when a start-up hires an engineering manager or someone on that management chain, they only really do so when the pain has gotten so bad that they want to throw money at the problem. So, I specialized in that for a little bit; very thankless job, but it was interesting because what happens is that every team fails in its own unique and beautiful sort of way. [laugh].

So, one of the first places where I did this, there was a person running product; he had learned his Agile methodology from being at Booz Allen Hamilton, which, I mean, it is a nightmare factory in every metric you can measure it on. But apparently, they specialize in Agile as well as the military-industrial complex, so great. [laugh]. He was running things on a one-week sprint. And it was a shippable system, so it had a cloud component but it also had a component that was forward-deployed into a customer network.

So, he was running this where basically everyone would work on a one-week sprint; they would then do a bug-bash every Friday, and it was very much a case of, you keep doing the same thing and you keep getting the same results, and you keep doing it to see if you get different results. And they were very much in that kind of mode. So, they would do this every single week. You would have a bug-bash where the same bugs came up every single time; they wouldn’t get fixed, no one would triage them. So, the same bug was in the system in maybe, like, 10 or 15 different tickets.

No one was triaging it. And it was just a mess. And so when I got there, part of my job was to just kind of break apart this crazy structure that was happening, and again, try to design something that would actually work, again, for the product. You have to design something that has to work for how the product gets deployed. So, as I said earlier, if you have cloud services, they can deploy whenever so structuring them around 
some sort of timeframe doesn’t really make a ton of sense.

However, when you have something that gets deployed into a customer network, like an agent, or some kind of desktop software, or anything that’s on machines that you do not have direct control over, you have to factor that into the speed at which you ship, you have to factor that into your engineering process. Because if you can only ship out that executable once every quarter, or—it’s like, how fast can your customer actually consume these things, right? Most places, if you give them updates every two weeks, they’re going to say, “What are you doing? Why are you making my life hard?” In a lot of places, the fastest—especially if you’re selling to an enterprise—the fastest they can consume a forward-deployed component is once a month at the very fastest.

Usually, they prefer on a quarterly or even a six-month basis. But if that’s the case, you have to design your engineering process to account for that. Then the other part is that when you land in a place like this, you can’t just pull the rug out from everybody immediately. It’s similar to saying, “Oh, we got to do a rewrite.” It’s like, well, you can’t just do a rewrite of your engineering process either; you have to incrementally make changes to it so that people are not confused about what they’re supposed to be doing, but you’re making changes towards things running in a more smooth fashion.

So, what I typically try to do is I try to design a process, and then get the team bought into it, and then hopefully get them moving faster. And the first time I tried this, it was a disaster. That was the company I was just talking about where they were running one-week sprints. I did not know what I was doing at the time; that was a very difficult situation. Landed at another place after that where this one was a two-week Scrum; similar problems around okay, frequency of testing, you have a component that gets deployed into a customer network, how fast can 
we deploy that?

And similar sorts of problems, and so now that I could see what the pattern was, I could now develop a—I had a much better time developing a process that actually worked and helped the team ship with confidence. Which is really what you want the process to do is you want the process to be something that takes burden off of the engineering team, as opposed to something that makes your job as the engineering manager easier, which I think a lot of engineering managers approach it from that perspective of, “Oh, I can get a report at Jira and then I don’t have to talk to everybody every day,” or whatever. If you’re trying to make your job easier through the process, you are necessarily putting more burden onto your team.

Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs.
The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download 
your copy of the report now!

Corey: It feels like this is almost the early version of a similar political machination playing out where, we see it now with—there are these large companies that, once upon a time, had these big mono repos, and they had 5000 developers, and every one wound up causing problems because a group of developers is collectively referred to as a merge conflict. Then they wound up building out, “Ah, we’re going to break the monolith apart into microservices and it solves that political problem super well.” And then you wind up with a bunch of startups with five engineers working there, and they have 600 microservices running in their environment, and it feels like someone took an idea outside of the context in which it was designed for and applied it to a bunch of inappropriate areas and just bred an awful lot of complexity while actively making everything worse. Please don’t email me if people disagree with that statement. But it feels like an echo of that, doesn’t it?

Cliff: Oh, absolutely, yeah. I mean, I think a lot of this is a reflection of our relative infancy as an industry. When you think about the amount of time software engineering has been around, and has been a going concern of itself, as opposed to other engineering disciplines, I think we are still very much in our infancy. Like I was saying, they don’t really teach this sort of stuff in school, and certainly not the theory behind why you would structure things this way versus that way. In fact, most people who get promoted as a manager, you get promoted from being an engineer—someone who codes all day, or codes and does design work, but basically, someone who works as an individual contributor—then you get promoted to being a manager, and very few places give you any training or education or anything at all about how do you even do this job. And so you either sink or swim.

Corey: It’s an orthogonal skill set that basically bears little relation to what you were doing before?

Cliff: Exactly. The thing that sort of gives you any sort of ability to swim in that type of job is having the clout or the respect of your former peers as you get promoted into that. And the people who do well with that, they basically learn on the job and rely on that inbuilt respect to basically screw up a lot until they can get the hang of it. But yeah, most of the time, you don’t get an education and management or any of the other things that are not just specific to people management, but people management for software engineering, which I do think is its own discipline.

Corey: And some, I guess, almost borderline ridiculous level, it feels like no one really knows what they’re doing when it comes to management, especially in engineering. In other disciplines, it seems that management is treated as a distinct key skill, but very often—the way my management strategy evolved—and those people think I’m kidding whatever I say this, but I assure you I’m not—it came out of looking at what my terrible managers had done in the past and what didn’t work for me, and what made me quit slash become demoralized slash convince others to quit, et cetera, et cetera; or, you know, steal office supplies. Whatever it is that—how it is that you act out, and then I just did the exact opposite of those things. And I’ve been told repeatedly, “Wow, you’re a great manager.” Not really. I just don’t do all the things I hated. It gets you surprisingly far.

Cliff: Well, yeah, absolutely. But that gets you far with your own reports. There’s a whole other side to being a manager, which is dealing with the outside world. And then that’s, especially if you’re in a large organization, even in some smaller ones as well, there’s a whole dimension to the job that you as an individual engineer, you don’t even see.

That’s the politics part of the job about how do you justify what you’re doing? How do you advocate for your team? How do you operate as a quote-unquote, “Shit umbrella” for your team? And all these sorts of other things where you provide a safe harbor within the company for your team to operate, and then try to procure resources and make sure that the decision-makers above you understand the importance of what you’re doing. And no one teaches you how to do that.

Corey: Oh, never. You’re absolutely right on this. I was mostly focused on managing my reports. I completely failed in those roles managing up and, in some cases, managing sideways as well, just because that was never clear to me when I was an independent contributor working on engineering problems. It’s an evolution, on some level, of figuring out what it is that the role really is.

And all this stuff is not that complicated to teach people, but for some reason, culturally, we don’t do it. We take the Hacker News approach to things and try and figure out complex forms of interaction from first principles. And it really feels like there are some giants upon whose shoulders we could stand.

Cliff: Yeah. I mean, I agree with that. I mean, there’s definitely people in the industry who’ve written books and who are starting to try to put down that first layer of institutional knowledge to share with other folks. You got people like Camille Fournier and other folks who’ve written books specifically for engineering managers who work in the software world. Which I think is a really great first step.

But yeah, when it comes down to it, it’s like, “Okay, we’re going to implement this process; we’re going to do these things; I don’t know why.” It’s almost like no one got fired for buying IBM; no one got fired as an engineering manager for implementing Scrum. But if you try to go and do some other weird stuff, you’re running the risk of getting fired, if you fail.

Corey: There is the question of whether someone at IBM will get fired for buying Red Hat, but that’s not the analogy that people always fall back on for the last 25 years. I think that there’s also the idea that people will try and build their own thing where it makes sense for them. In complex engineering areas, that often makes sense, and sometimes it doesn’t, but then they try and approach human interaction like it’s an engineering problem, and that can lead to a lot of, frankly, disastrous outcomes, on some level. I feel like this does tie into the, I guess, almost unthinking adoption of Agile and similar methodologies or perversions of those methodologies in many large enterprises. Do you see a fix for this, or is this something that we all more or less have to live with, and watch people continue to make the same mistakes for another ten years?

Cliff: I think, for the most part, you have to—I guess, change starts at home. [laugh]. What I would advocate for is that if you have problems or qualms with the process that your particular organization is following, and you have ways you think you could fix it or changes that you’d want to make to it, then start advocating for those. And you’d be surprised about how far you can get sometimes with just saying, “Hey, can we just stop doing this, or can we do this a different way?” But I would also say that, like—one of the things you just said sort of knocked something loose from my mind, which is that even when companies share, like, “Oh, we’ve done something amazing here. We’ve designed this amazing new process, it really works well for us.”

And they write a big blog post about it, turns out if anyone ever follows up on that, they either never did it or it was never as described. And they certainly don’t do it today. So, I think a good example of this would be like when Spotify put out there—this was a number of years ago—Spotify put out their big creed about, like, “Here’s how Spotify develops software.” And they had this whole bespoke thing about they’ve got these pods of people, and you’ve got a matrix management, and they reinvented a whole bunch of stuff. And then you talk to anyone who was at Spotify during that time, and they’re like, “Yeah, we tried that; it didn’t work.” But they still put out the blog post. So. [laugh].

Corey: And I think it’s still up and hasn’t been taken down yet. It’s, “Yeah, did this work for other people?” “No, absolutely not. But it might work for us.”

Cliff: [laugh]. Yeah, it’s the same kind of trick that companies do with open-source, which is you open-source something to a bunch of fanfare and try to get people to adopt it when it hasn’t even been adopted internally. And anyone who tries figures out it’s not the right thing, and they don’t even like it. And so, but it’s like, “Oh, yeah, we can open-source it and then it comes with the imprimatur of whatever company it comes from.” I mean, this is a pretty classic joke. It’s like that old movie, The Gods Must Be Crazy, you throw the Coke bottle out of the plane; someone on the ground picks it up, and eventually ruins your life, even though it’s just a Coke bottle. Same thing with open-source; same thing with management processes.

Corey: It seems like it’s going to be one of those areas that continues to evolve whether we want it to or not. Or at least I hope because the failure is, it doesn’t.

Cliff: Yeah, I mean, hopefully it evolves. And like I said, I would say change starts at home. Try to advocate for changes on your own team and think outside of the box; try to figure out what you can get away with and try to figure out, I guess, ways to break down the walls and the rituals that the Agile consultants have set up.

Corey: Ugh. [sigh]. I hope you’re right. If people want to hear more about your thoughts on these and many other matters, where can they find you?

Cliff: Yeah, so typically, I’m just usually tweeting. So, my Twitter account is @moonpolysoft, and that’s usually where I’m doing most of my stuff. Yeah.

Corey: And we will, of course, include a link to that in the [show notes 00:25:15]. Thank you so much for taking the time to chat with me today. I really appreciate it.

Cliff: Yeah, Corey. It was great, and thanks for having me.

Corey: Cliff Moon: absolutely everything except an Agile consultant. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment that you’re going to continue to iterate on and update every two weeks, like clockwork.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Announcer: This has been a HumblePod production. Stay humble.