‘But in IT, oh, it’s Tuesday. Everything’s about to change again.’ with Leon Adato

About Leon Adato
Leon Adato is a Head Geek and technical evangelist at SolarWinds®, and is a Cisco® Certified Network Associate (CCNA), MCSE and SolarWinds Certified Professional. His experience spans financial, healthcare, food and beverage, and other industries.Before he was a SolarWinds Head Geek, Adato was a SolarWinds user for over a decade. His expertise in IT began in 1989 and has led him through roles in classroom training, desktop support, server support, and software distribution.Funny:In my sordid career, I have been an actor, bug exterminator and wild-animal remover (nothing crazy like pumas or wildebeasts. Just skunks and raccoons.), electrician, carpenter, stage-combat instructor, American Sign Language interpreter, and Sunday school teacher.Oh, and I work with computers.Since 1989 (when you got a free copy of Windows 286 on twelve 5¼” floppies when you bought a copy of Excel 1.0) I have worked as a classroom instructor, courseware designer, desktop support tech, server support engineer, and software distribution expert.Then about 16 years ago I got involved with systems monitoring. I've worked with a wide range of tools: Tivoli, Nagios, Patrol, ZenOss, OpenView, SiteScope, and of course SolarWinds. I've designed solutions for companies that were extremely modest (~10 systems) to those that were mind-bogglingly large (250,000 systems in 5,000 locations). During that time, I've had to chance to learn about monitoring all types of systems – routers, switches, load-balancers, and SAN fabric as well as windows, linux, and unix servers running on physical and virtual platforms.Full LengthLeon Adato is a Head Geek and technical evangelist at SolarWinds®, and is a Cisco® Certified Network Associate (CCNA), MCSE and SolarWinds Certified Professional (he was once a customer, after all). His 27years of network management experience spans financial, healthcare, food and beverage, and other industries.Before he was a SolarWinds Head Geek, Adato was a SolarWinds user for over a decade. His expertise in IT began in 1989 and has led him through roles as a classroom instructor, courseware designer, desktop support tech, server support engineer, and software distribution expert.In the early 2000s, Adato got involved with systems monitoring and has since worked with a wide range of tools including Tivoli®, Nagios®, Patrol, ZenOss®, OpenView, SiteScope, and of course SolarWinds. He has designed solutions for companies that were extremely modest (approx. 10 systems) to those that were mind-bogglingly large (250,000 systems in 5,000 locations), through which he gained experience monitoring all types of systems – routers, switches, load-balancers, and SAN fabric – as well as Windows®, Linux®, and UNIX® servers running on physical and virtual platforms.His career includes key roles at Rockwell Automation®, Nestle, PNC, and CardinalHealth providing server standardization, support, and network management and monitoring.

Links Referenced

Transcript
Announcer: Hello and welcome to Screaming in the Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored by InfluxData. Influx is most well known for InfluxDB, which is a time series database that you use if you need a time series database. Think Amazon Timestream except actually available for sale and has paying customers. To check out what they're doing both with their SaaS offering as well as their on-premise offerings that you can use yourself because they're open source, visit influxdata.com. My thanks to them for sponsoring this ridiculous podcast. I'm Corey Quinn. I'm joined this week by Leon Adato. Leon, welcome to the show.

Leon: Hello there.

Corey: So you're a lot of things. You're a head geek at SolarWinds. In some circles, that would be considered an insult. In your case, it's a job title.

Leon: Absolutely. I took the job almost sight unseen when I found out that was the title. I'm really happy about having it.

Corey: We all gravitate for different things. You're also the cohost of the Technically Religious podcast.

Leon: Indeed.

Corey: You do identify as an Orthodox Jew.

Leon: Also indeed.

Corey: Excellent. I myself am such a good Jew I don't need to practice, which is neither here nor there. But we met a long time ago, I want to say, at one of the Devops Days events and got to talking and for some reason, never really stopped talking about a variety of different things despite the fact that you tend to operate in a world that I don't spend a whole lot of time in, namely on-premises and in data centers.

Leon: Yeah, I mean, the fact is is that it's not just on-prem, but it is a style of IT work that I think is particularly unsexy or seen as being unsexy. But the fact is is that plumbing is... Anyone who's ever done plumbing, and I do a lot of home repair, and that's my Achilles heel, plumbing is not sexy. There is simply no way to... Despite what pornography movies might have you believe, plumbing is not fun nor sexy, and yet routers and switches and the data must flow. And that's an area that I tend to focus on, and I tend to work with people to improve.

Corey: I think that the interesting part of this too is that we've set up a false dichotomy where, oh, you're either in data centers with the ancient buffalo, or you're entirely cloud native. In practice, almost no one is. I mean, as much as I want to start casting stones like that, I look at my NAS in the corner where I do local backups and things and realize, "Huh, okay, maybe there's some unfortunate truth I don't necessarily want to acknowledge here."

Leon: That's absolutely true. And it's more true for most of your medium and especially the large businesses that simply can't get a catapult and fling everything into the stratosphere as it were. First of all, as I mentioned before, the data must flow, that at a certain point you have people sitting at desks with compute devices, and those bits have to get there somewhere. So, you still have an investment of your networking gear if nothing else. And with the networking gear comes funny little things like firewalls. And you probably have a few proxy servers and some load balancers, and all of a sudden, you have a nice little stable of on-prem equipment that still has to be cared for.

And then even when you're talking about stuff in the cloud, again, the techniques still matter. You still have to know how to subnet. You still have to know how to set up a VPN. You still have to know how servers and applications and services and processes and events work regardless of whether they are in the cloud or, as my fellow Head Geek Thomas LaRock likes to say, earthed. Regardless, the same techniques work.

Sure, many applications are transitioning to full cloud stack. They're ephemeral, and they use containers, and they are microservices and all this. Yeah, many things do that. Many things are also simply lifted and shifted into an AWS instance, and they just sit there. And they still have to be managed the way we always manage them. It's just they're not there.

But here's the funny part, in the last... So, I've been in IT for 30 years. In the last 20 of those 30 years, I have never been permitted, not just that I haven't had a habit of it, but I've never been permitted into the room where the machinery that runs the stuff that I do was kept. It was remote. Remote in Amazon's data center was as remote, or perhaps in some ways, closer to me, than the data center down the hall. So I tend to work in that space of those same, I'll say it, old, techniques, processes, standards, systems, et cetera.

Corey: I would like to point out that those are your words, not mine.

Leon: Yes. Fair enough.

Corey: So, I periodically encounter you in your natural habitat, which is of course at a booth at an expo hall at a conference purporting to talk about technology, the conference, not you. The you, in turn, talk about what SolarWinds does, and this is not a sponsored episode. We're not going to go into the realm of everything your company does, but that's what I want to mention though is it seems like you folks do kind of a lot. It almost feels like a choose your own adventure story where no matter what you're into, there's something that you folks have to demonstrate that aligns with who the customer is.

And originally, this was, okay, great. So, I see you at some of the, shall we say, legacy vendor shows, fine, whatever. But then I see you at the cloud native shows, and you've still got interesting things to talk about. What did your, I guess, progression look like? You have been doing this in data centers far longer than I ever did. But now, you've migrated to be able to have reasonably intelligent-sounding conversations about things that touch on cloud. What did that transition look like for you?

Leon: Well, I think full disclosure, my degree's in theater, so I can act like I know what the hell I'm talking about most of the time and pull it off fairly well. But as I said, I've been in IT for 30 years, so hopefully, I've earned some credibility along the way. And I actually worked my way up the IT food chain. I started off in classroom training, teaching people how to use MultiMate and WordStar and continuing for about five and a half, six years as technology progressed from there and then desktop support to server support to network support.

And then I dovetailed into this really cool bizarre corner of the IT universe of monitoring and using a variety of the usual suspects, big blue and big red and all the other colors of the rainbow, I guess, doing monitoring. Basically, my shit would watch your shit and wait for it to break and tell the right person. That mindset has helped me transition because I really do believe that monitoring engineer... Okay, go ahead and laugh. Monitoring engineer is a thing in the same way that 10 years ago, infosec professional was not a thing. You just had people who really, really enjoyed ACLs or really loved to dumpster dive through log files on systems. And now, you have infosec professionals and red team, blue team, purple team, et cetera.

Right now and for the last, say, five or six years, people have been focusing on monitoring as a discipline, understanding the failure modes, not just, oh, I monitor storage, or I know how to monitor virtual, or I know... No, no, no. It's how things fail overall and how they all fit together. And when you're focusing on that in the way that I have for, like I said, about 20 years, you start to care less about the platform, "I'm a Windows admin." "No, Linux forever," whatever. And you start to focus on this is an interesting system, and this is the way it interacts with other systems. And this is, again, how it falls down or how it gets wobbly, and here's how we can catch it at this point.

So that combined with the fact that as Ecclesiastes says, "There's nothing new under the sun," you say containers, and I think LPARs running on AIX, which are literally exactly the same thing. You can talk about virtual storage, and I think DASD or shared drives or even timeshare on a mainframe. That's a little bit beyond my time. But the point is is that a lot of the things that are being used new are really reinventions of old techniques or technologies. In fact, I was just reading this morning about how Sun was trying to invent basically cloud-based desktop compute a couple of decades ago, and it never took off because people weren't ready for the concept. But these things keep coming around, and if you've been around long enough, you see those patterns.

So with all that said, I don't care what it is that you have. I don't care whether it's physically sitting on the ground, or it's hovering in the cloud, or whatever it is because I'm just interested in what it's doing and how I can get added and into it and figure out if it's happy or sad or somewhere in between.

Corey: The hard part I think is bridging the fundamental humanity of what this industry really is about with the technology bits. And for a long time, I think back in the early part of your 30 some odd years doing this, that wasn't really as, I guess, front and center. It seemed to be at that point, there was a serious jerk problem in our industry. Whether there still is or not is a subject for a different episode and an entirely separate debate.

But right now, I think that there's a certain empathy required to get in the front door, at least any place we'd want to work. Back then, that was far from true. If you knew how to write code and offend everyone around you, well, I stopped listening after the right code. Come on in. And what's strange is that you're one of the most empathetic people that I've had the privilege of speaking to about a number of things over the years. And I guess I'm wondering, how did you become that person? Or have you always been that person, and you just learned to walk amongst the jerks?

Leon: First of all, I think that you're right. In the '90s, which is when I started getting into IT, the lone genius was still very much the model. It was also the lone owner. Remember that in the '90s was when the single guy who invented PKZIP created his utility and sharewared it until somebody bought it, and he got his, forgive the euphemism, but his fuck you money. And he's out. And lots and lots of people did that with lots and lots of software.

So the idea of the brilliant loner who could do it all himself except run the business or hire people or even talk to people, but it wasn't necessary because it was, again, that loner mentality was very much in vogue. In corporations, you had the crystal tower. You had the mainframe operators or the data center or the data processing center. And again, this was a cobble of arcane wizards who could speak their own language but couldn't really relate to anybody else in the cafeteria.

But also remember that the flip side of it, that at that time, people who worked in IT were pretty heavily mocked and kept at arm's length as outsiders. Nowadays, for the last, let's say, 15, 20 years, it's become eminently evident that the business can't run without IT, and IT has no purpose without business. The second piece is taking a little while for IT people to grok, but we're getting it.

But businesses have realized that they really live and die based on how well they technology. And that's brought the jerk problem to the fore. And I think it is getting better. I don't think it's perfect everywhere. And to your point, the places where we want to work are the ones that have solved it. Sometimes, companies don't advertise, "We employ jerks here." So yeah, sometimes it's trial and error.

As far as how I got there, like I said, my degree's in theater, and I got into tech because at the time, the definition of a trainer was the person in the room who knew what was on the next page of the manual. They didn't have to know much else. I give a really funny DOS basics class, really thigh-slapping funny, and Windows basics and all that stuff. And I did a lot of that.

So when you're in a room with five to 10 people who used to be part of the typing pool on their IBM Selectrics, and they now know that if they cannot learn what they need to know in the next six hours, they will no longer have a job, and that's what comes in at nine o'clock in the morning when class begins, and it's on you. You do need to develop a bit of empathy. You do need to make sure that they know that you know what's on the line for them. And that continues. Again, desktop support, you're walking desk to desk, or you're working a help desk, you really do have to know what this person's going through, what they need, or else, you're going to hate them, and you're going to hate your job pretty quickly.

So I like to think that I'm a naturally empathetic person, but it hasn't hurt me. Also, being an extrovert has made me a flying pink unicorn of IT, especially back 30 years ago when it was populated by a lot of what I call green sock, blue sock people, the ones who didn't match their clothes and really didn't relate to other people particularly well. And as you can tell, I'm really shy. So that also has helped me to be in the kind of role I'm in now, which is, really, a storyteller and a story listener, listening to people and finding out what their challenges are and how they overcame them, or they haven't yet overcome them and putting them in touch with the right people, or helping to develop the right solutions that will get them on their way to fame, fortune, and success, or at least getting home on time.

Corey: I frequently said that multi-cloud is a stupid best practice, and I stand by that. However, if your customers are in multiple clouds and you're a platform, you probably want to be where your customers are unless you enjoy turning down money. An example of that is InfluxData. InfluxData are the manufacturers of InfluxDB, a time series database that you'll use if you need a time series database. Check them out at influxdb.com.

And that's really the trick is it seems that finding a way to meet people where they are is critically important. People sometimes ask how I wound up becoming the monster that I am now. And the piece that I think that I, I guess, give some of the most credit to has been I started off giving a bunch of terrible freaking conference talks in 2012, 2013 talking about SaltStack, which was a fantastic technology that soundly lost the configuration management wars. And that was fun for a while, and it turns out that putting your documentation on a slide and reading it to people is a terrible freaking way of building a reputation and learning how to give a fun talk. So in time, I asked people, "So how did it go?" And their response was, "Well, that was great. Now, here's how you make it even better." Yeah, people had the grace to approach me in that way.

And a year later, I wound up getting a traveling trainer job under contract for Puppet, which lost the configuration management wars in a radically different way. And that was a fantastic learning experience for lack of a better term because again, I knew what I was doing, but I also wasn't so confident about it that I felt like I could teach it. That changed rapidly because it turns out when you teach the same thing week after week, you learn how things break. And because of some of the dynamics we've already touched on, I was, in many cases, dealing with people who thought that I was the representative of the company that was coming to take their job away. And that was a interesting story. Then the demos would break periodically because hey, computers, and good luck, and oh, if they hate you enough, you don't have this job anymore.

So it was a really interesting story as far as learning to A, think on my feet, B, present to a potentially hostile audience, and C, amuse myself. And there was a weird progression there as the first one, I was freaking terrified. The second one, third one, was okay. I got it. The fourth one, no, I don't got it. But over time, I got better at it, and it went from exciting and terrifying to rote to routine, and then it was a matter of finding something new. But that was the training perspective.

What was interesting to me, it was talking to a lot of the students and seeing their transformation, their progression as they started wrapping their head around how a lot of this works. Now, that was Puppet, which was interesting and had its own DSL style of approach, but okay, fundamentally, all it was doing under the hood was managing some baseline primitives, some files, users, permissions, run random executions there and install packages, and you're more or less done. I can't imagine what it would look like today to be a trainer to, "Now, I'm here to teach you, pick your cloud provider of choice, how to use that in the world that you're operating with," and everyone just stares. How do you begin? How do you even start with something like that? I think it comes back to empathy. I think that trying to put yourself in that situation, even if you're not, is critical. I think the best teachers are great at walking that journey with their students.

Leon: You're a lovable, fuzzy, cuddly monster whom no one should ever cross because you will eviscerate them on Twitter if nowhere else.

Corey: Those are your words, not mine.

Leon: I understand. And you had a lot of what I think a lot of us call character-building experiences. Those are always good. But I think you're right. I think, first of all, you never understand something quite as well until you've had to teach it to somebody else. And the process of teaching not only forces you to build what I call technical empathy, the ability to see something that is familiar to you from a completely foreign perspective, but also, it gives you a chance to hear how others are using something with which you are somewhat familiar.

Again, once you get past that first class you taught that was terrifying, you were still familiar with it. You just didn't know what was going to happen. But you're taking something that you are familiar with and seeing it through their experience. And sometimes it's like, "Oh, wow, I've been using this completely differently than anybody else," which can either be, "Well, I'm a genius. I found this new use for dragon's blood," or it can be, "Wow, nobody understands what this product is supposed to do." So then you become an evangelist of showing people the one true way to Puppet or to whatever.

But at the same time, one of the best parts about training isn't, "Hey, I know this. Let me show you." The best part about training is, "Wow, that's a really interesting question. No one's ever asked that before. Let's find out."

Corey: Those were the best moments. The fun thing too is at some point when you're teaching the same curriculum pretty consistently again and again and again, you can find the common failure modes so frequently that you don't even need to walk around and look at their screen. You can tell them they dropped a comma, and they look at you like you can read their minds. And you can't. You just see the exact same thing manifest itself enough times that by the time it comes around there, it's old hat. The first time you see something, it's new and revolutionary. The sixth time you see something, oh, well, that's just how it works. I've seen it. There's no magic to it anymore. I've seen how the sausage gets made.

Leon: Right, so I, again, talking about that experience both from your side of the teacher's desk and the other side of the teacher's desk, one of the things I love about the Orthodox Jewish educational system called yeshiva is that the highest compliment you can be given by a teacher is not, "That's a really good answer." That's not the highest compliment, the highest compliment, and it's always given in Yiddish, Du fregst a gute kashe, you asked a really good question. Now, the funny part about it is that it may not be an original question. Remember that Jewish thought and Jewish conversations have been going on for at least 3,000 years.

Corey: Jewish arguments but only holidays.

Leon: Well, question, answer, argument about why that's the answer, et cetera, the dialogue, the debate has been going on for quite some time. So the odds of you asking an original question are fairly low. That's not going to happen. But the fact that you asked a question that somebody from 1100 or whatever also asked means, wow, you are thinking along the same lines as these great minds. That's a really good question. Now, let's go into the answer. Now, let's dig into where that line of thinking takes you.

And again, I think that's a very IT practitioner way of thinking. We don't get into this business because we want to stay in a steady state of I already knew that. None of us... Because if that's what you need, please, accounting, or I don't know, almost anything else. And you can say, "Yeah, I know that. Things have changed very little," but in IT, oh, it's Tuesday. Everything's about to change again.

So we revel in the idea of let me find out. I think that's our sweet spot as folks who build our careers in IT. So yeah, when you're teaching, what else is there, right? You've seen all the common failure modes. You can read their faces, if not their minds. Occasionally, they ask an interesting question. That's the part that you're looking for, for that spark.

Corey: I think that that is the single biggest divergence that I've seen from engineers and IT folk and the rest from the start of my career to the present day where... I'll use myself as an example. I started off my career as a subject matter expert in large scale email systems. I considered myself an expert in Postfix of all things because, not to toot my own horn, I was because it's a sad thing that not many people had to care about. But I did, and it was fun.

But I looked around the ecosystem and saw that things like managed email services were very clearly on the rise, and given at the time I was 23 years old, give or take, and I didn't have the confidence that this was going to be something that I could build an entire 40 some odd year career out of, I felt the world changing out from under me. And my answer to it for better or worse was, okay, well, if this isn't going to work, then what can I do instead? What do I pivot into? As we've already covered, my pivot was into configuration management, and then that changed, and now, I claim to know a lot of things about cloud, and I don't. But no one calls me on it because I'm confident and I'm loud on the internet as you've observed.

The fun part though is that you have to reinvent yourself. Let's say that AWS crashes itself into the sea through its own terrible naming of services that drive it out of business. I have two choices. I can either continue to shake my fist at a platform that is eroding out from under or I can pivot what I know into something that has more business relevance. And I say this now, and it sounds ridiculous, "Oh, how would Amazon ever go away?" Yeah, go back in time. How would Solaris ever go away? And you start to see that these shifts happen all the time. I don't think there is anything safe in the world of technology that you could start working on today as a new graduate and still be doing something even remotely related to it 40 some odd years later. And that's going to be a fascinating evolution point.

The question is, how will you reinvent yourself? Now, it's scary. Don't get me wrong, but it's also not what people think it is because I don't think you're coming back at this from a perspective of starting over at entry level. No, you take the things you know and parlay that laterally into something else that's tangentially related to the thing that you were working on and bring the baseline fundamentals with you. And it doesn't take long to develop some semblance of mastery in this new arena given a baseline level of experience and exposure where you started.

Leon: Yeah, it's amazing how many things dovetail, and it's also amazing how many things that are tangential to the work you do today become front and center later. But you realize that that tangential experience really gave you everything you need. And here's a simple but powerful example, as I said, one of the things that I taught early on in my career was word processors, and WordPerfect for DOS was a thing, right? And WordPerfect had a function called reveal codes. Bolding wasn't just a matter of highlighting because there was no mouse, and therefore, there was... You had to go in front of the word, and you had to say, "Make it bold," and then you go to the end of the word and make it turn off the bold.

And there were these little brackets, the greater than, less than symbol, and it would say... Sorry, no, they were square brackets. Let me go back there. And there were these little brackets, just the square brackets. And it would say bold at the beginning, and then you'd move to the end and say bold at the end and those kinds of things. And to see to make sure that you hadn't accidentally bolded an extra space or something like that, you would turn on reveal codes.

And so you'd see in reveal codes, words and these brackets, bold, paragraph, heading, underline. Sounds kind of like HTML, right? Because it kind of is. And when HTML came around six, seven years later for me, it took me zero time to get up to speed and start doing nascent web development because it's like, "Oh, it's just like reveal codes in WordPerfect." Now, reveal codes wasn't the be all and end all of WordPerfect. There was plenty of other things that that word processor did. And I am proud to say that I was a WordPerfect certified resource. And I fought really, really hard to get that certification, and I'm never giving it up. But reveal codes was not the most important piece of it. It was tangential. And yet, five, six years later, there was something where I was pivoting from doing desktop support to building a web design company on the side, part of my side hustle. And that one experience allowed me to really pivot quickly and smoothly into that new area.

So the mindset for those people who are listening, and they're thinking, "I don't have it in me. I can't do it," the mindset is not, "I'm going to reinvent myself. I have to completely transform myself." The mindset is to just remember that as IT practitioners, we have committed to being lifelong learners. And we learn things, whether it's to paraphrase an XKCD cartoon that one weekend I spent screwing around with Perl, or that one YouTube video I listened to, or that week I really dug deep into XML, or what have you. Those are the things that feed the next thing you're going to do. You will be amazed at where those tangential experiences crop up, and they end up becoming like a secret superpower that you didn't know you had.

Corey: If people want to hear more about the wise things you have to say, whatever they might be, where can they find you?

Leon: So you can find me... My blog is adatosystems.com, and there, I list out all the things I've written and all the videos I've done. But the other thing is I run a podcast also because I am a middle-aged white man, and we all have our podcasts.

Corey: A group of middle-aged white men is collectively known as a podcast.

Leon: Right, exactly, that's the grouping, right? Fish, they're in schools and-

Corey: Developers are merge conflicts, et cetera, et cetera.

Leon: Right, right. Exactly. Yeah. Head geeks, by the way, move in chaos. There's a chaos of head geeks. So anyway, so I have a podcast. And myself and Josh Biggley, who is Mormon, and let's see, Keith Townsend, who is Baptist, and Al Rasheed who is Muslim, and you're starting to get the theme here, we've all been in it for decades each. And there's a few more of us. And we've been in IT for decades each. We all have a very strong religious, moral, ethical point of view. We've got evangelical folks. We've got atheists. And the podcast is called Technically Religious, and we talk about the intersection between our strongly held religious and moral ethical views and our work in IT and how we make those two things, in fact, not conflict, but become synergistic between the two. And so you can find that at technicallyreligious.com or anywhere that fine podcasts are peddled to the masses.

Corey: Excellent. Thank you again for taking the time to speak with me today. I appreciate it as always.

Leon: Well, thank you for having me. This was amazing. I can't wait to hear future episodes of your stuff and to see you at another conference with old fogies with retired technology.

Corey: I don't believe we'd be able to recognize each other if we weren't wearing conference badges at the time.

Leon: Exactly.

Corey: Leon Adato, Head Geek at SolarWinds and all-around decent human being. I'm Corey Quinn. This is Screaming in the Cloud.

Announcer: This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com or wherever fine snark is sold.

This has been a HumblePod production. Stay humble.