When Data is Your Brand and Your Job with Joe Karlsson

Joe Karlsson (He/They) is a Software Engineer turned Developer Advocate at Tinybird. He empowers developers to think creatively when building data intensive applications through demos, blogs, videos, or whatever else developers need.

Joe's career has taken him from building out database best practices and demos for MongoDB, architecting and building one of the largest eCommerce websites in North America at Best Buy, and teaching at one of the most highly-rated software development boot camps on Earth. Joe is also a TEDx Speaker, film buff, and avid TikToker and Tweeter.

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Are you navigating the complex web of API management, microservices, and Kubernetes in your organization? Solo.io is here to be your guide to connectivity in the cloud-native universe!

Solo.io, the powerhouse behind Istio, is revolutionizing cloud-native application networking. They brought you Gloo Gateway, the lightweight and ultra-fast gateway built for modern API management, and Gloo Mesh Core, a necessary step to secure, support, and operate your Istio environment.

Why struggle with the nuts and bolts of infrastructure when you can focus on what truly matters - your application. Solo.io’s got your back with networking for applications, not infrastructure. Embrace zero trust security, GitOps automation, and seamless multi-cloud networking, all with Solo.io.

And here’s the real game-changer: a common interface for every connection, in every direction, all with one API. It’s the future of connectivity, and it’s called Gloo by Solo.io.

DevOps and Platform Engineers, your journey to a seamless cloud-native experience starts here. Visit solo.io/screaminginthecloud today and level up your networking game.

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn and I am joined today by someone from well, we’ll call it the other side of the tracks, if I can—

Joe: [laugh].

Corey: —be blunt and disrespectful. Joe Karlsson is a data engineer at Tinybird, but I really got to know who he is by consistently seeing his content injected almost against my will over on the TikToks. Joe, how are you?

Joe: I’m doing so well and I’m so sorry for anything I’ve forced down your throat online. Thanks for having me, though.

Corey: Oh, it’s always a pleasure to talk to you. No, the problem I’ve got with it is that when I’m in TikTok mode, I don’t want to think about computers anymore. I want to find inane content that I can just swipe six hours away without realizing it because that’s how I roll.

Joe: TikTok is too smart, though. I think it knows that you are doing a lot of stuff with computers and even if you keep swiping away, it’s going to keep serving it up to you.

Corey: For a long time, it had me pinned as a lesbian, which was interesting. Which I suppose—

Joe: [laugh]. It happened to me, too.

Corey: Makes sense because I follow a lot of women who are creators in comics and the rest, but I’m not interested in the thirst trap approach. So, it’s like, “Mmm, this codes as lesbian.” Then they started showing me ads for ADHD, which I thought was really weird until I’m—oh right. I’m on TikTok. And then they started recommending people that I’m surprised was able to disambiguate until I realized these people have been at my house and using TikTok from my IP address, which probably is going to get someone murdered someday, but it’s probably easy to wind up doing an IP address match.

Joe: I feel like I have to, like, separate what is me and what is TikTok, like, trying to serve it up because I’ve been on lesbian TikTok, too, ADHD, autism, like TikTok. And, like, is this who I am? I don’t know. [unintelligible 00:02:08] bring it to my therapist.

Corey: You’re learning so much about yourself based upon an algorithm. Kind of wild, isn’t it?

Joe: [laugh]. Yeah, I think we may be a little, like, neuro-spicy, but I think it might be a little overblown with what TikTok is trying to diagnose us with. So, it’s always good to just keep it in check, you know?

Corey: Oh, yes. So, let’s see, what’s been going on lately? We had Google Next, which I think the industry largely is taking not seriously enough. For years, it felt like a try-hard, me too version of re:Invent. And this year, it really feels like it’s coming to its own. It is defining itself as something other than oh, us too.

Joe: I totally agree. And that’s where you and I ran into recently, too. I feel like post-Covid I’m still, like, running into people I met on the internet in real life, and yeah, I feel like, yeah, re:Invent and Google Next are, like, the big ones.

I totally agree. It feels like—I mean, it’s definitely, like, heavily inspired by it. And it still feels like it’s a little sibling in some ways, but I do feel like it’s one of the best conferences I’ve been to since, like, a pre-Covid 2019 AWS re:Invent, just in terms of, like… who was there. The energy, the vibes, I feel like people were, like, having fun. Yeah, I don’t know, it was a great conference this year.

Corey: Usually, I would go to Next in previous years because it was a great place to go to hang out with AWS customers. These days, it feels like it’s significantly more than that. It’s, everyone is using everything at large scale. I think that is something that is not fully understood. You talk to companies that are, like, Netflix, famously all in on AWS. Yeah, they have Google stuff, too.

Everyone does. I have Google stuff. I have a few things in Azure, for God’s sake. It’s one of those areas where everything starts to diffuse throughout a company as soon as you hire employee number two. And that is, I think, the natural order of things. The challenge, of course, is the narrative people try and build around it.

Joe: Yep. Oh, totally. Multi-cloud’s been huge for you know, like, starting to move up. And it’s impossible not to. It was interesting seeing, like, Google trying to differentiate itself from Azure and AWS. And, Corey, I feel like you’d probably agree with this, too, AI was like, definitely the big buzzword that kept trying to, like—

Corey: Oh, God. Spare me. And I say that, as someone who likes AI, I think that there’s a lot of neat stuff lurking around and value hiding within generative AI, but the sheer amount of hype around it—and frankly—some of the crypto bros have gone crashing into the space, make me want to distance myself from it as far as humanly possible, just because otherwise, I feel like I get lumped in with that set. And I don’t want that.

Joe: Yeah, I totally agree. I know it feels like it’s hard right now to, like, remain ungrifty, but, like, still, like—trying—I mean, everyone’s trying to just, like, hammer in an AI perspective into every product they have. And I feel like a lot of companies, like, still don’t really have a good use case for it. You’re still trying to, like, figure that out. We’re seeing some cool stuff.

Honestly, the hard part for me was trying to differentiate between people just, like, bragging about OpenAI API addition they added to the core product or, like, an actual thing that’s, like, AI is at the center of what it actually does, you know what I mean? Everything felt like it’s kind of like tacked on some sort of AI perspective to it.

Corey: One of the things that really is getting to me is that you have these big companies—Google and Amazon most notably—talk about how oh, well, we’ve actually been working with AI for decades. At this point, they keep trying to push out how long it’s been. It’s like, “Okay, then not for nothing, then why does”—in Amazon’s case—“why does Alexa suck? If you’ve been working on it for this long, why is it so bad at all the rest?” It feels like they’re trying to sprint out with a bunch of services that very clearly were not conceptualized until Chat-Gippity’s breakthrough.

And now it’s oh, yeah, we’re there, too. Us, too. And they’re pivoting all the marketing around something that, frankly, they haven’t demonstrated excellence with. And I feel like they’re leaving a lot of their existing value proposition completely in the dust. It’s, your customers are not using you because of the speculative future, forward-looking AI things; it’s because you are able to solve business problems today in ways that are not highly speculative and are well understood. That’s not nothing and there needs to be more attention paid to that. And I feel like there’s this collective marketing tripping over itself to wrap itself in hype that does them no services.

Joe: I totally agree. I feel like honestly, just, like, a marketing perspective, I feel like it’s distracting in a lot of ways. And I know it’s hot and it’s cool, but it’s like, I think it’s harder right now to, like, stay focused to what you’re actually doing well, as opposed to, like, trying to tack on some AI thing. And maybe that’s great. I don’t know.

Maybe that’s—honestly, maybe you’re seeing some traction there. I don’t know. But I totally agree. I feel like everyone right now is, like, selling a future that we don’t quite have yet. I don’t know. I’m worried that what’s going to happen again, is what happened back in the IBM Watson days where everyone starts making bold—over-promising too much with AI until we see another AI winter again.

Corey: Oh, the subtext is always, we can’t wait to fire our entire customer service department. That one—

Joe: Yeah.

Corey: Just thrills me.

Joe: [laugh].

Corey: It’s like, no, we’re just going to get rid of junior engineers and just have senior engineers. Yeah, where do you think those people come from, by the way? We aren’t—they aren’t just emerging fully formed from the forehead of some god somewhere. And we’re also seeing this wild divergence from reality. Remember, I fix AWS bills for a living. I see very large companies, very large AWS spend.

The majority of spend remains on EC2 across the board. So, we don’t see a lot of attention paid to that at re:Invent, even though it’s the lion’s share of everything. When we do contract negotiations, we talk about generative AI plan and strategy, but no one’s saying, oh, yeah, we’re spending 100 million a year right now on AWS but we should commit 250 because of all this generative AI stuff we’re getting into. It’s all small-scale experimentation and seeing if there’s value there. But that’s a far cry from being the clear winner what everyone is doing.

I’d further like to point out that I can tell that there’s a hype cycle in place and I’m trying to be—and someone’s trying to scam me. As soon as there’s a sense of you have to get on this new emerging technology now, now, now, now, now. I didn’t get heavily into cloud till 2016 or so and I seem to have done all right with that. Whenever someone is pushing you to get into an emerging thing where it hasn’t settled down enough to build a curriculum yet, I feel like there’s time to be cautious and see what the actual truth is. Someone’s selling something; if you can’t spot the sucker, chances are, it’s you.

Joe: [laugh]. Corey, have you thought about making an AI large language model that will help people with their cloud bills? Maybe just feed it, like, your invoices [laugh].

Corey: That has been an example, I’ve used a number of times with a variety of different folks where if AI really is all it’s cracked up to be, then the AWS billing system is very much a bounded problem space. There’s a lot of nuance and intricacy to it, but it is a finite set of things. Sure, [unintelligible 00:08:56] space is big. So, training something within those constraints and within those confines feels like it would be a terrific proof-of-concept for a lot of these things. Except that when I’ve experimented a little bit and companies have raised rounds to throw into this, it never quite works out because there’s always human context involved. The, oh yeah, we’re going to wind up turning off all those idle instances, except they’re in idle—by whatever metric you’re using—for a reason. And the first time you take production down, you’re not allowed to save money anymore.

Joe: Nope. That’s such a good point. I agree. I don’t know about you, Corey. I’ve been fretting about my job and, like, what I’m doing. I write a lot, I do a lot of videos, I’m programming a lot, and I think… obviously, we’ve been hearing a lot about, you know, if it’s going to replace us or not. I honestly have been feeling a lot better recently about my job stability here. I don’t know. I totally agree with you. There’s always that, like, human component that needs to get added to it. But who knows, maybe it’s going to get better. Maybe there’ll be an AI-automated billing management tool, but it’ll never be as good as you, Corey. Maybe it will. I don’t know. [laugh].

Corey: It knows who I am. When I tell it to write in the style of me and give it a blog post topic and some points I want to make, almost everything it says is wrong. But what I’ll do is I’ll copy that into a text editor, mansplain-correct the robot for ten minutes, and suddenly I’ve got the bones of a decent rough draft because. And yeah, I’ll wind up plagiarizing three or four words in a row at most, but that’s okay. I’m plagiarizing the thing that’s plagiarizing from me and there’s a beautiful symmetry to that. What I don’t understand is some of the outreach emails and other nonsensical stuff I’ll see where people are letting unsupervised AI just write things under their name and sending it out to people. That is anathema to me.

Joe: I totally agree. And it might work today, it might work tomorrow, but, like, it’s just a matter of time before something blows up. Corey, I’m curious. Like, personally, how do you feel about being in the ChatGPT, like, brain? I don’t know, is that flattering? Does that make you nervous at all?

Corey: Not really because it doesn’t get it in a bunch of ways. And that’s okay. I found the same problem with people. In my time on Twitter, when I started live-tweet shitposting about things—as I tend to do as my first love language—people will often try and do exactly that. The problem that I run into is that, “The failure mode of ‘clever’ is ‘asshole,’” as John Scalzi famously said, and as a direct result of that, people wind up being mean and getting it wrong in that direction.

It’s not that I’m better than they are. It’s, I had a small enough following, and no one knew who I was in my mean years, and I realized I didn’t feel great making people sad. So okay, you’ve got to continue to correct the nosedive. But it is perilous and it is difficult to understand the nuance. I think occasionally when I prompt it correctly, it comes up with some amazing connections between things that I wouldn’t have seen, but that’s not the same thing as letting it write something completely unfettered.

Joe: Yeah, I totally agree. The nuance definitely gets lost. It may be able to get, like, the tone, but I think it misses a lot of details. That’s interesting.

Corey: And other people are defending it when that hallucinates. Like, yeah, I understand there are people that do the same thing, too. Yeah, the difference is, in many cases, lying to me and passing it off otherwise is a firing offense in a lot of places. Because if you’re going to be 19 out of 20 times, you’re correct, but 5% wrong, you’re going to bluff, I can’t trust anything you tell me.

Joe: Yeah. It definitely, like, brings your, like—the whole model into question.

Corey: Also, remember that my medium for artistic creation is often writing. And I think that, on some level, these AI models are doing the same things that we do. There are still turns of phrase that I use that I picked up floating around Usenet in the mid-90s. And I don’t remember who said it or the exact context, but these words and phrases have entered my lexicon and I’ll use them and I don’t necessarily give credit to where the first person who said that joke 30 years ago. But it’s a—that is how humans operate. We are influenced by different styles of writing and learn from the rest.

Joe: True.

Corey: That’s a bit different than training something on someone’s artistic back catalog from a painting perspective and then emulating it, including their signature in the corner. Okay, that’s a bit much.

Joe: [laugh]. I totally agree.

Corey: So, we wind up looking right now at the rush that is going on for companies trying to internalize their use of enterprise AI, which is kind of terrifying, and it all seems to come back to data.

Joe: Yes.

Corey: You work in the data space. How are you seeing that unfold?

Joe: Yeah, I do. I’ve been, like, making speculations about the future of AI and data forever. I’ve had dreams of tools I’ve wanted forever, and I… don’t have them yet. I don’t think they’re quite ready yet. I don’t know, we’re seeing things like—tha—I think people are working on a lot of problems.

For example, like, I want AI to auto-optimize my database. I want it to, like, make indexes for me. I want it to help me with queries or optimizing queries. We’re seeing some of that. I’m not seeing anyone doing particularly well yet. I think it’s up in the air.

I feel like it could be coming though soon, but that’s the thing, though, too, like, I mean, if you mess up a query, or, like, a… large language model hallucinates a really shitty query for you, that could break your whole system really quickly. I feel like there still needs to be, like, a human being in the middle of it to, like, kind of help.

Corey: I saw a blog post recently that AWS put out gave an example that just hard-coded a credential into it. And they said, “Don’t do this, but for demonstration purposes, this is how it works.” Well, that nuance gets lost when you use that for AI training and that’s, I think, in part, where you start seeing a whole bunch of the insecure crap these things spit out.

Joe: Yeah, I totally agree. Well, I thought the big thing I’ve seen, too, is, like, large language models typically don’t have a secure option and you’re—the answer is, like, help train the model itself later on. I don’t know, I’m sure, like, a lot of teams don’t want to have their most secret data end up public on a large language model at some point in the future. Which is, like, a huge issue right now.

Corey: I think that what we’re seeing is that you still need someone with expertise in a given area to review what this thing spits out. It’s great at solving a lot of the busy work stuff, but you still need someone who’s conversant with the concepts to look at it. And that is, I think, something that turns into a large-scale code review, where everyone else just tends to go, “Oh, okay. We’re—do this with code review.” “Oh, how big is the diff?” “50,000 lines.” “Looks good to me.” Whereas, “Three lines.” “I’m going to criticize that thing with four pages of text.” People don’t want to do the deep-dive stuff, and—when there’s a huge giant project that hits. So, they won’t. And it’ll be fine, right up until it isn’t.

Joe: Corey, you and I know people and developers, do you think it’s irresponsible to put out there an example of how to do something like that, even with, like, an asterisk? I feel like someone’s going to still go out and try to do that and probably push that to production.

Corey: Of course they are.

Joe: [laugh].

Corey: I’ve seen this with some of my own code. I had something on Docker Hub years ago with a container that was called ‘Terrible Ideas.’ And I’m sure being used in, like—it was basically the environment I use for a talk I gave around Git, which makes sense. And because I don’t want to reset all the repositories back to the way they came from with a bunch of old commands, I just want a constrained environment that will be the same every time I give the talk. Awesome.

I’m sure it’s probably being run in production at a bank somewhere because why wouldn’t it be? That’s people. That’s life. You’re not supposed to just copy and paste from Chat-Gippity. You’re supposed to do that from Stack Overflow like the rest of us. Where do you think your existing code’s coming from in a lot of these shops?

Joe: Yep. No, I totally agree. Yeah, I don’t know. It’ll be interesting to see how this shakes out with, like, people going to doing this stuff, or how honest they’re going to be about it, too. I’m sure it’s happening. I’m sure people are tripping over themselves right now, [adding 00:16:12].

Corey: Oh, yeah. But I think, on some level, you’re going to see a lot more grift coming out of this stuff. When you start having things that look a little more personalized, you can use it for spam purposes, you can use it for, I’m just going to basically copy and paste what this says and wind up getting a job on Upwork or something that is way more than I could handle myself, but using this thing, I’m going to wind up coasting through. Caveat emptor is always the case on that.

Joe: Yeah, I totally agree.

Corey: I mean, it’s easy for me to sit here and talk about ethics. I believe strongly in doing the right thing. But I’m also not worried about whether I’m able to make rent this month or put food on the table. That’s a luxury. At some point, like, a lot of that strips away and you do what you have to do to survive. I don’t necessarily begrudge people doing these things until it gets to a certain point of okay, now you’re not doing this to stay alive anymore. You’re doing this to basically seek rent.

Joe: Yeah, I agree. Or just, like, capitalize on it. I do think this is less—like, the space is less grifty than the crypto space, but as we’ve seen over and over and over and over again, in tech, there’s a such a fine line between, like, a genuinely great idea, and somebody taking advantage of it—and other people—with that idea.

Corey: I think that’s one of those sad areas where you’re not going to be able to fix human nature, regardless of the technology stack you bring to bear.

Joe: Yeah, I totally agree.

Corey: So, what else are you seeing these days that interesting? What excites you? What do you see that isn’t getting enough attention in the space?

Joe: I don’t know, I guess I’m in the data space, I’m… the thing I think I do see a lot of is huge interest in data. Data right now is the thing that’s come up. Like, I don’t—that’s the thing that’s training these models and everyone trying to figure out what to do with these data, all these massive databases, data lakes, whatever. I feel like everyone’s, kind of like, taking a second look at all of this data they’ve been collecting for years and haven’t really known what to do with it and trying to figure out either, like, if you can make a model out of that, if you try to, like… level it up, whatever. Corey, you and I were joking around recently—you’ve had a lot of data people on here recently, too—I feel like us data folks are just getting extra loud right now. Or maybe there’s just the data spaces, that’s where the action’s at right now.

I don’t know, the markets are really weird. Who knows? But um, I feel like data right now is super valuable and more so than ever. And even still, like, I mean, we’re seeing, like, companies freaking out, like, Twitter and Reddit freaking out about accessing their data and who’s using it and how. I don’t know, I feel like there’s a lot of action going on there right now.

Corey: I think that there’s a significant push from the data folks where, for a long time data folks were DBAs—

Joe: Yeah.

Corey: —let’s be direct. And that role has continued to evolve in a whole bunch of different ways. It’s never been an area I’ve been particularly strong in. I am not great at algorithmic complexity, it turns out, you can saturate some beefy instances with just a little bit of data if your queries are all terrible. And if you’re unlucky—as I tend to be—and have an aura of destroying things, great, you probably don’t want to go and make that what you do.

Joe: [laugh]. It’s a really good point. I mean, I don’t know about, like, if you blow up data at a company, you’re probably going to be in big trouble. And especially the scale we’re talking about with most companies these days, it’s super easy to either take down a server or generate an insane bill off of some shitty query.

Corey: Oh, when I was at Reach Local years and years ago—my first Linux admin job—when I broke the web server farm, it was amusing; when I broke part of the data warehouse, nobody was laughing.

Joe: [laugh]. I wonder why.

Corey: It was a good faith mistake and that’s fair. It was a convoluted series of things that set up and honestly, the way the company and my boss responded to me at the time set the course of the rest of my career. But it was definitely something that got my attention. It scares me. I’m a big believer in backups as a direct result.

Joe: Yeah. Here’s the other thing, too. Actually, our company, Tinybird, is working on versioning with your data sources right now and treating your data sources like Git, but I feel like even still today, most companies are just run by some DBA. There’s, like, Mike down the hall is the one responsible keeping their SQL servers online, keeping them rebooted, and like, they’re manually updating any changes on there.

And I feel like, generally speaking across the industry, we’re not taking data seriously. Which is funny because I’m with you on there. Like, I get terrified touching production databases because I don’t want anything bad to happen to them. But if we could, like, make it easier to rollback or, like, handle that stuff, that would be so much easier for me and make it, like, less scary to deal with it. I feel like databases and, like, treating it as, like, a serious DevOps practice is not really—I’m not seeing enough of it. It’s definitely, people are definitely doing it. Just, I want more.

Corey: It seems like with data, there’s a lack of iterative approaches to it. A line that someone came up with when I was working with them a decade and change ago was that you can talk about agile all you want, but when it comes to payments, everyone’s doing waterfall. And it feels like, on some level, data’s kind of the same.

Joe: Yeah. And I don’t know, like, how to fix it. I think everyone’s just too scared of it to really touch it. Migrating over to a different version control, trying to make it not as manual, trying to iterate on it better, I think it’s just—I don’t blame them. It’s hard, it really takes a long time, making sure everything, like, doesn’t blow up while you’re doing a migration is a pain in the ass. But I feel like that would make everyone’s lives so much easier if, like, you could, like, treat it—understand your data and be able to rollback easier with it.

Corey: When you take a look across the ecosystem now, are you finding that things have improved since the last time I was in the space, where the state of the art was, “Oh, we need some developer data. We either have this sanitized data somewhere or it’s a copy of production that we move around, but only a small bit.” Because otherwise, we always found that oh, that’s an extra petabyte of storage was going on someone’s developer environment they messed up on three years ago, they haven’t been here for two, and oops.

Joe: I don’t. I have not seen it. Again, that’s so tricky, too. I think… yeah, the last time I, like, worked doing that was—usually you just have a really crappy version of production data on staging or development environments and it’s hard to copy those over. I think databases are getting better for that.

I’ve been working on, like, the real-time data space for a long time now, so copying data over and kind of streaming that over is a lot easier. I do think seeing, like, separating storage and compute can make it easier, too. But it depends on your data stack. Everyone’s using everything all the time and it’s super complicated to do that. I don’t know about you, Corey, too. I'm sure you’ve seen, like, services people running, but I feel like we’ve made a switch as an industry from, like, monoliths to microservices.

Now, we’re kind of back in the monolith era, but I’m not seeing that happen in the database space. We’re seeing, like, data meshing and lots of different databases. I see people who, like, see the value of data monoliths, but I don’t see any actual progress in moving back to a single source of [truth of the data 00:23:02]. And I feel like the cat’s kind of out of the bag on all the data existing everywhere, all the time, and trying to wrangle that up.

Corey: This stuff is hard and there’s no easy solution here. There just isn’t.

Joe: Yeah, there’s no way. And embracing that chaos, I think, is going to be huge. I think you have to do it right now. Or trying to find some tool that can, like, wrangle up a bunch of things together and help work with them all at once. Products need to meet people where they’re at, too. And, like, data is all over the place and I feel like we kind of have to, like, find tooling that can kind of help work with what you have.

Corey: It’s a constant challenge, but also a joy, so we’ll give it that.

Joe: [laugh].

Corey: So, I have to ask. Your day job has you doing developer advocacy at Tinybird—

Joe: Yes.

Corey: But I had to dig in to find that out. It wasn’t obvious based upon the TikToks and the Twitter nonsense and the rest. How do you draw the line between day job and you as a person shitposting on the internet about technology?

Joe: Corey, I’d be curious to hear your thoughts on this, too. I don’t know. I feel like I’ve been in different places where, like, my job is my life. You know what I mean? There’s a very thin line there. Personally, I’ve been trying to take a step back from that, just from a mental health perspective. Having my professional life be so closely tied to, like, my personal value and who I am has been really bad for my brain.

And trying to make that clear at my company is, like, what is mine and what I can help with has been really huge. I feel like the boundaries between myself and my job has gotten too thin. And for a while, I thought that was a great idea; it turns out that was not a great idea for my brain. It’s so hard. So, I’ve been a software engineer and I’ve done full-time developer advocacy, and I felt like I had a lot more freedom to say what I wanted as, like, a full-time software engineer as opposed to being a developer advocate and kind of representing the company.

Because the thing is, I’m always representing the company [online 00:24:56], but I’m not always working, which is kind of like—that—it’s kind of a hard line. I feel like there’s been, like, ways to get around it though with, like, less private shitposting about things that could piss off a CEO or infringe on an NDA or, you know, whatever, you know what I mean? Yeah, trying to, like, find that balance or trying to, like, use tools to try to separate that has been big. But I don’t know, I’ve been—personally, I’ve been trying to step—like, start trying to make more of a boundary for that.

Corey: Yeah. I don’t have much of one, but I also own the company, so my approach doesn’t necessarily work for other people. I don’t advertise in public that I fix AWS bills very often. That’s not the undercurrent to most of my jokes and the rest. Because the people who have that painful problem aren’t generally in the audience directly and they certainly don’t talk about it extensively.

It’s word of mouth. It’s being fun and engaging so people stick around. And when I periodically do mention it that sort of sticks with them. And in the fullness of time, it works as a way of, “Oh, yeah, everyone knows what you’re into. And yeah, when we have this problem, reaching out to you is our first thought.” But I don’t know that it’s possible to measure its effectiveness. I just know that works.

Joe: Yeah. For me, it’s like, don’t be an asshole and teach don’t sell are like, the two biggest things that I’m trying to do all the time. And the goal is not to, like, trick people into, like, thinking I’m not working for a company. I think I try to be transparent, or if, like, I happen to be talking about a product that I’m working for, I try to disclose that. But yeah, I don’t know. For me, it’s just, like, trying to build up a community of people who, like, understand what I’m trying to put out there. You know what I mean?

Corey: Yeah, it’s about what you want to be known for, on some level. Part of the problem that I’ve had for a long time is that I’ve been pulled in so many directions. [They’re 00:26:34] like, “Oh, you’re great. Where do I go to learn more?” It’s like, “Well, I have this podcast, I have the newsletter, I have the other podcast that I do in the AWS Morning Brief. I have the duckbillgroup.com. I have lastweekinaws.com. I have a Twitter account. I had a YouTube thing for a while.”

It’s like, there’s so many different ways to send people. It’s like, what is the top-of-funnel? And for me, my answer has been, sign up for the newsletter at lastweekinaws.com. That keeps you apprised of everything else and you can dial it into taste. It’s also, frankly, one of those things that doesn’t require algorithmic blessing to continue to show up in people’s inboxes. So far at least, we haven’t seen algorithms have a significant impact on that, except when they spam-bin something. And it turns out when you write content people like, the providers get yelled at by their customers of, “Hey, I’m trying to read this. What’s going on?” I had a couple of reach out to me asking what the hell happened. It’s kind of fun.

Joe: I love that. And, Corey, I think that’s so smart, too. It’s definitely been a lesson, I think, for me and a lot of people on—that are terminally online that, like, we don’t own our social following on other platforms. With, like, the downfall of Twitter, like, I’m still posting on there, but we still have a bunch of stuff on there, but my… that following is locked in. I can’t take that home. But, like, you still have your email newsletter. And I even feel it for tech companies who might be listening to this, too. I feel like owning your email list is, like, not the coolest thing, but I feel like it’s criminally underrated, as, like, a way of talking to people.

Corey: It doesn’t matter what platforms change, what my personal situation changes, I am—like, whatever it is that I wind up doing next, whenever next happens, I’ll need a platform to tell people about, and that’s what I’ve been building. I value newsletter subscribers in a metric sense far more highly and weight them more heavily than I do Twitter followers. Anyone can click a follow and then never check Twitter again. Easy enough. Newsletters? Well, that winds up requiring a little bit extra work because we do confirmed opt-ins, for obvious reasons.

And we never sell the list. We never—you can’t transfer permission for, like that, and we obviously respect it when people say I don’t want to hear from your nonsense anymore. Great. Cool. I don’t want to send this to people that don’t care. Get out of here.

Joe: [laugh]. No, I think that’s so smart.

Corey: Podcasts are impossible on the other end, but I also—you know, I control the domain and that’s important to me.

Joe: Yeah.

Corey: Why don’t you build this on top of Substack? Because as soon as Substack pivots, I’m screwed.

Joe: Yeah, yeah. Which we’ve—I think we’ve seen that they’ve tried to do, even with the Twitter clone that tried to build last couple years. I’ve been burned by so many other publishing platforms over and over and over again through the years. Like, Medium, yeah, I criminally don’t trust any sort of tech publishing platform anymore that I don’t own. [laugh]. But I also don’t want to maintain it. It’s such a fine line. I just want to, like, maintain something without having to, like, maintain all the infrastructure all the time, and I don’t think that exists and I don’t really trust anything to help me with that.

Corey: You can on some level, I mean, I wind up parking in the newsletter stuff over at ConvertKit. But I can—I have moved it twice already. I could move it again if I needed to. It’s about controlling the domain. I have something that fires off once or twice a day that backs up the entire subscriber list somewhere.

I don’t want to build my own system, but I can also get that in an export form wherever I need it to go. Frankly, I view it as the most valuable asset that I have here because I can always find a way to turn relationships and an audience into money. I can’t necessarily find a way to go the opposite direction of, well have money. Time to buy an audience. Doesn’t work that way.

Joe: [laugh]. No, I totally agree. You know what I do like, though, is Threads, which has kind of fallen off, but I do love the idea of their federated following [and be almost 00:30:02] like, unlock that a little bit. I do think that that’s probably going to be the future. And I have to say, I just care as someone who, like, makes shit online. I don’t think 98% of people don’t really care about that future, but I do. Just getting burned so often on social media platforms, it helps to then have a little bit of flexibility there.

Corey: Oh, yeah. And I wish it were different. I feel like, at some level, Elon being Elon has definitely caused a bit of a diaspora of social media and I think that’s a good thing.

Joe: Yeah. Yeah. I hope it settles down a little bit, but it definitely got things moving again.

Corey: Oh, yes. I really want to thank you for taking the time to go through how you view these things. Where’s the best place for people to go to follow you learn more, et cetera? Just sign up for TikTok and you’ll be all over them, apparently.

Joe: Go to the website that I own joekarlsson.com. It’s got the links to everything on there. Opt in or out of whatever you find you want. Otherwise, I’m just going to quick plug for the company I work for: tinybird.co. If you’re trying to make APIs on top of data, definitely want to check out Tinybird. We work with Kafka, BigQuery, S3, all the data sources could pull it in. [unintelligible 00:31:10] on it and publishes it as an API. It’s super easy. Or you could just ignore me. That’s fine, too. You could—that’s highly encouraged as well.

Corey: Always a good decision.

Joe: [laugh]. Yeah, I agree. I’m biased, but I agree.

Corey: Thanks, Joe. I appreciate your taking the time to speak with me and we’ll, of course, put links to all that in the [show notes 00:31:26]. And please come back soon and regale us with more stories.

Joe: I will. Thanks, Corey.

Corey: Joe Karlsson, data engineer at Tinybird. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that I’ll never read because they’re going to have a disk problem and they haven’t learned the lesson of backups yet.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn and I am joined today by someone from well, we’ll call it the other side of the tracks, if I can—

Joe: [laugh].

Corey: —be blunt and disrespectful. Joe Karlsson is a data engineer at Tinybird, but I really got to know who he is by consistently seeing his content injected almost against my will over on the TikToks. Joe, how are you?

Joe: I’m doing so well and I’m so sorry for anything I’ve forced down your throat online. Thanks for having me, though.

Corey: Oh, it’s always a pleasure to talk to you. No, the problem I’ve got with it is that when I’m in TikTok mode, I don’t want to think about computers anymore. I want to find inane content that I can just swipe six hours away without realizing it because that’s how I roll.

Joe: TikTok is too smart, though. I think it knows that you are doing a lot of stuff with computers and even if you keep swiping away, it’s going to keep serving it up to you.

Corey: For a long time, it had me pinned as a lesbian, which was interesting. Which I suppose—

Joe: [laugh]. It happened to me, too.

Corey: Makes sense because I follow a lot of women who are creators in comics and the rest, but I’m not interested in the thirst trap approach. So, it’s like, “Mmm, this codes as lesbian.” Then they started showing me ads for ADHD, which I thought was really weird until I’m—oh right. I’m on TikTok. And then they started recommending people that I’m surprised was able to disambiguate until I realized these people have been at my house and using TikTok from my IP address, which probably is going to get someone murdered someday, but it’s probably easy to wind up doing an IP address match.

Joe: I feel like I have to, like, separate what is me and what is TikTok, like, trying to serve it up because I’ve been on lesbian TikTok, too, ADHD, autism, like TikTok. And, like, is this who I am? I don’t know. [unintelligible 00:02:08] bring it to my therapist.

Corey: You’re learning so much about yourself based upon an algorithm. Kind of wild, isn’t it?

Joe: [laugh]. Yeah, I think we may be a little, like, neuro-spicy, but I think it might be a little overblown with what TikTok is trying to diagnose us with. So, it’s always good to just keep it in check, you know?

Corey: Oh, yes. So, let’s see, what’s been going on lately? We had Google Next, which I think the industry largely is taking not seriously enough. For years, it felt like a try-hard, me too version of re:Invent. And this year, it really feels like it’s coming to its own. It is defining itself as something other than oh, us too.

Joe: I totally agree. And that’s where you and I ran into recently, too. I feel like post-Covid I’m still, like, running into people I met on the internet in real life, and yeah, I feel like, yeah, re:Invent and Google Next are, like, the big ones.

I totally agree. It feels like—I mean, it’s definitely, like, heavily inspired by it. And it still feels like it’s a little sibling in some ways, but I do feel like it’s one of the best conferences I’ve been to since, like, a pre-Covid 2019 AWS re:Invent, just in terms of, like… who was there. The energy, the vibes, I feel like people were, like, having fun. Yeah, I don’t know, it was a great conference this year.

Corey: Usually, I would go to Next in previous years because it was a great place to go to hang out with AWS customers. These days, it feels like it’s significantly more than that. It’s, everyone is using everything at large scale. I think that is something that is not fully understood. You talk to companies that are, like, Netflix, famously all in on AWS. Yeah, they have Google stuff, too.

Everyone does. I have Google stuff. I have a few things in Azure, for God’s sake. It’s one of those areas where everything starts to diffuse throughout a company as soon as you hire employee number two. And that is, I think, the natural order of things. The challenge, of course, is the narrative people try and build around it.

Joe: Yep. Oh, totally. Multi-cloud’s been huge for you know, like, starting to move up. And it’s impossible not to. It was interesting seeing, like, Google trying to differentiate itself from Azure and AWS. And, Corey, I feel like you’d probably agree with this, too, AI was like, definitely the big buzzword that kept trying to, like—

Corey: Oh, God. Spare me. And I say that, as someone who likes AI, I think that there’s a lot of neat stuff lurking around and value hiding within generative AI, but the sheer amount of hype around it—and frankly—some of the crypto bros have gone crashing into the space, make me want to distance myself from it as far as humanly possible, just because otherwise, I feel like I get lumped in with that set. And I don’t want that.

Joe: Yeah, I totally agree. I know it feels like it’s hard right now to, like, remain ungrifty, but, like, still, like—trying—I mean, everyone’s trying to just, like, hammer in an AI perspective into every product they have. And I feel like a lot of companies, like, still don’t really have a good use case for it. You’re still trying to, like, figure that out. We’re seeing some cool stuff.

Honestly, the hard part for me was trying to differentiate between people just, like, bragging about OpenAI API addition they added to the core product or, like, an actual thing that’s, like, AI is at the center of what it actually does, you know what I mean? Everything felt like it’s kind of like tacked on some sort of AI perspective to it.

Corey: One of the things that really is getting to me is that you have these big companies—Google and Amazon most notably—talk about how oh, well, we’ve actually been working with AI for decades. At this point, they keep trying to push out how long it’s been. It’s like, “Okay, then not for nothing, then why does”—in Amazon’s case—“why does Alexa suck? If you’ve been working on it for this long, why is it so bad at all the rest?” It feels like they’re trying to sprint out with a bunch of services that very clearly were not conceptualized until Chat-Gippity’s breakthrough.

And now it’s oh, yeah, we’re there, too. Us, too. And they’re pivoting all the marketing around something that, frankly, they haven’t demonstrated excellence with. And I feel like they’re leaving a lot of their existing value proposition completely in the dust. It’s, your customers are not using you because of the speculative future, forward-looking AI things; it’s because you are able to solve business problems today in ways that are not highly speculative and are well understood. That’s not nothing and there needs to be more attention paid to that. And I feel like there’s this collective marketing tripping over itself to wrap itself in hype that does them no services.

Joe: I totally agree. I feel like honestly, just, like, a marketing perspective, I feel like it’s distracting in a lot of ways. And I know it’s hot and it’s cool, but it’s like, I think it’s harder right now to, like, stay focused to what you’re actually doing well, as opposed to, like, trying to tack on some AI thing. And maybe that’s great. I don’t know.

Maybe that’s—honestly, maybe you’re seeing some traction there. I don’t know. But I totally agree. I feel like everyone right now is, like, selling a future that we don’t quite have yet. I don’t know. I’m worried that what’s going to happen again, is what happened back in the IBM Watson days where everyone starts making bold—over-promising too much with AI until we see another AI winter again.

Corey: Oh, the subtext is always, we can’t wait to fire our entire customer service department. That one—

Joe: Yeah.

Corey: Just thrills me.

Joe: [laugh].

Corey: It’s like, no, we’re just going to get rid of junior engineers and just have senior engineers. Yeah, where do you think those people come from, by the way? We aren’t—they aren’t just emerging fully formed from the forehead of some god somewhere. And we’re also seeing this wild divergence from reality. Remember, I fix AWS bills for a living. I see very large companies, very large AWS spend.

The majority of spend remains on EC2 across the board. So, we don’t see a lot of attention paid to that at re:Invent, even though it’s the lion’s share of everything. When we do contract negotiations, we talk about generative AI plan and strategy, but no one’s saying, oh, yeah, we’re spending 100 million a year right now on AWS but we should commit 250 because of all this generative AI stuff we’re getting into. It’s all small-scale experimentation and seeing if there’s value there. But that’s a far cry from being the clear winner what everyone is doing.

I’d further like to point out that I can tell that there’s a hype cycle in place and I’m trying to be—and someone’s trying to scam me. As soon as there’s a sense of you have to get on this new emerging technology now, now, now, now, now. I didn’t get heavily into cloud till 2016 or so and I seem to have done all right with that. Whenever someone is pushing you to get into an emerging thing where it hasn’t settled down enough to build a curriculum yet, I feel like there’s time to be cautious and see what the actual truth is. Someone’s selling something; if you can’t spot the sucker, chances are, it’s you.

Joe: [laugh]. Corey, have you thought about making an AI large language model that will help people with their cloud bills? Maybe just feed it, like, your invoices [laugh].

Corey: That has been an example, I’ve used a number of times with a variety of different folks where if AI really is all it’s cracked up to be, then the AWS billing system is very much a bounded problem space. There’s a lot of nuance and intricacy to it, but it is a finite set of things. Sure, [unintelligible 00:08:56] space is big. So, training something within those constraints and within those confines feels like it would be a terrific proof-of-concept for a lot of these things. Except that when I’ve experimented a little bit and companies have raised rounds to throw into this, it never quite works out because there’s always human context involved. The, oh yeah, we’re going to wind up turning off all those idle instances, except they’re in idle—by whatever metric you’re using—for a reason. And the first time you take production down, you’re not allowed to save money anymore.

Joe: Nope. That’s such a good point. I agree. I don’t know about you, Corey. I’ve been fretting about my job and, like, what I’m doing. I write a lot, I do a lot of videos, I’m programming a lot, and I think… obviously, we’ve been hearing a lot about, you know, if it’s going to replace us or not. I honestly have been feeling a lot better recently about my job stability here. I don’t know. I totally agree with you. There’s always that, like, human component that needs to get added to it. But who knows, maybe it’s going to get better. Maybe there’ll be an AI-automated billing management tool, but it’ll never be as good as you, Corey. Maybe it will. I don’t know. [laugh].

Corey: It knows who I am. When I tell it to write in the style of me and give it a blog post topic and some points I want to make, almost everything it says is wrong. But what I’ll do is I’ll copy that into a text editor, mansplain-correct the robot for ten minutes, and suddenly I’ve got the bones of a decent rough draft because. And yeah, I’ll wind up plagiarizing three or four words in a row at most, but that’s okay. I’m plagiarizing the thing that’s plagiarizing from me and there’s a beautiful symmetry to that. What I don’t understand is some of the outreach emails and other nonsensical stuff I’ll see where people are letting unsupervised AI just write things under their name and sending it out to people. That is anathema to me.

Joe: I totally agree. And it might work today, it might work tomorrow, but, like, it’s just a matter of time before something blows up. Corey, I’m curious. Like, personally, how do you feel about being in the ChatGPT, like, brain? I don’t know, is that flattering? Does that make you nervous at all?

Corey: Not really because it doesn’t get it in a bunch of ways. And that’s okay. I found the same problem with people. In my time on Twitter, when I started live-tweet shitposting about things—as I tend to do as my first love language—people will often try and do exactly that. The problem that I run into is that, “The failure mode of ‘clever’ is ‘asshole,’” as John Scalzi famously said, and as a direct result of that, people wind up being mean and getting it wrong in that direction.

It’s not that I’m better than they are. It’s, I had a small enough following, and no one knew who I was in my mean years, and I realized I didn’t feel great making people sad. So okay, you’ve got to continue to correct the nosedive. But it is perilous and it is difficult to understand the nuance. I think occasionally when I prompt it correctly, it comes up with some amazing connections between things that I wouldn’t have seen, but that’s not the same thing as letting it write something completely unfettered.

Joe: Yeah, I totally agree. The nuance definitely gets lost. It may be able to get, like, the tone, but I think it misses a lot of details. That’s interesting.

Corey: And other people are defending it when that hallucinates. Like, yeah, I understand there are people that do the same thing, too. Yeah, the difference is, in many cases, lying to me and passing it off otherwise is a firing offense in a lot of places. Because if you’re going to be 19 out of 20 times, you’re correct, but 5% wrong, you’re going to bluff, I can’t trust anything you tell me.

Joe: Yeah. It definitely, like, brings your, like—the whole model into question.

Corey: Also, remember that my medium for artistic creation is often writing. And I think that, on some level, these AI models are doing the same things that we do. There are still turns of phrase that I use that I picked up floating around Usenet in the mid-90s. And I don’t remember who said it or the exact context, but these words and phrases have entered my lexicon and I’ll use them and I don’t necessarily give credit to where the first person who said that joke 30 years ago. But it’s a—that is how humans operate. We are influenced by different styles of writing and learn from the rest.

Joe: True.

Corey: That’s a bit different than training something on someone’s artistic back catalog from a painting perspective and then emulating it, including their signature in the corner. Okay, that’s a bit much.

Joe: [laugh]. I totally agree.

Corey: So, we wind up looking right now at the rush that is going on for companies trying to internalize their use of enterprise AI, which is kind of terrifying, and it all seems to come back to data.

Joe: Yes.

Corey: You work in the data space. How are you seeing that unfold?

Joe: Yeah, I do. I’ve been, like, making speculations about the future of AI and data forever. I’ve had dreams of tools I’ve wanted forever, and I… don’t have them yet. I don’t think they’re quite ready yet. I don’t know, we’re seeing things like—tha—I think people are working on a lot of problems.

For example, like, I want AI to auto-optimize my database. I want it to, like, make indexes for me. I want it to help me with queries or optimizing queries. We’re seeing some of that. I’m not seeing anyone doing particularly well yet. I think it’s up in the air.

I feel like it could be coming though soon, but that’s the thing, though, too, like, I mean, if you mess up a query, or, like, a… large language model hallucinates a really shitty query for you, that could break your whole system really quickly. I feel like there still needs to be, like, a human being in the middle of it to, like, kind of help.

Corey: I saw a blog post recently that AWS put out gave an example that just hard-coded a credential into it. And they said, “Don’t do this, but for demonstration purposes, this is how it works.” Well, that nuance gets lost when you use that for AI training and that’s, I think, in part, where you start seeing a whole bunch of the insecure crap these things spit out.

Joe: Yeah, I totally agree. Well, I thought the big thing I’ve seen, too, is, like, large language models typically don’t have a secure option and you’re—the answer is, like, help train the model itself later on. I don’t know, I’m sure, like, a lot of teams don’t want to have their most secret data end up public on a large language model at some point in the future. Which is, like, a huge issue right now.

Corey: I think that what we’re seeing is that you still need someone with expertise in a given area to review what this thing spits out. It’s great at solving a lot of the busy work stuff, but you still need someone who’s conversant with the concepts to look at it. And that is, I think, something that turns into a large-scale code review, where everyone else just tends to go, “Oh, okay. We’re—do this with code review.” “Oh, how big is the diff?” “50,000 lines.” “Looks good to me.” Whereas, “Three lines.” “I’m going to criticize that thing with four pages of text.” People don’t want to do the deep-dive stuff, and—when there’s a huge giant project that hits. So, they won’t. And it’ll be fine, right up until it isn’t.

Joe: Corey, you and I know people and developers, do you think it’s irresponsible to put out there an example of how to do something like that, even with, like, an asterisk? I feel like someone’s going to still go out and try to do that and probably push that to production.

Corey: Of course they are.

Joe: [laugh].

Corey: I’ve seen this with some of my own code. I had something on Docker Hub years ago with a container that was called ‘Terrible Ideas.’ And I’m sure being used in, like—it was basically the environment I use for a talk I gave around Git, which makes sense. And because I don’t want to reset all the repositories back to the way they came from with a bunch of old commands, I just want a constrained environment that will be the same every time I give the talk. Awesome.

I’m sure it’s probably being run in production at a bank somewhere because why wouldn’t it be? That’s people. That’s life. You’re not supposed to just copy and paste from Chat-Gippity. You’re supposed to do that from Stack Overflow like the rest of us. Where do you think your existing code’s coming from in a lot of these shops?

Joe: Yep. No, I totally agree. Yeah, I don’t know. It’ll be interesting to see how this shakes out with, like, people going to doing this stuff, or how honest they’re going to be about it, too. I’m sure it’s happening. I’m sure people are tripping over themselves right now, [adding 00:16:12].

Corey: Oh, yeah. But I think, on some level, you’re going to see a lot more grift coming out of this stuff. When you start having things that look a little more personalized, you can use it for spam purposes, you can use it for, I’m just going to basically copy and paste what this says and wind up getting a job on Upwork or something that is way more than I could handle myself, but using this thing, I’m going to wind up coasting through. Caveat emptor is always the case on that.

Joe: Yeah, I totally agree.

Corey: I mean, it’s easy for me to sit here and talk about ethics. I believe strongly in doing the right thing. But I’m also not worried about whether I’m able to make rent this month or put food on the table. That’s a luxury. At some point, like, a lot of that strips away and you do what you have to do to survive. I don’t necessarily begrudge people doing these things until it gets to a certain point of okay, now you’re not doing this to stay alive anymore. You’re doing this to basically seek rent.

Joe: Yeah, I agree. Or just, like, capitalize on it. I do think this is less—like, the space is less grifty than the crypto space, but as we’ve seen over and over and over and over again, in tech, there’s a such a fine line between, like, a genuinely great idea, and somebody taking advantage of it—and other people—with that idea.

Corey: I think that’s one of those sad areas where you’re not going to be able to fix human nature, regardless of the technology stack you bring to bear.

Joe: Yeah, I totally agree.

[midroll 00:17:30]

Corey: So, what else are you seeing these days that interesting? What excites you? What do you see that isn’t getting enough attention in the space?

Joe: I don’t know, I guess I’m in the data space, I’m… the thing I think I do see a lot of is huge interest in data. Data right now is the thing that’s come up. Like, I don’t—that’s the thing that’s training these models and everyone trying to figure out what to do with these data, all these massive databases, data lakes, whatever. I feel like everyone’s, kind of like, taking a second look at all of this data they’ve been collecting for years and haven’t really known what to do with it and trying to figure out either, like, if you can make a model out of that, if you try to, like… level it up, whatever. Corey, you and I were joking around recently—you’ve had a lot of data people on here recently, too—I feel like us data folks are just getting extra loud right now. Or maybe there’s just the data spaces, that’s where the action’s at right now.

I don’t know, the markets are really weird. Who knows? But um, I feel like data right now is super valuable and more so than ever. And even still, like, I mean, we’re seeing, like, companies freaking out, like, Twitter and Reddit freaking out about accessing their data and who’s using it and how. I don’t know, I feel like there’s a lot of action going on there right now.

Corey: I think that there’s a significant push from the data folks where, for a long time data folks were DBAs—

Joe: Yeah.

Corey: —let’s be direct. And that role has continued to evolve in a whole bunch of different ways. It’s never been an area I’ve been particularly strong in. I am not great at algorithmic complexity, it turns out, you can saturate some beefy instances with just a little bit of data if your queries are all terrible. And if you’re unlucky—as I tend to be—and have an aura of destroying things, great, you probably don’t want to go and make that what you do.

Joe: [laugh]. It’s a really good point. I mean, I don’t know about, like, if you blow up data at a company, you’re probably going to be in big trouble. And especially the scale we’re talking about with most companies these days, it’s super easy to either take down a server or generate an insane bill off of some shitty query.

Corey: Oh, when I was at Reach Local years and years ago—my first Linux admin job—when I broke the web server farm, it was amusing; when I broke part of the data warehouse, nobody was laughing.

Joe: [laugh]. I wonder why.

Corey: It was a good faith mistake and that’s fair. It was a convoluted series of things that set up and honestly, the way the company and my boss responded to me at the time set the course of the rest of my career. But it was definitely something that got my attention. It scares me. I’m a big believer in backups as a direct result.

Joe: Yeah. Here’s the other thing, too. Actually, our company, Tinybird, is working on versioning with your data sources right now and treating your data sources like Git, but I feel like even still today, most companies are just run by some DBA. There’s, like, Mike down the hall is the one responsible keeping their SQL servers online, keeping them rebooted, and like, they’re manually updating any changes on there.

And I feel like, generally speaking across the industry, we’re not taking data seriously. Which is funny because I’m with you on there. Like, I get terrified touching production databases because I don’t want anything bad to happen to them. But if we could, like, make it easier to rollback or, like, handle that stuff, that would be so much easier for me and make it, like, less scary to deal with it. I feel like databases and, like, treating it as, like, a serious DevOps practice is not really—I’m not seeing enough of it. It’s definitely, people are definitely doing it. Just, I want more.

Corey: It seems like with data, there’s a lack of iterative approaches to it. A line that someone came up with when I was working with them a decade and change ago was that you can talk about agile all you want, but when it comes to payments, everyone’s doing waterfall. And it feels like, on some level, data’s kind of the same.

Joe: Yeah. And I don’t know, like, how to fix it. I think everyone’s just too scared of it to really touch it. Migrating over to a different version control, trying to make it not as manual, trying to iterate on it better, I think it’s just—I don’t blame them. It’s hard, it really takes a long time, making sure everything, like, doesn’t blow up while you’re doing a migration is a pain in the ass. But I feel like that would make everyone’s lives so much easier if, like, you could, like, treat it—understand your data and be able to rollback easier with it.

Corey: When you take a look across the ecosystem now, are you finding that things have improved since the last time I was in the space, where the state of the art was, “Oh, we need some developer data. We either have this sanitized data somewhere or it’s a copy of production that we move around, but only a small bit.” Because otherwise, we always found that oh, that’s an extra petabyte of storage was going on someone’s developer environment they messed up on three years ago, they haven’t been here for two, and oops.

Joe: I don’t. I have not seen it. Again, that’s so tricky, too. I think… yeah, the last time I, like, worked doing that was—usually you just have a really crappy version of production data on staging or development environments and it’s hard to copy those over. I think databases are getting better for that.

I’ve been working on, like, the real-time data space for a long time now, so copying data over and kind of streaming that over is a lot easier. I do think seeing, like, separating storage and compute can make it easier, too. But it depends on your data stack. Everyone’s using everything all the time and it’s super complicated to do that. I don’t know about you, Corey, too. I'm sure you’ve seen, like, services people running, but I feel like we’ve made a switch as an industry from, like, monoliths to microservices.

Now, we’re kind of back in the monolith era, but I’m not seeing that happen in the database space. We’re seeing, like, data meshing and lots of different databases. I see people who, like, see the value of data monoliths, but I don’t see any actual progress in moving back to a single source of [truth of the data 00:23:02]. And I feel like the cat’s kind of out of the bag on all the data existing everywhere, all the time, and trying to wrangle that up.

Corey: This stuff is hard and there’s no easy solution here. There just isn’t.

Joe: Yeah, there’s no way. And embracing that chaos, I think, is going to be huge. I think you have to do it right now. Or trying to find some tool that can, like, wrangle up a bunch of things together and help work with them all at once. Products need to meet people where they’re at, too. And, like, data is all over the place and I feel like we kind of have to, like, find tooling that can kind of help work with what you have.

Corey: It’s a constant challenge, but also a joy, so we’ll give it that.

Joe: [laugh].

Corey: So, I have to ask. Your day job has you doing developer advocacy at Tinybird—

Joe: Yes.

Corey: But I had to dig in to find that out. It wasn’t obvious based upon the TikToks and the Twitter nonsense and the rest. How do you draw the line between day job and you as a person shitposting on the internet about technology?

Joe: Corey, I’d be curious to hear your thoughts on this, too. I don’t know. I feel like I’ve been in different places where, like, my job is my life. You know what I mean? There’s a very thin line there. Personally, I’ve been trying to take a step back from that, just from a mental health perspective. Having my professional life be so closely tied to, like, my personal value and who I am has been really bad for my brain.

And trying to make that clear at my company is, like, what is mine and what I can help with has been really huge. I feel like the boundaries between myself and my job has gotten too thin. And for a while, I thought that was a great idea; it turns out that was not a great idea for my brain. It’s so hard. So, I’ve been a software engineer and I’ve done full-time developer advocacy, and I felt like I had a lot more freedom to say what I wanted as, like, a full-time software engineer as opposed to being a developer advocate and kind of representing the company.

Because the thing is, I’m always representing the company [online 00:24:56], but I’m not always working, which is kind of like—that—it’s kind of a hard line. I feel like there’s been, like, ways to get around it though with, like, less private shitposting about things that could piss off a CEO or infringe on an NDA or, you know, whatever, you know what I mean? Yeah, trying to, like, find that balance or trying to, like, use tools to try to separate that has been big. But I don’t know, I’ve been—personally, I’ve been trying to step—like, start trying to make more of a boundary for that.

Corey: Yeah. I don’t have much of one, but I also own the company, so my approach doesn’t necessarily work for other people. I don’t advertise in public that I fix AWS bills very often. That’s not the undercurrent to most of my jokes and the rest. Because the people who have that painful problem aren’t generally in the audience directly and they certainly don’t talk about it extensively.

It’s word of mouth. It’s being fun and engaging so people stick around. And when I periodically do mention it that sort of sticks with them. And in the fullness of time, it works as a way of, “Oh, yeah, everyone knows what you’re into. And yeah, when we have this problem, reaching out to you is our first thought.” But I don’t know that it’s possible to measure its effectiveness. I just know that works.

Joe: Yeah. For me, it’s like, don’t be an asshole and teach don’t sell are like, the two biggest things that I’m trying to do all the time. And the goal is not to, like, trick people into, like, thinking I’m not working for a company. I think I try to be transparent, or if, like, I happen to be talking about a product that I’m working for, I try to disclose that. But yeah, I don’t know. For me, it’s just, like, trying to build up a community of people who, like, understand what I’m trying to put out there. You know what I mean?

Corey: Yeah, it’s about what you want to be known for, on some level. Part of the problem that I’ve had for a long time is that I’ve been pulled in so many directions. [They’re 00:26:34] like, “Oh, you’re great. Where do I go to learn more?” It’s like, “Well, I have this podcast, I have the newsletter, I have the other podcast that I do in the AWS Morning Brief. I have the duckbillgroup.com. I have lastweekinaws.com. I have a Twitter account. I had a YouTube thing for a while.”

It’s like, there’s so many different ways to send people. It’s like, what is the top-of-funnel? And for me, my answer has been, sign up for the newsletter at lastweekinaws.com. That keeps you apprised of everything else and you can dial it into taste. It’s also, frankly, one of those things that doesn’t require algorithmic blessing to continue to show up in people’s inboxes. So far at least, we haven’t seen algorithms have a significant impact on that, except when they spam-bin something. And it turns out when you write content people like, the providers get yelled at by their customers of, “Hey, I’m trying to read this. What’s going on?” I had a couple of reach out to me asking what the hell happened. It’s kind of fun.

Joe: I love that. And, Corey, I think that’s so smart, too. It’s definitely been a lesson, I think, for me and a lot of people on—that are terminally online that, like, we don’t own our social following on other platforms. With, like, the downfall of Twitter, like, I’m still posting on there, but we still have a bunch of stuff on there, but my… that following is locked in. I can’t take that home. But, like, you still have your email newsletter. And I even feel it for tech companies who might be listening to this, too. I feel like owning your email list is, like, not the coolest thing, but I feel like it’s criminally underrated, as, like, a way of talking to people.

Corey: It doesn’t matter what platforms change, what my personal situation changes, I am—like, whatever it is that I wind up doing next, whenever next happens, I’ll need a platform to tell people about, and that’s what I’ve been building. I value newsletter subscribers in a metric sense far more highly and weight them more heavily than I do Twitter followers. Anyone can click a follow and then never check Twitter again. Easy enough. Newsletters? Well, that winds up requiring a little bit extra work because we do confirmed opt-ins, for obvious reasons.

And we never sell the list. We never—you can’t transfer permission for, like that, and we obviously respect it when people say I don’t want to hear from your nonsense anymore. Great. Cool. I don’t want to send this to people that don’t care. Get out of here.

Joe: [laugh]. No, I think that’s so smart.

Corey: Podcasts are impossible on the other end, but I also—you know, I control the domain and that’s important to me.

Joe: Yeah.

Corey: Why don’t you build this on top of Substack? Because as soon as Substack pivots, I’m screwed.

Joe: Yeah, yeah. Which we’ve—I think we’ve seen that they’ve tried to do, even with the Twitter clone that tried to build last couple years. I’ve been burned by so many other publishing platforms over and over and over again through the years. Like, Medium, yeah, I criminally don’t trust any sort of tech publishing platform anymore that I don’t own. [laugh]. But I also don’t want to maintain it. It’s such a fine line. I just want to, like, maintain something without having to, like, maintain all the infrastructure all the time, and I don’t think that exists and I don’t really trust anything to help me with that.

Corey: You can on some level, I mean, I wind up parking in the newsletter stuff over at ConvertKit. But I can—I have moved it twice already. I could move it again if I needed to. It’s about controlling the domain. I have something that fires off once or twice a day that backs up the entire subscriber list somewhere.

I don’t want to build my own system, but I can also get that in an export form wherever I need it to go. Frankly, I view it as the most valuable asset that I have here because I can always find a way to turn relationships and an audience into money. I can’t necessarily find a way to go the opposite direction of, well have money. Time to buy an audience. Doesn’t work that way.

Joe: [laugh]. No, I totally agree. You know what I do like, though, is Threads, which has kind of fallen off, but I do love the idea of their federated following [and be almost 00:30:02] like, unlock that a little bit. I do think that that’s probably going to be the future. And I have to say, I just care as someone who, like, makes shit online. I don’t think 98% of people don’t really care about that future, but I do. Just getting burned so often on social media platforms, it helps to then have a little bit of flexibility there.

Corey: Oh, yeah. And I wish it were different. I feel like, at some level, Elon being Elon has definitely caused a bit of a diaspora of social media and I think that’s a good thing.

Joe: Yeah. Yeah. I hope it settles down a little bit, but it definitely got things moving again.

Corey: Oh, yes. I really want to thank you for taking the time to go through how you view these things. Where’s the best place for people to go to follow you learn more, et cetera? Just sign up for TikTok and you’ll be all over them, apparently.

Joe: Go to the website that I own joekarlsson.com. It’s got the links to everything on there. Opt in or out of whatever you find you want. Otherwise, I’m just going to quick plug for the company I work for: tinybird.co. If you’re trying to make APIs on top of data, definitely want to check out Tinybird. We work with Kafka, BigQuery, S3, all the data sources could pull it in. [unintelligible 00:31:10] on it and publishes it as an API. It’s super easy. Or you could just ignore me. That’s fine, too. You could—that’s highly encouraged as well.

Corey: Always a good decision.

Joe: [laugh]. Yeah, I agree. I’m biased, but I agree.

Corey: Thanks, Joe. I appreciate your taking the time to speak with me and we’ll, of course, put links to all that in the [show notes 00:31:26]. And please come back soon and regale us with more stories.

Joe: I will. Thanks, Corey.

Corey: Joe Karlsson, data engineer at Tinybird. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that I’ll never read because they’re going to have a disk problem and they haven’t learned the lesson of backups yet.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.