Avery Pennarun on Tailscale’s Evolution: From Mesh VPN to AI Security Gateway

Transcript

Avery: What's very strange about tail scale, uh, and very strange in the security world in general, is that when you use tail scale to solve that problem, you accidentally make your system more secure. And also the easiest thing for all of your engineers and people inside your company to do becomes the secure thing instead of the insecure thing.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's been a while since I've had Avery Pener run on the show. Thank you for joining me. Again, you are still the co-founder and CEO of tailscale, which at this point is getting pretty darn close to. You've heard of this company when I bring it up in almost every conversation I'm in.

Avery: That is pretty exciting. I think. I can't remember when I was on your show last time, but it was at least a couple years ago and we've been growing really fast in the last couple years. This episode is sponsored in part by my day

Corey: job Duck. Bill, do you have a horrifying AWS bill? That can mean a lot of things.

Predicting what it's going to be, determining what it should be, negotiating your next long-term contract with AWS, or just figuring out why it increasingly resembles of. Phone number, but nobody seems to quite know why that is. To learn more, visit duck bill hq.com. Remember, you can't duck the duck bill.

Bill, which my CEO reliably informs me is absolutely not our slogan.

I'm seeing you in more and more places. I've been using you in my personal environment for many years now, and the stuff that I set up once upon a time is still working. You're rolling out new stuff that continues to add, be additive to this at at at work.

I'm paying you now, which was a big problem I had with you previously of there's no good way for me to give you money. Could you maybe fix that? Good job. You fix that. So things are all up and to the right, which is kind of amazing.

Avery: It is kind of amazing. It's amazing how long we can keep doing it.

Although I've been informed that if you keep doubling revenue at a hundred percent year over year, then in 10 years you'll be a thousand times bigger, and that might not be realistic. But it might

Corey: at some point you hit population limits. Uh, last year I gave the opening keynote at Nano 91, and the whole theme of what I was talking about back then was that there's been a rising tide in the level of what clouds could take.

From, from folks who are working in on-prem environments, networking is becoming something of a lost art. When you find someone who works as a network engineer, they're usually my age and not new grads who are playing around with these things. Tailscale is in some ways an answer to some of this, where you, you're taking things away from the traditional network, uh, switch and router world and into just make a big flat network.

And then we'll wind up handling this through policy files for access control. Even recently this year, you folks wound up redoing your policy, uh, policy, uh, format as far as making it a lot easier to do grants with access grants, as well as now creating a visual builder, which I've not yet played with because I haven't found a way to make it work in Vim yet.

Avery: One of my fixations as a CEO is I insist that every change to the policy file get run by me. Almost nothing else in the whole company runs by me, but I'm like, no, if you're changing the policy, policy and tax, I wanna see it first. So we went through a lot of iterations of the ACL grant syntax, uh, before we finalized it, and I'm really excited about what we came up with.

I realize it's a little strange to be really excited about a file syntax, but I actually am really excited and I think in a little understood feature of ACL grants is it's really extensible. Like you can grant stuff to applications that are provided by people that are not tailscale, that are running on your tailscale network.

And when you connect to that application over tailscale, it has visibility. Into the grants that you gave it based on your groups, the tags, the blah, blah, blah, whatever's going on in your tail, tailnet routing. And it doesn't have to know about, it doesn't have to know what group you're in. It doesn't have to have its own business logic about what group you're in.

It can just say like, tailscale says this connection should be allowed to do this thing on this thing. And you can change all that in a central place. And so easy example is like Grafana, you can say. Today, everybody in the production group should have admin access to Grafana. So when they connect to Grafana, they get admin access.

You don't have to set anything up in Grafana. If you change your mind later or you change who's in that group, then next time they connect to Grafana, even if it's like three seconds later, their permissions are gonna change. Right? And that was not possible before we had this ability to just sort of like pass these things through.

And so it gives you this ability to just build on top of tailscale and just stop worrying about all that stuff.

Corey: You also have a great feature where you can effectively disallow people from modifying things in the console without going through a whole bunch of very scary warnings, uh, and mandating effectively a GI ops flow, which is fantastic, especially combined with the fact that you have test cases

Avery: built into your policy files.

Exactly. Well, it's super fun, right? Because you, you just said like, you can't use our ACL editor because you like Vim, and like, that's not actually true because we did this another, another. Very nerdy thing I'm super excited about is that you can round trip the JSON of the policy file to the GUI editor and back with no loss of anything.

And it's not just regular JSON, it's our special weird hug, JSO, that has comments and extra commas. Uh, which means you can actually have comments, your js o describing what goes on. And then when you go to the GUI and then back, the comments don't get lost. And so you, and also GI Ops can like take this text, store it in GitHub.

Um. And then push it back when you're done. And then of course it's not very good to have the GUI edit it in tailscale if you're using GI Ops. But you can go to the gui, come up with a rule that you want, or an edit that you want, and it'll tell you what Text to paste back into your Git repository to get the results you want.

So it's this very nice flow where like everybody who likes everything gets to have what they want.

Corey: It's weird. It's just thinking back. It's been a bunch of small releases, but they add up to almost a, that completely different product that still does the underlying baseline thing. It always did, which is flattening the network to make it work like we all used to think networks did until we knew better.

It's a, it's been a very. Fun evolution. I think it was last year you did that partnership with Mulva, where I think for five bucks a month now I can get, I get access to the Mulva VPN stuff. It's a couple of clicks of a house and I'm suddenly emerging from anywhere else I want to be, which is super handy for me and my brother who lives in Brussels.

We are at poe, both have EU and US citizenship, so, but there's an awful lot of government sites that, oh, you're not physically here. Clearly you could never wanna access these things for no apparent reason. Trivial,

Avery: easy.

Corey: Al

Avery: Also my bank, uh, in Canada, whenever I go traveling anywhere that is not Canada, they're like, oh my God, nobody outside Canada could possibly need to access a Canadian bank.

And they kick me out. Uh, but I, you know, I could do that by, what I do is I use a, an exit node on my Apple TV at home and I just bounce through my Apple tv. Uh, but I also use Malva for experimenting and stuff.

Corey: I did that originally and then the raspberry pie I sent with my brother to his place wound up dying.

And that is not, he is a government functionary there. He's not really the, uh, the technical type as far as, Hey, now log into the Linux console and tell me what you see.

Avery: Y no, that's why I went with the Apple tv. 'cause they, you know, they're five times as expensive, at least as a raspberry pie, but they have five times at least as much quality control as a raspberry pie in the manufacturing process.

So,

Corey: and a warranty service that is comprehensible to humans.

Avery: Yeah. And a gui where you can just tell your brother like, Hey, can you go to the app store and pig tailscale, uh, as opposed to going to the console.

Corey: Yeah. I, I also like things that are, that have changed or some things have not changed in tailscale that are still somewhat annoying and I understand why.

Uh, I'd love to be available to connect at two tailnets at the same time. Now you can be logged in and toggle between them, but yeah, a device that talks between two networks is generally considered a bridge and corporate security would like a word if you start doing that. There are ways now to share nodes between tailnets that start making that a lot more straightforward.

I would still love, on some level, the ability to set a custom domain for the tailnet domain that I can control the certs for. I, I get that that is a hard thing to do. I'm sure some big customer somewhere has it, but Yeah.

Avery: Yeah. It's, it's surprising how it's, well, that particular feature. It's a little hard to do.

I would say the difficulty of doing it is not actually the thing holding us up. What's the, the thing holding us up is the phishing potential when you start doing it. Uh, 'cause you combine that with tailscale funnel and people register some arbitrary domain that looks suspiciously like, but is not quite.

google.com and next thing you know, you're hosting phishing sites for Google. Do. Right. If everything ends in blurb, butty blurb ts.net, then you don't have that problem and it's like remarkable how much trouble that saves us. So we really want to get to the custom domain thing. We just need to like very carefully control who gets to have custom domains and minimize the abuse potential.

One way is like to attach it to not easy. All the people that pay you. That was one of the things we've been thinking. It's actually, it's, I mean, it's, it's a pretty good start. We should probably do that. Exchanging money for goods and services. That's wild. Yeah. Yeah. It's just the, do we need to limit it to only those people?

It's kind of sad to have to do that. I wish we had a better idea, but like, you know, nevertheless, uh, yeah, it's definitely on the list. Similarly with sharing, we've been in the same state with node sharing since I think like 2021. And a bunch of internal changes have been going on, uh, architecturally to finally like enable way more kinds of interesting sharing.

Um, but I really see like there's so much potential to newer kinds of, uh, node sharing. I don't think you ever want to be in two tailnets at the same time. I realize that everybody at first thinks you would want to do that 'cause it would be really tempting. But it is this bridge between tailnets and it like really confuses.

Like, as an example, I would like to be in two tailnets at the same time. I have a personal account with my family stuff on it, and I have a work account with all my work stuff on it. And where I'm the CEO, uh, that has access to a bunch of sensitive things. Right now, if I'm at a computer with my, and maybe your children should not have access to those same things.

Yeah. Maybe they shouldn't. Right? So if I have a device that my children borrow, right? An iPad or something like that. I really should not be logged into that device using my tailscale account. But if I'm on my corporate device, I really would like to have access to my private stuff. 'cause why not? Right?

But if I'm logged into both tailnets at the same time, now I'm inadvertently creating a bridge between my corporate account. So the security team should lock me out. Uh, and my personal account, right? And the security team incidentally almost locked me out a few days ago because I wasn't on the MDM yet.

So I like forcibly enrolled into the MDM, which forced me to upgrade my Mac Os. And there's a bunch of new features in Mac OS that I was missing. So I guess that's good. And that yak is getting nicely shaved. Yep, exactly. So I, I'm pretty far down this path, but, you know, um, anyway, what I think people want, and what I want to give people is the ability to log into each device using exactly one account.

And for you to be able to share many or all, or a good subset of the nodes from another account into your account. You are almost taking the GitHub identity model. Yeah, I guess so.

Corey: Yeah, I, I have a GitHub account, but I can, I be outta different organizations that do different things. My personal account is also what I use for work, but you can gate access to things and

Avery: yeah, that part makes me nervous.

Like when I log into GitHub, I have access to all my corporate stuff and my personal stuff. So if I log into my personal GitHub account. When I'm not on a work computer, I'm like putting work at risk, which is scary. So what I think we should do is still have the two accounts, but on my personal devices, I log into my personal account that doesn't have access to my corp stuff, but I log in my work computer, I've access to all my corp stuff, and my corporate user also has outgoing access to my personal stuff.

Corey: Yeah. For the last eight years, I haven't really had anything personal because my entire life has become work Right around the time that, uh, shit posting on social media became a job.

Avery: Yeah, I guess that makes sense. So yeah, I mean, yeah, I, I'm really talking about the experience for other people. Um, but yes, I mean, I have an Apple tv.

Does the corporation want my Apple TV on the corporate network? Like not really. So little things like that, and I think we can do it. We're getting very close to being able to do it. We just keep, like doubling in size a lot. Uh, and so most of the engineering that we do, uh, actually ends up being just like, Hey, you, you now have like a tailnet with hundreds of thousands of nodes on it with like thousands of nodes churning per minute because someone is using it in a gigantic.

CICD cluster. Uh, did you know that's an n squared algorithm? Uh, did you know that the whole system is gonna crash? 'cause you did that? I'm like, oh, I didn't know that. Um, but then we had to fix it.

Corey: We learn exciting things, uh, through other people's use cases.

Avery: Exactly. So some of this stuff keeps getting delayed, uh, but it's gonna be really good when it finally comes out.

Corey: Yeah. And, and you're, you have a great list of customer references that are, that are doing all sorts of fascinating stuff, some of whom I know reasonably well. And what I've, I also like the fact that there are options if tailskill isn't right for people, if you want one that is a lot more confusing, a lot less capable, and much more expensive.

I mean, AWS has launched VPC Lattice and then they've marketed it so poorly that people don't know if I'm making that up or not.

Avery: Yes, I actually had not heard of them. Uh, that is maybe embarrassing.

Corey: Now, Nope, this is par for the course. I thought it was great when it came out and then I forgot it existed, and then it just goes years without being mentioned by anyone until I encounter.

It's like, oh, right, that, that, that exists. That's kind of neat. I should look into it and every time I do, I come away with, or I could just use tailcale and save myself a lot of heartache. So I do honestly, on some level, your next Go to Market for enterprise, you'd just be offering people a free month of VPC latice.

Avery: We've actually had that a few times and there's like a comparison and we're like, can we please be like first while you're doing the comparison and then you can, you know, install the other ones later. And they do, you know, they're done with tailscale in like 15 minutes and then they go off and they try to install the next one.

But it, but if they try to install the next one first, they might never get to tailscale. Right. 'cause they don't finish. That's the, that's the dark secret of POCs. Yep.

Corey: You've done a fair number of things that are, it's hard to even describe what tailscale is. You, you have tail drop, which is effectively an end-to-end, uh, file sharing option.

Uh, it feels like you are flirting with becoming almost a service discovery tool. Uh, one of the, we have enough service niche in the world, but it feels like this one makes a strong contention for being one.

Avery: Well, I have, uh, we're, we're trying out new versions of the mission statement 'cause previous ones were too complicated.

I will, I will present a preliminary version, uh, that we've been trying out. Uh, it is a new layer three for every device everywhere. It's like maybe too simple. You have to be a network person to know even what I'm talking about by layer three, I tried like new internet protocol. Sometimes people are afraid of that 'cause it's not like IPV seven, but it does the job of what layer three, the internet protocol was supposed to do.

And let me, let me try to explain what that means. So like way back in the day when I logged into the internet, I could connect to any device anywhere that was on the internet by using its IP address. That is, has not been the case for now decades, right? It's now gotten to the point where, in fact, the only things I can really connect to by IP address are.

Maybe my wifi router, if I can remember what the IP address is, and I'm in my house, uh, or cloud providers who own like most of the public IP space at this point. And that's like kind of weird. That defeats a lot of the purpose of the internet. Another thing that happened is if you have a, even if you had that connectivity, imagine you had IPV six rolled out everywhere, which requires a bit of a big imagination.

But let us imagine. IPV six was everywhere. If I switch to a different network, like between wifi and cellular, my IP address changes. And now the connection breaks. Uh, and I actually can't find that device unless I use DNS. Everyone's best friend, DNS, the thing that is not anywhere in the OSI stack, but is somehow playing some job, making some of the layers of the OSI stack work together, right?

So now I'm like dynamic DNS, I'll just update it every time my phone jumps between wifi and cellular, like not likely, right? And so like the actual interneting part of the internet stack does not work anymore. It's not location independent. And it doesn't make everything in the world addressable to me, right?

It's actually layer two. It's just a replacement for ethernet addresses because every time my interface changes. The address is a different thing. It might as well be an ethernet port, right? And it hasn't done this job that's like missing from the stack. And so tailscale jumps in there and it's a tunnel, but it's like, hey, it works the way it's supposed to work.

Like obviously the world has changed. You don't want everyone in the world to be able to access you, but everyone I want to be able to access me gets an it. It know it can find out my name, and I get a fixed ip. And I'll make it work everywhere. And it doesn't change when my device moves around. So tailscale, all the stuff you can talk about.

But the thing that it does is it actually pro produces layer three of the OOSI stack for the first time in decades.

Corey: That's nothing short of magical. Uh. It's, it's weird in that because this gets highly technical, highly, quickly, and goes very deep, but it is stupid simple to get set up. We were just traveling in France, my wife and I, and she wanted to access something that was only available from home.

Great. Hand me your iPad a second. I didn't even bother to have her set up an account. I just logged it into my tailnet so now she can get access to my shit posting nonsense if she really wants it. And suddenly it worked when I turned it on as an exit node. Uh. I've also found, and this is what really sparked the idea of having this conversation now, is with now I have a test Kubernetes cluster that mostly works.

I have your provisioner that auto the operator that automatically gives access to any service I put on the thing. It's got some drama when the nodes themselves are on the tailnet and that becomes their magic. DNS becomes their resolver. It tries to pass those out to containers and that becomes a little bit of a, uh, let's patch cord and s to make it not do that.

But once I do, I can spin up arbitrary containers. Not have to worry about security, which sounds like a wild thing to say, but the only place that those things are available is on the tailnet. I'm the only person except for my wife's iPad on the tailnet, and even then I could restrict it down further via ACL grants.

Suddenly I'm doing the thing that a lot of people used to do on the open internet of, oh, I'm not big enough to find, no one will find this weird port. I've bound it to only there is security. It's not just pretend security.

Avery: Right. And that's another thing that like, you know, if it's, again, IPV six, if it had been fully rolled out today, still wouldn't solve that problem because it was invented 30 years ago and there's been 30 years of new problems since then, right?

So it's like time for like a thing past IPV six, if we could move past it ourselves psychologically. But like there has to be identity, there has to be security, there has to be a concept of like, which things. Are allowed to connect to which other things, not just the dream of the late 1990s of like, you know what, if everybody could just talk to everybody, the whole world would be happier and we'd have world peace and stuff.

And we sort of learned from the internet that like world peace doesn't happen when everybody can like chase you around and harass you all day, right? And so you just need that level of security. But you want the feeling that we had on the small internet before, you know, most of the really bad people showed up.

Corey: I think that's the right path. It's. You have, I keep forgetting this because of course in your case you have to deal with a, especially with a free way to get started here, you have to deal with a tremendous amount of abuse concerns on this. But it's, it's also not traffic necessarily passing through you.

One of the smarter things you've done from pure cloud economics perspective is you're the coordination central point, but the actual heavy duty traffic is point to point.

Avery: Yep. Exactly. So tailscale splits in, in network terms. We call it the control control plane and the data plane, right? The control plane is decides like how to distribute the keys, how do you log in, um, who should be allowed to talk to which other people.

And then it like sends those instructions to every device in your tailnet. And then the devices themselves, uh, handle the data plane, which is sending the data direct whenever possible directly point to point between. Between themselves. So it doesn't cost us anything to transport your data. And it costs us very little to be the simple coordination point between the notes.

And this is what makes it extremely scalable. And a lot of this stuff is based on some of the original concepts of the internet, right? It's like, look, it should be extremely scalable. You can't have like one company that is routing all the traffic for you, such as at and t back in the day with the telephone network, right?

Like you just, you know, it works, but you shouldn't have that. You should build a system where that doesn't happen. Uh, and tailscale is very much, uh. Moving along those lines and it, it, it is kind of magical, especially because if you get two devices sitting right next to each other on your local network, they get direct connections to each other on your local network.

Right. Almost any other thing will try to beam it up to the internet and back, which is pointless in situations where they're side by side. And so if you've got a data center or a VPC filled with containers and they want to talk to each other, it's really silly to send all those things to the internet and back to say nothing of like the egress fees you'll incur.

Corey: It. What, what's weird to me is also how effective you are at routing money to other companies, uh, uh, through tailscale. I use moad, uh, as we've discussed. I also pay for next DNS because that's where I do most of my ad blocking, which makes it super handy when I try and hit something like a link in an email that gets blocked.

Great. I could special case it. Why would I do that? I'll just toggle off tailscale, hit the thing I need to and turn it back on. I do that multiple times. Every day you have become, uh, something I use constantly, but also almost never think about. Which is the, honestly, the, the Val Howa of infrastructure.

Avery: Yep. Infrastructure is really tricky. 'cause we have, you know, we're trying to balance, uh, word of mouth. 'cause you want everyone to brag about how they use tailscale. And simultaneously the best infrastructure is the infrastructure you never think about. So it reminds me, I forget the name of this, that this trendy workout.

Uh, campaign from like 10 years ago where like the, the joke was like, how do you know someone's on this trendy workout campaign is like, they won't stop talking about it. Uh, so tailscale people, people love their infrastructure so much that they will not stop talking about it, which is a very strange situation to be in.

Uh, I did not see that coming when we started the company, but it's more or less what like drives the adoption of tail.

Corey: This

Avery: episode is sponsored

Corey: in part by my day job Duck. Bill, do you have a horrifying AWS bill? That can mean a lot of things. Predicting what it's going to be, determining what it should be, negotiating your next long-term contract with AWS, or just figuring out why it increasingly resembles.

Phone number, but nobody seems to quite know why that is. To learn more, visit duck bill hq.com. Remember, you can't duck the duck bill. Bill, which my CEO reliably informs me is absolutely not our slogan.

Yeah, every time I see weird questions on come through on the AWS subreddit, which I keep a loose eye on, it's like, oh, that sounds like a tailscale usage.

And sure enough, it's always the first comment someone has there. Have you considered using tailscale for this, like a sensible person, which

Avery: Yeah, exactly. Yeah. And you mentioned like these partners that we work with and roading money to them, like tailscale is increasingly, uh, it's a little among. You know, in the, in the entrepreneur world, you have to be really careful with this word, but we are increasingly a platform.

And what is a platform? It's like the base layer of something that people build on top of, right? And uh, I was talking to our investors the other day and someone said like, look, the, the advice, uh, or the most important thing to know about building a platform, and the biggest mistake almost everybody makes is trying to do it.

Uh, and especially doing it too soon, like almost no company ever actually builds a platform. And if you are wrong and you go and build one anyway, you waste a ton of time and energy. And so we've been a little bit dragged into building a platform. I've started talking about last year how maybe someday tailscale's gonna evolve into a platform.

Form. Uh, and then this year we made a feature that's called the TailNet's, API. So a completely automated way to create a new tailnet, add devices to it, and then spin down the tailnet, share it with other people and stuff, just entirely API based. And so now we have big cloud providers that are like, you know what, I'm gonna make my InterCloud.

Connections, just use tailscale in the background and our customers don't even have to know about it. And I'm gonna do it all using the tailnets A API, right? So we're kind of like, well, this is way ahead of schedule now we're a platform. Um, and I don't even,

Corey: can you cheat it under the hood to take specific decisions on the path traffic takes to get from point A to point B?

Avery: Yeah, exactly. I mean, they're basically, um, well the big, the, the problem space that these people are mostly in is like they're, you know, lower tier cloud providers. They provide, you know, the biggest thing is usually GPUs, right? Uh, at better prices than the big cloud providers have. And then customers like ignore the prices.

They actually have them for rent. Yeah. Or more availability, et cetera, right? Or the right ones at all kinds of things. Uh, but then the same customers wanna run the rest of their stuff in a more mature cloud provider. Now you've got a connection problem between like, kind of weird GPU Cloud provider and the top tier provider, right?

And so how do you connect between cloud providers? Well, it's actually hard. Almost nobody makes a product for that at all. Um, these cloud providers, they could tell you like, go use tailscale, but then you have to go figure out a third product that kind of slows down their marketing. So they're just like, you know what?

We will provide the service of connecting you to anything. Uh, don't even worry about it. Uh, and they just like set up a tailnet and suddenly their VPC on that cloud is actually connected to the VPC on the other cloud

Corey: and it's the right path. But what I have found that is, so I guess. Compelling about all of this has just been that over the years it has, it has solved so many weird problems and I continue to watch the logos on your site continue to expand, uh, to going from small companies to mid-size companies like, I don't know, Microsoft.

Avery: Yeah, Microsoft, uh, recently got added to our logo list. Uh, there's a bunch of other, you know, there's subsidiaries of Microsoft, there's a bunch of other big names. Most of our biggest names are still not actually in our logo list because we didn't get logo rights for them. Uh, people often that is always the way that it works.

It's especially true in the security world, uh, because security people are like, wait, I don't want to advertise what our infrastructure is using for security. That's just like painting a sign on our back. Yeah. Do you view yourself as a security product tradition? Well, so I've ex uh, well, I'm, I'm stumbling on this because the correct answer is sort of, uh, or Yes.

Corey: Well, who, whose cost center is our purchase, this contract coming out of? Sure. We're a security platform. I get it. Go, go where the money is. I hear you. Are you an analyst? No. Unless you have analyst budget, then? Yes.

Avery: Yeah. So tailscale. I think the best term I heard for it is a mesh VPN firewall. Right. Um, and the reason for that is most people who end up adopting tailscale, adopt tail scale 'cause it solves a connectivity problem that they have right now, and they just, it becomes the easiest way to connect things.

What's very strange about tailscale, uh, and very strange in the security world in general, is that when you use tailscale to solve that problem, you accidentally make your system more secure. And also the easiest thing for all of your engineers and people inside your company to do becomes. The secure thing instead of the insecure thing, and nobody really sees that coming.

Um, but then once it gets there, the security people are like, wow, how come I'm not the bad guy? I'm always the bad guy. I don't wanna be the bad guy. Uh, we love tailscale. Uh, most of the time today, tailscale is not adopted through the security team because the burning problem is not like blocking people from connecting to things.

The burning problem is usually connecting to things. Uh, but you get both at the same time. And that was like from the very beginning. At tailscale, usually you have to buy like a connectivity thing, like a router or a VPN and a firewall. And they're run by different teams and they fight with each other all day.

Corey: Honestly, I found that the most people I talk to the most who are the biggest champions of tailscale are the ones that are empowered to do the thing that they wanna do. It's, oh, uh, the policy is because I said so the end, this doesn't feel like it's something that's gonna be instituted top down just because it's not painful enough for the user.

Avery: We have a new experimental thing that we're working on, uh, that I think is really gonna appeal to security teams specifically as buyers. And I wanna run it by you and like hopefully get feedback from everybody else who's listening. Um, you can, you can post my email address or my blue sky or whatever you want.

Um, so people get you say that and yet. You know, I, I get hate mail. I've, I've received hate mail at this point. I, as a CI get hate mail from my own employees. Um, and so, you know, the, the skin gets thicker over time. Um, but yeah. So here's the thing. Uh, ai, I think we've all heard about it. People are deploying it.

Um, in their companies, uh, and often carelessly, believe it or not, uh, they don't always think about all the consequences before roll out ai. Uh, and yet, some many companies, and some of them we've heard about, uh, more than others, but many companies have directives from the top down to roll out more ai. So the CISO is sitting here and it's like, wow.

Everything you guys are doing is horrible. Uh, and this is a ticking time bomb. And I can't believe that I have to say yes to this because my job is not just to block progress in the company. My job is to like ensure success or ensure security as much as we can. But if they say yes, it's like there's gonna be a breach.

And if they say no, they're probably gonna get fired because they're blocking progress. Right. I think a solution to this is when you want to, and, oh, sorry, I forgot another part of the story, which is that. When you're bringing AI into the company, that's one thing. The new trend in AI is this MCP protocol model, context protocol that you can use to connect your favorite AI agent to your favorite data source, no matter what it might be, or all of your favorite data sources, right?

When you do that, all kinds of terrible and exciting things can happen. And if you Google around a bit, you can find examples of like someone hooking. Hooking.

Corey: Oh yeah. The, the attack vector now is quite literally telling the computer, trust me bro, in those words,

Avery: and it's so, so exciting, the kinds of problems you can have, like some people hooked at GitHub up to this and like the repo that it looked at contained instructions to the LLM that then convinced it to take the rest of the data in GitHub and send it to somebody else.

And he is like, wow, that's, that's a super neat attack. Uh, as a security person, I can appreciate super neat attacks, but also like, wow, what are you gonna do to defend against this kind of thing? Right? And I think the answer is the LLM has gotta be supervised, just like any, uh. Any person or any weird thing that you put into your network, you've gotta have auditability control, acls, identity encryption, uh, all that stuff that you should always have, that you actually don't have today when you hook an AI ops to stuff.

Right? The way to do that is to funnel your AI traffic into a thing that has the ability to audit log. Um. And and control and filter and decide what can connect to which other things and then forward it on through. Right? And of course, tailscale is a connectivity and security layer that makes it easy to build such a thing and deploy such a thing.

But then you have a really interesting other problem. And I apologize if this is getting like weirdly deep, but I hope your audience loves weirdly deep things once you've got a proxy. Once you've got a proxy that is forwarding traffic from, like it's acting on behalf of Avery, say, on its way to Salesforce, right?

Avery goes into the proxy. The proxy then wants to go to Salesforce. The Salesforce says is like, okay, you're a proxy. You have a like service account. What did we do? Do we set up the service account to have, we have global access to Salesforce, and then the proxy needs to be trusted to only give Avery the stuff Avery should have access to.

Well, that sounds like a terrible idea, but. It can't act as Avery by default, because it's not Avery. It's running as proxy and it had incoming connection from Avery that doesn't give it rights to Salesforce. So you have to have this little interchange

Corey: to avoid confused deputy that way.

Avery: Yeah, exactly. So you have to have this interesting interchange where Avery makes a connection to this proxy, and the proxy has the right to exchange that, that identity for a token that allows it to access Salesforce.

As Avery with a little note on it that says, by the way, it's Avery's ai, don't give it too much stuff. So it's like Avery minus minus. To do that you can use an OAuth protocol that I won't go into, but it's like there's, you know, originally when the MCP standard came out 10 months ago, I think, uh, there was like almost literally at this page intentionally left blank in the security section.

Uh, since then there has been an improvement where they, they said actually OAuth should be the way you do this. And then people started implementing that and now they're at the stage where like. It tries to o off to like 10 different things and each of those things leads you to a click through, uh, to granted permission to do some stuff.

So with tail scale, we have this neat feature where like every connection that happens on the tailscale network has your identity already attached. You don't have to click through anything. It's just like inside your tail net, everything knows who you are.

Corey: Every request inherently becomes authenticated.

Avery: Exactly. So the trick we did is we wrote this new tool on top of tailscale called T-S-I-D-P, uh, the tailscale identity provider. Uh, it's open source, by the way. You can look at the, uh, GitHub repository and fork can do whatever you want. It's only a few hundred lines, and what it does is it's a complete OAuth server, but the user side is just, I already know who you are, right?

So when you try to access a service, the service redirects you to your IDP, which says, I already know who you are, and then redirects it back. No click throughs. But it's controlled by the ACL grant policy we talked about earlier. 'cause it's just a tool on top of tailscale. We didn't have to modify tailscale to make any of this work.

It decides which kinds of tokens it's willing to exchange on behalf of this proxy running inside your tailnet. Right, but this proxy, the TS IDP server, can be accessible over tailscale funnel to the outside world. So you can even use TS IDP with any service on the internet that supports custom IDP or custom OIDC.

So you have this really interesting situation where. From the very beginning, tailscale is like, I'm not gonna be an IDP. We're not doing usernames and passwords. Get outta my way. That's the past. Let's live in the future. Use a real IDP. You should still do that, but you can use that to get into tailscale.

And after that, you can use TS IDP to connect to everything else. And this MCP thing means your AI can do the same thing. Right. And all of it can be zero click because you can set a policy on your administrator for you. Company can set a policy on T-S-I-D-P to decide which things can be zero click.

Right? And if you're worried about sort of privacy, I know a lot of people who like use Google, uh, log in with Google are like, ah, Google's tracking me all over the internet now. 'cause I use login with Google every time I log into a service. They know every service I use. Now Google only knows that you use tailscale, right?

Because your instance of TS IDP that you ran, that is open source, is the one doing all the rest of your authentication. And so you have access to all. You're the only one that has access to all that private information. Even we don't. 'cause it's just a tool, right? Built on top of tailscale. And so the combination of all that stuff allows you to like control your AI access, but it also lets you have zero click authentication to like everything on the internet if you want.

And it also lets you have zero click authentication to things on your tailnet that don't understand tailscale. All they need to understand is custom OAuth. So I think Home Assistant is a really popular one. Grafana is another one, et cetera. So I apologize for that monologue. I'm still working on the short version.

Corey: No, please. It's, it, it's a, it's a fascinating approach because we are definitely in a post network world. It used to be that once upon a time you had breaches where I'm gonna go and I'm going to go and take things out of your system and then send it to a different system somewhere else. Now you can do all of that just by hitting the same single endpoint.

That's just the AWS Control plane. And it just a question of what the content of those requests are. So you effectively have to, I don't think we call it this anymore, but you need to, uh, man in the middle, everything that is being passed through for deep packet inspection, which in turn then becomes, if you can see all the payloads, well, you now have a central point of attack for that, but people have already accepted you in a security facing role.

I think that. It is a more novel approach that is likely to get further than the current security posture, which is putting the No seriously, bro, be secure in all caps in the system. Prompt.

Avery: Yeah. Well, exactly. And the best thing about this MCP proxy thing, first of all, you can have it right. We have a little default one.

It's open source. Again, you can like build your own if you want, and it can run on your private tailnet and it can access stuff that's on your private tailnet. It can be accessed by your favorite LLM that might or may not be running on your private tailnet. And also it can access things outside your private tailnet so there's no people coming in trying to beat on your MCP server to find the security holes, right?

It's only the content that matters. And for that, you can have something filtering the content and watching what's going on to make sure the AI doesn't go wildly off track. Yeah, I think that's

Corey: the, that is the right path. It's, it's part of a defense in depth approach.

Avery: Exactly. We're aiming for this like, again, convenience, where like the easiest way to roll out AI in your company is the tailscale way, and also coincidentally, it's gonna be way more secure.

If we can get that, then I think we'll really, like, we'll be on the.

Corey: The two problems I can see, you're gonna have one, you use the Salesforce, Salesforce example, but everything has to start supporting this on some level at an application level.

Avery: So they need to support OAuth. They don't need to support any of the rest of the stuff.

And that's what's really neat because everybody who makes an MCP server has to support OAuth now as part of the standard and like where APIs were kind of hard to get access to before the trend is that, look, everyone's gonna be mad at us as a vendor if we don't support OAuth for getting API keys.

Right. As long as you have that, all of the rest of this magic is happening behind the scenes, the gateway has to understand all this. Tis IDP and everything, everybody else just sees to know server.

Corey: Yeah. The, the other challenge that you're gonna have, and this is trivial of course, is you have to come up with a few, uh, reference implementations of this that are basically click, click done, and to show folks how it works.

They can modify to their own approach. But historically, my big problem with. Uh, early with early stage products is the documentation's not there. You've gotta basically read the code, come up from first principles, how you want to tell it to actually do the thing that you do. A little bit of documentation goes a long way and not for nothing.

Increasingly, that documentation is being written for LLMs so that they can then explain how to do this to folks. So there's a, there's a bit of a lead time. It has to be absorbed into the models before it starts spinning out.

Avery: Yep. Yeah. The best we have right now, uh, we have a, a YouTube personality that works for us that runs the tailscale YouTube channel, Alex, and he's got at least one video about T-S-I-D-P.

Uh, that's from before we added this MCP layer, but it's actually pretty well done, was like many, many people in their personal tailnets are already using T-S-I-D-P for their own stuff. Uh, so I think there's gonna be some, some growth there. But yeah, we're gonna have to document it. We're gonna have to do all that work.

This is all pretty, pretty early stage, but we're really, we're interested in like talking to people who think this is gonna be interesting to them and like kind of working with them on making the product better and also integrating into the open source world, uh, because tailscale. Personal plan is free.

Um, and it's unlimited, essentially, uh, unlimited time. Lots and lots of devices. You can do all kinds of stuff with it. And it would be nice to make people, or have people who are using this in their home lab already, they can take advantage of this thing as well.

Corey: Oh yeah. I, I do a lot of testing in my home lab for this exact sort of thing.

I, I still haven't gotten quite to a level of comfort where I'm putting production nodes independently On the tailnet, I tend to use subnet routers, and, and that is for now the way that I approach it, just because it, it feels like taking anything into a critical path, past a certain point, has risk attached to it.

That's how we built it and that's how it works for now. If I were doing it today, I don't know that I would be as cautious given the conversations I've had since then with customers who are working with it in that way.

Avery: Yep. There are, there are some very big name customers, some of which I can name and some of which I can't, that are like all in on, like, we're gonna run Kubernetes in every single pod, in every single cluster, in every single store.

Um, and turning 'em like crazy 'cause that's what Kubernetes does. Uh, and they, they seem to be pretty happy. It means we have to have like pretty high up tam on our control server Tailscale is designed so that even if the control server went down for. A while in, in fact, it could go down for hours. The data plane keeps on working.

So there's only certain things that stop working if the control plane is like out of touch for a while. So you have this like, pretty high level of resilience that people don't expect, and it comes from us not routing your traffic. Uh, for the most part,

Corey: that that is the bridged cross. And you've, you've hit a point now where there's enough of a community around tailscale.

That if someone's trying to do something that no one else has really done before, it is no longer likely that they're doing something correctly. I, I don't mean to be unkind, but in the early days, I would, I was talking to your team near constantly with what? How do I do this thing? Oh, we hadn't considered that.

Now, whenever I ask any of those questions that come up like, oh, here's a giant blog post on how to do that, or, here's the GitHub issue where we explain exactly how you're holding it wrong, and so on and so forth, which is just. It's, it's a, it's a maturing of the product.

Avery: Yep. Yeah. We've been putting a lot of work into maturing it.

I think one of the hardest things as CEO, uh, is just con convincing everybody to not build a. Everything they want. And just like, let's focus on refining the core. Let's do everything we can to run this business so that the core gets better and better and better. And that's how we're gonna make money.

Not like building tons of stuff on top. Uh, which I know is a pretty unusual, especially in the security world, is not the normal way to do it. The normal way to do it's collect. Uh, I know it's like collecting Pokemon cards or whatever. Well, I need a DLP and I need this and I need this and I need this and I need this and I need this.

And now you can buy it from one vendor and it's gonna be a collection of like sort of. Halfheartedly integrated tools, right? And tailscale is like, look, we're not that. We have this one thing, it works super well and it's gonna work with all the other stuff you buy from other people, but it means we spend all our time just like, you know, writing docs like those or fixing the bugs that led to the need for docs like those.

Corey: It's really neat. Any, any last words on what we can expect in the somewhat near future? Anything fun and exciting? Uh, coming down the pike, which I know is a weird thing to say about a networking infrastructure tool, and yet.

Avery: Um, I think the, the two most interesting things are happen are that are gonna happen.

One of them is more and more stuff is gonna be buildable on top of tailscale or include tailscale as an option in it. So we're starting to see more and more things like, Hey, if you run my program, it's linked with the tailscale library. Just paste your offkey here and that thing is just going to work.

Um, a similar one is, I think. Think, I don't know if we've announced it or not, we're gonna announce it. If not, this is the announcement. Um, the workload identity feature that allows, if you're using tailscale with GitHub actions, for example, to just like, not even use Offkey because you can set it up. It's like, oh, this is your account on GitHub.

I believe GitHub when it says it's running under this account. So now everything just has access to your tailnet automatically. That's super slick way to do it. You don't have to manage rotating off keys and stuff like that. And I guess the third one is, uh, for direct connectivity. You know, life is not always perfect.

Sometimes firewalls are weird. Um, so we have this new thing called, you mean there are times where they're not. Well, some are weirder than others. So we, we get through almost all the weird firewalls, but there's some extremely weird firewalls out there. Uh, we have this new thing called the peer relay also in alpha, but if you're interested in choir within, uh, should be in beta sometime soon.

But you can have really access if you, if somebody asks, um. It allows, basically, if you remember the old days of Skype and Supernodes, it allows you to build supernodes that will route the traffic in situations where direct connections are not possible. So you can still get full speed if you put your Supernodes in the right places, um, including behind a firewall if you want.

So then. Even when things can't manage to get direct connections, 'cause your internal firewalls are too weird, if they can connect to the supernode behind your firewall, you can still avoid the egress traffic. Uh, so this is something that our biggest customers with, of course, the weirdest firewalls and the most firewalls, uh, are gonna benefit from humongously.

Corey: I would love to hear the story about which firewalls that are doing this and how they're configured. 'cause that is such a rare occurrence in the modern era, but,

Avery: oh yeah, we actually, we actually sponsored a patch to free BSD to finally fix this problem, 'cause for a while, free BSD, any free BSD based firewall.

Of course, it's pf. Yeah, well, uh, it was, it's in, it's intended as a security feature to be blocking this stuff. It just turns out when you do the whole like, decision tree, it turned out that didn't increase security at all and just made everyone's life miserable and in instead of, uh, of, 'cause it makes it so secure that to get anything done, people start rolling out UPNP.

Uh, and UPMP is never a good choice. Uh, security wise, and yet it's the only workaround to this problem. So we finally convinced them of this. We sponsored free BSD to like, Hey, can you at least make it a flag? Why does AHI take up all my CPU core? Yeah, yeah, yeah. So we made, uh, now there's a flag and I think the flag is now the default.

Just not be silly, but there are a few other firewall vendors that are doing the same thing, but I'm hoping we can talk them out of it. 'cause it's actually a relatively simple, it's called a hard net versus a. Easy net in tailscale terminology, and they make their hard net hard for like it turns out. No good reason, and it's avoidable if you change your code just a little bit.

But unfortunately, sometimes it's our competitors making the firewall so they don't always super eager to do that. Yeah,

Corey: I, I really wanna thank you for taking the time to speak with me. If people wanna find out more, where should they go?

Avery: Uh, well, there's tailscale.com. We have a blog. Uh, sometimes I post in the blog.

I also have an account on Blue Sky. I have a little used account on the system for Blue known as Twitter. Uh, and I have my own blog on appware.ca,

Corey: which has been a recurring presence on the newsletter. Whenever you put something interesting out there, and we'll put links to all of this in the show notes.

Thank you so much for taking the time to speak with me. I appreciate it. Thank you very much. It's

Avery: always a pleasure.

Corey: Avery Penan, CEO, and co-founder of TailScale. I'm cloud economist Cory Quinn, and this is Screaming In the Cloud. If you've enjoyed this podcast, please, we have a five star review on your podcast platform of choice, whereas if you've hated this podcast, please, we have a five star review on your podcast platform of choice along with an angry comment that isn't going to post properly because you once again have misconfigured your crappy firewall.