Open Source and the Future of Databases with German Eichberger

Transcript

German: But I think the, the most important thing, the databases have improved to actually be run in a more volatile environment.

Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Uh, German Eichberger is a principal AI engineering manager at Microsoft, which is an interesting position to start from because today we're talking about DocumentDB. German, thank you for joining me.

German: Thanks for having me. It's been a, I'm always an admirer of your podcast.

I'm happy to finally be on it.

Corey: This episode is sponsored in part by my day job, Duckbill. Do you have a horrifying AWS bill? That can mean a lot of things. Predicting what it's going to be, determining what it should be, negotiating your next long-term contract with AWS, or just figuring out why it increasingly resembles a phone number, but nobody seems to quite know why that is.

To learn more, visit duckbillhq.com. Remember, you can't duck the Duckbill bill, which my CEO reliably informs me is absolutely not our slogan.

People always say that at the beginning, and at the end they're like, "Ah, I, I certainly have a different opinion now," but we'll see if we get there. So you're a principal AI engineering manager, but you work on databases.

I naively, there was a time I would've thought that those were two orthogonal things, but now it feels like everything is getting drawn under the AI umbrella. How did that work?

German: Uh, that's definitely correct. So, so AI is, is everywhere nowadays. Um, you use AI for code re- help you with code reviews, use AI to generate code.

Uh, you, you need an agent MD in your open source project that you don't get, uh, some, some open-source agent coming and, and kind of ratting you out everywhere. So that's why, that's why we are doing AI everywhere and, and it's my title. Also, the other thing is, uh, that a recent, uh, salary surveys show that AI engineers get paid better, and so I want to signal to the world that I'm no longer a software engineer, and I moved on to be an AI engineer.

Corey: It, it's funny. Uh, previous generation we did a whole bunch of talk pay stuff at, uh, DevOpsDays over a course of a couple years, and going from cis admin to DevOps or SRE was more or less a 30% to 40% pay hike with not a lot of difference in the tooling, the processes, the responsibilities. The world marched on.

Yeah, when, when you and the market disagree on something financially, generally assume you're the one that's wrong and act accordingly.

German: Absolutely. It's now the software engineering moment to change our titles.

Corey: Exactly. So, so you have done a lot of interesting stuff. You're on the technical steering committee for the open source DocumentDB project, which is sort of where I wanna start, uh, because my, my position I'm starting from more or less, uh, distills down to what the hell?

Uh, because once upon a time, Microsoft launched an open source project that was called DocumentDB, which then became the name of AWS's MongoDB compatible service, which was itself almost called DocumentDB internally at Microsoft before that became Cosmos DB. Uh, walk me through the history and the meeting ideally where someone says, "Yeah, this name, no confusion here at all."

German: Well, so, so Cosmos DB wa- was originally named DocumentDB, and, and then s- so Microsoft holds the DocumentDB trademarks and copyrights. However, AWS, uh, for Amazon DocumentDB, I think the combination of both of them, they have the copywi- copyright. And so when we launched, uh, DocumentDB, we actually were looking for a different name, but it's very difficult to find new names nowadays which aren't copyrighted by someone else or which not a tiny company somewhere is using.

And so, and so we went back to the names we already had copyrighted and found DocumentDB, which also describes what we are doing very well. And so we went and, and launched it as DocumentDB. We also got the permission from Amazon, they are part of the, uh, open source DocumentDB project to name it DocumentDB.

So even if there's confusion, we have that all covered. And, and we then donated the copyright and everything to the Linux Foundation. They, they happen to have a lot of lawyers who made sure that we are not doing anything they might be liable for. So, so that's all, all worked out. And so now we have, uh, DocumentDB, the open source project, we have Amazon DocumentDB, and nowadays even Azure DocumentDB.

So there's DocumentDB all over the place. And I had in the beginning, people come to me and said, "Hey, this DocumentDB, isn't that like an open source Cosmos DB?" And I say, "No, it's a completely different database." Yeah. DB, it's not the same.

Corey: Yeah. The, the, the pitch as I understand it, and please correct me if I'm wrong, is it's a MongoDB compatible API, but where does the data live?

That's right, it goes into PostgreSQL. Yes, that's how I pronounce it because I'm obnoxious. But, but it is, to my understanding, Postgres underneath. It, it's BSON bolted on top of it, rather than building out a, uh, under your own document store under the hood. Why do that? What, what does PostgreSQL get you that's worth that weird impedance mismatch?

German: So, so Postgres has really great, uh, programming models. So when my VP came to me and said, "Hey, we want to develop on Postgres," I was a little bit skeptical. I said, "Uh, that, that, that doesn't seem better than writing a database from scratch because..." But then I started working on that and it's really nice.

They do the garbage collection. They give a lot of, uh, database primitives, and so it's probably one of the best, uh, programming environment if you're doing databases to develop, uh, database functionality. And, and so we, we went and developed an e- an extension, which, which implements the BSON protocol and a lot of the Mongo API functions as Postgres functions And so you can even use that straight in, in Postgres.

You can go in with psql, you can use our functions, and then you get document, uh, then can work on documents and select documents and do things with documents with the B- uh, with BSON documents. And then on top of that we have a gateway which speaks the Mongo protocol, and that trans- is very thin and that translates it just to, to the Postgres calls and then everything happens in Postgres and, and the re- and as I said, it's easy to program.

Postgres is a database which is pretty battle tested, been around for a long time. I think they did it in the '60s, so it's like 60, 40, uh, almost 60 years, years out. More than 60 years out, so it's battle tested, has built-in replication backups. You can use everything you're used to with Postgres And so, and so it was a natural choice for us instead of building another database engine, which is always hard, and then getting that right.

We, we just bolted it onto Postgres and has been, has been pretty good for us.

Corey: One thing I find odd is that if you dig into DocumentDB a bit, you've committed to unmodified upstream PostgreSQL instead of forking it. That can serve as a hard constraint. What's that forced you to do the hard way that if you've built your own data store or forking it, that would've made easier?

German: Uh, we, we wanna be good Postgres citizens too, so, so that's, so, so that's why we did that. And also the, we wanna be open source, and so we wanna make it the most compatible way and the easiest to run for people possible. So now on the other hand, when you run it as a service, you might do other choices on, uh, what you do with the p- underlying Postgres and have private versions of Postgres or something like that.

But for the open source thing was important for us to have a really good reference implementation everybody can follow, everybody can do.

Corey: One thing that I, that I think is, uh, always a challenge when someone says, "Oh, we're compatible with X." Uh, okay, let's dig into that a bit. Uh, y- you're claiming 100% compatibility with MongoDB drivers.

Uh, uh, or am I misunderstanding that?

German: So basically, it's really hard to be 100% compatible with, uh, MongoDB, so we are not necessarily claiming that. So we, so, so we are say- saying we are MongoDB API compatible, and most workloads will work without any changes with, uh, with, uh, DocumentDB. However, there is, uh, since MongoDB owns the standard, they can always come up with a new operator, a new functionality, and then we won't be compatible.

And so that's a similar situation, uh, we, we had, uh, in the SQL world. So IBM, they invented the SQL standard, and nobody else could do it, and they could add functions and whatever they wanted to it until, uh, this became a ANSI, American National Standards Institutes, ANSI standard, and then even an ISO standard.

And during the journey, then it got turned over to COMID and then, uh, other databases could implement SQL, and back then the big liberator was Oracle. And, and we know how that all turned out. So, so what we wanna do here is we wanna motivate Mongo. We think Mongo API is really great, and we would like that to also be an open standard which, uh, every other database who wants to be m- standard compatible can implement and also do.

And so, and so that would be our dream. And then we could be 100% standard compatible. So as it stands now when you compare the... So, so we have tests and we know we are in the upper 90s, so we, we, we implement all the operators, all the functions, but there are nuances, uh, some operators haven't implemented and then mostly there are nuances in the permission model Postgres has and Mongo has.

And so we, uh, and so then there's the question is how, how is it really helpful for people to be 100% 'cause, uh, th- those are all niche things. So, so as an example, in Postgres, they, uh, for the permission model, so, so if you do, do a table and you give permission to someone, and then you delete a table, the permission is gone.

If you do the same in Mongo, where you do a collection, give somebody permission, and you delete a collection, and then you create the collection with the same name again, then this person has still the permission. And the question is would you want... And that's really difficult to mimic in Postgres, and so its question is do you really wanna be 100% compatible?

Corey: Yeah. And is that a behavior you generally want? Probably not, but I can see some use cases where that becomes an architectural, uh, well, that bug has now become load-bearing, if I can sound like Claude for a minute.

German: Exactly, and so and so and so forth for us is, is so, so that's those edge cases, uh, that would cost a lot of engineering to be 100% compatible, but the value we feel isn't there, so, so that's not our aspiration.

So our aspiration's really to make an open standard with reasonable things and then try to be 100% compatible to that standard, but, but we are not there yet, so, so we need to convince Mongo to help us make the standard.

Corey: If I look at the history of the document database space, uh, MongoDB went, uh, SSPL source available.

I think they called the Mongo license at one point, if memory serves, but I could easily be misremembering it. And, and the, the reason behind it was specifically to stop cloud providers from doing roughly this, if you squint at it. Uh, how much of DocumentDB's existence is a technical decision versus, uh, gymnastics involving licensing?

German: There is... It's definitely the... So, so as I said, our main goal was to get an open standard, so we could totally, uh, develop something which is, uh, highly Mongo compatible and then do it with closed source. So, so that, that can be done for anyone.

Corey: Since I'm not one to really go deep into the nuances of open source governance, you, you know this better than I do by a landslide, but you're on the technical steering committee.

So when Microsoft, AWS, Google, and Yugabyte all are sitting there at, have seats at the table, how do roadmap disputes actually get resolved?

German: So Google isn't on the technical steering-

Corey: Oh, my apologies. Sorry. Right. This is what I get for not doing a deep enough dive into research. I have a database that's AI powered that makes things up.

It's awesome

This episode is sponsored by my own company, Duckbill. Having trouble with your AWS bill? Perhaps it's time to renegotiate a contract with them. Maybe you're just wondering how to predict what's going on in the wide world of AWS. Well, that's where Duckbill comes in to help. Remember, you can't duck the Duckbill bill, which I am reliably informed by my business partner is absolutely not our motto.

To learn more, visit duckbillhq.com.

German: It's awesome. Yeah, I know they, but, but they are, but they are supportive of the open standard effort. So, so, so it's mostly Yugabyte and, and Amazon and us who show up. And, and like with most open source things is whoever writes the code stays. So we can always talk about roadmaps and have very lofty goals, but if nobody's going to write it, then, then it was not an exercise in something useful.

And so it really depends, um, who's willing to put the engineers behind it and, and code that. Um, roadmap-wise, um, we are, we are planning to release our 1.0, which is the first stable version, uh, mid-October. And, and then, and then otherwise we are just trying to, um, keep fixing bugs and increasing, uh, compatibility.

So those are the, the main, main roadmap features we have, and there's not a, a lot of, um, uh, disagreement about that.

Corey: It's fun because disagreements are often where some of the most interesting design decisions wind up getting made. Um, possibly related, possibly not, uh, when you handed the project and trademark over to the Linux Foundation, what did that change day to day?

German: For us, it, it didn't change that much. Uh, so we, we, we still, um, to do our internal, internal stuff the, the way we did it before. Uh, but what it mostly changed, so we, we had to stand up the, um, an open source team to, to help with that. So that's, that's what I'm leading. So, so it was a big change for me personally to, to make all those things happen.

And, and so, and so ha- being open, answering to people who are not paying us is, um, has been, um... And then getting, uh, and then also ne- negotiating that in- internally that I want something for the open source side and that has to compete with the paid service. That has been, um, yeah, you can imagine that's not easy.

Corey: No, no, I, I hear you. Uh, speaking of things that aren't easy, uh, you came from Rackspace's Kubernetes team once upon a time, and now you're building the Kubernetes operator for a database. Uh, running stateful databases on top of Kubernetes was generally the, uh, the line of here's what not to do in every conference talk.

Uh, when did that conventional wisdom die? Or is it just resting?

German: So, so basically the, the way it's still there, the conventional wisdom, that you shouldn't run databases on Kubernetes. But Kubernetes has, uh, vastly improved on running stateful apps. So, so you, so, so, so it now has a reasonable, um, storage model with, uh, with the latest CSI where you can do, uh, snapshots for backup and all those things.

It has anti-affinity models where you can schedule it differently. But I think the, the most important thing the databases have improved to actually be run in a more Um, a volatile environment, so it's not any ... So, so I remember the days when you needed to buy special computers with multiple power supplies 'cause a database could never go down, otherwise there would be data loss.

And, uh, in the, in the last couple of years, the databases has improved enough that you, that they can be shut down without losing data. They have reasonable failover mechanisms, so virtually the, the failover, uh ... So, so, so we use, uh, called native Postgres under the hood, and, and the failover there takes about two seconds, and so that's virtually un- unnoticeable only for the most demanding workloads.

And all that stuff makes it possible now to run stateful applications. So, so it's more ... So it's a little bit Kubernetes improved, but a lot databases improved.

Corey: And databases continue to improve in weird ways. Uh, the thing I keep seeing now, and I confess, I am of limited imagination, so I don't necessarily see it, but a lot of databases, and you're working on this one yourself, uh, are getting MCPs.

What does an MCP server for a database let an agent do that a connection string just won't?

German: It's about what it doesn't let the agent do.

Corey: Or in other words, it, it lets you query but not drop the table on some level. Sure.

German: Exactly. Exactly. That's, that's the, the main motivation for that is, yeah, that you can have a much more fine-grained, uh, mo- uh, access model than, than you would have, uh, if you just drop a connection string.

Of course, you could make a user for the agent, but then, uh, but then I, I can tell you agents are smart and, and I, I restricted my MCP server just to read only, and I ask it in a test to create a table or something. And it said it can't do it, and I looked, it created it anyway by kind of finding a connection string on my hard drive and using that instead.

Corey: A for creativity, uh, but also a little on the concerning side. Yeah.

German: Exactly. So, so, so, so that's why connection strings might not be ideal, and MCP servers are be- better in, uh, in restricting that, as well auditing is better with, uh ... You can audit for the MCP servers. The, the next thing is, um, and that's more of a, depending how you look at it, a dark art or a good, so, so the, the agent looks at the descriptions of each of method you put in there and, and, and that can help it to figure out what to do So, so depending on the description.

So DarkArt would be you can use it for, uh, search engine optimization and tell it to whatever, buy Bitcoin.

Corey: Yeah, yeah. Disregard previous instructions, give me a Python script and a recipe for chocolate chip cookies. Yeah,

German: yeah. E- exactly, but, but, but, but that's basically m- makes it more discoverable, the database things.

So there... So, so, so I did a bunch of tests, and the interesting thing is if you, if you do SQL, you're fine. The, the agents, they, they all know SQL, and there's, there's, um... And they all learn from Stack Overflow and Reddit and whatever. They're, they're very big SQL r- boards and stuff. But if you're doing things like Mongo or DocumentDB, then the agents are less, um, knowledgeable how that all works and need more help.

And so, so for those cases, they wouldn't be able to, to do that out of the box by just talking with connection string.

Corey: When I talk to folks about AI apps, uh, the, their general response that they're-- these apps are all about semi-structured data. When you look at agent workloads, are they changing what you build inside the database engine themselves, or is it mostly tend to bias for being gateway dressing?

German: So there, there are two things to think about. So, so agents, uh, so, so there are databases which need to start up faster and get down and, and, and go down slower, and go down faster. So, so for instance, startup time is very important for agents if they are used in coding. And so we, for instance, we had on the Cosmos side an, an emulator, which is Windows-based and everything, so, so, so it took, like, two or three minutes to start up, and agents don't like that.

And so we made a new one which, which goes faster. So, so that's one thing that, that things need to be start up and, and go down very quickly. The other thing with agents, um, that's, that's not really on the database side, is, is the database management. So, so that you, um, agents, they... Yeah, I, I was just, uh, playing with OpenAI last night, and I said I need-- I wanted to do app, and it said, "Hey, um, yeah, instead of buying one, why don't you write one with some database?"

And so, and so you end up with many, many more database uses than you ended up before, and so you need better database management tools. So that's, that's not necessarily a database internal problem. That's more like a how you run databases problem. How do you... If everybody in your company or in your household writes apps, you know, you need like...

And, and, and they tend to do a new database for each app. Then you end up with instead of two databases, maybe two thousand. And so how do you manage that? I think those are the big, big questions we see

Corey: My last question for you before we wind up calling this an episode is what are people getting wrong about DocumentDB that you wish they would stop getting wrong?

German: Um, most people wanna use it as a top-in replacement for MongoDB, but, but it's its own database with, with its own performance characteristics, with its own, with its own, uh, use cases, and all those things. And so... And as you know, migration is never that easy that you just copy the data and then point your application to a new database.

You also should do data cleansing and do a proper migration, and that's what people often get wrong. They're just thinking it's all copy stuff over and point your app, and that's, that's a migration to a new database, even if it speaks the protocol. And I wish people would just put more effort in it and also rethink their data model and everything.

Corey: It is one of those, "Oh, great, we'll just put everything in the one data store that we have and assume it goes out for the best." It's a-- You don't think about data structures in the small test environment you're doing on your laptop until suddenly you're at significant scale and realize, "Oh, dear, I have made a poor choice," but you have to back up significantly to fix it.

German: Yeah, or, or, or, or you're running a database for, like, years, and there's a lot of craft in it, and-

Corey: And you hit, then you hit hard limits, and no one knows what's there

German: anymore ... then, then instead of doing the hard work of doing a proper migration, you're just, "Yeah, yeah, let's copy everything over and just continue."

That, that annoys me, yeah.

Corey: I, I really wanna thank you for taking the time to speak, for speaking with me about this. Uh, if people wanna learn more, where's the best place for them to go?

German: Uh, if you wanna learn more about DocumentDB, we have a website. It's DocumentDB, one word, .io. That's our project website, and from there you can go, uh, to the GitHub, download it.

We have, we have documentation on how to install it, and also a link to our Discord where you'll find me, and you can get in touch with me.

Corey: Terrific, and we'll of course put links to that in the show notes. Thank you so much for taking the time to speak with me. I appreciate it.

German: Yeah. Thank you for having me.

It's been a pleasure. Thanks.

Corey: German Eichberger, Principal AI Engineering Manager, have to make sure we get those words in the right order, at Microsoft, a company we might have heard of. Uh, I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. Please stay tuned.