SlateDB: An Embedded Storage Engine Built on Object Storage
We open sourced SlateDB a week ago. Let's look at where we're at and where we're headed.
Apologies to those who follow me on Twitter and LinkedIn—this is probably old news to you—but I’m going to talk about SlateDB this week.
For those not following me, SlateDB is a cloud-native embedded storage engine built as a log-structured merge-tree (LSM tree) on object storage. That’s a lot in one sentence, so let’s unpack it. The cloud-native and object storage parts refer to the fact that SlateDB writes all its data to object storage (S3, GCS, and the like). SlateDB is a embedded, so it’s a library that runs in your process. It’s a storage engine that exposes a key-value API similar to RocskDB. And like RocksDB, it’s built as an LSM tree.
Object storage allows SlateDB to provide bottomless storage capacity, high durability, and easy replication. The trade-off is that object storage has a higher latency and higher API cost than local disk. I’ve written extensively about these tradeoffs before.
In fact, my cloud storage triad post was the genesis of SlateDB. The tail end of the post pitches the idea and asks for contributors to help me work on it.
Shortly after the post, Rohan Desai (cofounder at Responsive), Vignesh Chandramohan (manager at Azure Streaming), Paul Butler (CEO at Jam Socket), and several others all got in touch. We have been working together for the past 4 months to get SlateDB ready to open up. Last week, we got there.
We built SlateDB because we felt an open source LSM on object storage would be useful for stateful stream processing, serverless functions, durable execution, and more. We had a lot of inspiration. Companies like WarpStream﹩ and Turbopuffer set zero disk architecture on the map. And open source projects like Tonbo and Monotone are blossoming. But we couldn’t find an LSM that was purpose-built for object storage (see our FAQ for discussion on RocksDB-cloud, RocksDB on EBS, and RocksDB on EFS). So we built it.
In the past week, we’ve been overwhelmed with the positive feedback, questions, contributions, and new users showing up on Github and Discord. We’ve hit 327 stars as of this writing (please star us on our Github repository!).
Several people have asked me if I (or anyone else) has commercial plans for SlateDB. I can only speak for myself: I am explicitly not going to do anything with SlateDB other than help make it a successful open source project. I am not going to start a company around it or try and monetize it in any way. To that end, we’ve chosen the permissive Apache 2.0 license, and I’m hoping we can get it into CNCF; we’ve submitted a sandbox application.
We’ve also got a lot of work to do on the project itself. We’re working hard to add on-disk and in-memory caches to speed up reads. There’s a lot more to do on compaction. And we haven’t yet implemented snapshots, garbage collection, transactions, and range queries—all must-have’s. We’ve also got some clever ideas up our sleeves, including writable snapshot clones, remote compaction, and a lot more. If this sounds like fun, we’d love to have your contributions. Our Github issues have “help wanted” and “good first issue” labels to get you started.
If you want to learn more, check out slatedb.io (especially the architecture page and design documents). You can also register for P99 CONF (it’s free and virtual). Rohan and I signed up to do a talk there. We’ll be discussing SlateDB’s history, architecture, performance, learnings, and more.