I'm Still Here! Let's Catch Up.
The latest on Designing Data-Intensive Applications, SlateDB, AI, Materialized View Capital, and forthcoming newsletter posts.
My apologies for the lack of posts recently. I’ve been hard at work on a few things over the summer, which has left little time for this newsletter. I’m getting my bearings now. More posts are back on the docket. My first post—this one—will be a quick catch-up.
Martin and I have been working on Designing Data-Intensive Applications’s batch and streaming chapters for its second edition. The batch chapter has been published to Safari Online as an early release; it required a full rewrite. The streaming chapter is still a work-in-progress. Kafka: The End of the Beginning, my previous post on this newsletter, reflected on how stagnant the streaming ecosystem has been over the past 10-15 years. The updates to the streaming chapter reflect this; they’ll be more minimal. I do plan to add a section on incremental view maintenance (IVM), which I’ll base off of my IVM newsletter post.
Everything You Need to Know About Incremental View Maintenance
Incremental view maintenance has been a hot topic lately. Materialize has been around for a while, but newcomers like PostgreSQL’s (semi-working) pg_ivm extension, Feldera, Epsio, Bytewax, and many others are starting to make noise. In the data warehousing space,
Meanwhile, SlateDB work continues apace. Li Yazhou has added serializable snapshot isolation (SSI) support. He’s in the process of adding transactions. The RFC is here if you’d like to learn more. We also have both Python and Go bindings now. I have been focusing on refactoring and stability; I added a basic deterministic simulation tester recently. Sujeet Sawala is working on an RFC to persist compaction progress. We’re starting to see some real adoption. More projects are launching in the near future, too.
The biggest news with SlateDB, though, is Pierre Barre’s ZeroFS project. ZeroFS provides network filesystem (NFS), network block device (NBD), and Plan 9 Filesystem Protocol (9P) implementations on top of SlateDB. The filesystem is also POSIX compliant—no small feat. On the performance front, check out Pierre’s AWS EFS, AWS Mountpoint-S3, JuiceFS, and Azure Files benchmarks.
ZeroFS is a young project, but I’m very excited about it. SlateDB’s branching and forking features mean ZeroFS will be able to provide zero-copy filesystem forking—an important feature for AI and many other use cases.
Speaking of AI, I’m still getting my bearings with it. I’ve been reluctant to post about the topic because I don’t feel I’m an expert in the subject. (Then again, it’s so new that very few are.) I use coding agents constantly, though. As a user, I’ve begun to form some opinions around model context protocol (MCP), agent adoption in the enterprise, and its impact on developer tooling and infrastructure. I plan to write more on AI in the near future.
I’ve continued to invest in startups throughout the summer. Materialized View Capital is now 75% deployed and will be fully deployed ~18 months from its inception. One usually targets a 3 year deployment. An 18 month deployment for a smaller fund like MVC is not unheard of. I’m quite pleased with our portfolio, which includes Bauplan, Dosu, Fiveonefour, Gauge, Loophole Labs, ParadeDB, Reboot, Signadot, Spiral, Tigris, Tensorlake, and many more.
Starting a fund has been rewarding. I plan to take a few months off after the fund is deployed. I’d like to evaluate what’s next for my startup investing adventure.
And that pretty much sums it up! I’m sure I’ve missed a few things. Let me know if there’s something specific that’s worth noting. In the meantime, expect an interview next week with Xorq’s﹩ CEO, Hussain Sultan.
Book
Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.
Disclaimer
I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.