Developers write MCP servers that provide resources, prompts, and tools to the LLM. The MCP site discusses these concepts in detail, but the Core MCP Concepts section provides a summary:

1. Resources: File-like data that can be read by clients (like API responses or file contents)

2. Tools: Functions that can be called by the LLM (with user approval)

3. Prompts: Pre-written templates that help users accomplish specific tasks

These categories are arbitrary and confusing. At first blush, it seems resources are read-only and tools are write-only. But the MCP server documentation uses searchFlights as their tool example—a read-only operation. Even more baffling, they later show flight searching as a resource. Here’s their tool definition:

{
  name: "searchFlights",
  description: "Search for available flights",
  inputSchema: {
    type: "object",
    properties: {
      origin: { type: "string", description: "Departure city" },
      destination: { type: "string", description: "Arrival city" },
      date: { type: "string", format: "date", description: "Travel date" }
    },
    required: ["origin", "destination", "date"]
  }
}

And here’s their resource definition:

{
  "uriTemplate": "travel://flights/{origin}/{destination}",
  "name": "flight-search",
  "title": "Flight Search",
  "description": "Search available flights between cities",
  "mimeType": "application/json"
}

Prompts are simply static JSON definitions that describe potential user activities.

The whole protocol feels off to me. Prompts are just static documentation, resources are static URL definitions, and tools look like remote procedure call (RPC) definitions. I asked ChatGPT 5 Thinking to convert the searchFlights tool definition to an OpenAPI definition (an actual RPC definition). Unsurprisingly, it worked just fine:

openapi: 3.0.0
info:
  title: Flights API
  version: "1.0.0"
paths:
  /searchFlights:
    get:
      operationId: searchFlights
      summary: Search for available flights
      description: Search for available flights
      parameters:
        - in: query
          name: origin
          required: true
          description: Departure city
          schema:
            type: string
        - in: query
          name: destination
          required: true
          description: Arrival city
          schema:
            type: string
        - in: query
          name: date
          required: true
          description: Travel date
          schema:
            type: string
            format: date
      responses:
        "200":
          description: Search results
          content:
            application/json:
              schema:
                type: array
                items:
                  type: object

This begs the question: why do we need MCP for tool definitions? We already have OpenAPI, gRPC, and CLIs. ChatGPT understands OpenAPI definitions. Millions of web services already provide OpenAPI definitions, too. And ChatGPT is actually better at CLIs than most humans—just watch Codex fly through sed, awk, and grep commands. A friend recently informed me that they successfully taught ChatGPT to use tmux. MCP vs CLI: Benchmarking Tools for Coding Agents reflects this sentiment:

MCP vs CLI truly is a wash: Both terminalcp MCP and CLI versions achieved 100% success rates. The MCP version was 23% faster (51m vs 66m) and 2.5% cheaper ($19.45 vs $19.95).

I’ve seen several arguments made to justify MCP’s existence:

1. LLMs have a limited context window. OpenAPI documentation takes up too much context space.

2. Many services are not well documented; they don’t come with an API spec.

3. LLMs need a way to discover what tools are available to it.

I’m skeptical. Perhaps MCP does allow us to squeeze a few more tools into the context window. Perhaps it does let models run a little faster. OpenAPI documents are indeed somewhat verbose. But for how long will this matter?

Last year we were talking about 1 million token models. Now, we have 2 million token models in OpenRouter. Smaller models, fine tuning, and other research also continues apace. Viewed from this lens, MCP seems like a temporary kludge.

The final argument that many services have poor documentation is true for internal enterprise services. Jake Mannix makes this point in his recent thread. I hear that MCP is most widely adopted for this segment.

Customer-facing SaaS APIs are a different story. Most are remarkably well documented. Some are complex, but these are often inherent complexities (I can attest to this after working with WePay’s API).

Moreover, the argument seems to be that the same people that wrote the bad OpenAPI specs are going to write good MCP specs. I don’t buy it. (And again, even if they could do this, why not write the endpoint as an OpenAPI endpoint or a CLI tool?)

As for discovery, I accept that LLMs need to know where tools are and how to use them. But this is static content. We already have AGENTS.md, .github/instructions, openapi.json, and so on.

Developers are waking up to this. Bruin﹩ is using MCP just to expose documentation; their tool calls are done through normal CLI commands. Donobu﹩ simply provides an OpenAPI spec. Both solutions work. Meanwhile, MCP’s Google trend looks bleak.

I don’t blame the MCP authors for this mess. Things happen fast in the AI ecosystem. MCP is a victim of its own success. MCP: An (Accidentally) Universal Plugin System does a good job explaining the situation.

We need to take a step back and think about what we’re trying to accomplish. There’s no law that LLMs need a new protocol to interact with software. In most cases, we have what we need. Where we don’t, we should write CLIs, web services, and documentation using existing standards.

My takeaway? Maybe instead of arguing about MCP vs CLI, we should start building better tools. The protocol is just plumbing. What matters is whether your tool helps or hinders the agent's ability to complete tasks.

—Mario Zechner, MCP vs CLI: Benchmarking Tools for Coding Agents

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

While working on the DataFrame section, I was surprised to find how bifurcated the ecosystem remains. On one side, you have single-node libraries like R and Pandas. On the other side, you have bolt-on Pandas-compatible APIs for distributed execution engines and cloud data warehouses such as Apache Spark’s pyspark and Snowflake’s Snowpark. It seems to me that a more unified approach is needed.

This post is an interview with Hussain Sultan, the co-founder and CEO of Xorq﹩. We discuss the fragmented DataFrame ecosystem and how Xorq bridges the gap with its multi-engine DataFrame library built on Ibis, Apache DataFusion, and Apache Arrow. Hussain has a diverse background that spans electrical engineering, digital signal processing (DSP), data science, machine learning, and field engineering. I’m so excited about what Hussain and Xorq are doing that I’ve joined their most recent funding round.

C.R: Thanks for taking the time to talk. I'm interested in your career path. I noticed you started as a hardware engineer before falling into the data space. How did you go from a background in digital signal processing to founding Xorq?

H.S.: Appreciate you asking—this has definitely not been a straight line. I’ve always been a builder. I had early access to a computer, picked up a Visual Basic for Applications (VBA) book before I knew what code really was, and hacked on WordPress sites in college. But knowing I could always fall back on software gave me the confidence to dive into other domains—hardware, embedded systems, DSP—where the problems felt more physical, more constrained.

Electrical engineering gave me the tools to think in systems. I worked on turbine blade monitoring using LabVIEW and Field-programmable gate arrays (FPGAs)—very visual, very declarative. It was kind of like building a directed acyclic graph (DAG), just with wires. That led me into DSP, and eventually electroencephalogram (EEG) denoising in grad school. EEGs were fascinating because they were a deeply human signal—full of noise, uncertainty, context. But in MATLAB everything felt so perfect, so sandboxed. I wanted data that was messier, more grounded in the real world, perhaps even macro behavior.

That’s what pulled me toward other types of data. I got fascinated by the idea that how people transact and spend money reveals something real about human behavior—at scale. Purchase behavior is just another noisy signal, but at a macro level. So I went deep into consumer models, credit risk, customer behavior etc.

That’s when I ran into real-world noise: SAS jobs, VBA macros, SQL you couldn’t trace, pipelines nobody owned. The business logic—net present value (NPV) models, Monte Carlo simulations, decision rules—was always trapped in brittle systems. And I knew there had to be a better way.

I turned to open source Python out of necessity: Pandas, Dask, and Arrow. When I found the PyData community, it felt like WordPress again: smart people solving hard problems together, in the open.

Over the years, I took that mindset into big tech, avionics systems, and consulting for banks trying to modernize. We saved millions just by building reusable, inspectable pipelines. But I kept running into the same issue: compute wasn’t composable, but was rapidly being commoditized by things like DuckDB and DataFusion. You couldn’t trace them, reuse them, or move them across engines…easily. It wasn’t an infra problem—it was an abstraction problem. Data has structure. Compute didn’t.

That’s what led to Xorq. With Arrow, Ibis, and DataFusion, the primitives were finally ready. Xorq is a compute catalog: a place to put the logic itself. Expressions become portable, observable, and reusable. Multi-engine. Storage-agnostic. Versioned by default.

So no, this wasn’t some grand plan. I just kept chasing messy signals, building tools to make sense of them, and leaning on the communities that made that work possible.

C.R.: What does the compute logic in Xorq’s “compute catalog” look like? Are we talking about operators that can be run in a physical query plan executor—things like a “sort-merge join” operator? Or are these more bespoke pieces of logic? Some examples might help me understand more.

H.S.: Short answer: we don't deal with physical operators—you won’t see “sort-merge join” in our plan. What we store is a relational + functional + semantic plan that’s declarative, portable, and able to run across different engines.

You can think of this compute logic as a “high-level plan” that stitches together many logical plans for engines running in different places—DuckDB, DataFusion, Pandas, and so on. The bespoke logic shows up as Python UDFs and UDAFs, and we leaned on DataFusion because it treats Arrow-based functions as first-class. At this level we also track tags, caching, and pipeline operators that move data between engines. Once the plan is executed into a backend, that’s when the physical choices—hash vs. sort-merge join, vectorization, parallelism—get made. We generally leave that up to the engine. Things like the into_backend operator to go from one engine to another and flight_udxf nodes for arbitrary computations appear often instead.

Since multi-engine workloads heavily lean on deterministic caching, we build a stable hash of the plan that can be used with a cache operator. If two subgraphs are the same, they map to the same cache key and return identical results, regardless of which engine runs them.

A concrete example: take a simple scikit-learn pipeline. We lift it into Xorq’s intermediate representation (IR), split it into train/test, add cache boundaries, and tag each step with metadata (step name, parameters, features, target). Under the hood, fit lowers to a UDAF that produces state (weights, scalers, encoders), and apply lowers to a scalar UDF that consumes that state row by row (normalize, lookup, predict). Xorq uses Ibis heavily for its expression system and its the de facto user-facing IR.

So in this plan, the “predicted” column is literally a scalar UDF parameterized by the fitted model executed in the DataFusion engine. Reads flow through DuckDB, ML bits get routed to DataFusion and UDFs, and from the developer’s side it’s just Python (`expr.py` with `.fit(...)`, `.predict(...)`, tagging, and caching). Xorq then emits a build directory so the whole thing can be reproduced.

Because the entire workflow is captured in this relational algebra, we can lower ordinary Python pipelines into SQL engines, or extend them with arbitrary compute via `flight_udxf`. That’s what makes it a “high-level plan”—broader than a database logical plan, and able to tie together engines outside the SQL world (vector databases, graph systems like KuzuDB, etc.) and of course, coupled with a user-facing expression system in Ibis.

A quick recording is available if you’d like to see it in action; here’s a gist with the complete build outputs, which might help readers understand it more concretely.

C.R.: Wow, OK, this is pretty slick. You mentioned that you re-use identical results if two subgraphs are the same. Does this work across queries, workflows, or even users as well? Also, how do you handle cache invalidation, where a subsequent query might want fresher data than the cache contains?

H.S.: It does—across queries, across workflows, and, within a workspace and its permissions, across users. The way we make that safe and predictable is by tackling the old, thorny problem of “naming things.” If you can name a computation deterministically, you can look it up later and recover its full lineage to the source. This isn’t a new idea; you see versions of it in content‑addressed storage and in Nix.

My co‑founder, Dan, and I have lived through this pain for years. Every time we’d run a new experiment, we’d save the training dataset and the splits, and then were left juggling with naming datasets and—even harder—when to invalidate the cache. Nix gave us a clean mental model: it hashes, builds deterministically and leverages binary caches. If two people produce the same build, Nix can tell you whether a valid artifact already exists or whether it needs to be rebuilt.

Xorq applies that principle to data compute. We derive a cache key for any expression by hashing the contents of the expression itself along with the relevant source descriptors. Depending on the caching strategy, we either incorporate a change signal—like a table’s last‑modified time—or we pin to a snapshot. If the upstream table changes and you’re in a “follow the source” mode, the stamp changes, the hash flips, and the cache is naturally bypassed so the computation is re‑materialized. If you’re in snapshot mode, the cache key stays stable until you explicitly ask for something fresher.

To make reuse work across users without surprises, the hash has to be stable. That means we strip away environment‑specific noise and avoid anything brittle like absolute local file paths. The result is a content‑addressed artifact that’s consistent regardless of who computes it, so long as they’re looking at the same logical expression and the same view of the source.

Because "cache" is just another node in the relational graph, you can attach it wherever it makes sense. You might cache a heavy transformation to accelerate iteration but choose not to cache a subsequent training step. The choice is surgical and composable.

An added benefit of stably hashing every node is discovery. We can compare expression nodes across different graphs—even across teams—and detect shared subgraphs. That lets us surface high‑value candidates for caching and reuse across an organization, not just within a single project.

On freshness guarantees, especially for near‑real‑time features like “activity_last_5m,” we combine strategies. We can materialize the feature in snapshot mode first—so we’re not chasing source modification times constantly—and then apply a time‑to‑live at retrieval. When the TTL expires, we recompute. That pattern will feel familiar to anyone who’s worked with feature stores for ML workloads.

So the short version is: reuse works across queries, workflows, and (with proper access controls) users, because everything is deterministically named. Invalidation is built into that identity—either via source‑change stamps for automatic freshness or via TTL‑driven policies when you want precise control.

C.R.: Hah, you walked right into my next question! I am curious if you’ve considered how streaming and realtime data fit into Xorq. Many other processing frameworks (Spark, Apache Flink, Feldera) are working towards a unified API for both modalities. It sounds like Xorq is pretty batch-focused. Are there any plans to add streaming or “micro-batch” support?

H.S.: You’re right—most of our current focus is on batch engines. I usually describe Xorq’s architecture as out-of-core i.e. we stream Arrow RecordBatch fragments between operators instead of materializing everything up front. That lets us handle datasets larger than memory while still feeding engines directly. The key distinction here is bounded vs. unbounded. Out-of-core workloads are bounded—they terminate— whereas micro-batch or real-time streaming engines operate over unbounded streams. DataFusion, our embedded engine, has also been a solid building block for unified models and even some emerging streaming use cases.

We’ve been actively experimenting with streaming engines like Flink because our expression system, through Ibis, can already describe watermark- and time-aware operations. The real difference comes at the source: if you attach a bounded table, you’re still in a mini-batch world; if you attach an unbounded source like Kafka or CDC, the exact same expression can be evaluated incrementally by the engine. Official Flink support is on our mid-term roadmap, but even in these early experiments it’s clear how naturally our model extends to streaming SQL. This is slated for mid-term priority on our roadmap and would be an indicator that we feel confident with batch use-cases and ready to tackle streaming/real-time engines. I am also super psyched about engines like Feldera, since it turns the SQL that we know for batch cases and makes it incremental without changes. A unified API for both modalities is most welcomed!

What’s exciting is that the same caching and reuse story still holds. We can incorporate Flink’s operator state into our hashing model, which makes streaming computations just as stable and discoverable as batch ones.

This ties directly into how we think about freshness guarantees in research vs. production. In a research notebook, you might snapshot a feature like “activity_last_5m,” attach a TTL, and let the cache refresh when it expires. It’s reproducible and easy to debug. But in production, you want that feature continuously updated without extra orchestration.

And that’s really the arc: in research, you rely on explicit snapshots and cache nodes; in production, you lean on streaming engines and policies that manage freshness automatically.

C.R.: Similarly, how do you see Xorq fitting in with AI training and inference workloads?

H.S.: Xorq works great with AI/ML training and inference, but for tabular data. We model opaque compute with a flight_udxf node, executed as an Apache Arrow Flight do_exchange endpoint. This allows us to remain in a relational authoring surface—DataFrames and SQL—while offloading to tensor runtimes as opaque operations, all without leaving the plan.

Apache Arrow provides the typing system and transport layer. Crossing the boundary is efficient: Arrow arrays convert to DLPack and back with minimal overhead. Because training and inference typically run in mini-batches, our out-of-core streaming backbone remains unchanged, and relational engines compose seamlessly with arbitrary compute.

For scikit-learn–style machine learning, our goal was to unify training and inference into a single relational graph instead of two ad-hoc scripts. We achieve this with UDF/UDAF machinery in DataFusion: “fit” lowers to an aggregate UDF that emits state (weights, scalers, vocabularies) as bytes or artifact references; “apply” lowers to a UDF that consumes that state row-wise for predictions. Because DataFusion operates directly on Arrow RecordBatches, the pattern is fast in practice. And we can make these UDFs portable with Arrow Flight.

However, one key lesson is that DataFrame APIs excel for feature engineering and early-stage preprocessing, but late-stage preprocessing nearly always shifts to tensor operations; think nn.Module/TensorDict, dense normalization, tokenization etc.. The next step is to make this “last mile” feel like PyTorch, instead of DataFrame-ey, while keeping the entire workflow declarative, optimizable, and reproducible. This will also unlock new workloads with unstructured data, videos, images etc.

Long-term, we are inspired by the Declarative Sub‑Operators paper: expose a lower‑level, declarative layer beneath relational algebra so we can express things like tensor-prep. I am excited for the declarative future, where Tensor and DataFrame APIs can be used interchangeably.

C.R.: You’ve described so many directions that Xorq can go in. There’s clearly many years of work ahead; I look forward to seeing it evolve. In the meantime, it seems like a pretty solid batch multi-engine system. I really appreciate you taking the time to talk with me. I’ll give you the final word. Anything you’d like to add? Where should folks go if they’re interested in learning more?

H.S.: It’s a big vision, made possible only by standing on the shoulders of Apache Arrow, Ibis, and DataFusion. Across the ecosystem, teams are building semantic layers, generating request-time dynamic SQL, and with Xorq, making ML accessible to data and analytics engineers. In the multi-engine batch processing world, our early design partners report speeding up dynamic dbt-style workloads by 5-15x versus Jinja-based SQL model generation, cutting infrastructure costs by >10x compared with Snowpark, while providing end-to-end column-level lineages and greater transparency. This fits since our target audience tends to be multi-stakeholder organizations that span tech, business, and data. This means, we need to thread value across various stakeholders i.e. cost of ownership, time to value and policy-based governance.

Our catalog-first approach makes reuse, optimization, and governance first-class, while staying minimally invasive to how teams already run on SQL engines. We are super keen to get feedback from the community on the "compute catalog" concept. If this resonates, kick the tires and tell us what’s missing:

• GitHub: https://github.com/xorq-labs/xorq

• Docs: https://docs.xorq.dev

• Website: https://xorq.dev

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Martin and I have been working on Designing Data-Intensive Applications’s batch and streaming chapters for its second edition. The batch chapter has been published to Safari Online as an early release; it required a full rewrite. The streaming chapter is still a work-in-progress. Kafka: The End of the Beginning, my previous post on this newsletter, reflected on how stagnant the streaming ecosystem has been over the past 10-15 years. The updates to the streaming chapter reflect this; they’ll be more minimal. I do plan to add a section on incremental view maintenance (IVM), which I’ll base off of my IVM newsletter post.

Meanwhile, SlateDB work continues apace. Li Yazhou has added serializable snapshot isolation (SSI) support. He’s in the process of adding transactions. The RFC is here if you’d like to learn more. We also have both Python and Go bindings now. I have been focusing on refactoring and stability; I added a basic deterministic simulation tester recently. Sujeet Sawala is working on an RFC to persist compaction progress. We’re starting to see some real adoption. More projects are launching in the near future, too.

The biggest news with SlateDB, though, is Pierre Barre’s ZeroFS project. ZeroFS provides network filesystem (NFS), network block device (NBD), and Plan 9 Filesystem Protocol (9P) implementations on top of SlateDB. The filesystem is also POSIX compliant—no small feat. On the performance front, check out Pierre’s AWS EFS, AWS Mountpoint-S3, JuiceFS, and Azure Files benchmarks.

ZeroFS is a young project, but I’m very excited about it. SlateDB’s branching and forking features mean ZeroFS will be able to provide zero-copy filesystem forking—an important feature for AI and many other use cases.

Speaking of AI, I’m still getting my bearings with it. I’ve been reluctant to post about the topic because I don’t feel I’m an expert in the subject. (Then again, it’s so new that very few are.) I use coding agents constantly, though. As a user, I’ve begun to form some opinions around model context protocol (MCP), agent adoption in the enterprise, and its impact on developer tooling and infrastructure. I plan to write more on AI in the near future.

I’ve continued to invest in startups throughout the summer. Materialized View Capital is now 75% deployed and will be fully deployed ~18 months from its inception. One usually targets a 3 year deployment. An 18 month deployment for a smaller fund like MVC is not unheard of. I’m quite pleased with our portfolio, which includes Bauplan, Dosu, Fiveonefour, Gauge, Loophole Labs, ParadeDB, Reboot, Signadot, Spiral, Tigris, Tensorlake, and many more.

Starting a fund has been rewarding. I plan to take a few months off after the fund is deployed. I’d like to evaluate what’s next for my startup investing adventure.

And that pretty much sums it up! I’m sure I’ve missed a few things. Let me know if there’s something specific that’s worth noting. In the meantime, expect an interview next week with Xorq’s﹩ CEO, Hussain Sultan.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

The pace of change in the batch processing ecosystem stands in stark contrast to its streaming counterpart. Apache Kafka remains the near-universal solution for streaming data, while Apache Flink remains the dominant stream processing system. Both projects were started well over 10 years ago. Flink 1.0 was released in 2016, while Kafka 0.9 (widely adopted in production) was released in 2015. Even Debezium started around 10 years ago; we adopted 0.2 at WePay in 2016.

Undeniably, there has been some innovation. The separation of control, data, and compute planes has led to object store-backed streaming systems like WarpStream﹩, Bufstream, and Redpanda. The object store transition also led to streaming lakehouse integrations like Tableflow. And Frank McSherry et al.’s work on timely and differential dataflow is nothing short of is brilliant.

Yet the architecture that Confluent was founded on—an enterprise service bus built on Apache Kafka—remains largely the same as it was 10 years ago. Moreover, the actual technologies we’re using are the same. Much of the story of the past 10 years has been about adoption rather than innovation. We’re still using Kafka (or at least its protocol), we’re still using Flink, and we’re still using Debezium.

I’ve been thinking about this stagnation for a while. I don’t believe we’ve solved everything. It’s still very difficult to write and deploy stream processing jobs, for example; it honestly feels like it’s gotten worse, not better. I’d like to believe that these challenges are not fundamental—that there are better ways.

This malaise seems to be growing. Yaroslav Tkachenko posted a summary of last week’s Current 2025 conference in London. The post is worth a read, but his comment at the top of the post mirrored my own view:

Finally, I feel like the data streaming industry is still in a tough spot. The growth is slow, and the sales cycles are long. One person I spoke with said that “80% of the companies in the Expo hall will be dead in two years”. I don’t want to believe them, but it might be true.

I was speaking with someone just yesterday that said something similar: it’s the same people, same companies, and same technology. There are no new ideas and no new users. This person said they would not attend future Current conferences; it wasn’t worth it; they didn’t learn anything new.

Yaroslav’s post also says that Redpanda was banned from the conference. I don’t know the details, nor do I really care. But if true, it’s hard not to read this as an indication of scarcity and fear, not one of abundance.

It’s worth reflecting on Apache Hadoop. Hadoop began at Yahoo! in 2006. It took about 10 years for companies such as Hortonworks and Cloudera to drive adoption through the enterprise. By 2016, it was widespread, but dated. This is when dataflow engines and cloud data warehouses began to replace Hadoop. Confluent was founded in 2014. Eleven years later, it feels like we might be at a similar turning point with Kafka; it’s widely adopted, but dated.

Yet, I don’t see the challengers that Hadoop saw in 2016. I suspect this is because Kafka’s protocol has become the de facto standard. HDFS, including its protocol, was largely replaced by object stores like S3. Kafka, on the other hand, managed to adopt object stores while keeping its protocol and client rebalancing architecture intact.

A de facto standard protocol is great for enterprises; they don’t need to migrate and vendors are all forced to compete on a commoditized storage layer. But what happens when the protocol needs to change? Entrenched vendors aren’t going to ask their customers to migrate.

Concepts like partitions and leadership are baked into Kafka’s protocol (especially its clients). Broker awareness of time—fundamental to stream processing—is largely absent. Kafka is not designed for millions (or billions) of streams. Gunnar Morling, of Debezium fame, has an excellent post where he asks, “What if We Could Rebuild Kafka From Scratch?”

One bright spot is S2. Shikhar and his team seem to be rethinking what a modern, cloud-native ecosystem could look like. Notably, of all the modern streaming solutions—Redpanda, AutoMQ, WarpStream, Confluent Freight, and Bufstream—S2 stands alone as the only system that doesn’t (yet) use Kafka’s protocol; they are unconstrained.

While writing my incremental view maintenance (IVM) post, I wondered what timestamps in Kafka might look like. A month later, S2 announced their take. In, “What if streams had the primacy of objects?,” they discuss supporting per-user streams. This is the fresh thinking we need. Unfortunately, adoption is a real challenge. Kafka’s protocol network effects are strong.

Meanwhile, the stream processing side of the house seems somehow both worse and better. I am no fan of Flink, but it has won the stream processing race (outside of Google Cloud, which has Dataflow). Unspoken amongst vendors, the true winner is really just regular old Kafka consumers and producers.

I do see some glimmers of hope. Compact libraries that incorporate differential dataflow, such as D2﹩, are appearing. And Feldera seems like a genuinely better computation layer than Flink. Such systems might become the next Spark or Snowflake of the streaming ecosystem (and batch ecosystem, in Feldera’s case).

The next decade of stream processing could look like the last decade of batch processing: one of growth and innovation as we transition out of legacy on-prem architectures, protocols, and systems into a truly cloud-native streaming era. I hope so; it would be great for everyone.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

This post will provide a high-level overview of what incremental view maintenance is, why it’s useful, and how it can be implemented. We’ll then look briefly at three (relatively) recent research papers: timely dataflow, differential dataflow, and DBSP; these influential papers have influenced many (but not all) of the products listed above.

What is IVM?

(If you already know what an IVM is, you can skip this section.)

To understand incremental view maintenance, we must first understand views. In this context, a view is a projection of data in a specific way. Pivot tables are a view: you have a spreadsheet with data, and you create a pivot table to filter and aggregate the data from the source spreadsheet. Similarly, a table in a database can be thought of as a spreadsheet, and a database view can be thought of as a pivot table. Views in databases are defined using SQL, rather than the UIs you’re used to in Excel and Google Sheets.

Continuing with our pivot table analogy, when a user updates data in their spreadsheet, the pivot table needs to be updated to reflect changes in its filters and aggregates. The same thing applies for database tables and views.

There are many approaches to view updates. The simplest way is to re-execute the query every time the view is accessed. In Excel, this would mean every time the pivot table is viewed, it must re-run its query on the source data; the same applies for database views.

Refreshing a view on every query can be slow and costly for large datasets, though. A spreadsheet with millions of rows might halt a user’s progress for several seconds. A database with billions (or trillions) of rows might block things for much longer. An alternative is to use materialized views. A materialized view is a cached result of the view. Rather than re-computing views on every read, they are refreshed periodically. Once the computation is complete, the new version of the view is used.

This approach—periodically reprocessing the entire dataset—trades data freshness for lower read latency. Queries on views will be faster since their data has already been computed, but the data will reflect an older version of the data from when it was last refreshed. Reprocessing the entire dataset can also be wasteful. A change to a single row in the source data requires reprocessing all the source data to generate the new materialized view. SELECT COUNT(*) has to recount every row to discover that only one new row was inserted.

Incremental view maintenance addresses these problems. Rather than reprocessing the entire dataset every time a change occurs, IVM reprocesses only the view data affected by the change (the delta). This approach dramatically decreases the cost of maintaining the materialized view. The DBSP paper illustrates this well:

Informally, QΔ built by our algorithm, is faster than Q by a factor of O(|DB|/|ΔDB|). In practice this may be an improvement of several orders of magnitude. For our example above |DB|≅10⁹ and |ΔDB|≅10², this can make QΔ 10 million times faster!

Cheap updates allow us to refresh materialized views more frequently. In doing so, we can keep materialized views more up to date with their source data, thereby reducing data latency. Best of all, read latency can remain largely unaffected since readers continue to query pre-computed data.

How do IVM engines work?

Now that we understand what IVM is, let’s discuss how they have historically worked. There are two things to consider:

• When do incremental updates occur?

• How do we know what data needs to be updated?

The answer to the first question is fairly straightforward. Incremental materialized views are computations like any other; they can run on a fixed schedule, on an ad-hoc basis, or with a trigger. In practice, the last option—a trigger—is usually used to watch for newly changed data. In a data warehouse, Airflow’s triggers and sensors detect changes. Similarly, developers can write triggers in OLTP databases to update views whenever source tables are mutated. OLTP databases that have built-in IVM support also usually update views as source data changes. Stream processors follow the same pattern: state is updated as new events arrive.

The second question—knowing what data needs to be updated—is trickier. The most intuitive approach is to write code or ad-hoc queries to react to triggers and update data. For example, we might have an orders table and a customer_order_totals view. We can then write a trigger to update customer_order_totals whenever data is inserted into order.

CREATE OR REPLACE FUNCTION orders_insert_trigger_fn()
RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO customer_order_totals (customer_id, total_amount)
    VALUES (NEW.customer_id, NEW.amount)
    ON CONFLICT (customer_id)
    DO UPDATE SET total_amount = customer_order_totals.total_amount + EXCLUDED.total_amount;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

Triggers and ad-hoc code work for simple cases. But complex incremental queries that use joins, window functions, or recursive algorithms are very difficult to write by hand. A more systematic approach is needed.

The systematic approach many databases have relied on uses bag algebra (relational algebra). Users start by defining an incrementally maintained materialized view as a SQL query. The database then translates the SQL query into relational algebra operations such as select, project, union, intersect, join, difference, and so on. Once in relation algebra form, there is a bunch of math that shows inserts and deletes (called deltas) from source tables can be fed into the relational expression to compute the difference that must be applied to the materialized view. PostgreSQL’s incremental view maintenance wiki is a good starting point if you want to learn more.

Problem solved, right? Developers can continue to define materialized views in standard SQL, and databases can use fancy math to operate on just the deltas rather than recomputing everything. Unfortunately, not. It turns out, the bag algebra approach does not work well for complex and computationally expensive queries, especially with recursive or nested structures. While we’ve increased our expressiveness and ease of use, our IVM is no longer as cheap as before.

A Modern Approach

This leads us to the papers three papers I mentioned at the beginning of this post: Naiad: a timely dataflow system, Differential dataflow, DBSP: Incremental Computation on Streams and Its Applications to Databases. These papers collectively present a different way to build an IVM engine that is:

• Fast enough to update materialized views on every data change

• Expressive enough that developers can use query languages such as SQL or Datalog to define views

• Flexible enough to be used with stream processors as well as databases

Frank McSherry, one of the foremost contributors to this space, describes these new innovations as a set of tools with different levels of flexibility and opinions. The lowest level, and most flexible, is timely dataflow. Differential dataflow is built on top of timely dataflow, and enforces some requirements on the user in order to calculate incremental updates. DBSP enforces still more requirements on the user in exchange for a simpler implementation. Let’s look at how these three systems build off one another.

Timely Dataflow

Timely dataflow provides a model of time that makes it easier to do IVM computation without sacrificing expressiveness. Time is modeled as a vector that includes both an epoch as well as a loop counter.

As the name implies, loop counters keep track of what iteration a specific event is on. I won’t go into detail on why this is important (see the paper), but it means timely dataflow can support deeply nested loops very easily. Graph processing algorithms, in particular, benefit from this. Indeed, the paper spends a large amount of time talking about this use case.

The next innovation in timely dataflow is how it handles time. Traditionally, time updates (“punctuations”) are sent as special events or metadata in the data stream; this forces time to cascade sequentially through a dataflow. A task receives a new time update, and forwards that outputs downstream. Such an approach can cause bottlenecks.

Timely dataflow uses out-of-band watermark broadcasts. This sounds fancy, but it just means time is tracked outside of data streams—the data and control planes are separate. Timely dataflow has every task tell every other task in the dataflow how far it’s processed outside of the data streams. Armed with this information, tasks can get a global view of time and determine when they can make forward progress. Tasks in the same dataflow can even be processing different points in time, or multiple points in time. The takeaway is that the computation can happen concurrently (and thus faster) than the bag algebra approach.1

Practically speaking, the API for timely dataflow is fairly simple. It has four methods: SendBy, NotifyAt, OnRecv, OnNotify. The send/recv methods are to send and receive events, and the notify methods are for advancing time. I mention this because I like to think of timely dataflow a lot like Hadoop’s Map/Reduce: a powerful, but low-level framework for data processing.

Differential Dataflow

Differential dataflow introduces “differential computation”:

The novelty of differential computation is twofold: first, the state of the computation varies according to a partially ordered set of versions rather than a totally ordered sequence of versions as is standard for incremental computation; and second, the set of updates required to reconstruct the state at any given version is retained in an indexed data-structure, whereas incremental systems typically consolidate each update in sequence into the “current” version of the state and then discard the update.

The paper is very technical (I don’t understand the math in it). The gist is that it keeps track of multiple versions of data states, arranged in a partial order, rather than just a linear sequence (see the paper for examples). Timely dataflow timestamps are used to track the order. The arrangement allows the system to selectively reuse prior computations, significantly reducing the amount of work when the data is updated.

Once differential computation is defined, the paper shows (in section 4.3) that standard SQL-like operators can be built above the computation engine. The result is that developers can express incremental views using high-level query languages like SQL or Datalog. The engine handles joins, aggregations, and even recursive computations efficiently. This automation significantly simplifies building and maintaining IVM systems, especially for complex scenarios such as graph analytics or deeply nested computations. Continuing with the Hadoop ecosystem metaphor, you might describe this as the Pig or Cascading layer.

DBSP

I had to take one electrical engineering class in college. I remember being blown away when I learned that any circuit could be expressed using nothing but NAND gates (not-and’s). This property is known as functional completeness, a property that it shares with NOR gates.

…any other logic function (AND, OR, etc.) can be implemented using only NAND gates. An entire processor can be created using NAND gates alone.

I raise this idea because DBSP is based on digital signal processing (DSP). The authors realized that incremental view maintenance looks somewhat similar to differentiation and integration in DSP. And much like the NAND gate phenomenon I learned about in college, the DBSP paper presents four operators (lift, delay, and two operators for recursive programs) that are the foundation for all relational operations in SQL. To get a flavor, here’s how the lift operator is depicted:

This is simply a map operator. The delay operator is also simple. It’s represented in the paper as z⁻¹, and it delays output by one step.

Using circuit design and relational algebra, the paper is able to show that arbitrary SQL queries can be translated into DBSP circuits. Once converted into a DBSP circuit, the paper then shows that the circuit can be converted to an incremental DBSP circuit. This is a really powerful idea. DBSP can convert any batch-based SQL to an incremental query.

DBSP does make some tradeoffs when compared to differential dataflow. It simplifies the programming model by constraining how time and state management occur. This simplification limits some of the concurrency gains we see in timely and differential dataflow. In exchange for this, DBSP offers a simpler, more accessible framework for typical database and stream processing workloads.

Putting it all Together

Ultimately, the progression from timely dataflow to differential dataflow and then to DBSP shows a clear trajectory: moving from highly flexible, low-level frameworks toward more structured, to easier-to-use incremental computation systems.

The ideas pioneered by these three papers underlie many of the recent incremental view maintenance implementations. Materialize leverages differential dataflow directly, while newer entrants like Feldera are built on DBSP. Even frontend tools are shifting toward differential-like incremental maintenance paradigms, demonstrating the broad applicability and utility of this research.

Yet, IVM engines still have a long way to go. IVM in stream processors is still a work in progress. Many databases have missing or incomplete IVM implementations. A PostgreSQL implementation would be a very big deal. Perhaps ParadeDB will build this next? Nikhil Benesch, former CTO of Materialize, apparently contemplated this early on:

Differential dataflow is just really complicated. A few years ago, Jamie Brandon asked, Why isn't differential dataflow more popular? His post is worth reading. The first Hackernews response has a ring of truth to it:

Indirectly answering the question - I've skimmed through the git README, the abstract and all the pictures in the academic paper that it references.

I have no idea what this thing does. Can someone explain in simple terms what it does?

But systems like those I list at the top of this post make it more accessible. Rather than complex APIs and semantics, developers can use SQL. DBSP could unlock more IVM solutions, too. Getting this technology into everyone’s hands will be a very big deal.

A special thanks to Frank McSherry, Lalith Suresh, and Gunnar Morling for feedback on early drafts.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

I’ve avoided talking about AI on my newsletter thus far. The space is moving so rapidly. It feels futile to try and keep up, and anything written becomes stale very quickly. Still, I have been using AI daily for the past few years, primarily for coding. First with ChatGPT, then with Windsurf. I’m going to break my embargo for this post to cover some recent discussions I’ve had about AI IDEs like Windsurf and Cursor.

For the non-developers out there, Windsurf is an AI-powered agentic integrated developer environment (IDE) akin to Cursor. In layman’s terms, these are application that developers use to write code along side an AI. Both Windsurf and Cursor are built on Visual Studio Code, and both are quite effective. I primarily use Windsurf’s AI features when writing unit tests and small scripts, be it Python, Bash, or Github Actions YAML. A few weeks back, I decided to check in and see if I should switch to Cursor.

There was no consensus. My impression from the comments was that Windsurf was slightly better than Cursor. I decided to stay with Windsurf since I was already using it. Fast forward a few weeks, and this post on Twitter caught my eye:

Tom’s post validates my impression, but it also raises an interesting question about moats. As Tom says, since both Windsurf and Cursor are forks of VS Code with nearly identical interfaces, the switching cost is nearly zero. When I switched from VS Code to Windsurf, it took all of 30 minutes. I haven’t looked back. Switching from Windsurf to Cursor, I’m confident, would be the same.

It’s easy to think these IDEs are destined to be commodities. I don’t think so. The fact that Windsurf is unseating Cursor right now is a signal that, even with commodity LLMs, Windsurf is offering a better product than Cursor. It does make the space highly competitive, however. As Steve’s post says, the IDEs have to be more than API calls to OpenAI and Anthropic. What might this look like?

Last week, Josh Wills and I were discussing AI IDEs over coffee (he’s hiring at DatologyAI, by the way!). Josh made an off-hand comment that LLMs really struggle with dataframe libraries that aren’t Pandas; they hallucinate and assume you’re using Pandas. This resonated with me. I had the exact same experience when trying to learn Zig in 2023. The language was very new, and similar enough to Go and Python that ChatGPT would intermingle the three languages together.

This dynamic is interesting: LLMs struggle with new infrastructure and tools. The same is true for proprietary software—both internal company code and closed source vendor software. There isn’t enough public data on the internet to train the LLMs on such software.

A poor LLM experience with new and proprietary software makes for an interesting virtuous cycle. Developers get a better experience working with legacy software that has thousands of stack overflow questions and github repositories. A better experience increases the adoption cost for new software (or decreases the cost of using legacy software). Developers will stick with the existing software, which will give the AI IDEs and LLM models yet more data to train on. This, in turn, will further improve the developer experience for the existing software.

Proprietary vendors and new open source projects need to figure out how to break this cycle. Offering software that is far better than incumbents could justify the increased switching cost. Vendors could also build their own plugins that help with prompt engineering and retrieval-augmented generation (RAG). Anthropic’s Model Context Protocol (MCP), llms.txt, and agents.json standard might also be adopted. Indeed, Nile ﹩ announced an MCP server for its product as I write this post.

AI IDEs like Windsurf and Cursor seem well positioned to help with the adoption problem. IDE companies could partner with vendors to integrate tightly into the IDE’s agents. The companies might also help with fine-tuned models, smaller LLMs purpose-built on curated data, custom integrations, VS Code plugins, or even preferred placement in the IDE—all for specific languages, tools, and vendors. A partnership between Windsurf and a company bringing a new software product to market would be symbiotic. Windsurf would build a better (more sticky) IDE while the company it partners with would gain a better developer experience. Windsurf might even be able to charge for such services.

AI IDE companies could extend this work to internal company software as well. This is where Sourcegraph (the company that makes Cody) got its start. Internal company codebases are not typically publicly available, so LLMs know less about them. Both Cursor and Windsurf are already selling into the enterprise; they should be able to offer a wide range of tools for this use case.

And then there is Github. In the Bluesky thread above, Qian Li pointed out that Github already has access to many proprietary (private) repositories. They also offer Copilot. They, too, seemed well positioned to compete as an AI IDE. I’m less confident in their ability to execute, however. Copilot has lagged behind its competitors despite strong competitive advantages.

So yes, AI IDEs are a very competitive space, but I believe they’ll develop moats around specialized-models, partnerships, proprietary training data, and more.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Earlier this month, I joined The Infra Pod to chat about all things infrastructure. Give it a listen:

Finally, I re-did my personal blog, cnr.sh, as well. I am planning to do some non-infrastructure writing this year. If you’d like to keep up to date, please do follow my newsletter or RSS feed there. I just published my first post, Comparing Apache, CNCF, and Commonhaus.

The blog launch led to a couple of fun Python libraries: markupdown and python-docstring-markdown. Markupdown is the static site generator I’ve always wanted. It acts more like a build system rather than a classical SSG. python-docstring-markdown was just scratching an itch I had: I wanted to generate a simple DOCUMENTATION.md file from Python docstrings in my Python libraries.

Recap is a project I started working on a few years ago. It started as a tiny data catalog for machines and morphed into a type system. It provides a single schema definition language (SDL) that can describe database schemas, event schemas such as Avro, and web service schemas such as JSON schema and Protobuf. The project has now been adopted by Gable.ai; they use it as the SDL for their data contracts.

One of Recap’s features was its client framework. I implemented readers for BigQuery, Confluent’s schema registry, and a slew of databases and filesystems. I began to realize that these systems were all data catalogs. Some called themselves registries, others were information_schema, and many called themselves catalogs. But they were all handling metadata about your data; they seemed 80% the same. Yet they were being used for different things.

Registries are typically used to store event schemas for streaming systems in Avro, Protocol Buffers, and JSON Schema. Service catalogs such as Backstage and Apicuriou serve a similar purpose for web services. information_schema-style metadata is used in both OLTP and OLAP databases. “Traditional” data catalogs such as Collibra, Atlan, Alation, Open Metadata, and Datahub began with data discovery use cases.

Each use case has a distinct yet overlapping set of requirements. Registries must be low latency and highly available since realtime infrastructure often queries them in production. They must also track schema compatibility to prevent breaking changes. Similarly, OLTP metadata needs to be very fast since it’s being used by query engines serving production traffic. OLAP metadata, on the other hand, need not be as low latency. Instead, tracking things like lineage, schema evolution, access controls, and sensitive information are important. Service catalogs are concerned with discovery, but also schema compatibility.

I’m not convinced that each of these use cases deserves its own distinct catalog. Each use case has separate requirements, but it seems to me that one system could service them all.

Indeed, I’m beginning to notice a lot of convergence in this space. Backstage, which started as a microservice catalog, now covers data pipelines, machine learning models, and more. Datahub, which began as a data discovery tool, is going to adopt Iceberg’s REST API. Confluent’s Tableflow now speaks Iceberg’s REST API and converts schemas for their registry. LakeKeeper, an Iceberg REST-compatible data catalog, has adopted data contract features similar to those you find in Confluent’s schema registry or Gable’s data contract product. Datahub has also tried to adopt data contracts.

Catalog convergence makes sense. We spent the last 10-15 years doing data integration. Data pipelines are supply chains now. All of our data (and metadata) flows through OLTP, streaming systems, batch, and data warehouse systems. It no longer makes sense to have a separate catalog for each use case.

The optimist in me hopes the convergence continues; “data catalog” should mean only one thing. If Iceberg’s REST API evolves fast enough, it might become the lingua franca that we need. The realist in me knows that this is unlikely. Instead, we’ll probably end up with catalog gateways or proxies.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Before co-founding Epsio, Gilad worked as a product manager for Axis Security and was an R&D team lead for Israeli Military Intelligence.

C.R.: Let's start with the basics to get everyone on the same page. Can you give me a brief definition of what streaming SQL is and what it's useful for?

G.K: Unlike "traditional" SQL that processes a dataset at a specific point in time, streaming SQL refers to constantly processing a stream of changes and understanding how a given change in the underlying data affected the previously outputted result—without ever re-processing the entire dataset.

For example, if you are running a query that calculates the count of items in a warehouse per category, a streaming SQL engine will output the initial result of that query and just do "plus one" to the relevant category whenever an item gets added and "minus one" when an item gets removed. Although this specific example is a fairly easy one to imagine, the same concept could be applied to much more complex queries containing many joins, subqueries, window functions, and so on.

In scenarios where an application runs a predefined set of queries on a dataset (e.g. in-app analytics, data modeling, reporting, etc.), streaming SQL can help you get faster and cheaper results by orders of magnitude compared to reprocessing the entire dataset each time you run the query.

C.R.: There are so many streaming SQL offerings out there: Materialize, Flink SQL, Feldera, RisingWave, and others. I tend to split solutions into one of two categories: streaming databases (RisingWave, Materialize) and stream processing systems (Flink SQL, kSQL). Does this match your mental model? And what differentiates Epsio from these systems?

G.K.: This model makes a lot of sense. Stream processing systems like Flink SQL and kSQL focus on outputting changes in query results to downstream systems (e.g., Kafka), leaving users responsible for consuming and processing these changes later. On the other hand, streaming databases go a step further by materializing up-to-date query results based on the stream of changes and serving them directly to clients, much like traditional databases. Unlike stream processing systems, streaming databases often allow users to connect via SQL clients (usually PostgreSQL) to interact with the data, run ad hoc queries, create indexes, and perform other typical database operations.

At Epsio, we’re huge fans of PostgreSQL and MySQL. Our goal is to bring the benefits of streaming databases to the existing PostgreSQL/MySQL deployments that organizations already use and love. Epsio achieves this by natively consuming the CDC stream from these databases and updating result tables directly within the original database. With this architecture, creating a new “streaming query” is as simple as calling a stored procedure in your current PostgreSQL/MySQL instance and querying the resulting table—essentially transforming your existing database into a streaming database.

Since Epsio's results sit within the original database, organizations get the best of both worlds — a robust streaming SQL engine, with all the amazing features MySQL/PostgreSQL already have to offer (diverse indexing options, table partitioning, constraints, etc.) and without needing to migrate to a new database.

C.R.: Streaming SQL without having to add a new database is very appealing. Though, it sounds like Epsio still needs to run as a separate process on a separate machine. How would you compare your solution to PostgreSQL’s pg_ivm extension?

G.K.: When we originally built Epsio, we started out by building it as a PostgreSQL extension (similar to pg_ivm). Pretty quickly, we changed the architecture to what we currently offer based on pretty strong feedback we got from initial users/customers.

Other than the fact most managed database offerings (RDS, Cloud SQL, and so on) don't allow users to install unauthorized extensions, adopting new database technologies is a pretty scary endeavor. We found that asking companies to install a new extension (that could potentially crash) on their production database was a pretty big ask to make. By sitting "behind" the existing database, reading CDC logs, and writing back results to the original database, users can integrate Epsio without worrying about affecting anything other than the results tables it needs to maintain. We even actively recommend not giving Epsio permissions to anything other than that.

Additionally, in a world where PostgreSQL and MySQL doesn't scale out elegantly, we found that being able to offload some computations to a separate instance using Epsio was actually a benefit to many customers who were already reaching the limits of what a single PostgreSQL or MySQL instance could do.

Regarding pg_ivm — other than the mentioned points above regarding being an extension, I believe it is currently still a bit of a "batch processor in disguise" and not ripe yet for real-world use cases. Doing streaming SQL at scale for real-world use cases is no easy task, and I believe the way pg_ivm stores data (PG tables vs. storage engine built for streaming), passes data between nodes ("static rows" and not "changes in data" that can be consolidated), and executes queries (still missing support for basic operators like OUTER JOIN) just isn't enough yet for the use cases we see companies needing.

C.R.: The isolation Epsio offers by running on a separate instance is compelling. Ultimately, though, it has to write back into the database. In my experience, it’s possible for stream processing systems to overwhelm OLTP systems that they write into.

One example would be an Epsio instance that is offline for an extended period of time, then comes back and has to catch up by reading all of the CDC logs it missed. In such a scenario, I could imagine Epsio inserting data very frequently for a burst of time. How does Epsio prevent such issues?

G.K.: One of the exciting things in streaming SQL technologies is the concept of consolidations. To avoid redundant calculations/writes back to the DB, consumed changes are "consolidated" internally before writing back to database. This means that, for example, if you have 1M changes that affect a specific COUNT aggregations you have, instead of updating the result COUNT 1M times, all the changes will be consolidated into a single UPDATE to reflect the latest count after processing all the changes. Another example is if a row gets updated 100 times — the changes will be "consolidated" into a single update of the last value of the row.

Additionally, since the engine ensures that the outputted results are transactionally correct (i.e., a single transaction in the PostgreSQL/MySQL base tables is never split across multiple transactions updating the results table), Epsio actually only uses up to a single connection per results table to update results, significantly limiting the amount of stress a single view can generate in the database (since both PostgreSQL and MySQL don't use more than one core per INSERT/DELETE operation).

Having said all the above, every piece of software has its edge cases. If you have hundreds of large views that need to be fully rewritten after a long downtime, it might still be smart to limit the number of connections the `epsio_user` (the user with limited permissions created for Epsio to have access to the database) can create on the database!

C.R.: There seem to be a few different ways to implement incremental materialized views. Companies like Materialize and Feldera use timely dataflow, differential dataflow, and DBSP. Can you give me a 10,000 foot view of these approaches, and which you chose for Epsio?

G.K.: Sure! So basically, both timely/differential dataflow and DBSP are pretty awesome new theories/frameworks/libraries that allow you to create incremental materialized views with a handful of benefits that the more "legacy" (e.g., Flink) stream processors don't offer:

• Being highly parallel while promising high consistency - Both differential and DBSP allow you to process in parallel a stream of changes, while always ensuring the result is consistent (for example, if a user adds $100 to their bank balance and then withdraws $90 — their bank balance would never reflect the withdrawal before the addition).

• Supporting "iterative" syntax - Both differential and DBSP support "iterative" syntax, meaning they can efficiently handle recursive queries or computations that depend on previous results.

Apart from the above benefits, they offer a new (and very elegant!) way to look at changes in data and process them much more efficiently compared to more "old-school" ways.

Although not directly based on differential dataflow/DBSP, Epsio's approach is probably more similar to the differential dataflow approach, although not completely similar (we are disk-based and have approached a couple of things differently given the use case Epsio is trying to optimize for).

C.R.: Users sometimes have a hard time understanding where incremental view maintenance (IVM) fits into their stack. I most often see it used for fraud detection, where fraud detection models need up to date information. What other use cases and verticals do you see it being adopted in? What problems can you solve with it that users might not be aware of?

G.K.: Since the "Streaming SQL" world evolved out of the streaming world, most people initially associate it with "streaming" use cases like fraud /anomaly detection, notification, and so on. However, we've seen a pretty big shift in the last couple of years from those use cases to more SQL-related use cases.

Such use cases, which run complex and non-ad hoc queries on changing datasets, are common in areas like:

• Customer facing analytics (e.g. dashboards)

• Data transformations an enrichments (e.g. incremental DBT)

• Real time reporting

In classic "streaming" use cases, the main benefit of IVM was the ease of writing SQL rather than writing custom code. In the use cases above, the benefits are more about query performance and cost—how easy it is to deliver performant, cost-effective queries. No matter how fast or efficient a traditional database is, if you are running a heavy query and most of the dataset hasn’t changed since the last run, there is a lot of wasted compute. This translates into either higher cost, higher latency, or both.

We've seen companies make customer-facing analytics 100x faster, drop infrastructure costs by an order of magnitude, and turn reports that took hours to execute into ones that complete in under a second.

C.R.: Thanks for taking the time to talk with me. Let’s wrap things up. Where do you see IVM headed in the next few years. Any other parting thoughts? Where can readers get in touch with you?

G.K.: Happy to!

Regarding the upcoming years in IVMs, I think exciting times are coming. Whether it's in the use cases we earlier discussed or new use cases that emerge as IVM technologies evolve (like what the folks at ReadySet are working on with database caching), I believe IVMs offer a pretty fundamental change in how organizations can work with their data (i.e., building data structures around specific queries vs. trying to store data to be "generically" fast for any query) and that we're going to see them much more widely used!

For anybody interested in learning more about Epsio, you can head over to our website at epsio.io or just ping us at contact@epsio.io. We’re always happy to chat!

Warpstream → Confluent
Rewriting Kafka in golang but then making it spill to S3. I'm happy for the Warpstream team, but Confluent could have done this themselves.

As Andy says, WarpStream﹩ is a Kafka re-implementation that uses S3 rather than local disks (a Zero Disk Architecture). The tradeoff with this design is that writes take a little bit longer to be durably sent—usually 100ms-500ms. It turns out, a little bit of extra latency is tolerable for many use cases, especially when you factor in cost savings, operational simplicity, and the ease of BYOC support.

Andy’s comment begs the question, if Confluent could build WarpStream, why didn’t they? One could offer many theories: it was cheaper to buy than to build, to eliminate competition, to capitalize on WarpStream’s branding, to acquire talent, or something else.

I believe Confluent didn’t build WarpStream because it wasn’t set up, as an organization, to build such a product. To understand why, let’s first consider what WarpStream did: they built a Kafka-compatible BYOC solution with a handful of engineers in less than 18 months. They were also in the process of rolling out WarpStream SaaS and adding transaction support when they were acquired.

Next, let’s consider what percentage of total Kafka usage in the world would have been satisfied with a 100ms-500ms SaaS or BYOC Kafka offering. My guess is a significant portion; well over 50% of all usage.

This is just my guess, but it sounds right based on my experience. All of our Kafka usage at WePay would have been completely fine with a 250ms-500ms P99 latency. All of our data integration, our ETL, our service-to-service queuing would have worked just fine. And we would have loved to be able to run Kafka as a stateless service in our own cloud on Google Cloud Storage.

Another thing to consider is revenue: WarpStream can be run very cheaply. This let WarpStream undercut Confluent on cost. They went hard on this messaging, talking about how S3 allowed them to bypass Amazon Web Service’s (AWS) inter-availability zone costs.

WarpStream was also moving into the platform layer. They launched Orbit to compete with Confluent Replicator, Managed Data Pipelines to compete with Kafka Connect, and a BYOC schema registry to compete with Confluent Schema Registry.

Let’s pause to let this sink in. WarpStream was in the process of building a cheaper (and arguably better) version of Confluent’s platform with a handful of engineers that could service the majority of Confluent customers’ use cases. That’s a pretty staggering statement.

To compete, Confluent would have to build WarpStream. Indeed, they tried to do this with Confluent Freight. Shortly before the WarpStream acquisition, Confluent announced Confluent Freight as Kafka re-implementation that writes to S3, ABS, or GCS. It’s nearly identical to WarpStream, but there’s one key difference:

Freight clusters utilize the latest innovations in Confluent Cloud’s cloud-native engine, Kora, to deliver low cost networking by trading off ultra low latency performance.

Freight is built on top of Confluent Kora. The very first paragraph of Kora’s announcement post says:

Kora isn’t something you can download, or even something you could run outside our control plane and the rest of our cloud

And there-in lies a very significant difference. Freight can’t be run outside of Confluent’s cloud. WarpStream can. Why would Confluent build Freight without BYOC support? I think the answer is that it was too encumbered by its legacy tech stack, its customers (the innovator’s dilemma), and its organization (Conway’s law).

Apache Kafka was built before cloud native architectures existed. It had to solve its own replication, handle its own leadership election, and manage its own storage. As with all legacy systems, this means it carries baggage with it. Many design decisions would have been made differently if it started from scratch. This is why Confluent has forked Kafka and why WarpStream wrote their implementation from scratch.

Kafka’s original design did, however, come with some flexibility that newer cloud-native designs didn’t have. Namely, it could handle low-latency use cases that systems like WarpStream couldn’t. I’m not making the claim that it’s impossible to build a low-latency cloud-native Kafka (Confluent has done so with Kora). But I’m saying that, even if you did that, it would look different from Kafka.

Regardless, Confluent picked up some large, high-margin customers that depend on some of Kafka’s more exotic features, including its low latency support.

[The Innovator’s Dilemma] describes how large incumbent companies lose market share by listening to their customers and providing what appears to be the highest-value products, but new companies that serve low-value customers with poorly developed technology can improve that technology incrementally until it is good enough to quickly take market share from established business.

Confluent can’t easily abandon these customers and their use cases. Such customers tend to be big name companies with large contracts (and presumably, higher margins). Confluent needs this revenue; they now have over 3,300 employees on six continents (according to leadIQ). Some of Confluent’s size could be attributed to its maturing during zero interest rate policy (ZIRP) when money was cheap. Some could also be attributed to its obligation to support such use cases with more complex systems. Regardless, WarpStream was unencumbered by both.

Finally, even if Confluent decided that building Freight with BYOC support was the right thing to do, they still had to deal with entrenched teams in their organization whose raison d'être was in direct conflict with a BYOC Freight solution. If Confluent could service the majority of their users with a product that looked like WarpStream, what happens to the Kora team, the legacy Kafka teams, the entire sales motion, the marketing, and so on? Such a pivot would be seismic. Many teams would need to be eliminated or reorganized.

You need not look far to see what such a change would look like. Elon Musk’s shake up at Twitter is a reflection of very similar headwinds.

It’s simply much easier to build very lean, very efficient companies now. When you compound cloud native efficiencies with AI-enabled developers, things get even more pronounced. New companies are adding engineers at a rate that accounts for the AI multiplier that these engineers come with. ChatGPT, Claude, Windsurf, Cursor, and many other AI tools have improved engineering productivity.

Legacy companies built their teams before the AI shift happened. It’s harder and more painful to turn the ship. Salesforce recently announced it won’t hire engineers in 2025. Meta is looking forward to replacing mid-level engineers with AI. LinkedIn had layoffs, and is running very lean (from what I hear). New competitors don’t have to deal with the damage that these changes cause. This has left legacy companies in a tight spot.

Given all this, it’s no surprise that Confluent decided to build Freight on top of Kora. WarpStream fits nicely in the product offering as its BYOC option. But as I say in my post above, I worry this strategy is delaying the inevitable.

Kafka, itself, is commoditized. Bufstream, AutoMQ, WarpStream, Redpanda, and S2 show this. When these companies move up the platform stack, as WarpStream was doing, as Bufstream is doing with its schema registry, Confluent will be competing with much more efficient organizations.

Much of this post has been about Confluent and WarpStream, but that’s simply the angle I’ve taken. I don’t mean to pick on them. It’s easy to see this pattern across the industry. Many new startups are going after Elastic, for example. Datadog’s Quickwit acquisition is a strong signal there. Legacy infrastructure companies are going to have a tough time competing with these new cloud native, AI-enabled, post-ZIRP companies.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

One of the less glamorous data processing tasks payment providers do is shuffle files between their vendors and partners. Such shuffling usually involves Secure File Transfer Protocol (SFTP). Simply put, SFTP allows you to securely upload or download a file from a server. Much of the US banking system still runs on SFTP servers. Banks and credit card processors exchange files over SFTP to signal when fund transfers complete, when accounts close, and so on. These files are often written in CSV, XML, or fixed-length file formats. How ACH works: A developer perspective is a good read for those that are interested.

When I joined WePay, the company was running as a PHP monolith. One of the first microservices that I built was responsible for reliably syncing files to and from our bank and credit card processors SFTP servers. The service proved very useful, and was in heavy use when I left the company seven years later.

Separately, another of my teams was responsible for WePay’s data pipeline. We used Airflow to extract data from our production databases and load it into our data warehouse. We also used Kafka and Kafka Connect to stream changes from production databases into Kafka and other downstream systems.

One day, our business analytics team approached me to ask for a BigQuery dataset that could be shared with one of our larger customers. The sales team was trying to close a deal, and the customer wanted data access included in the contract. The team’s plan was to write Airflow jobs that would output customer-specific data to the tables to a BigQuery dataset and grant the customers access to query it.

I hit the roof: our data pipeline (and team) wasn’t built for this. The pipeline was stable enough to be used for internal reporting, debugging, and modeling use cases, but it was not reliable enough to expose to end customers. Our pipelines would break when upstream developers changed their schema, when Kafka Connect decided to rebalance, or when our single Airflow machine failed. We did not have enough SREs and on-call engineers to provide a pipeline suitable for end-users to depend on.

SFTP never came up when talking with the business analytics team about their use case, yet the pattern is similar. We were trying to share data with an external integration partner (our customer, in this case).

The customer data sharing use case stuck with me. It was not an unreasonable request. I started to notice this pattern everywhere. We received many such data requests. The fascinating thing was that different customers requested their data in different forms. Some wanted periodic batch data dumps (like the SFTP integrations our payments team worked on). Others wanted a realtime HTTP/JSON API. Still others simply wanted a button on the website that would generate and email a PDF or CSV file. Data usage varied, too: financial reconciliation, reporting, user-facing dashboards, data products, monitoring, and so on.

Despite all this demand, “reporting”, as we called it, was rarely invested in. It was a rather murky area—the reporting data was owned by the payments product managers, yet the reporting team was part of our frontend group. And of course, my team actually managed the data pipeline. There was no real owner. WePay’s reporting and data export features languished.

I spent several weeks investigating vendor solutions for this mess. I wanted a white-label data platform that we could send our data to. The ideal product would then expose this data as a WebSocket API, HTTP/JSON API, data warehouse, CSV, SFTP, dashboard, or whatever other whacky format our customers wanted. To my dismay, I found nothing that met our needs.

I’ve never given up on this vision, and I continue to see use cases for such a platform everywhere. Stripe Sigma, Stripe Data Pipeline, Nango, Snowflake Data Cleanrooms, every reverse ETL platform, Quill﹩; these are all parts of a whole. Customers want data from their SaaS vendors. Yet, for the past decade (longer, really), SaaS vendors have been left to roll their own solutions. That is, until now.

Several startups, including Prequel, Bobsled, and General Folders now seem to be taking this use case seriously. The one I’m most excited about is Prequel. They call themselves a, “data export platform.” The product marketing on their website looks rather reverse ETL-ish, except that the destination’s connections are meant to be an external organization rather than internal tools such as Salesforce or Zendesk (as is usually the case with reverse ETL). But when I spoke to Prequel, they were quite focused on one particular pattern: exporting data to S3 in Parquet files.

The idea is actually fairly straightforward: modern data processing is centralizing around data lakehouses using S3, Apache Iceberg, Apache Parquet, and data lake query engines such as DuckDB and Trino. SaaS vendors can upload Parquet files to a shared S3 bucket managed by an Iceberg catalog. Customers can then query the files simply by adding an external table to their existing data warehouse or by querying the data directly using a data lake query engine.

Offering customers data in a data lake format maintains some of the benefits that SFTP has historically provided: data transfers are very fast, access can be managed centrally in the catalog, and files can be atomically exposed to customers (no partial uploads). This architecture comes with some additional benefits, too. Centralizing around a strongly typed, columnar format like Parquet and table format like Iceberg makes managing data models much easier. And query engines already support this format, so users can query their data directly without having to ETL it.

Schema evolution challenges still remain. Application engineers might decide to drop a field that customers are using. This problem has existed for a long time, but it becomes a critical outage when it impacts customers. Fortunately, data contracts are getting attention. Gable.ai﹩and its founder, Chad Sanderson, have been doing a lot of evangelism in this area, which helps. But much like the reverse ETL companies, data contract companies seem to be focused on internal use cases. So when I recently spoke to Vakamo, the creators of Lakekeeper, I was very heartened to hear that they’re thinking about data contracts for data catalogs. They directly called out customer data integration as a use case for this feature.

I’m cognizant that a customer data lake doesn’t address all the use cases I enumerated above. Data is loaded in batch, rather than streamed on web sockets or polled through HTTP APIs. You can have any data format you want, so long as it’s Parquet. SaaS vendors are still left to roll their own visualization features if they’re needed. But maybe this is OK. I suspect a data lakehouse with periodic batch loads might be enough.

Customer data lakes might actually end up being a driving force for lakehouse adoption. Many companies are still in the, “What is this, why do I need it, and how do I use it,” phase of lakehouse adoption. These same companies are very motivated to get access to their SaaS data.

Giving a customer’s analysts DuckDB and pointing them at an S3 bucket is pretty compelling. It could drive a land-and-expand transition for lakehouse architectures as companies initially adopt lakehouse query engines to get access to their SaaS data, then realize the benefits for their own data over time.

These lakehouses could also give analytics engineers something to do—something that drives revenue. I recently lamented that the role didn’t drive enough revenue to justify its existence. But someone’s going to need to curate these customer data lakehouse data models and make sure the files are loaded on time. Seems like a good fit.

It’s still early days for all of this, but I see a lot of tailwinds. Customers want their data, lakehouses seem like a good fit, and we have analytics engineers that are eager to help.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Alas, we didn’t have such a system at the time. But object storage systems like S3 have grown to give us all the properties we need; they are insanely scalable and provide atomic operations needed for transactional workloads. Object stores are still missing POSIX semantics, though. You can’t take any old system that uses filesystem I/O libraries and use S3 as its storage layer.

In the absence of a POSIX API for S3, two approaches have emerged to leverage what object stores have to offer . The first is building S3 directly into the systems, which most database and streaming companies are doing. Neon, WarpStream, Turbopuffer, and Responsive﹩ are all in this category. The other is to wrap S3 in a filesystem in userspace (FUSE) interface or an NFS-based implementation. Amazon S3 File Gateway, JuiceFS, s3fs, and Goofys are examples of such an approach. (If you squint, Apache OpenDAL fits in this category, but inverts the relationship by wrapping everything in its own API.)

Direct integration requires a storage-layer rewrite. I/O calls must be converted to S3-compatible API calls. Moreover, each system needs to figure out how to deal with higher object storage latencies, a subject I’ve written about before.

Direct object storage integration makes sense for systems like databases, whose primary job is to store and query data. But for systems, frameworks, and libraries that just need to read and write files as part of a broader workload, rewriting the storage layer for object storage is too burdensome.

FUSE and NFS-based implementations present their own challenges. s3fs and other FUSE-based systems implement only a subset of the POSIX interface, often missing features such as random access writes or appends, metadata operations, atomic renames, hardlinks, and inotify features. Such limitations simply won’t work for many systems and libraries. It’s not clear if RocksDB, for example, can safely be run on EFS. And some implementations, such as JuiceFS, store files in their own block format, which limits interoperability.

The need for scalable filesystems has grown, too. Database and streaming use cases have been around for a long time, but the growth of Kubernetes and AI workloads are new. Kubernetes has made every system distributed. And with AI, multi-modal data such as audio, text, and video is a critical ingredient. Many ML and AI libraries are built with local filesystems, and those that support object storage often don’t have good caching to speed up workloads.

I think we’re finally getting to the point where we have both the technology and the demand to get us what we need: a distributed POSIX-compatible filesystem. Regatta Storage is building in this direction, with a stated goal of replacing EFS and elastic block store (EBS) general purpose (gp3) use cases.

Though Regatta doesn’t have complete POSIX compatibility, their offering is compelling. They are using NFS now, and moving to their own protocol. This should give them a lot of flexibility as they try to implement more obscure POSIX features. Even if they never get complete POSIX support, they should be able to get closer than current solutions. Regatta also provides a generic cache to reduce latency, a key ingredient for a disk-like experience. Such an architecture is akin to a generic version of Neon’s Safekeepers and Pageservers, something I’ve been dreaming of for a while. Plus, unlike JuiceFS, files appear in object storage as normal files rather than opaque blocks that must be accessed only through the JuiceFS interface.

I expect this area to get more competitive. Much as competition with S3 has driven AWS to improve its offering, I anticipate more EFS features in the future. Other object store providers such as Tigris ﹩are also well positioned to pursue this area. JuiceFS and Alluxio will also continue to make progress, too. AI-specific offerings could also emerge. Fortunately, I think the TAM is big enough (and the use cases diverse enough) to support winners for different use cases, even if the underlying technology is largely the same.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Artemis Blog

Fragmentation in the Data Stack and cost structure with Chris Riccomini

Read more

a year ago · 3 likes · Josh Gray

I also did a podcast interview on The Joe Reis Show. Joe and I had a leisurely talk about all things data infrastructure. Best enjoyed in a hammock with a beverage of choice.

I posted a Bluesky thread this weekend arguing that analytics engineers and data engineers should be folded back into a single role. I decided to make the argument after coming across ’s post, Disband the analytics team. The post wrestles with the struggles that analytics teams face and offers the choice: disband or rebrand.

In short, my answer is that analytics—not as an industry or as a technology ecosystem, but as a discipline—might not work. The average company may never be able to make better decisions by hiring a team of average analysts. We can make dashboards and be operational accountants. But the fun, exploratory, “valuable” work may always be an indulgent, empty dessert, and never the entrée we want it to be. — , Disband the analytics team

I’ve long held that creating the “analytics engineer” role was a mistake. dbtLabs says, “Analytics engineers provide clean data sets to end users, modeling data in a way that empowers end users to answer their own questions.” I don’t believe that this set of activities is enough value to justify a full headcount; it’s is too limited in scope and too far removed from revenue generation.

Extracting, transforming, and loading (ETL’ing) data used to be handled by one team: the data warehouse team. But several trends have encouraged a schism in warehouse teams. Some—data engineers—now work on data pipelines (extract and load) while others work on data marts (“clean data sets”, as dbtLabs calls them). In short, data engineers do the E and L, and analytics engineers do the T. Many trends contributed to this bifurcation.

• We switched from ETL to ELT when we adopted data lake architectures. Dumping garbage into an object store made it easy for data engineers to ignore transformations and gave analytics engineers something to do.

• Similarly, adopting data integration with Kafka and Kafka Connect greatly expanded the number (and importance) of data pipelines in an organization, which gave the data engineers something to do as well.

• Shift-left became a data philosophy that encouraged everyone to be their own analyst, which left analysts squeezed.

• ZIRP ended, which made CFOs take a hard look at the cost of analyst and data teams, which further squeezed analysts.

• dbtLabs, Motherduck, and other MDS vendors were all too willing to create a new role to sell their products, which dovetailed nicely with analyst’s desire to be engineers and get paid more.

• LLMs are replacing analysts in some cases. Screech all you want, but it’s happening. There are dozens of data chatbots now (Cimba.ai﹩, DataChat, Julius.ai), and LLMs write pretty good SQL.

We’re in a new world now, though. ZIRP is gone, most of the connectors that data engineers were working on have now been built, and there are many vendors you can pay to run your data pipeline, and chatbots can answer data questions. It’s time to merge data engineers and analytics engineers back into a single data team that’s responsible for E, T, and L.

I’m happy to see companies and projects showing up to ease this transition. The most notable one is dltHub, which adopts SDLC best practices for data pipelines much as dbt did for transformations. Tools such as this should make it easier for analytics engineers to take ownership of data pipelines. I’ve also seen several tools like tabsdata﹩ that merge ETL back into a single tool for analytics and data engineers, rather than having both dltHub and dbt. I expect to see a collapse of data engineering and analytics engineering back to a single team in the next few years.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Let’s begin with social media. I’ve moved to Bluesky 🦋. Follow me @chris.blue if you’ve enjoyed my Twitter posts over the past 15 years. You can crosspost with Fedica or Buffer if you like. There are some great starter packs to bootstrap your feed, too. Here are a few:

• Infrastructure Engineers (mine)

• Distributed Systems on 🦋

• AI, ML, Data Science, and Security

• ML, Data, & Tech

• Data People Starter Pack

• AI and Data

I don’t know what this means for my Twitter account. All I can say is that I’ve been using Bluesky exclusively for the past week and it’s absolutely buzzing. It feels like the good old days. I haven’t missed Twitter at all.

Next, Materialized View turned one on October 31 🎃. It’s been an incredible year for the newsletter. I’ve published 50 posts and the newsletter just passed 4,000 subscribers. I’ve also received a lot of positive feedback. Almost everyone I meet mentions Materialized View. Thanks again for all the support and encouragement.

Finally, Rohan Desai and I presented at P99 CONF and the video is now online. Along with my The Geek Narrator interview, our talk is a great starting point to learn about SlateDB’s internals.

A consequence of drinking from the dataBS firehose is that you will get a lot of DuckDB chatter. For the unfamiliar, DuckDB is essentially SQLite for columnar data. It has a number of interesting properties. It’s very portable: it runs locally on your laptop, inside an application, or even in a browser. It’s also very fast (though, I’m told that’s not enough). Most importantly, it can connect to remote storage to read Apache Parquet files and Apache Iceberg tables.

These properties have made DuckDB a favorite among analytics and data engineers. All kinds of creative DuckDB uses have popped up. Okta uses DuckDB to cheaply transform data before it enters Snowflake. MotherDuck also showcases ETL examples. Rill and Mode have both adopted DuckDB as their in-memory query engines. PostgreSQL has been overrun with DuckDb extensions such as pg_duckdb, pg_mooncake, and pg_analytics. You can even use DuckDB to query New York City taxi data straight from your laptop (or from Modal﹩).

Given that DuckDB is an online analytical processing (OLAP) database, you might expect to see stories of DuckDB replacing Snowflake, Redshift, BigQuery, or Databricks as a data warehouse. There are some, but not many. I’ve always been skeptical of the idea that DuckDB is a viable solution for an enterprise data warehouse. (of fame) posted an observation that resonated with me:

DuckDB’s deployment model and limited scalability are what I struggle with. If you’re in an enterprise, your data warehouse users are going to include product managers, customer support, risk analysts, business analysts, finance teams, operations teams—virtual everyone at the company. I don’t see how DuckDB can be deployed in such an organization. It’s untenable to install DuckDB on everyone’s laptop, grant everyone access to data lake buckets, and ask them to run queries from the CLI.

Even if a company wanted to use DuckDB as their data warehouse, they couldn’t. DuckDB can’t handle the largest queries an enterprise might wish to run. MotherDuck has rightly pointed out that most queries are small. What they don’t say is that the most valuable queries in an organization are large: financial reconciliation, recommendation systems, advertising, and others. These are the revenue drivers. They might comprise a minority of all the queries an organization runs, but they make the money. DuckDB just can’t handle such queries.

To be a viable data warehouse, DuckDB needs a centralized deployment model, a better UI, and a way to scale. This is exactly what MotherDuck is building, and it sounds a lot like Snowflake or BigQuery. As much as MotherDuck would like to be the DuckDB vendor, they’re a cloud data warehouse that just happens to use DuckDB.

This begs the question: why should I switch from my current data warehouse to MotherDuck? It seems like the answer right now is cost. Cloud data warehouses are expensive. MotherDuck saves money by running DuckDB on small data sets. But it’s also really expensive to change data warehouses. It’s often easier to cut costs in your existing data warehouse by auditing queries and data retention.

Smaller companies can adopt DuckDB or MotherDuck and scale cheaply as they grow. This is a reasonable story for SMBs, but not for enterprises that already have a warehouse. But SMBs can also adopt the PostgreSQL extensions that I mentioned earlier. If I were tasked with rolling out DuckDB in an organization, that’s probably how I’d do it.

So, on the one hand, MotherDuck has picked a fight with some of the nastiest apex predators out there: Snowflake, BigQuery, and Databricks. On the other, they’re getting squeezed by PostgreSQL extensions and DuckDB on the laptop. This is a tough environment. MotherDuck has raised a lot of money, so perhaps they can find enough SMB customers and wait for them to scale.

As for DuckDB itself, I think Pedram and Erik have it right in the Tweet above. It’s amazing middleware, much like SQLite. I don’t see it as a data warehouse, though.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Microservices, serverless functions, AI, batch computation, stateful services, and other use cases all need special features to work well. Serverless functions need fast start times, AI and LLM workloads need GPUs, and many use cases could benefit from better state snapshotting technology. Yet, serverless functions suffer from high cold starts between 100ms and 1 second. Snapshots are slow, and only migrate certain pieces of state such as memory, while leaving other pieces, such as GPU state or network addresses behind. And GPU support in these execution models is spotty at best.

VM snapshots are useful when recovering from a failure or migrating execution to a new machine. Moving your workload to a different machine (or cloud) could save money, unlock better GPUs, or speed up training and inference. Cedana﹩, Modal﹩, and Microsoft have been working on this problem for some time.

This is why I’m excited to see a spate of recent developments that target cold start, snapshot, and GPU requirements.

GPU features are evolving quickly. gVisor added GPU support last year, NVIDIA recently open sourced their cuda-checkpoint tool, and Firecracker had a meeting on October 9th, 2024 to discuss GPU support.

NVIDIA’s cuda-checkpoint is particularly important. CUDA offers GPU APIs meant for generic computation (not gaming). Such APIs are widely used in AI models and LLMs. As developers execute operations on a GPU, the GPU’s memory accumulates state. This GPU data is very difficult to read directly, which poses a problem if you wish to snapshot a machine’s state. Now cuda-checkpoint offers a simple, bare-bones, free tool to do GPU checkpoint and recovery.

For serverless functions, unikernels such as Unikraft now boast single digit boot times and fast snapshotting. This should enable faster cold starts and scale-to-zero, which will result in cost savings. Many unikernels tout increased security in multi-tenant environments, as well. As unikernels add Kubernetes support, I expect adoption to increase, so non-serverless workloads like microservices will benefit. I’ve also heard there are benefits for specific verticals such as gaming.

Meanwhile, Loophole Labs has launched Architect to simplify VM migrations. They’ve built some really slick tech that incrementally snapshots and migrates both memory state and network bindings between machines. Architect purports to migrate faster than a spot instance preemption occurs, which would allow for huge cost savings for many workloads.

These developments will have a big impact on how we think about the cloud. We will, for example, see more multi-cloud adoption as state is easier to migrate and GPUs are harder to come by. Yingjun Wu (Founder of RisingWave) pointed me to SkyPilot, a UC Berkeley project that offers multi-cloud deployment specifically for AI, LLM, and batch workloads. Serverless functions might supplant service oriented applications, too. Vercel has been doing great work here. These are big shifts with big implications.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

Speaking of P99 CONF, I did a little speaker spotlight with them last week. I talk about the projects I’m working on and which talks I’m most excited about at P99 CONF.

In 2019, I gave a QCon talk entitled The Future of Data Engineering. It discussed the evolution of our data pipeline at WePay over the years. The talk presents a series of steps that most organizations go through when building pipelines:

• Stage 0: None

• Stage 1: Batch

• Stage 2: Realtime

• Stage 3: Integration

• Stage 4: Automation

• Stage 5: Decentralization

I won’t discuss each one of these (see the video or read the blog), but one point I make in the talk is that it’s reasonable, if not suggested, to take your time with this evolution. It’s possible you won’t even need the latter stages. In fact, WePay had a batch data pipeline—stage 2—running on Airflow nearly the entire time I was there. It ran at 15m intervals and it was remarkably effective.

These days, the ETL ecosystem is very diverse. An enterprise can still stand up an ETL pipeline like WePay’s using Airflow, Prefect, or Dagster. Or an enterprise might choose to adopt a data integration platform such as Airbyte, Fivetran, or Kafka Connect. Many cloud providers offer one-click integration between their cloud SQL services and their data warehouses as well.

For many smaller organizations, these tools are overkill. Though I don’t fully agree with MotherDuck’s Big Data is Dead thesis, they’re right that a lot of organizations just have a few terabytes of data. Developers are thinking: if I can get by with a small query engine (DuckDB), maybe I can get by with a small ETL tool.

Pipeline ownership has shifted as well. WePay had a team of data engineers to manage our data pipelines. These days, we’re asking ML, AI, analytics, and product engineers to build data pipelines. These users are used to interacting with data in a different way. They like to transform data in batch through SQL and Python with tools like dbt, Jupyter notebooks, and Pandas data frames. They want an ETL tool that integrates with this flow.

The closest thing that I’ve come across so far is data load tool (dlt). As the name suggests, dlt is philosophically similar to dbt; it’s trying to give these new users a tool that fits into their flow, but that incorporates software development life cycle (SDLC) best practices. Unlike dbt, which focuses more on transformations, dlt offers sources and sinks and a simple API to work with:

The proliferation of Python libraries such as Pandas, Jupyter notebooks, NumPy or PyTorch revolutionised the ML/AI space by allowing millions of practitioners to actively build the ecosystem.

We aim to bring such revolution to the data space. dlt is a pip-installable, minimalistic library that anyone writing Python code can use. dlt enables those people to create new datasets and move them to tools they use - be it other Python projects or engines or tooling from the Modern Data Stack.

dlt fits nicely into many so-called modern data stack (MDS) tools out there, but also runs on its own. It’s a nice stepping stone from step 1 to step 2 in my stage-list, above.

There are challenges to this approach. If everyone is managing their own data pipelines, there will duplication. There is also likely to be inefficiency as engineers opt for simpler solutions rather than cheaper ones. Moreover, the pipelines could become a tangled mess with no clear ownership as employees come and go. These challenges are probably familiar to anyone working with dbt in a large organization.

But I think dlt has recognized something: data consumers want control of their data pipelines. I’m excited to see where this goes. There’s a lot of work to be done to get this right.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.

I was a bit skeptical of Confluent’s messaging at first—acquisition motivation and communication are rarely the same. But I’m coming around to the idea. Freight can’t support BYOC deployment because it’s built on Kora, the proprietary system confluent built for its SaaS offering. Kora is not easily deployed in a customer’s environment, whereas WarpStream is.

Confluent can now offer self-hosted, BYOC, and cloud versions of their product. I suspect their customers were asking for BYOC, a trend I expect to increase (see Bring Your Own Cloud, Nuon﹩, and Hosted SaaS Challenges With Jon Morehouse). Confluent’s product offering is indicative of what customers want: flexible deployment options. There’s no one-size-fits-all deployment offering.

But Confluent is now running (and supporting) three different versions of Kafka: self-managed open-source Kafka, Confluent Cloud with Kora, and WarpStream. If Confluent were starting from scratch now, I don’t think they would have built things this way. While there’s no one-size-fits-all deployment offering, I suspect we can still build a single system that can accommodate flexible deployment. Doing so would provide a much better user (and vendor) experience.

ClickHouse is an example of such a system. I really didn’t understand why everyone was so excited about ClickHouse until I dug into it in my recent post, Unpacking the Buzz around ClickHouse. Then it clicked: you can run ClickHouse as a single node. In hindsight, it’s a comically obvious gap in the realtime online analytical processing (OLAP) space.

ClickHouse is more than a single-node binary, though. It can also be run in a cluster, or embedded as an in-process library with chDB. The company also offers a SaaS cloud, and BYOC deployment products. This is an incredible amount of flexibility for one system. It scales from a single in-process library all the way to a distributed cluster. And it can be deployed as a self-managed service, BYOC, or as SaaS. Its highly efficient single-node binary can also scale up nicely, too. This is the future.

Once this clicked, I started seeing the trend everywhere. PostgreSQL now has pglite.dev for client-side and edge deployment. Neon has added an object storage tier (granted, it’s a fork when I last checked). MotherDuck is trying to stretch DuckDB from analyst laptops to the cloud. Daft﹩’s next-generation query engine is going after not only Spark-scale workloads, but DuckDB as well.

There are many reasons for this shift. On the product side of the fence, we’re finally understanding what infrastructure customers want: flexible deployment. On the tech side, new trends such as zero-disk architecture, Rust and C++ adoption, composable data systems, and WebAssembly’s (Wasm) slow but continued growth have enabled us to actually build such systems.

Infrastructure vendors need to ask themselves: does our system deploy self-managed, BYOC, and as a SaaS cloud? Can we support in-process embedded, single-node, and clustered execution? If not, you’re cooked.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile for a complete list.

C.R.: Let's start with how you got into formal methods. It looks like you spent 12 years at Google before moving on to Lyft and Clumio. At what point did you first come across formal methods, what problem were you solving, and which tools did you use?

JP: Before joining Google, the systems I worked with were relatively small-scale, and we managed consistency through centralized databases. However, at Google, everything we built had to be large-scale, highly available, and eventually consistent. This shift required a deep understanding of how to maintain data integrity and fault tolerance in a decentralized environment.

As I worked with systems like GFS, Bigtable, and Spanner, and delved into their design papers, I saw firsthand the importance of ensuring system correctness, especially in scenarios involving data consistency and failure recovery. These papers often included rigorous proofs, highlighting the complexity of building robust systems. This experience sparked my interest in exploring more rigorous methods for verifying correctness.

After leaving Google, I noticed a gap in the broader industry, where microservices and NoSQL databases were often adopted without a solid theoretical foundation. At both Lyft and Clumio, I encountered consistency issues that often led to customer escalations and the need for refunds.

Having a habit of reading papers on various topics, I explored distributed systems classics and came across works by Leslie Lamport and Nancy Lynch. This led me to discover TLA+, which opened up a whole class of formal methods tools. Among these, TLA+ stood out as the most accessible and widely used.

My experience with formal methods, particularly TLA+, is relatively recent, spanning just a couple of years. The first project I used TLA+ for was at Clumio, where we needed to charge customers for their backups. The previous system had design issues, and fixing them with code changes like retries or locks didn't address the root cause—it often just moved the problem elsewhere. So, when I took over the project, I wanted to prove the system's correctness. Initially, I explained the design, but then I wrote the TLA+ specs as well. Unfortunately, I couldn't convince anyone to review the TLA+ specs.

C.R.: I think I can guess, but why didn’t anyone want to review the TLA+ spec?

J.P.: The main reason was the complexity of TLA+'s syntax and semantics. Its syntax isn't intuitive—it relies heavily on ASCII representations of mathematical symbols, which can feel alien to most programmers. The semantics also require a level of mathematical reasoning that many developers aren't used to.

Moreover, by the time I had learned TLA+ in my spare time and modeled the system, a few weeks had passed, and the spec was no longer a priority for the team. I did get my proposal accepted, but it wasn’t the smoothest process. If there had been a more accessible formal verification tool, I could have clearly demonstrated the advantages of my proposal without risking misunderstandings or tension.

C.R.: I suppose this is where FizzBee enters the picture. How did you get from TLA+ to FizzBee, and can you give a brief overview of your new project?

J.P.: While working with TLA+, I found myself instinctively translating its specs into a more familiar, Java-like syntax to make sense of them. This led me to realize that a tool with a syntax closer to programming languages would be more intuitive. I explored various options and ultimately decided on a Python-like syntax because it felt the most natural and concise for expressing complex ideas.

For these methods to be widely adopted—not just for mission-critical applications but for everyday software engineering—I believe they must be simpler and easier to use. When there’s a tradeoff between quality and time to market, time to market almost always takes precedence.

Our goal with FizzBee is to enable developers to ship high-quality software faster. It’s designed to make it easier for everyday software engineers to ensure system correctness and performance. FizzBee uses a Python-like syntax to make the process of modeling and verification more intuitive and concise. The time taken to model a system in FizzBee is often less than the time it takes to write a traditional design document. 

Additionally, FizzBee automates the generation of visualizations, like block and sequence diagrams, directly from the model, making it easier to communicate and review designs. This approach not only simplifies the design process but also helps catch potential issues early, allowing teams to ship more quickly.

C.R.: What does FizzBee’s architecture look like? How is it implemented?

J.P.: FizzBee’s current implementation is fairly straightforward.

The parser built with Antlr converts the specification into an abstract syntax tree (AST). The model checker is an interpreter implemented in Go, starting with a breadth-first search from the initial state, evaluating next states while following FizzBee-specific rules for forking, context switching, crashes, and more. It uses the Starlark interpreter to evaluate expressions, checks assertions, and eventually evaluates liveness.

Once the checking is complete, states are stored on disk, which we then use for performance modeling in Python. The online playground and visualizations are implemented in Node.js/JavaScript.

Currently, the model checker runs in a single process and thread, with all states kept in memory. Distributed model checking is in the plans, and I expect to start on it in a month. It will be exciting to make this the first project within FizzBee to use FizzBee itself to model its distributed design.

C.R.: Did you consider writing a transpiler to convert FizzBee code to TLA+ so that you could leverage TLA+’s battle-tested proof system? Or do you have any other thoughts about TLA+ interoperability?

J.P.: Yes, I did consider writing a transpiler from FizzBee to TLA+, mainly to leverage its model checker. But I quickly realized that differences in syntax—like TLA+’s use of 1-indexed arrays—would make this impractical and error-prone. In fact, building a transpiler would have been more complex than just creating an interpreter directly.

Additionally, TLA+ wouldn’t support many of the features I wanted. For example, TLA+ tracks only state transitions, FizzBee needs to capture more detailed information, like decisions made during those transitions. This is essential to supporting features like performance modeling and interactive visualizations.

As for TLAPS, while it’s an impressive proof system, very few people actually use it. Even among TLA+ users, I don’t see it gaining widespread adoption. When I talk to other developers, they’re more focused on verifying that the implementation works correctly, rather than proving the design is flawless. So, a proof system isn’t a priority for FizzBee. If we ever do build one, it would more likely involve direct use of solvers, rather than relying on TLAPS.

C.R.: Given that you’re using Starlark, do you see any opportunity to somehow integrate the code in the design with the code that eventually gets shipped?

J.P.: That’s an interesting idea. I haven’t looked into it deeply yet, but I’m generally cautious about code generation systems where developers need to tweak the generated code. The gap between high-level models and actual implementation can be quite large, so keeping everything in sync is challenging.

However, embedding Starlark code snippets directly into the implementation could be feasible in some cases. I’d need to think through it more to identify where it might be practical. It’s definitely something worth exploring further.

C.R.: Let’s look to the future a little bit. Paint me a picture of what success looks like for FizzBee. You mentioned a distributed model checker. What other tools and integrations do you want to implement, and where do you see adoption coming from?

J.P.: FizzBee’s goal is to help teams ship high-quality software faster. By making formal methods practical and easier to use, we're addressing the stagnation in system design validation. Our recent release of interactive visualizations, for example, enhances communication of designs within teams. Ultimately, we aim to directly link these improvements to faster software delivery. Ensuring that the implementation matches the design is the critical missing link.

That’s our current focus. With FizzBee, from the model and its explored states and transitions, we can automatically generate comprehensive tests before any code is written. This approach delivers the benefits of Test-Driven Development without extra effort. We’re also exploring lightweight deterministic simulations to identify specific types of bugs. This would expand FizzBee to continuous builds.

As AI-assisted coding becomes more prevalent, validating that the code performs as intended will be essential for AI adoption. FizzBee will be instrumental in building that confidence.

C.R.: The AI angle is interesting; I hadn’t considered that. How do you envision FizzBee working with AI generated code? If an LLM generates code, how would a developer validate it using FizzBee. Have you thought at all about using an LLM to generate code from a FizzBee model?

J.P.: That’s a great question. When it comes to validating AI-generated code, FizzBee can provide value in two key ways: model-based testing and model-based monitoring.

First, with model-based testing, once a FizzBee model is built and validated, we can automatically generate comprehensive tests based on the model’s states and transitions. Whether the code is written by humans or AI, these tests ensure that the implementation behaves as expected, providing strong validation before deployment.

Second, FizzBee can enable model-based monitoring for post-production validation. FizzBee could monitor real-world behavior by analyzing databases, data lakes, logs, and other systems to detect anomalies. This would require developers to create a projection function that maps the runtime state of the application back to the model’s state. While this doesn’t prevent bugs from reaching production, it helps mitigate their impact by identifying them early on.

As for using an LLM to generate code from a FizzBee model, that’s trickier. FizzBee focuses on modeling complex, distributed systems logic, but a full application often includes a lot of business-specific logic that isn’t captured by the model. So, generating complete production code directly from a FizzBee model wouldn’t work—though it could generate basic scaffolding. We could introduce auxiliary structures to capture the additional requirements, but that’s beyond FizzBee’s current focus.

LLMs could, however, assist in other ways, such as helping generate FizzBee specs from natural language descriptions or enhancing FizzBee’s review platform by answering "what if" questions about system behavior.

Looking forward, LLMs have made tremendous progress in recent years, but for full code generation, they still need to handle intricate logic more reliably. I’m optimistic that future breakthroughs, perhaps in areas like reinforcement learning combined with LLMs, will bring us closer to that.

C.R.: Thank you so much for taking the time to talk with me. It’s been a really thought provoking conversation. There’s so much more to cover, but in the interest of time, let’s call it here. I’ll give you the final word. Anything you’d like to share?

J.P.: Thanks, Chris, I really enjoyed our conversation! I’m excited about the future of software development and the potential of formal methods.

We’re currently collaborating with early adopters and design partners through hands-on proof-of-concept projects. If you’re working on distributed systems and want to see how FizzBee can improve your development process, feel free to reach out at jp@fizzbee.io. You can also explore more at fizzbee.io or check out our GitHub repo at github.com/fizzbee-io/fizzbee.

In other news, my Prefect Summit 2024 keynote is up. Check out 4 infrastructure trends in 20 minutes to learn about primary persistence on object storage, composable databases, PostgreSQL's renaissance, and durable execution.

It’s a story as old as time. A tech startup is born. Early engineers work night and day to build a product that customers want. They iterate furiously—adding new features and repurposing old code. No time to refactor; they need revenue. And then, if they’re lucky, miraculously, the startup gets customers. Product market fit is achieved, and it’s time to put the pedal to the metal. More customers, more features, more scale, and more engineers.

Somewhere along the way, the codebase goes from 100,000 lines to 10,000,000. The application that the early engineers built—a monolithic application in a single repository—is now a house of cards. Every change breaks something. It’s taking longer to build, test, and deploy. Even checking the code out is cause for a coffee break.

And then the FAANG engineers arrive. What are you even doing, they ask. You need scale, you need isolation, you need to decouple. You have a fever, and the only prescription is more microservices. And so the “escape the monolith” death march begins.

Scaling a monolith is hard, no doubt. To date, the only real tool we’ve had in our toolbox has been to switch to a service oriented architecture. Services can be built and deployed independently. This isolation keeps build times small, tests passing, and makes deployment easier.

At least that’s the pitch. In practice microservices can be just as tough to wrangle as monoliths. Services get tightly coupled; deploying a change in one can break another. Deploying 1000s of services independently requires an immense amount of tooling. Developers now need to spin up dozens of services to test locally, or manage their own cloud environment. Remote procedure call (RPC) frameworks need tooling to enforce compatible schema changes. Operations has to wrangle service meshes, distributed tracing, Kubernetes, Terraform, and so much more.

I’ve lived this story twice. First as an understudy at LinkedIn, then as an instigator at WePay. Both projects were brutal. At LinkedIn, we had to halt much of engineering for several months while we tore our monolith apart. Kevin Scott, our SVP of engineering wrote a great write-up about this in, When Your Tech Debt Comes Due. At WePay, the monolith migration project never ended. I wrote WePay’s second microservice when I joined in 2015. When I left in 2021, the monolith still housed much of WePay’s core logic. I have microservice fatigue.

Yet, I don’t see a lot of alternative solutions being offered. So I’m pleased to see the modular monolith trend growing. The idea is to build a monolithic application as a series of modules, each responsible for a portion of business logic. At first blush, this sounds silly. Isn’t this just good engineering? I was skeptical

But the more I think about modular monoliths, the more excited I get. Yes, writing modular code is a good idea, and yes it’s obvious. Yet, we’re not doing it. Our monoliths always end up as spaghetti code. Rather than fixing the monolithic code, we’ve jumped straight to microservices. Why? I think the answer is tooling.

What might modular monolith tooling look like? Let’s start with the benefits that microservices confer: they can be built, tested, and deployed independently; they have isolated databases; and they have clear public APIs. To get similar characteristics from a monolith, developers need:

• Incremental build systems

• Incremental testing frameworks

• Branch management tooling

• Code isolation enforcement

• Database isolation enforcement

Incremental build systems speed up monolithic build times. Rather than rebuilding an entire application, only the portions that change are rebuilt. Similarly, incremental testing allows developers and continuous integration (CI) systems to run tests only for the portion of the monolith that’s changed (including upstream and downstream dependencies). Bazel is doing a lot of work in this area.

Since all developers are committing to the same codebase, branch management is also important. There are many options here: GitHub Flow, GitLab Flow, Trunk-Based Development, and so on.

By extension, CI tooling is important. If a developer breaks the monolith, they’ve broken it for everyone. Breaking builds must be quickly detected and fixed. Tools to predict whether a change is risky, to detect which change in a batch of commits broke the build, and to manage reverting or fixing forward are all required. Unlike incremental build and test, I find many teams are rolling their own scripts to manage such activities.

Changes that introduce new cross-module dependencies must be detected. Calls to non-public interfaces must be actively rejected. And code owners must be notified when new modules depend on their own modules. Code ownership files that define owners and approvers for each module must be added, and tooling built around them. Gauge﹩ is doing interesting work here with their open source dependency management tool, Tach.

Finally, monolithic databases need to be broken up. This is often the biggest chunk of work for any monolith migration. All parts of the codebase share a single ORM and assume they’re interacting with a single transactional database. Transaction boundaries must be defined and separated between modules. Tables must be grouped by module and isolated from other modules. Tables must then be migrated to separate schemas either on the same database or a separate one. I am not aware of any tools that help detect such boundaries and enforce isolation right now.

For monoliths just starting out, it would be great if full stack frameworks like Next.js, Redwood.js, Rails, and others began to adopt modular concepts in their codebase. This would help developers writing new software to do the write thing from the get-go.

Best of all, I don’t see the modular monolith vs. microservices as an either-or choice. I see it as a stepping stone that can extend a monolith’s life. For some, the modular monolith might be all that’s needed. For others, it can provide a more natural transition to a service based architecture. Modular monolith tooling could even facilitate such a migration. And I suspect the result of a migration would be a well maintained monolith with 7 ± 2 services. I find such an architecture far more appealing than 1000s of services. We’ve spent decades building tooling for microservices. It’s time to give monoliths the same respect.

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile for a complete list.

Before ParadeDB, Noël was the co-founder of Whist, a cloud-hybrid browser that accelerated web applications.

C.R.: You and I have been talking for the past year, give or take. I'm excited to finally have you on Materialized View. Let's start with the basics. What is ParadeDB and how did it come to exist?

P.N.: Hey Chris! I've been a reader since day 1 and am excited to be a part of it.

ParadeDB is a Elasticsearch alternative for Postgres. We build the core feature set of Elasticsearch (the data store), full-text search and fast analytics, in Postgres via Postgres extensions. The idea is that with ParadeDB, Postgres users can avoid needing to ETL data to Elasticsearch or similar tools in order to power user-facing search and analytics, and can instead stay within Postgres and keep their infrastructure simple. ParadeDB is an open-source project and is compatible with any existing Postgres deployment, including on managed Postgres services like AWS RDS, etc.

We came up with the idea while operating Postgres+Elasticsearch ourselves as part of a previous project, Retake. We were frustrated with needing to maintain an Elasticsearch cluster and an ETL pipeline, and by the lack of real-time and transactional guarantees in our search workload. We then went around asking our friends and fellow YC batchmates, and realized that this frustration was shared by a large number of them. Realizing the gap between people's love for Postgres and hate for Elastic convinced us to launch into this and thus ParadeDB was born.

C.R.: The overlap between search and analytics is something I’ve written about recently in my 15 Years of Realtime OLAP series (part 1 and part 2). These use cases are distinct, but they have remarkably similar infrastructure. Faceted search has always looked a lot like columnar aggregation to me, for example. And both have shifted from batch to realtime ingest.

How do you think about these use cases? ParadeDB has pg_search, pg_lakehouse, and pg_analytics extensions. You seem to be suggesting that they don’t warrant different infrastructure (i.e. Elastic), but you’ve implemented different extensions for each.

P.N.: We think of these use cases from the perspective of the customer. While it is true that they are distinct, they are fundamentally the same: enable users to filter data to derive insights/take actions. Every major customer who comes to us for full-text search is also interested in faceted search, which as you point out is very similar to columnar aggregations. In fact, our faceted search uses a custom columnar implementation built by Tantivy.

These customers typically build products that contain both user-facing search and live dashboards. You can get very far powering real-time dashboards with faceted search, but sometimes you need to fetch data that's stored in object storage, or you need to perform aggregations over the JOINed tables, which inverted indexes aren't as optimal for.

We believe customers should pick a product by thinking about their business need(s) rather than about the infrastructure of the product itself. ParadeDB pg_search delivers BM25 full-text search and faceted search in Postgres. We've also built pg_analytics to offer on-disk columnar analytics in Postgres, and finally pg_lakehouse to offer analytics in Postgres over data in object storage like AWS S3. We're now combining pg_lakehouse and pg_analytics into a single extension, pg_analytics, which will offer both capability in a single extension to streamline development and adoption.

The use case is fundamentally the same: Customers want to offer user-facing search and analytics. They build their backend on top of Postgres, and ParadeDB enables them to offer these features on top of Postgres, no matter where their data lives.

C.R.: That last comment—building on PostgreSQL—seems to be the real differentiator for ParadeDB. PostgreSQL has been experiencing a real renaissance lately. Why do you think that is, and how has your experience been working with it?

P.N.: In my view Postgres isn't having a renaissance but is rather "coming of age". The last 30 years of smart design decisions have slowly made it the best open-source RDBMS. It's extremely reliable and very extensible, which is a really big deal.

The development of Postgres is open-source (not backed by any single company) and moves slowly. Maintainers are very opinionated and getting any patch in core is a serious task. This is good, as it guarantees quality. However, it means development is slow and Postgres could be left behind faster-moving database projects. pgvector was able to be developed very quickly as it sits outside of core, as an extension. Extensions enable Postgres to stay current with innovations in the data space while also maintaining extreme care to keep core as small and robust as possible. Every now and then, some extensions become so ubiquitous that it’s merged into core, like pg_stat_statement. The clear separation of work granted by extensions allow Postgres to uniquely accomplish the best of both worlds: Stay relevant while being rock-solid.

Working with Postgres has been wonderful. The extension ecosystem and API is very mature. Thanks to Eric Ridge (ZomboDB)'s pgrx project, we can build our extensions in Rust and benefit from the entire Rust open-source data community, without which ParadeDB couldn't exist. Oftentimes we find ourselves reading Postgres source and mailing lists, which is dense but the quality of code in core Postgres is quite amazing. Hacking on Postgres is really hard. Robert Haas wrote a big post about it, for the curious. It's hard to find developers who know Postgres core, though. We didn't before we started. We're hiring, so if a reader is looking to get into Postgres internals come say hi!

C.R.: You mentioned that you’re combining pg_analytics and pg_lakehouse. Hydra and MotherDuck recently announced pg_duckdb alongside DuckDB Labs, Neon, and Microsoft. The Postgres analytics ecosystem seems really hot and really fast changing right now.

How is pg_duckdb different from what ParadeDB is doing, why are you combining your two analytics extensions, and how do you see things shaking out in this space?

P.N.: Analytics in Postgres should be a single extension, whether it's pulling data from object storage or from Postgres tables. That is the reason we’re consolidating pg_analytics and pg_lakehouse.

pg_duckdb is a cool project! I'm very excited to see it come to life. I had previously talked with DuckDB Labs and MotherDuck about donating our analytics work to DuckDB Labs to be the foundation of pg_duckdb, but others seem to have beaten us to it. I'm excited and cautiously optimistic it will flourish. If it does, it would be very good for everyone in Postgres, including for ParadeDB.

Building analytics in Postgres is really hard. Doing it properly is much harder than what ParadeDB could do today. My hope is that many companies work on pg_duckdb so it becomes a good foundation that we can rebase pg_analytics on top of. The vision is for many Postgres companies to build on top of pg_duckdb to easily create an analytics-in-Postgres extension for their platform.

For now, though, pg_duckdb is still very early. It also appears to be more DuckDB-centric than Postgres-centric, which concerns me. I am worried that MotherDuck, who seems to be the primary company driving this, does not care enough about Postgres to really make the project what it needs to be. Hopefully Neon gets involved, as they are one of the few players who have the Postgres expertise to really build it correctly. It's too early to tell, but we are following along.

On our side, ParadeDB is building an Elasticsearch alternative. We have plenty to do with search and already have a strong analytics offering with faceted search and our existing pg_analytics extension. If pg_duckdb evolves in the right direction, we'll be very eager to adopt it and to contribute to it. Fingers crossed!

C.R. On the Elasticsearch front, what challenges have you found in implementing search in Postgres, and what does the roadmap look like for pg_search?

P.N.: ParadeDB pg_search is built on top of a Lucene-inspired library called Tantivy. Tantivy stores data in its own file format. Because Tantivy uses its own format, we do not store data inside Postgres block storage (yet!). This means we have had to implement various of Postgres' native functionalities, like backup, WALs, etc. manually on top of this new storage. This is a lot of work, and we’re continuously improving upon it.

Today, pg_search is built via a multithreaded Postgres background writer and dynamic functions via the Postgres `CALL` API. It's quite uncommon, but very elegant. This enables us to play inline with Postgres as much as possible and has made our work integrating Tantivy much cleaner.

Roadmap-wise, we're working on features like:

• BM25 indexes over partitioned tables

• BM25 indexes over JOINed tables (highly requested, but very difficult)

• Full transaction isolation for faceted search

Eventually, we're also planning to integrate with Citus to offer a horizontally-scalable version of ParadeDB.

C.R.: Partnering with Citus is an interesting twist. ParadeDB’s extensions are licensed under the AGPL. I know you’ve written about your license choice in the past. You also don’t have a cloud offering right now. What’s your strategy for bringing ParadeDB to market?

P.N.: Today, we sell commercial licenses of ParadeDB containing a few enterprise-relevant features on top of our open-source offerings. Our customers self-host ParadeDB clusters from those licenses.

We're working on a bring-your-own-cloud (BYOC) solution to make this process smoother, and are going to announce partnerships with Postgres platform providers to power a ParadeDB Cloud.

C.R.: Wonderful. There’s plenty more to say, but I’ll save that for a future interview! Let’s call it here. Any parting thoughts?

P.N.: For anyone interested in our project, you can find (and star, hihi) our repo here: https://github.com/paradedb/paradedb. We welcome community contributions and are hiring for an early engineer role. If working on search and analytics in Postgres interests you, shoot me a note.

Thanks for having me, Chris! 🙂

Subscribe now

Share

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile for a complete list.

I also had a very pleasant conversation in the inaugural episode of Tech on the Rocks. We discussed stream processing, Rust, SlateDB, and more. Give it a listen here:

My two previous posts, 15 Years of Realtime OLAP (part 1, part 2), documented my experience with home-grown realtime OLAP systems, Apache Druid, and Apache Pinot. I also discussed the use cases I had for these systems: user facing product analytics and fraud detection. My intent was to lay the foundation for this post, where I investigate the buzz around ClickHouse.

ClickHouse has been appearing a lot in some of my recent interactions. Tinybird, a user-facing analytics product, uses ClickHouse as its database. Several startups I’ve talked to are working on ClickHouse-related products. My Twitter feed has a lot of ClickHouse mentions, too.

What’s more, much of the feedback about ClickHouse appears to be quite positive, too. This caught my eye—we engineers tend to be a critical bunch. Moreover, when I looked at ClickHouse, I saw another realtime OLAP system like Druid or Pinot. Why all the attention?

The responses to this post were interesting. The feedback seems to boil down to three things: speed, ease of install, and ease of operations.

Speed

ClickHouse is indeed very fast. A friend told me, “They have the mentality of a team on a budget. Like they didn’t have 1000s of machines to throw at the problem. They had to make it work on what they had.” This might or might not be true—ClickHouse comes from Yandex—but the software definitely has this vibe.

As proof, ClickHouse runs a benchmark project. Database vendors may submit their databases to see how they fair. ClickHouse has been dominating its competition (at least until 6 months ago when Umbra showed up). One can quibble over the workloads tested, but ClickHouse is clearly a very fast database.

Installation

Though speed is nice, it’s not as important as it used to be. Where ClickHouse really shines is its installation experience. Realtime OLAP systems are notoriously annoying to get running. Apache Druid and Apache Pinot both use bash scripts that spawn multiple local JVM-based services to get the system up and running. Either that or you’ve got to run Docker and use Helm charts, as is the case with StarRocks.

ClickHouse, by contrast, is a single cURL command to clickhouse.com. The server is smart enough that it recognizes the lack of user-agent in the HTTP request and automatically gives you a bash script to install a native binary for your host. It just works.

They’ve occupied a really nice spot between embedded OLAP systems like DuckDB and distributed realtime OLAP systems like Apache Pinot and Apache Druid. It’s surprising to me that there aren’t more single-process realtime OLAP systems out there; it seems obvious in hindsight. ClickHouse seems unique in this regard, aside from recent PostgreSQL OLAP developments (more on this later).

Another way of phrasing all this is that the developer experience (DX) is really nice. And a great developer experience leads to a lot of rave reviews on Twitter. I suspect this is where a lot of the buzz is coming from.

Operations

A great DX is nice and all, but does it scale and is it easy to operate? Here too, the feedback is positive, but more mixed. ClickHouse’s speed and efficiency mean it can scale up quite nicely—you can continue to run it on one big machine for quite a while.

Once you’re ready to move beyond one machine, you’ll need to introduce another ClickHouse service: ClickHouse Keeper. Here, too, the developer experience is excellent. ClickHouse used to rely on ZooKeeper to coordinate its nodes in a distributed environment. Running ZooKeeper is tough, so ClickHouse wrote their own drop-in replacement, which they bundle bundle into their binary.

The operational flexibility to scale up or out without adding a bunch of services is really valuable. And it’s run on some very large workloads. Tinybird has some customers doing 300K-600K events per second. Uber adopted ClickHouse for their log analytics platform (more on this later, too). And I assume Yandex’s usage is still fairly large.

Of course, there will always be operational challenges. Javi Santana, Tinybird’s co-founder, says it’s, “super hard to run at scale.” I also noticed some discussion about disk usage and imbalances. And its XML configuration files are clunky.

Challenges

As nice as ClickHouse appears to be, I see a few challenges. The first and most significant is cost. Remember Uber’s ClickHouse log system I mentioned above? They’re moving off ClickHouse. Yupeng Fu presented an excellent talk at StarTree’s﹩ RTA Summit 2024 called Evolution of OLAP at Uber. The talk discusses how Uber is replacing several pieces of infrastructure, including ClickHouse, with Apache Pinot.

Yupeng says that Uber’s log analytics platform migration in 2020 resulted in 50% cost savings when compared to their previous ELK-based log analytics system. But ELK is very expensive to run on large datasets. A 50% gain isn’t really that much. Since the migration, the team has hit cost and performance challenges. Stories like these are somewhat alarming for large-scale enterprises.

Another more subtle (and perhaps more minor) challenge with ClickHouse is its behavior with materialized views. Materialized views are important for many realtime analytics use cases. By updating aggregates when a write occurs, reads become very fast. Entire systems like Materialize and Feldera are built around this concept. ClickHouse supports materialized views, but updates are only triggered when the “main” table—the first table in a join—is written to. For many queries, especially those without joins, this is perfectly acceptable. But for more sophisticated use cases, it simply isn’t good enough.

And finally, the elephant in the room: PostgreSQL is becoming an OLAP system. Hydra recently published pg_duckdb with backing from MotherDuck, Microsoft, Neon, and others. Hydra’s extension integrates DuckDB (an even more buzzy project than ClickHouse) with PostgresSQL. And ParadeDB has seen a lot of adoption with its pg_lakehouse, pg_analytics, and pg_search PostgreSQL extensions.

As PostgreSQL’s OLAP extensions mature, it will be a great solution for the exact space that ClickHouse shines: single-node scale-up realtime OLAP with a great DX. If PostgreSQL takes ClickHouse’s single-node and small-scale usage, and systems like Pinot and Druid take its large-scale market, there’s not much left. This is the biggest long-term threat that I see for ClickHouse.

Still, as things stand now, ClickHouse is a robust system, and a reasonable solution for many use cases. I look forward to seeing how things shake out; I have a real soft spot for realtime analytics.

Subscribe now

Share

Book

Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.

Buy Now

Disclaimer

I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile for a complete list.