The Surprising Simplicity of Prometheus's Architecture, Discover FrostDB's LSM Tree for Observability, and more...
Why you can't offer Prometheus as a service; I try to find an embeddable metrics database; and I stumble over Parca, an open-source CPU profiler in an eBPF filter.
Why Is There No Cloud Hosted Prometheus?
Apurva, CEO of Responsive [$], recently asked about timeseries databases (TSDBs) for Responsive’s observability product.
The responses included the usual suspects: Clickhouse, TimescaleDB, InfluxDB, and Prometheus.
I was surprised to find that there isn’t any Prometheus cloud hosting. There are only managed offerings from GCP and AWS, or Prometheus protocol-compatible offerings from VictoriaMetrics, Grafana Mimir, and so on.
After some digging, I realized I’d been misunderstanding Prometheus’s architecture. Prometheus is intentionally designed with only single-node storage:
Prometheus's local storage is limited to a single node's scalability and durability. Instead of trying to solve clustered storage in Prometheus itself, Prometheus offers a set of interfaces that allow integrating with remote storage systems.
I love this! Getting started is easy because you don’t have to set up a distributed system. Users can defer the scaling decision until later. When it’s time to scale, users get to decide whether to scale up or out. Users that decide to scale out have dozens of remote stores to choose from:
That final note at the bottom—prom-migrator—means even migration is easy. This architecture is all about user choice. Kudos to the Prometheus designers.
Project Highlight: FrostDB
Prometheus’s on-disk layout documentation got me thinking about embedded TSDBs. The last embedded TSDB I used was RRDtool (c.f. Ganglia, LinkedIn’s InGraphs). RRD has been around for a long time; the repo’s favicon.ico is 18 years old. Surely things have evolved in the last 20 years.
It looks like many people are repurposing embedded OLAP DBs like DuckDB or Clickhouse’s engine. OLAP DBs should work, but I was really looking for something purpose-built for common metric characteristics:
High writes, low reads
Wide and unpredictable data (unstructured, many dimensions/labels, and so on)
Immutable data
I came across a project called frostdb from PolarSignals.
From the README:
FrostDB is similar to many other in-memory columnar databases such as DuckDB or InfluxDB IOx.
FrostDB may be a better fit for you if:
Are developing a Go program
Want to embed a columnar database in your program instead of running a separate database server
Have immutable datasets that don't require updating or deleting
Your data contains dynamic columns, where the number of columns in the schema may increase at runtime
The README also has a lovely write-up on frostdb’s on-disk structure. It’s built exactly as I’d hoped—a log-structured merge-tree (LSM). The first level stores Apache Arrow records. Data is then rewritten into Parquet for subsequent levels.
If you’re looking for a local TSDB that’s built for metrics, frostdb is worth a look!
Bonus: frostdb was built for Parca, another cool project. Parca does, “Continuous profiling for analysis of CPU, memory usage over time, and down to the line number.” It’s built as a single eBPF filter.
More Awesome Infrastructure
Keep up with new infrastructure projects as they’re added to awesome-infra. New submissions are welcome!
No updates for this post. Send me PRs!
I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a [$] in this newsletter. See my LinkedIn profile for a complete list.