It's Time to Merge Analytics and Data Engineering (Again)
Data pipelines are commoditized and analytics engineers don't provide enough value.
Josh Gray, CEO of Artemis﹩, interviewed me about data stack fragmentation, dbt, cost savings, and more. Josh’s questions really got me thinking, and is the subject of today’s post. Here’s the interview:
I also did a podcast interview on The Joe Reis Show. Joe and I had a leisurely talk about all things data infrastructure. Best enjoyed in a hammock with a beverage of choice.
I posted a Bluesky thread this weekend arguing that analytics engineers and data engineers should be folded back into a single role. I decided to make the argument after coming across
’s post, Disband the analytics team. The post wrestles with the struggles that analytics teams face and offers the choice: disband or rebrand.In short, my answer is that analytics—not as an industry or as a technology ecosystem, but as a discipline—might not work. The average company may never be able to make better decisions by hiring a team of average analysts. We can make dashboards and be operational accountants. But the fun, exploratory, “valuable” work may always be an indulgent, empty dessert, and never the entrée we want it to be. —
, Disband the analytics team
I’ve long held that creating the “analytics engineer” role was a mistake. dbtLabs says, “Analytics engineers provide clean data sets to end users, modeling data in a way that empowers end users to answer their own questions.” I don’t believe that this set of activities is enough value to justify a full headcount; it’s is too limited in scope and too far removed from revenue generation.
Extracting, transforming, and loading (ETL’ing) data used to be handled by one team: the data warehouse team. But several trends have encouraged a schism in warehouse teams. Some—data engineers—now work on data pipelines (extract and load) while others work on data marts (“clean data sets”, as dbtLabs calls them). In short, data engineers do the E and L, and analytics engineers do the T. Many trends contributed to this bifurcation.
We switched from ETL to ELT when we adopted data lake architectures. Dumping garbage into an object store made it easy for data engineers to ignore transformations and gave analytics engineers something to do.
Similarly, adopting data integration with Kafka and Kafka Connect greatly expanded the number (and importance) of data pipelines in an organization, which gave the data engineers something to do as well.
Shift-left became a data philosophy that encouraged everyone to be their own analyst, which left analysts squeezed.
ZIRP ended, which made CFOs take a hard look at the cost of analyst and data teams, which further squeezed analysts.
dbtLabs, Motherduck, and other MDS vendors were all too willing to create a new role to sell their products, which dovetailed nicely with analyst’s desire to be engineers and get paid more.
LLMs are replacing analysts in some cases. Screech all you want, but it’s happening. There are dozens of data chatbots now (Cimba.ai﹩, DataChat, Julius.ai), and LLMs write pretty good SQL.
We’re in a new world now, though. ZIRP is gone, most of the connectors that data engineers were working on have now been built, and there are many vendors you can pay to run your data pipeline, and chatbots can answer data questions. It’s time to merge data engineers and analytics engineers back into a single data team that’s responsible for E, T, and L.
I’m happy to see companies and projects showing up to ease this transition. The most notable one is dltHub, which adopts SDLC best practices for data pipelines much as dbt did for transformations. Tools such as this should make it easier for analytics engineers to take ownership of data pipelines. I’ve also seen several tools like tabsdata﹩ that merge ETL back into a single tool for analytics and data engineers, rather than having both dltHub and dbt. I expect to see a collapse of data engineering and analytics engineering back to a single team in the next few years.
Book
Support this newsletter by purchasing The Missing README: A Guide for the New Software Engineer for yourself or gifting it to someone.
Disclaimer
I occasionally invest in infrastructure startups. Companies that I’ve invested in are marked with a ﹩ in this newsletter. See my LinkedIn profile and Materialized View Capital for a complete list.