On Monday, at TC Disrupt Colin Zima CEO of Omni, Jordan Tigani CEO of Motherduck, Daniel Svnova CEO of Superlinked & Toby Mao CTO of Tobiko Data who are leading the evolution of the Post Modern Data Stack discussed the trends they are seeing.

Here are some of the themes & predictions from the group.

Customers are excited about new architectures that significantly reduce cost. In the last 10 years, investments in big data have become increasingly expensive & focused on very large data volumes. Most data workloads are quite small, about 100MB. Also, data warehouses particularly in large teams are used very inefficiently - with about half of the Snowflake bill spent on inefficient data transformations.

AI is changing the structure of data teams. In the past, software engineering teams and data teams haven’t collaborated, but data pipelines & AI endpoints rapidly becoming essential parts of software, they now work together much more closely. In a parallel to the DevOps Fusion which joined Software Engineering and Set Reliability Engineering, there is a movement that is fusing data and software teams together.

There’s a broad desire across data teams to empower analysts, marketers, product managers, and sales teams to create their own metrics while balancing the data team’s need for centralized governance of data. New BI systems will enable both.

Vectors power AI systems. We use vectors to find similar documents & images & content to help AI answer questions better or generate inspiring images. In the future, most data will be vectorized as AI permeates our workflows.

Enterprise adoption of Iceberg is slower than expected. We discussed some of the potential reasons : the lack of immediate cost cutting, the desire of the incumbents to retain that data for their own revenue.

Snowflake and Databricks will compete less in the future than they have today as they refocus on their core areas of expertise, namely structured data and likely applications built on top of that structured data for Snowflake and large data pipelines feeding AI for Databricks.

The data world is evolving rapidly. I’m grateful to the panelists for joining me to share their views.

Continuing this theme, I’ll be revealing my predictions for the Post Modern Data Stack at the Monte Carlo IMPACT event on November 14.