In the world of data, it’s been a busy couple of weeks: Salesforce acquired Informatica, Servicenow acquired data.world, Databricks acquired Neon, Snowflake acquired Crunchy Data, and Collibra acquired Raito. Snowflake and Databricks had their user conferences making a slew of announcements. AI is clearly accelerating the need for data, and we shouldn’t expect things to slow down.
Let’s take a step back and look at the bigger picture. This is where we are:
- The decoupling between storage and compute is here to stay: Open table formats like Iceberg let data platforms run data workloads wherever the data is stored. It’s more cost effective, provides more flexibility, and solves the age-old data silo problem. With Databricks’ launch of LakeBase, this architecture is now available for Postgres as well. With this, the need to move data across platforms is quickly going away. The ETL market is shrinking, data movement is becoming more natively built-in into data platforms (eg Snowflake Openflow and Databricks Lakeflow) and zero-copy is the new norm. Is this why Informatica sold?
- The age of multi-structured data has arrived: Structured and unstructured data used to be managed separately — those times are over. AI applications and AI agents now combine structured and unstructured data. Data platforms are quickly expanding their capability to process this multi-structured data. But governing unstructured data is an unsolved problem. Organizations need to discover, catalog, curate, monitor, govern access, and ensure the quality and trust of unstructured data, and do it at scale. Everything that has been built for structured data over the last 10 years now has to be built for unstructured data.
- The modern data stack is dead: Massive venture capital has flowed to the modern data stack. Every micro-category, from catalog to transformation, orchestration, observability, access and lineage… now has a long list of small point-products. Very few of these vendors have any scale. More importantly, it’s not what organizations are looking for. The result is a fragmented, expensive data stack that’s hard to manage and doesn’t deliver enough business value. Some of these vendors will continue to consolidate (we’ve seen 4 catalog companies be acquired in the last few months), and many will go away. Organizations need simpler data stacks based on fewer, but broader platforms that maximize business value for the lowest TCO.
- The value of data is moving up the stack to the semantic and governance layer: At Snowflake’s conference, Sam Altman said “the perfect AI is a very tiny model with superhuman reasoning and 1 trillion tokens of context.”At Databricks’, Ali Ghodsi said that whoever owns governance in the enterprise owns the future, because of AI. We agree! For AI and agents to work in production, context is king. For data, that means business context must be codified through a semantic layer. The semantic layer combines rich business context with active, executable governance. We are seeing it with customers — data products are quickly becoming the best-practice to deliver trusted data, and knowledge graphs are seeing a resurgence. The major business application platforms also understand the value of their business semantics: SAP is going all-in on data products with Business Data Cloud and Salesforce is betting on Data Cloud to help deliver on its agentic vision.
The big question is – where does this take us?
1.The fight for data and AI workloads intensifies.
As data platforms continue to decouple storage from compute, the fight for that compute is only intensifying, especially as compute workloads move from data to AI. The recent announcements at Databricks’ and Snowflake’s user conferences made that very clear. It is highly unlikely that Microsoft, AWS and Google will just give those data and AI workloads away.
In this fight, governance is quickly becoming a key differentiator. AI is driving an exponential increase in data consumption while fueling new applications that are often autonomous. This requires having confidence in that data to run these workloads at scale in production. It’s no surprise that Ali Ghodsi started his keynote with governance. We fully agree with his message that “governance needs to go across all data, and needs to be unified across every use-case”. All of this is a good thing for organizations as competition drives innovation and cost efficiencies.
2. The risk and cost of governance fragmentation.
History is repeating itself. The same years of pain we lived through dealing with data silos and data fragmentation will now move up the value chain to governance silos and governance fragmentation. As data and AI workloads are deployed across multiple platforms, each with their own governance features of variable maturity, fragmentation becomes almost a certainty. The need for Unified governance across all data, every use-case, made usable for all users can quickly become a nightmare. The stakes with respect to quality, security, privacy, visibility, and efficiency are too high for governance to be an add-on or an after-thought.
Luckily, we know the solution. We’ll deal with governance fragmentation in the same way we dealt with data fragmentation – by decoupling governance from compute. Databricks has led the industry by using open standards to deliver an architecture that is more scalable, more cost-effective and more flexible. Their approach won! At Collibra – we’re doing the same: we’re building a tightly integrated, but decoupled unified governance layer for data and AI, to deliver a more scalable, more cost-effective and more flexible data estate. We’re also equally leaning into open standards such as OpenLineage and Open Data Contracts to make this a reality.
At Collibra, we continue to innovate with intention — to help our customers confidently use data and AI, anywhere, at any scale. It’s never been a more exciting time to be in data!
