Accelerating enterprise AI with high-trust outcomes

The enterprise race to productionize generative AI has entered a new phase: the shift from passive LLM experimentation to autonomous, agentic systems. Today, organizations are deploying AI agents capable of executing multi-step workflows, generating native code and taking real-world corporate actions. Yet, as these systems scale, an uncomfortable truth has emerged: without a unified semantic layer, autonomous agents don't solve our data consistency problems—they amplify them.

Without verified enterprise context, autonomous agents hallucinate, misuse sensitive data and pull conflicting definitions from fragmented systems. The business risk is material; an unguided agent operating outside corporate controls represents an immediate compliance and financial liability. Highlighting this systemic gap, Gartner projects that by 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance failures identified only after production incidents occur.

To eliminate this friction and empower organizations to scale production AI with absolute transparency, traceability and control, Collibra and Databricks have significantly expanded their strategic partnership. We are also deeply honored to announce that Databricks has named Collibra its 2026 ISV Data Governance Partner of the Year.

This recognition highlights our shared technical vision: while Databricks provides the ultimate high-performance processing engine for enterprise data and AI, Collibra delivers a unified control plane that governs context across the entire enterprise ecosystem, synthesizing structured data from SAP, BI tools and databases, alongside unstructured content from platforms like SharePoint, to make this trusted intelligence available for Databricks agents.

The architectural challenge: Platform silos vs. the enterprise control plane

Modern enterprise governance cannot stop at the boundaries of the lakehouse. In a heterogeneous data landscape, key business metrics and core data pipelines are inherently distributed across multi-cloud environments, BI dashboards like Tableau, PowerBI, legacy mainframes, CRMs like Salesforce and ERPs like SAP.

Platform-native catalogs are highly effective at managing technical assets within their own compute boundary. However, when an enterprise relies solely on siloed catalogs, governance fragments. The same financial metric or data contract can be defined differently across separate platforms, breaking downstream trust and confusing autonomous models.

Furthermore, non-engineering stakeholders—including data stewards, privacy officers, legal counsels and risk managers—do not work inside developer-centric code catalogs; they require purpose-built workflow environments to orchestrate enterprise policy, track regulatory compliance and audit data lineage from source to the boardroom.

Collibra complements Unity Catalog by enriching it with an overarching enterprise context layer. By serving as a platform-agnostic trust layer, Collibra synchronizes business definitions, quality scores and compliance metrics directly into Databricks. This ensures that the exact same governed understanding travels seamlessly across the full lifecycle of an AI model or data product.

Three stages of enterprise context

To seamlessly weave corporate governance into active data execution, Collibra has engineered a multi-tiered maturity model that transitions metadata from a passive catalog asset into an active, runtime guardrail. This strategy progresses through three core architectural stages: Connected Context, Actionable Context and Embedded Context.

Stage 1: Connected context (bi-directional Unity Catalog sync)

The foundation of the integration relies on a continuous, automated metadata exchange between Collibra and Databricks Unity Catalog. This bi-directional loop unifies the technical metadata gathered from active execution with the logical context curated by business stakeholders.

Core architecture: Powering the bi-directional exchange

At the center of the expanded partnership is a new, bi-directional integration that keeps Collibra and Databricks continuously aligned. Governed metadata, semantic definitions and policies move from Collibra into Databricks, while technical metadata, lineage and observability signals move from Databricks back into Collibra.

This integration ensures that Collibra’s business context and governance are instantly available within Databricks, giving users direct access to critical information like data sensitivity and ownership without switching platforms. It streamlines compliance and empowers responsible data use at the point of consumption. Syncing technical metadata and lineage back to Collibra delivers holistic oversight and traceability, even for users without Databricks access, resulting in unified governance and a consistent view of data across both environments.

• Governance and policy enrichment (Collibra → Databricks): Business descriptions, data ownership, data quality certifications and regulatory classifications (e.g., GDPR, PII) authored in Collibra are automatically pushed down into Unity Catalog. This ensures that business context is visible natively inside Databricks at the exact point of data consumption.
• Automated traceability (Databricks → Collibra): Technical metadata, structural table alterations and runtime column-level lineage are ingested directly from Databricks into the Collibra AI Command Center. This automated collection allows risk managers to verify end-to-end data provenance, tracing corporate metrics out of the lakehouse and back through upstream operational systems.
• Operational trust from Agent Bricks into AI Command Center (Databricks → Collibra): The AI Command Center already gives teams a trusted, continuously governed view of every AI agent—and now it captures how agents actually behave in production, too. By bringing live operational signals from Databricks Agent Bricks into the AI Trust Score, it enriches existing governance and assessments with real-time evidence of agent quality: daily pass rates from LLM judges checking groundedness and safety, AI UC-1 assessment status, and token-consumption trends that catch agents drifting off the rails before they cause harm. A fleet-wide dashboard surfaces what matters most—average Trust Score, overall pass rates and unmonitored agents—so leaders spot risk early and scale AI with confidence
• Access governance (Collibra - Databricks): Stewards can construct plain-English privacy and data entitlement policies inside Collibra. The integration automatically maps and enforces these rules down into Databricks as native access controls and dynamic data masking parameters.
• Semantic enrichment for AI (Collibra → Databricks): Data specifications, business context and semantics governed in Collibra can be deployed into Databricks to enable business-ready metric views and to ground Agent Bricks agents in verified, enterprise-approved definitions. AI systems inherit the curated meaning of the data, improving accuracy and reducing the risk of misinterpretation in natural language analytics and autonomous decision-making.
• Unstructured data ingestion via Deasy Labs: Unstructured text files (PDFs, call logs, corporate documentation) represent a massive volume of modern AI ingestion. Deasy Labs (a Collibra company) automatically qualifies, deduplicates and evaluates raw files for safety and freshness before they land in the lakehouse. It then extracts an inherent structural taxonomy from the unstructured content and writes it directly to Unity Catalog as the schema for the corresponding Databricks volume, capturing data anomalies or sensitive content before it enters the model pipeline.

Stage 2: Actionable context (the Model Context Protocol server)

As enterprises transition from static analytics to autonomous workflows, context must be accessible in real time at model runtime. Passive metadata documentation cannot prevent an active agent from making an erroneous operational decision.

To solve this, Collibra has introduced its Model Context Protocol (MCP) Server natively available within the Databricks Marketplace. MCP acts as an open standard protocol that exposes curated governance data as structured API tools that autonomous systems can query on-demand.

Core architecture: Powering the MCP interactions:

When a Databricks Agent Bricks agent is triggered to perform a business task—such as compiling an executive financial statement or calculating customer churn—it does not query raw tables blindly. Instead, the supervisor agent calls the Collibra MCP Server tool.

The agent passes the requested attributes, and the MCP Server returns real-time context validation: verifying the mathematically approved definition of the KPI, retrieving the latest data quality health scores and enforcing active access boundaries. This architecture provides a robust, programmatic guardrail that forces autonomous agents to operate strictly within enterprise parameters.

Stage 3: Embedded context (governed Genie Spaces & Lakebase integration)

The final stage of our tech strategy represents a deeply embedded state where Collibra’s enterprise context layer is infused directly into the user interfaces and native storage paradigms of the Databricks environment.

Core architecture: Powering the embedded context:

• Governed MetricViews and Genie Spaces (coming soon): Databricks Genie enables users to query data lakes using conversational, natural language. However, natural language interfaces are highly prone to semantic ambiguity. By injecting Collibra's curated Business Glossary and Business Context for MetricViews and directly into Genie Spaces, we provide the foundational semantic layer required to anchor conversational queries.
• Collibra Insights integration with Databricks Lakebase (coming Oct/Nov): To bring governance analytics closer to where data engineers and data scientists naturally work, Collibra is developing an integration with Databricks Lakebase. This architecture pipeline will replicate historical Collibra analytical and operational data directly into your native Databricks Lakebase environment without degrading the operational performance of your main Collibra cluster. Data teams can immediately join Collibra governance metrics with native Databricks tables to generate custom quality dashboards, track internal data compliance KPIs or leverage Agent Bricks to discover hidden process bottlenecks.

Move from AI experimentation to production reality

The expanded engineering integration between Collibra and Databricks directly solves the toughest blocker preventing modern enterprises from scaling generative and agentic AI systems: the structural gap between rapid engineering execution and institutional governance. By embedding automated traceability, shared semantic intelligence and cross-platform visibility directly into Unity Catalog, Genie and Agent Bricks, we provide a clean, auditable path to true enterprise-grade AI production.

Join us at the Databricks Data & AI Summit (DAIS) 2026:

• Visit the live demos: If you are on the ground at the summit in San Francisco, visit the Collibra booth #656 to see live, end-to-end demonstrations of our bi-directional Unity Catalog synchronization, Governed Genie Spaces and live Agent Bricks workflows with our MCP Server.
• Partner of the Year celebration: Come meet our technical engineering teams and product experts to celebrate our recognition as the 2026 Databricks Data Governance Partner of the Year.
• Apply for private preview access: The advanced integrations outlined above are currently available for private preview participants. To request immediate pilot onboarding for your organization's data estate, please contact your dedicated Collibra or Databricks account team.

To explore deeper technical architecture guides, customer implementation case studies and reference deployment blueprints, visit our dedicated Databricks and Collibra Partnership Hub.

The era of Data Intelligence is here. Let’s build it right, together!

See you in San Francisco!

• 

  Collibra

  Collibra

  Enterprise AI Control Plane