From fire drills to faster, secure and auditable remediation

Share on:

Critical quality issues often stay hidden until they hit your bottom line. Forrester warns that inaccurate data fuels poor decisions costing organizations between $5 million and $25 million annually. As complexity scales, reactive data quality is no longer an option. Ensuring every record is trustworthy isn’t just a governance checkbox, it’s a prerequisite for operational efficiency and protecting your brand's reputation.

What’s new: Archive break records for Data Quality and Observability

Collibra is proud to introduce break records archiving, a critical enhancement to our Data Quality & Observability (Cloud) offering designed to provide flexibility and scalability for data remediation. This new feature enables the permanent storage of records that fail specific data quality rules directly within the customer's data source where processing occurs. Unlike traditional live previews that offer a fleeting glimpse of current errors, this creates a persistent historical record of exactly which rows failed a DQ job and why.

By allowing users to archive these "broken" records, Collibra facilitates a more robust remediation process. This launch ensures that quality issues are documented and accessible for long-term reporting and auditing, rather than being limited by the temporary memory of a query result. This persistent storage is the first step toward a metadata-enhanced break record archive that will eventually aggregate issues across all monitors and connections into a single source of truth.

How this feature helps

Managing data quality at scale often feels like a race against time, especially when technical limitations prevent a full view of failure points. Organizations frequently struggle with limited metastore capacity or the inability to export full rule records needed for complex remediation. We solve these challenges by moving the storage burden away from the platform and into the scalable environment of the customer’s choosing. This feature helps:

Eliminate remediation bottlenecks: Provides full rule records needed to fix source data rather than just limited previews
Scale with your data: Removes metastore limitations by archiving records directly to the data source
Enable historical reporting: Supports roll-up style reporting to track how issues evolve over time in BI dashboards
Improve flexibility: Offers a scalable solution for remediating records outside the platform
Reduce manual overhead: Automates the persistence of break records, ensuring users have the information needed for remediation without manual exports

How it works

The technical architecture is built for high-performance efficiency by leveraging the same data source where Pushdown processing occurs. When a DQ job is executed on a platform like Snowflake, the feature writes the records that fail custom rules directly into a designated table—Collibra_DQ_Break_RECORDS—within that same data source. This ensures that sensitive data remains within your secure environment while still being accessible for governance oversight.

Administrators have granular control through a new toggle in the DQ connection and job settings. When enabled, the system automatically creates the break records table using a standardized schema consistent with the Platform terminology.

Integration with the broader Collibra Platform is seamless, with the feature respecting all existing user permissions for data previews. Users can access these archived results directly through the UI on a per-job-run basis or programmatically via the break records API. This API allows external systems to retrieve specific versions of failure data by UUID, supporting automated downstream remediation workflows or third-party reporting tools.

Break records visible within the application

Why you should be excited

Archive break record helps the whole business from data stewards, to data engineers, to compliance managers across numerous use cases:

Data Stewards: Gain a permanent, auditable trail of data failures, allowing you to prioritize remediation efforts based on historical impact and regulatory risk
Data Engineers: Reduce administrative friction with automated storage of failure records directly in your native data environment, such as Snowflake or Oracle
Compliance Managers: Ensure total defensibility during audits with a "clean" overview of policy adherence and persistent evidence of identified and resolved data issues

It helps these users solve for:

Source data remediation: Directly use the archived records in your Snowflake instance to trigger automated "fix" scripts that address recurring data entry errors at the golden source
Trend analysis via BI dashboards: Connect Tableau or PowerBI to the archived break records table to visualize data quality health trends and the effectiveness of remediation across different business units
Regulatory reporting: Document compliance with data integrity mandates by maintaining a versioned history of every broken record identified during a reporting cycle

Key takeaways

This launch is a foundational piece for trust by ensuring that the evidence for every rule violation is easily accessible and actionable. By bridging the gap between identifying a problem and storing the evidence needed to fix it, Collibra empowers organizations to move from reactive firefighting to proactive, automated data quality. Here are the three things you should not forget about archive break record:

Join Collibra’s Spring Product Premiere to learn how our new Data Quality and Observability feature can help you:

Persistence over previews: Move beyond temporary views to permanent, archived failure records that support long-term remediation and auditing
Secure and scalable: Keep your sensitive failure data within your own environment while leveraging the power of Collibra’s governance controls
Automated accountability: Use standardized schemas and API access to ensure that every broken record is accounted for and ready for resolution