Data Quality & Observability

Data quality is a measure that refers to whether data is fit for use to drive important business decisions. Measuring data quality can help business analysts and data scientists decide whether the data they have access to is suitable for use in decision-making or if any errors must be fixed before the data can be further processed.

For data to be considered high quality, it must be consistent, unique, valid and complete. The data must be relevant to the organization, easy to reference and reflect the real-world needs of the business in terms of what data is collected and how it's formatted. As businesses become increasingly digital and find themselves collecting data on multiple platforms from disparate sources, it becomes increasingly important to ensure all the data is in a consistent format and easy to cross-reference or integrate. Data remediation tools and data quality monitoring solutions assist with this process, enabling companies to truly utilize the power of the data they collect.

Data quality is a complex issue. Every organization has differing needs, but there are common strategies that can help improve quality.

• Formal data quality dimensions, rules and thresholds can help you ensure consistency and quality across data sources.
• Data observability capabilities can help you proactively identify and address issues such as schema changes, missing data, duplicate data and missing records.
• Data lineage can help you identify the causes and downstream impacts of data issues and notify relevant stakeholders.
• Mapping data quality scores to data catalog assets and policies can help you validate the compliance of your data quality with your policies.
• Mapping data quality scores to data catalog assets and policies can help you validate the compliance of your data quality with your policies.

Collibra Data Quality and Observability can help you with these activities and be tailored to your specific needs.

Data quality is measured using a variety of metrics including:

• Accuracy
• Completeness
• Timeliness
• Duplication
• Validity
• Consistency
• Uniqueness
• Availability
• Lineage

High-quality data should have been accurate when it was collected and recent enough that it's still likely to be current. Records should be complete and have a unique identifier, so they're easy to reference. Duplicate data should be avoided whenever possible, and pieces of data (such as addresses or dates) should be in a consistent format for ease of comparison. Data stores should be consistently accessible, and there should be a way of tracing the lineage of each record to assist with diagnosing quality issues.

Data quality focuses on measuring and maintaining the reliability and accuracy of the data. In contrast, data governance focuses on data asset management, control and utilization. Maintaining a high level of data quality is required so that you can make the most out of the data that your organization collects.

Data governance is essential to ensure the data you gather is stored and processed in a way that's compliant with local or national regulations and prevents data breaches or misuse of data. The two practices are closely related, and large organizations commonly use data governance and data quality tools.