You're responsible for data quality. How do you make sure your governance tools are up to the task?

Our mission is to enhance data infrastructure and promote data-driven decision-making. Before we begin developing a comprehensive data governance system, we must conduct a thorough assessment of our data quality and maturity levels. This evaluation, based on best practices and well-established principles, will help us understand our strengths and areas for improvement. Armed with this knowledge, we can align our governance strategies with our broader objectives and goals, ensuring that data governance is not just a compliance requirement but a driving force for data-driven success.

Data Quality and Data governance aren't the same. Data Governance is about having standards, controls and right policies in place which fundamentally is a key factor in enhancing overall maturity of the data quality for an enterprise. Data governance tools serve an entirely different purpose to that of Data Quality. Irrespective of what tools are used to implement Data Governance and manage Data Quality, they are simply enablers of the overarching policies, standards and business requirements/rules around critical data. Hence, before embarking on choosing the right tool, it is imperative to identify and formalise critical data elements, rules and standards that they should abide along with data quality requirements.

NASA’s Data Maturity Framework is a great example of a successful method to measure data quality maturity. The framework itself helps determine if a product is ready to launch or scale.

The primary outcome expected from a data governance program is Data Quality, essential for fostering reliable data and improving decision-making processes. Conducting a maturity assessment helps in understanding your organization's standing in this aspect. While standard Data Quality maturity models may not align perfectly with your organization's approach, they provide valuable reference points. By customizing criteria and indicators, you ensure relevance to your specific context. Assessments offer clarity on your current position, guiding strategic planning for improvement. They pave the way for defining objectives, methods, and timelines, thereby creating a roadmap for enhancing Data Quality.

Tools are tools, data quality maturity does not happen until the organization itself understands the value of data and becomes a data drive culture. However, what I would suggest before trying to adopt/ adapt a tool, is to ensure the right processes are in place to start improving data and to see if possible, there is a way to govern the data intake process.

a possible approach for defining and implementing data quality requirements: 1. Identify Key Data Quality Dimensions 2. Consider Data Characteristics: - Analyze your data sources, types, formats, volumes, and flows to tailor requirements accordingly - Differentiate between critical and non-critical data elements 3. Define Roles and Responsibilities 4. Develop Processes for Data Quality: - Outline processes for data profiling, cleansing, validation, and monitoring - Establish workflows for addressing data quality issues promptly and efficiently 5. Align with Business Objectives 6. Implement Monitoring and Reporting 7. Continuous Improvement: - Implement a feedback loop - Regularly review and update data quality requirements

A Data Governance tool will not do much good on its own, rather it has to be engineered based on the data quality requirement and rules specific to meeting the identified gaps from the maturity assessment. A well-defined and captured requirement also forms the basis for measuring how effective the data governance tool is. Continuous monitoring will further show areas where the requirements need to be modified, rules tweaked and the data governance tool fine-tuned for better results. It is safe to say that how well a data governance tool performs is highly dependent on the data quality rules defined as well as other processes established within the context of the organization where it is implemented.

Defining data quality requirements is crucial. It’s about aligning these goals with your organisation’s strategic objectives. For example, a healthcare provider realised that their main focus should be on data security to protect patient information.

Many companies don’t have a solid view of the impact of data quality issues (data downtime) on their operations and digital initiatives. If you’re in that situation it can be useful to perform a data quality vulnerability assessment first, creating a map of direct (eg single points of failure in a crucial pipeline) and indirect (eg. organizational flaws) vulnerabilities to data quality problems.

Clearly articulating your data quality requirements is a pivotal step in ensuring your governance tools are effective. This involves setting specific standards and metrics for data accuracy, completeness, consistency, and reliability. These requirements should align with your organization's overall objectives and data strategy. By defining these criteria, you can better evaluate and select data governance tools that meet your specific needs.

These are not really functions data governance or its tools: data profiling, data cleansing, data validation, data monitoring, data lineage, data catalog, and so on. DAMA DMBOK 2.0. Data Governance represents an inherent separation of duty between oversight and execution. Data Governance is exercise of authority, control and shared decision-making over the management of data assets. Tools that execute data profiling, data cleansing, data validation, and such are data quality management tools. Tools that execute data lineage tracking, data asset cataloging, dictionaries, and similar are metadata management tools. Governance makes decisions on and monitors how these tools are applied -> oversight but not execution of management.

Lo primero es definir lo que espera la compañía por un dato gobernado, desde el punto de vista técnico y de negocio. Realizar una mesa colavorativa con el equipo de gobierno y los stakeholders correspondientes. Teniendo un catastro de lo que se espera de la gobernanza de datos, realizar una matriz con los posibles proveedores considerando dimensiones tales como: -Precio (instalación, licencias, ambiente DEV, entre otros) -Tiempo aproximado de instalación (considerar otras herramientas de la compañía que puedan integrarse con la herramienta) -Funciones de gobierno: roles de datos, gestión de dominios de datos, glosario de negocio, indicadores, extracción de metadata de las BBDD, perfilamiento de datos, reglas de calidad, umbrales, otros.

When tasked with ensuring data quality, it's crucial to verify that governance tools are up to the task. This involves comparing different data governance tools available in the market. By conducting a comprehensive comparison, considering factors such as features, scalability, user-friendliness, and compatibility with organizational needs, data professionals can make informed decisions to select tools that best align with their data quality objectives.

When exploring data governance tools, note Collibra's standing as a top player, excelling in comprehensive data governance. Alation, however, stands out with its open DQ framework, seamlessly integrating with diverse DQ solutions. Each tool brings unique strengths—Collibra for holistic governance, and Alation for adaptable DQ frameworks. Embrace a mindful approach, acknowledging each tool's varied strengths, and ensuring the chosen one aligns harmoniously with your organization's specific data governance needs.

Al momento de decidir qué herramienta utilizar para nuestra estrategia de gobierno de datos, debemos comparar diferentes opciones, evaluamos características como perfilamiento de datos, limpieza, validación, linaje, catálogo y diccionario de datos, glosarios de negocios, etc. así como herramientas de administración y que todo se encuentre centralizado, facilitando la integración y uso de la misma. También es importante considerar aspectos como funciones, beneficios, costos, escalabilidad, integración, seguridad y facilidad de uso, para ello podemos utilizar una matriz de comparación y un sistema de puntuación para clasificar las herramientas y reducir nuestras opciones.

To ensure that governance tools meet the demands of maintaining data quality, a rigorous testing and selection process is essential. Data professionals should subject potential data governance tools to thorough testing, evaluating their performance against predefined criteria. This process helps in selecting tools that align with the organization's data quality requirements, ensuring their effectiveness in maintaining and enhancing data quality.

Conducting pilot tests, seeking feedback from stakeholders, and reviewing case studies or references from other users can provide valuable insights into the effectiveness and suitability of different data governance tools for the organization's use case. Ultimately, the goal is to select a tool that empowers the organization to establish and maintain effective data governance practices, enhance data quality and integrity, and support data-driven decision-making across the enterprise.

Before fully implementing a tool, it's crucial to conduct thorough testing to ensure it meets your data quality and governance needs. This might involve a pilot phase where you test the tool’s functionality, assess its integration capabilities with your existing systems, and evaluate its ease of use and effectiveness. Based on the results of these tests, you can make an informed decision on which data governance tool is the best fit for your organization.

My personal takeaway from my experience is the importance of looking beyond the surface-level features and functionalities. The success of a data governance tool lies in its ability to adapt to the intricacies of your data landscape and the support it offers. Consideration of factors such as maturity, industry recognition, and vendor support ensures a more holistic and effective approach to selecting data governance tools that truly enhance data quality.

After selecting the right tools, the focus shifts to effective implementation and optimization. This involves integrating these tools into your current data systems, ensuring they work seamlessly with your processes and workflows. It's important to continuously monitor their performance, making adjustments as needed to maximize their effectiveness. Proper implementation and ongoing optimization of these tools are key to maintaining high data quality standards in your organization.

Once the designated individuals have been assigned as data owners and stewards, and the scope of the data governance initiative has been accurately outlined, along with the selection of pertinent metrics aligned with business objectives to monitor data quality implementation and progress, the final step involves ensuring the chosen tools are compatible with the enterprise ecosystem. This entails verifying that the selected metrics can be effectively tracked using these tools and ensuring comprehensive training for the company to adeptly utilize them.

Making sure governance tools are up to the task of maintaining data quality involves not only selecting the right tools but also implementing and optimizing them effectively. After the selection process, organizations must implement the chosen data governance tools and continuously optimize their usage. Regular reviews and adjustments are necessary to ensure that these tools evolve with changing data quality needs, maintaining their relevance and effectiveness over time.

In my experience, requirements change over time due to changes in business structure, data needs, or regulatory requirements. This means the initial data quality requirements are bound to change at some point. A corresponding change to the tool in use or its implementation will then be required. It is therefore necessary to put in place methodologies for continuous monitoring from within the organization. This will ensure the data governance tools are managed and kept up to date and relevant to the business needs.

In my experience evaluate how well the tools tackle issues through effective data profiling and cleansing. See if they provide a clear path to trace where your data comes from and where it goes – like a digital map for your information. Consider how easily your team can understand and use the tools, as user-friendliness is key. Based on my journey, regularly reassessing and upgrading these tools is crucial to keeping pace with the evolving demands of maintaining high-quality data.

The feedback is the goal of a quality process. In data context, could be the difference between the company data environment be a value or a cost source.

Data Quality is everyone's shared responsibility, yet in many organizations, data engineers are expected to clean and maintain poorly written SQL or fix pipelines they didn't break. No one wants to serve as the company's designated 'data janitor.' A path forward: tackle data at the source (proactive) and provide tools to improve tools to maintain data quality without this is an all-consuming-job (Data quality co-pilot can help here)

There are many aspects to Data Quality but I would narrow it down to two 😊 : - Technical controls: in a nutshell, anything happening during the ETL process to harmonize, normalize or enrich data (data format, numbers …) - Business controls: where intelligence resides. For instance, age of a policy subscriber should => 21 and address should be filled out along with social security number. This is where a business centric data catalog will allow any part of the business to document their quality controls in natural language. Then the IT team takes over converting this in code/controls. The results are then sent back to the DataCatalog for business information or remediation. In conclusion, you need both to be truly efficient.

Identify the data with the highest impact and work on improving data quality that has business impact. Data lineage needs to be a living process and must be updated as systems and processes change. Ability to track and audit data lineage should be available on demand and be built into the data quality solution. Business rules both pre-built generic & domain specific are an important part of the data quality process and should be defined by the business users. Improving data quality is an ongoing process – not just when the data is initially loaded. Data is constantly changing. Data profiling solution that can post daily, weekly or monthly reports automatically, after each run

It’s not just about tools! It’s better to establish data quality practices as part of your data operating model, then choose the right DQ tool. DQ will always need tooling, as you need to profile and measure/monitor data quality and show trends over time, but the tool will not help you if you don’t have accountability for issue management and data remediation.

The question is written as if to suggest that data quality is one person’s responsibility in a company. In fact, everyone in an organization has a role in contributing to data quality to better inform decision making. From defining data quality to collecting, storing, analyzing, managing, and using quality data, everyone in the organization has some role in building a robust organizational culture and infrastructure for data collection and management. It is important for those responsible for managing data governance to work with the entire organization to understand good data practices that can improve and sustain data quality. We work mostly with the data generators and users to help improve our data governance tools and policies.