What is the most common conflict in data engineering?

Data Quality and Quantity both play a crucial and conflicting role for data engineers. It's never recommended to compromise on data quality for quantity, but as we as data engineers are expected to ingest a huge volume of data which is error free and meets all the quality standards. Rigorous methods to achieve this in a timely manner is a challenge to ensure data is appropriate and consistent and well validated before handing over to the end user. This can be well handled by defining clear objectives and educating the stakeholders with the best practices. Ensuring quality and quantity compliments their existence can be ensured by following data governance standards, adapting to continuous monitoring and improvement within the systems.

Data leads to decisions and compromising your decisions with bad quality of data will always lead to bad decisions. But on the other hand, good quality data which is too small in size is also difficult to make a decision. It is very important to have a right trade-off but priority should be always clean data as this is what leads to business decisions and strategy building. We say data is the new oil because it needs days of sourcing, mining, refining and then transporting to make it suitable to use. Same goes with the data.

Existem clientes e necessidades diferentes e para lidar com cenários distintos e por vezes complexos é recomendado um desenho de solução mais elaborado que possibilite mais de um caminho para os dados com critérios diferentes de verificação de qualidade dos dados e robustez da arquitetura. Um cliente que realiza acompanhamento criterioso de um mercado, por exemplo um órgão regulador como Banco Central do Brasil, precisa que os dados entregues sejam de altíssima qualidade no tempo certo. Já clientes que trabalham com indicadores de score ou indicadores baseados em análise e modelos estatísticos podem aceitar um pequeno percentual de dados com qualidade inferior e descartá-los.

Scalability vs Cost: The clash between scalability and cost is like walking a tightrope between ambition and budget. Building a system that can handle expanding data loads without bankrupting the company requires a delicate balance. Maybe it's finding more efficient algorithms to process data, or perhaps it's cleverly leveraging cloud services to scale up only when needed. Engineers must innovate to optimize scalability while keeping a watchful eye on expenses. As for data engineers, mastering this balancing act is the ultimate thrill, like conquering the high seas of scalability while keeping a keen eye on the treasure chest of cost efficiency.

One of the most common conflicts in data engineering revolves around the trade-off between data quality and data quantity. Data engineers often face the challenge of balancing the need for comprehensive data sets with the imperative to ensure data accuracy, consistency, and reliability. Sometimes, stakeholders gives importance to gathering large volumes of data to fuel analytics, ML models, or BI initiatives. This push for data quantity lead engineers to prioritise data collection and aggregation efforts, at the expense of quality. However, collecting and processing vast amounts of data without adequate quality controls can result in inaccuracies and biases that undermine the reliability and trustworthiness of analytical insights.

Data governance vs data democratization: * Conflict: Data governance aims to establish controls and policies to ensure data accuracy, security, and compliance. On the other hand, data democratisation seeks to provide broader access to data across the organisation. Balancing governance with democratisation can be challenging, as increased access may pose risks to data integrity and security. * Resolution: Define clear data governance policies and guidelines. Implement role-based access controls to regulate data access based on user roles and responsibilities. Foster a culture of responsible data usage through training and awareness programs to align both governance and democratisation goals.

In general, the common problem of the conflict is about how data will be available to work; but the root of the conflict could be based in 2 points: 1) The business want to be productive over anything: don't want to wait. 2) No understand the impact of bad usage of data, processes or working platform. Like data engineering team, you must check that all stakeholders give its approved to the solution design will implement before do it to check they solve these points: this is part of development lifecycle. These stakeholder can give their approved: data steward (owner of data), business representation (consumer of data), security team (keep it sure), legal team (avoid legal problems) & data governance Analyst (comply with policies).

Alavancar o uso dos dados nas organizações está diretamente relacionado ao conhecimento das áreas de negócios sobre os dados, sua capacidade de gerar insights relevantes, a disponibilidade e acuracidade do universo de dados necessários. Do meu ponto de vista, dentre os pontos citados, o mais importante para acelerar a democratização de dados é a capacitação. Esta dará autonomia e conhecimento para as áreas de negócios. Em contrapartida, habilitará recursos que permitam a criação de silos de dados, gerando assim ofensores a governança. O monitoramento do uso dos dados e a definição de políticas de acesso serão capazes de garantir um "limite seguro" entre autonomia e a "ordem dos dados".

Pick democratization first and gradually implement governance as needed. Your beautifully governed data is useless if nobody is using it. Data engineers should stimulate data literacy in their colleagues. Help them find the right data, query it, and create visualizations. Anno 2024, working with data is an essential skill for everyone - not just people with "data" in their titles. When everyone in your company sees the value in data-driven work, it's much easier to get them on board for solid data governance.

Data(lake/house) serves mostly Data Scientists. They usually want it all. Always. On demand. From other side, there are security, governance, GDPR, <you name it> considerations. Hence we have emerging technology solutions, such as Ranger, Iceberg, CloudFormation, etc. , which allow (or would allow in relatively near future) to separate concerns. Let’s put it straight. Security considerations are there. No matter state or business. The more it is transparent and fluent in the triangle of Data-User-Security, the more efficient will be process in general. And we are going there in general.

Data engineering is the backbone of data science. People often give too much importance to data science but they forget it cannot learn and cannot work without the data engineers. Analogy is , If data science is brain and data engineering is the body. both complement each other and useless without each other

Let's simplify it: Data engineering is about setting up systems to collect and organize data, like building the pipes for water in a house. For example, think about setting up databases for a ride-sharing app like Uber to keep track of all the rides people take. Data science is about analyzing that data to find useful information, like being a detective looking for clues. So, for Uber, data scientists would use the data to figure out things like how much to charge for rides in different areas or when people are most likely to need a ride, making the app work better for everyone.

Conflicts between data engineering and data science often arise from differences in priorities, tools, communication, data quality, and project ownership. Data engineers focus on infrastructure and scalability, while data scientists prioritize insights and model accuracy. Bridging these gaps requires effective communication, clear roles, and shared goals, enabling organizations to leverage both disciplines for successful data-driven initiatives.

I think the most common conflict thing about data engineering is defining the role responsibilities itself. some company combine a 3 roles responsibilities under the umbrella of data engineering. I saw many juniors data engineers who do the ingestion phase, the transformation phase and then do the work of data analysis and data science. Also the relationship between data teams needs to have more effective communication, as data engineers are the backbone of any data solution for each company, they provide data for end users such ad DA, DS and ML engineers. to have clear communication between these teams and clear SLA is crucial to any company which wants to be data-driven company.

Imagine a government where data engineering is akin to infrastructure development and data science is the policy-making body. To serve the common people (business), roles and responsibilities are defined – infrastructure builds the roads (data pipelines), while policymakers (data scientists) design effective governance. Cross-functional teams (govt. departments) collaborate, using shared tools (govt. platforms) and common language. Agile governance adapts to evolving needs, ensuring efficient service delivery. Leadership (govt. officials) promotes a culture of teamwork, recognizing each organ's value. By this, like a well-coordinated government, data engineering and data science can jointly serve the common business interests effectively.

In my experience, proactive communication and early involvement of all stakeholders in the planning phase can prevent many data engineering conflicts. For example, when our team integrated a cross-departmental data flow, we held workshops to align everyone's understanding and expectations. This preemptive step helped us uncover potential issues and address them before they escalated into conflicts. I agree with the need for a robust data governance strategy, but it's also essential to have a rapid conflict resolution mechanism for when issues do arise, ensuring they're handled efficiently and constructively.

Effective communication and collaboration with stakeholders is key to navigating data engineering conflicts and finding the right balance for each situation.

Establishing clarity in roles, goals, and processes while emphasizing data literacy and ethics, fostering collaboration, and promptly addressing conflicts can significantly mitigate data engineering conflicts.

Dentro da função de data engineer e existem alguns papéis: aquele que é mais próximo da área de negócio, num papel muito próximo de um analytics engineer mas direcionando ingestões específicas para o negócio! Tem outra persona que atua na parte de governança dos dados e outra na parte de suporte as ferramentas utilizadas para ingestão e análise! O segredo para evitar os possíveis conflitos citados, está na padronização dos processos e recursos! Uma área de plataforma de dados por exemplo, pode ser a responsável em dar esse direcionamento pois é a área que é capaz de pensar e oferecer soluções que o capítulo de engenharia necessita para o dia a dia!

Data Governance is a Key : With out governance, it lead to inconsistencies in key data and its definitions, makes it potential for bad quality. Data democratization will help users to explore more and make biz. decisions faster. Data Quality Vs Quantity: Data quality is a critical aspect and poor data quality can lead to inaccurate results on biz. decisions. Best practices to overcome : To have a data governance strategy in place. Implement automated data validation and cleansing processes to ensure data quality. Implementing security measures such as access controls and encryption to protect sensitive data. To add more roles like data steward, data governance analyst, data security professionals etc.

In the realm of data engineering, the impact of storytelling and human-centric approaches is profound. By incorporating user stories and feedback into the design and execution of data projects, we foster a deeper connection between technical teams and end-users. This not only humanizes the data, making it more relatable and impactful, but also encourages a more collaborative and inclusive data culture. Emphasizing the real-world applications and benefits of data work can transform technical challenges into opportunities for innovation and meaningful engagement.

Consider what's more important: data availability or processing resiliency. For the latter, a decoupled architecture is often the best, where data ingestion and data persistence or consumption are completely isolated from each other via some sort of store and forward mechanism, often using streams. The issues on one do not impact the other, making the overall pipeline resilient. However, it also makes the data less immediately available and makes it hard (and often impossible) to sequence properly which is needed for transactional systems. There is no perfect solution; you just need to decide which one has priority and then design the data engineering process accordingly.

Balancing Other Priorities: Data engineers juggle many other trade-offs besides data quality and quantity. Here are a few examples: - Performance vs. Cost: Balancing how fast data pipelines run against how much they cost to maintain can be tricky. - Data Timeliness vs. Data Quality: There can be pressure to get data to users quickly, even if it means sacrificing some quality checks. - Scalability vs. Maintainability: Building systems that can handle growing data volumes is important, but complex architectures can be hard to maintain over time. - Centralization vs. Decentralization: Decentralized data storage can be faster for some use cases, but centralized storage offers better governance and control.

Another major conflict that data engineering teams face is with the business stakeholders. No matter how hard you try, there always remains some gap between what the business or the downstream application team requirements and the data your team provides. What worked best for me is to keep a tight coupling between the business stakeholders and data engineers involving them in regular UAT approvals. This eventually saves a lot of time for both the business well as the developers significantly reducing these conflicts.

Evolving Technologies:  The rapid pace of change in the data technology landscape can create challenges. Data engineers might need to convince stakeholders to invest in new tools and platforms while also acquiring the necessary skills to use them effectively.