We started working with the Dext team last year when they came to us with an interesting challenge: How to get data lineage across their multiple dbt projects at a column level and integrate with dbt. The team then debated building this in-house but quickly realized they would have to build many internal tools, such as custom alerting, testing, and lineage, and decided to bring on SYNQ. "SYNQ’s automated data lineage has been a game-changer. It has significantly reduced the time we spend tracing data lineage, allowing us to focus more on analysis and insights. It has also improved our data quality by enabling faster identification and resolution of data issues.” - Yordan Ivanov 📈 Read the full story here: https://lnkd.in/eREX77j2
SYNQ’s Post
More Relevant Posts
-
In a world where data drives decisions, ensuring its integrity is non-negotiable. My latest article delves into how DBT's testing framework ensures the reliability and quality of data transformations.
Ensuring Data Integrity with DBT Testing
medium.com
-
𝐓𝐡𝐞 𝐦𝐨𝐝𝐞𝐫𝐧 𝐝𝐚𝐭𝐚 𝐬𝐭𝐚𝐜𝐤 𝐭𝐨𝐨𝐥𝐬 𝐝𝐨𝐧'𝐭 𝐭𝐚𝐥𝐤 𝐭𝐨 𝐞𝐚𝐜𝐡 𝐨𝐭𝐡𝐞𝐫 𝐞𝐧𝐨𝐮𝐠𝐡! Modern data stack tools are fighting to push boundaries, and this fight sometimes gets them in conflict with each other. You may have heard of a recent disagreement between Tobiko and dbt Labs, competitors in the T layer, around Coalesce. In happier news, Tobiko team are working on a SQLMesh + dlt metadata handover that enables SQLMesh to generate scaffolds with built in incremental models for dlt pipelines. Why is this awesome and pushing boundaries? Because metadata handovers make open source tools highly interoperable. Open apis on those tools enable other vendors or developers to use them as de facto standards and build better pipelines. Check out the amazing work being done by Tobiko team on this project: dlt-SQLMesh generator: https://lnkd.in/eSNFBBcC
dlt-sqlmesh generator: A case of metadata handover
dlthub.com
-
Declarative data transformations at columnar level have been an elusive dream of many data engineering practitioners. They drastically reduce the complexities of modern data transformation pipelines and make them easy to understand, maintain and extend with clear columnar lineage. Explore our latest blog to discover how the DataForge Core framework makes this vision a reality.
Very excited to publish Part 2 of the Introduction to DataForge for developers! The blog builds on the innovative high-level concepts of Part 1 and shows how a fully-flushed out Class structure and framework empowers data developers to build faster and scale-out their code base efficiently. DataForge is the only platform breaking down the core problems with modern data engineering code to allow teams to grow their data pipeline architecture and keep up with the ever accelerating demands for data and insights. https://lnkd.in/e8zPgPpv Vadim Orlov
DataForge Declarative Data Transformation Part 2 — DataForge
dataforgelabs.com
-
💥 Exciting News! Part 2 of the Introduction to DataForge series is now live! 🚀 This blog dives deeper into the advanced concepts we introduced in Part 1, showcasing how DataForge’s comprehensive class structure and framework can help data developers build faster and efficiently scale their codebase. Check it out and share your thoughts!
Very excited to publish Part 2 of the Introduction to DataForge for developers! The blog builds on the innovative high-level concepts of Part 1 and shows how a fully-flushed out Class structure and framework empowers data developers to build faster and scale-out their code base efficiently. DataForge is the only platform breaking down the core problems with modern data engineering code to allow teams to grow their data pipeline architecture and keep up with the ever accelerating demands for data and insights. https://lnkd.in/e8zPgPpv Vadim Orlov
DataForge Declarative Data Transformation Part 2 — DataForge
dataforgelabs.com
-
Two weeks ago, Georg Heiler and I shared a blog post about our vision for the possible future of data processing pipelines with open source tools. We are so grateful that so many people reacted and showed that we are up for something meaningful and valuable. What is next? We want to iterate on this idea and improve. If you are interested in this topic, we would like to get in touch with you and hear your feedback, questions and possible use cases. #Dagster #DBT #DuckDB
Big time for me, my first blog post! Co-authored with Georg Heiler. We write about how Dagster, dbt Labs and DuckDB integration reshapes the current data transformation process and propose a direction for the future of modern data engineering. https://lnkd.in/dw9WKZh9
Dagster, dbt, duckdb as new local MDS | Georg Heiler
georgheiler.com
-
We love seeing companies embrace code-driven practices in data. In this article, Y42 explains the benefits of analytics-as-code for data versioning, modeling, and pipelines - along with a hands-on tutorial. It's a great approach that pairs well with tools like Evidence
In our most recent blog post, we’re excited to explore the topic of “Analytics-as-Code”. We are focusing on the “Great Divide" between data and code and highlighting an approach that integrates code, configurations, and data in a version-controlled environment, enabling consistency and scalability. It's a practical read for data engineers aiming to enhance the reliability of their data pipelines through stateful and declarative systems. Read on here: https://lnkd.in/gg_J4C4F
Analytics as Code - Code, configs, and data version-controlled | Y42
y42.com
-
Unlock dbt's potential beyond local setups with essential production deployment strategies. Explore high, medium, and low-effort options tailored to diverse business needs. Optimize your process for superior data modeling in production. #dbtDeployment #DataModelingInProduction #DataOpsEfficiency #StreamlineYourStack 🔍🔧📊
dbt Deployment Options - Datacoves
datacoves.com
-
I love seeing best practices in action like this. It really validates what we're trying to do with Recce by providing a platform to perform these data checks as part of standard practice when opening a data project PR.
The Cal-ITP data-infra dbt project is the perfect case study for the application of pull request best practices! In this article, Dave Flynn and Even Wei take a look at what practices Cal-ITP uses to ensure that changes are detailed, data is properly validated, and the team can react swiftly when a bad merge does occur. Great work from the team who handled some of the PRs we looked at. Laurie Merrell, Soren Spicknall, Evan Siroky, Tiffany Chu etc. or anyone else at the Cal-ITP: California Integrated Travel Project data team, we'd love to chat about your PR process and see what else we can learn! https://lnkd.in/ggjr23mg #DataEngineering #AnalyticsEngineering #Data #dbt #BestPractices #Recce #OpenSource #DataValidation
dbt best practices in action at Cal-ITP’s data-infra project | by Dave Flynn | In the Pipeline | Apr, 2024 | Medium
medium.com
-
Have you ever wondered about the differences between Datafold's Data Diff and #dbt tests? Do data teams really need both? In this post, Elliot G. and Leo Folsom explore: - Why dbt tests prevent some data quality issues, but not all - How the two tests answer fundamentally different data quality questions - And why having both in your CI pipeline is essential for complete data quality coverage https://lnkd.in/gsdjcmF8
Three key differences between Datafold tests and dbt tests | Datafold
datafold.com
-
Testing all of your data over and over can cost you more than data transformations themselves! This is where we were two years ago - we wanted to stay safe and identify issues early in our data pipeline but running thousands of tests per day is very costly. Luckily, Dominik Golebiewski found a simple and elegant way to implement batch testing in dbt by dbt Labs. With a straightforward assumption we were able to test reliably and reduce costs by 80%. Dive into the post below to find out more and get the macros we used. https://lnkd.in/d-NFQSNU
Mastering data testing with dbt — part 2
medium.com
