Modern Data Stack France

REPLAY - Meetup DCube, architecture de référence Azure, Snowflake, dbt, Power BI Meetup #moderndatastack organisé, animé et hosté par DCube. On parle d'architecture de référence autour des composants Microsoft Azure, Snowflake, dbt Labs et Power BI Frédéric Brossard, Julien FOURRE, Nicolas Bailly et Fabien TANQUEREL experts data, DevOps et Infra/Sécurité présentent une architecture type cloud native déployée chez plusieurs clients et décrivent les meilleures pratiques pour collecter les données, les stocker, les transformer, les réconcilier et offrir un pilotage analytique adapté aux métiers. Ils traitent tous les aspects d'une architecture data moderne, de la zone bronze à la zone gold, du raffinage et de la qualité des données, en passant par la sécurité, et détaillent l'utilisation des composants suivants : - Azure Data Factory, Azure Function, Azure Container App, Azure Service Bus, Azure Logs Analytics - Azure DevOps, pipelines CI/CD yaml et terraform - Power BI : modèles sémantiques et rapports/dashboards - Snowflake : compute et stockage des données - dbt : transformation Ils présentent la genèse de cette architecture en justifiant le positionnement et l’utilisation de ces services par différents cas clients utilisateurs.

I’ve been using AWS Step Functions extensively recently. The service has evolved has significantly improved the last year, and I must say it’s now one of my top AWS services now. In this week’s article, I’ll share what I’ve learned and what I appreciate—and find challenging—about it Interested ? Checkout the link in the comments: #data #dataengineering

New #SQLMesh video just dropped! If you're a dbt user and want to try out SQLMesh, you can without having to rewrite your project. Just use our dbt adapter which gives you column level lineage, virtual environments, and breaking/non-breaking change detection out of the box! https://lnkd.in/g7V7Bucv #dataengineering

👀 Polaris github moved https://lnkd.in/gwWyjhms

⚠️ New ASF Project Spotlight ⚠️ Apache DolphinScheduler is built for data scientists, data engineers, data analysts, and data practitioners who need to solve big data workflow scheduling problems with a simple and easy-to-use approach, accelerating the efficiency of the data ETL development process. Read more here: https://bit.ly/4dNZOgH #opensource #ASF25Years #bigdata

Data observability and data quality are two closely related concepts. While #dataquality focuses on ensuring data reliability, #dataobservability goes further by offering a broader view of your data ecosystem's health. Here's how the two concepts are interlinked: 👉 Data Quality as a foundation: Data quality is an essential element of data observability. To ensure the reliability of data used for analysis and decision-making, it's crucial to make sure it is accurate, consistent, and complete. 👉 Evolution towards observability: With the exponential growth in data volume, sources and use cases, traditional data quality methods such as manual testing or scripts become insufficient. Data observability addresses this challenge by offering tools capable of continuously monitoring data status at each stage of its lifecycle. 👉 Real-time monitoring: Data observability translates into the implementation of real-time monitoring of data pipelines, allowing for quick detection of anomalies, drifts, and incidents. This goes beyond simple quality tests by providing a contextual understanding of problems and their potential impact. 👉 Data catalog as a crossroads: A robust data catalog, integrating both data quality and lineage information, is essential for effective data observability. This allows tracing the origin of data, understanding its transformation, and quickly identifying potential sources of problems. 👉 Data lineage for an overview: Data lineage, by tracing the journey of data from its creation to its use, is a pillar of data observability. It allows understanding the impact of changes, identifying dependencies between systems, and facilitating incident diagnosis. In summary, data observability can be considered an evolution of data quality, adapted to the complexity of current data ecosystems. It provides the tools and information necessary to ensure data reliability, trust, and traceability.

Vous cherchez des datasets pour vos projets de data ? 🔍 En cherchant des datasets pour mes projets, j'ai découvert une ressource incroyable : Google Dataset Search. Ce moteur de recherche spécialisé est parfait pour les Data scientists, les scientifiques, journalistes de données, passionnés de données, et quiconque cherche des datasets. Avec Google Dataset Search, vous pouvez trouver des datasets sur presque tout en utilisant une simple recherche par mot-clé. Il indexe des milliers de datasets, des plus vastes aux plus spécifiques, grâce aux métadonnées fournies par les éditeurs de datasets. Découvrez Google Dataset Search et trouvez les données dont vous avez besoin ! 📊✨ Lien : https://lnkd.in/eGbeTnAw #Données #DataScience #Google #Recherche #Datasets

We started a series of blog posts called “DuckDB tricks”. In these posts, we will present some useful shorthands, hacks, and design patterns for DuckDB. In the first part of the series, Gábor Szárnyas explains some techniques for handling floats, fine-tuning the CSV reader, and shuffling data sets. If you have a favorite DuckDB trick, let us know in the comments! https://lnkd.in/dwXd2b_s

#Airflow 2.10 is here! 🎉 Julian LaNeve, CTO at Astronomer, spoke with VentureBeat's Sean M. Kerner about what's new in #Airflow210, our latest Astro Platform release that adds #dbtsupport, and an early look into priorities for #Airflow3 in the age of #GenAI 💫 Check out the full article to learn about how #ApacheAirflow 2.10 will improve #dataorchestration with hybrid execution and why #datalineage in data orchestration matters for #AI 🤖

Is DuckDB's PostgreSQL extension the makeshift, hybrid OLAP-OLTP (also called HTAP) that developers have been waiting for? PostgreSQL is the leading open source RDBMS engine for transactional workloads. DuckDB is a lightweight, embeddable OLAP database for lightening fast query performance on a single node. Now with DuckDB's PostgreSQL extension, you can attach DuckDB to PostgreSQL and execute analytical queries against it on the same instance with minimal overhead. While Change Data Capture from PostgreSQL to data warehouses is best for scalable workloads, DuckDB-PostgreSQL seems like a great fit for ad-hoc, lightweight analytics on your database. #dataengineering #postgresql #duckdb