Collate’s Post

I was recently asked what I thought the most underrated Snowflake feature is, and without hesitation, I'd say more of our clients should be leveraging Snowflake Dynamic Tables. Dynamic Tables provide a seamless way to manage data transformations right within Snowflake by defining transformation logic in a simple SQL query. Snowflake takes care of the heavy lifting, ensuring your data is always up-to-date. Here’s why I think they’re awesome: - No more managing complex data dependencies or refresh schedules - Near real-time data insights without creating streams/tasks or using 3rd party tools - Simplified ETL pipelines with automated, continuous data refreshes However, Dynamic Tables might not be the best fit if: - You need to retain historical data not present in source tables - You rely heavily on self-joins or non-deterministic functions To make the most of Dynamic Tables, take a few minutes to read the following docs link and give them a try! https://lnkd.in/egTkkmDu #Snowflake #DataTransformation #DynamicTables #RealTimeData #DataAutomation

🚀 Automate Your CI/CD Pipeline with Snowflake and GitHub Actions 🚀 Our Senior Data Engineer, Jaime Salas, shares an insightful guide on integrating GitHub Actions with Snowflake to streamline your CI/CD pipeline. Learn how to: 🔹 Set up key pair authentication 🔹 Create and manage Snowflake users and schemas 🔹 Automate tests and merges using GitHub Actions Enhance your development workflow with these powerful automation techniques! https://hubs.ly/Q02BWXC10 👉 Read the full article #DataEngineering #Snowflake #GitHubActions #CICD #Automation

I recently completed a project that shows how to build an ELT pipeline using dbt, Snowflake, and Airflow. Here's a quick rundown: - Set Up: Start by setting up your environment. Configure dbt to connect to Snowflake. - Create Models and Macros: Use dbt to create your data models and transformations. Macros help simplify repetitive tasks. - Testing: Add tests to make sure your data is accurate and reliable. - Deploy with Airflow: Use Airflow to automate and orchestrate your dbt models. I was pleasantly surprised by how straightforward it was to set up dbt. The documentation and initializing commands, along with the clear file structure, made it easy to get started. Additionally, Airflow's intuitive interface simplifies defining tasks, setting dependencies, and managing credentials for Snowflake, making the whole process efficient and straight to the point. For more details and the full code, check out my GitHub repository: https://lnkd.in/duuPsGRz

🚀 Snowflake unveiled Snowflake Trail at Summit 2024, an advanced #observability solution. #Snowflake #Trail captures and stores logs, metrics, and traces directly within Snowflake accounts, streamlining data management and analysis.. 💡 Using #OpenTelemetry standards, Snowflake Trail integrates with observability tools like #Datadog, #Grafana, #MonteCarlo, #PagerDuty, ...etc. It supports real-time monitoring of Snowpark applications and data pipelines, providing insights into their behavior and performance.. 🔍 Centralizing observability data within Snowflake allows for comprehensive analysis and troubleshooting, enhancing the efficiency and reliability of data operations. Fore more about this feature : https://lnkd.in/erCg_Ydc #datacloud #observability #snowflake

"Excited to share my latest project on GitHub: an ELT pipeline using DBT, Snowflake, and Airflow! 🚀 From setting up the Snowflake environment to deploying on Airflow, each step was meticulously crafted for efficient data transformation. Highlights include creating source and staging files, implementing macros for DRY coding, and building transformative models and tests. Check out the repository here: https://lnkd.in/eRCQGAks for a detailed walkthrough and feel free to connect for discussions and collaborations! #ELT #DataEngineering #DBT #Snowflake #Airflow"

Latest additions to Snowflake’s features is REST APIs. 😍 ❄️ In case you are building API-first applications, integrations, or data pipelines, you can now interact with Snowflake resources and data in the programming language that suits your project. This feature wide range of #Snowflake objects, including databases, tasks, roles, users, virtual warehouses, Snowpark container services, Cortex features! #data #ai #ml #api #Snowflake #squadmember #community

Check out my latest monthly Snowflake Release Blog Post. This was again a super busy month, and here are some notables: ☑ GA: Dynamic Tables ☑ GA: Authentication enhancements ☑ GA: Clean rooms ☑ GA: new regions Zurich ☑ Preview: Git integration ☑ Preview: Copilot https://lnkd.in/gVEz4S_V #snowflake #infostrux #newreleases #datasuperhero #snowflake_advocate

Snowflake has just unveiled its Git integration to the public so i've written a blog discussing an innovative approach to harness this integration with Kestra for orchestrating data pipelines. The potential here is immense, and i'm thrilled to share insights on how to leverage it effectively. If you find it intriguing, show your support with a "clap" on Medium! #Snowflake #GitIntegration #DataOrchestration #Kestra #DataPipelines

Did you know Snowflake can now connect directly to Databricks Unity Catalog? 💡 I’ve been diving into this integration recently, I explored using Apache Iceberg to enable smooth interoperability for reading Databricks tables directly from Snowflake. 📌 Key Benefits: Simplifies data sharing between platforms. Leverages Iceberg’s schema evolution and time-travel features. Enhances query performance across both ecosystems. #Databricks #Snowflake #ApacheIceberg #DataEngineering #DataIntegration https://lnkd.in/gMFcv4gr

🚀 Build a Scalable Data Pipeline with Apache Airflow, Snowflake & dbt! Looking to streamline your data engineering workflows? Follow along with this video to learn how to: Orchestrate tasks with Apache Airflow Transform data efficiently with dbt in Snowflake Use Docker for easy setup and portability This guide will help you build a robust, modern data pipeline. Perfect for anyone looking to enhance their data engineering skills! 💡 #DataEngineering #Snowflake #ApacheAirflow #dbt #Docker #DataPipelines