Collate’s Post

I was recently asked what I thought the most underrated Snowflake feature is, and without hesitation, I'd say more of our clients should be leveraging Snowflake Dynamic Tables. Dynamic Tables provide a seamless way to manage data transformations right within Snowflake by defining transformation logic in a simple SQL query. Snowflake takes care of the heavy lifting, ensuring your data is always up-to-date. Here’s why I think they’re awesome: - No more managing complex data dependencies or refresh schedules - Near real-time data insights without creating streams/tasks or using 3rd party tools - Simplified ETL pipelines with automated, continuous data refreshes However, Dynamic Tables might not be the best fit if: - You need to retain historical data not present in source tables - You rely heavily on self-joins or non-deterministic functions To make the most of Dynamic Tables, take a few minutes to read the following docs link and give them a try! https://lnkd.in/egTkkmDu #Snowflake #DataTransformation #DynamicTables #RealTimeData #DataAutomation

I recently completed a project that shows how to build an ELT pipeline using dbt, Snowflake, and Airflow. Here's a quick rundown: - Set Up: Start by setting up your environment. Configure dbt to connect to Snowflake. - Create Models and Macros: Use dbt to create your data models and transformations. Macros help simplify repetitive tasks. - Testing: Add tests to make sure your data is accurate and reliable. - Deploy with Airflow: Use Airflow to automate and orchestrate your dbt models. I was pleasantly surprised by how straightforward it was to set up dbt. The documentation and initializing commands, along with the clear file structure, made it easy to get started. Additionally, Airflow's intuitive interface simplifies defining tasks, setting dependencies, and managing credentials for Snowflake, making the whole process efficient and straight to the point. For more details and the full code, check out my GitHub repository: https://lnkd.in/duuPsGRz

🚀 Automate Your CI/CD Pipeline with Snowflake and GitHub Actions 🚀 Our Senior Data Engineer, Jaime Salas, shares an insightful guide on integrating GitHub Actions with Snowflake to streamline your CI/CD pipeline. Learn how to: 🔹 Set up key pair authentication 🔹 Create and manage Snowflake users and schemas 🔹 Automate tests and merges using GitHub Actions Enhance your development workflow with these powerful automation techniques! https://hubs.ly/Q02BWXC10 👉 Read the full article #DataEngineering #Snowflake #GitHubActions #CICD #Automation

🚀 Snowflake unveiled Snowflake Trail at Summit 2024, an advanced #observability solution. #Snowflake #Trail captures and stores logs, metrics, and traces directly within Snowflake accounts, streamlining data management and analysis.. 💡 Using #OpenTelemetry standards, Snowflake Trail integrates with observability tools like #Datadog, #Grafana, #MonteCarlo, #PagerDuty, ...etc. It supports real-time monitoring of Snowpark applications and data pipelines, providing insights into their behavior and performance.. 🔍 Centralizing observability data within Snowflake allows for comprehensive analysis and troubleshooting, enhancing the efficiency and reliability of data operations. Fore more about this feature : https://lnkd.in/erCg_Ydc #datacloud #observability #snowflake

Latest additions to Snowflake’s features is REST APIs. 😍 ❄️ In case you are building API-first applications, integrations, or data pipelines, you can now interact with Snowflake resources and data in the programming language that suits your project. This feature wide range of #Snowflake objects, including databases, tasks, roles, users, virtual warehouses, Snowpark container services, Cortex features! #data #ai #ml #api #Snowflake #squadmember #community

"Excited to share my latest project on GitHub: an ELT pipeline using DBT, Snowflake, and Airflow! 🚀 From setting up the Snowflake environment to deploying on Airflow, each step was meticulously crafted for efficient data transformation. Highlights include creating source and staging files, implementing macros for DRY coding, and building transformative models and tests. Check out the repository here: https://lnkd.in/eRCQGAks for a detailed walkthrough and feel free to connect for discussions and collaborations! #ELT #DataEngineering #DBT #Snowflake #Airflow"

Snowflake has just unveiled its Git integration to the public so i've written a blog discussing an innovative approach to harness this integration with Kestra for orchestrating data pipelines. The potential here is immense, and i'm thrilled to share insights on how to leverage it effectively. If you find it intriguing, show your support with a "clap" on Medium! #Snowflake #GitIntegration #DataOrchestration #Kestra #DataPipelines

🚀 Build a Scalable Data Pipeline with Apache Airflow, Snowflake & dbt! Looking to streamline your data engineering workflows? Follow along with this video to learn how to: Orchestrate tasks with Apache Airflow Transform data efficiently with dbt in Snowflake Use Docker for easy setup and portability This guide will help you build a robust, modern data pipeline. Perfect for anyone looking to enhance their data engineering skills! 💡 #DataEngineering #Snowflake #ApacheAirflow #dbt #Docker #DataPipelines

Have you noticed the recent updates to Snowflake's UI? Now with a separate CREATE button, you can easily create Tables, Worksheets, Notebooks, Dashboards and more. Add data is also now in General Availability, allowing for multiple options to load data in Snowflake. Excitingly, Notebook is now in Preview, posing tough competition to Databricks. Additionally, GIT Repository Integration from UI is also in Preview mode. Looking ahead, I would love to see some more connectors in the upcoming months, such as a direct connector from BIGQUERY to SNOWFLAKE, REDSHIFT to SNOWFLAKE, or SYNAPSE to SNOWFLAKE. What are your thoughts on these updates, Sachin Mittal? #snowflake #connectors #notebook #dataengineering #updates

It’s easier than ever to run ML on Snowflake! Choose a Container Runtime with your next Snowflake Notebook: get a managed, preconfigured enviro built to accelerate ML. 🤩 Bring your code into Snowflake with ease ⚡️ Use runtime features: optimized data ingestion, distributed training Full details: https://okt.to/daoqUC