Astronomer

Astronomer 🤝 Snowflake ❄️ We have recently deepened our partnership with Snowflake, and are now a Snowflake Premier Technology Partner. Astro is the best way to orchestrate data transformations in Snowflake in a scalable, observable, and cost-effective way. We provide an easy Snowflake integration that helps you build pipelines faster, run at scale with confidence, and make sense of your entire data ecosystem, without vendor boundaries. With Astro's integrated development tools, you can even build ELT pipelines without any boilerplate Python code and make orchestrating Snowflake queries more accessible for all data practitioners. Learn more about the Astro and Snowflake integration: https://bit.ly/480JxTW

Building a company that scales quickly requires tools that handle growth and maintain reliability. In this episode, Devin Stein, Founder of Dosu, explains how Airflow became a crucial part of their infrastructure. Devin talks about why Airflow was the perfect fit for their needs, from its ability to handle large-scale operations to its ease of use. Devin also shares how Dosu balances innovation with maintaining data reliability to drive long-term success. Listen to the full episode, linked in the comments below. #AI #Automation #Airflow #MachineLearning

In today’s fast-paced tech environment, managing data at scale can be a significant challenge. Ben Tallman from M Science discusses how Airflow has become a critical tool for their data operations, allowing them to focus on innovation rather than getting bogged down in workflow management. Click the link in the comments below to listen. #AI #Automation #Airflow #MachineLearning

Generative AI is revolutionizing data utilization. Julian LaNeve, CTO at Astronomer, discusses how their customers excel in collecting and centralizing data, and how generative AI can elevate this to the next level. David Xue, Machine Learning Engineer at Astronomer, joins Julian in highlighting the transformative impact of AI on data orchestration. They explore how targeted AI applications are driving innovation and efficiency in data workflows. This episode offers deep insights into leveraging AI for meaningful data use and improving operational outcomes. To access this episode, follow the link in the comments below. #AI #Automation #Airflow #MachineLearning

Kenten Danas, Cody Rich and Jacob Roach will be speaking at @IBM TechXchange Conference 2024 on October 21 and 22 in Las Vegas, in two different sessions: - Introduction to Apache Airflow: Write data pipelines for any use case - October 21, 2:00 PM-2:30 PM PDT with Kenten, Cody and Jake - Unlocking Enterprise-Grade Orchestration with Apache Airflow and Astro - October 22, 11:30 AM-12:30 PM PDT with Kenten and Cody Register now and check out the full conference schedule: https://ibm.co/4eXjQpH

[New webinar announcement!] A situation data engineers face all the time. The data is somewhere, in some format but it needs to be somewhere else, in another format (preferably a little cleaner). Time for ETL! Or ELT? ETLT? And can I pass the data through XCom? How do I make my pipeline dynamic? Are you looking for some best practice guidance on how to write ETL and ELT pipelines with Apache Airflow? 😊 We have a webinar for you! The Best practices for writing ETL and ELT pipelines webinar on October 17th at 11am PT / 2pm ET/ 8pm CEST/ 11:30pm IST gives a beginner friendly overview of how to plan and write ETL and ELT pipelines with lots of template code. Register now to either attend live or get the recording after the event. Link in the comments!

"The barrier to entry to interact with and extract insights from data will be significantly lower, so establishing a strong data culture and practices are extremely important." - Julian LaNeve, CTO, Astronomer Julian is quoted in InfoWorld in this article "5 ways data scientists can prepare now for GenAI transformation," sharing that data science teams should expect increased stakeholder interest and participation because of GenAI capabilities. He also recommends developing a proper data platform based on data engineering best practices and well-cataloged data dictionaries for non-technical colleagues. Another role is consulting on proper governance and guardrails for end users. https://bit.ly/4dC4AgN

🆕 Astro now separates the scheduler and DAG processor for medium, large and XL scheduler sizes, which improves security and reliability because the scheduler job and DAG processor job can run on separate hosts. Consider the following details when choosing your scheduler and DAG processor sizes: 🤔 To use the separate scheduler and DAG processor, you must use at least version 9.7.0 of Astro Runtime. If your Deployment uses a lower Runtime version, then the scheduler and DAG processor run on the same host, and the Extra Large Deployment size is not available. 👫 For Small Deployments, the scheduler and DAG processor run on the same host. 2️⃣ Extra Large Deployments have two DAG processors allocated per Deployment. Read more on Astronomer Docs: https://bit.ly/4gXLlRs

How does AI boost productivity for engineering teams? In this episode of The Data Flowcast: Mastering Airflow for Data Engineering & AI, we speak with Devin Stein, Founder of Dosu, about how AI and Airflow automate knowledge management. Devin shares how Dosu is streamlining processes and driving innovation in the open-source community. Visit Dosu at https://dosu.dev/ and click the link in the comments to listen to the full episode. #AI #Automation #Airflow #MachineLearning

Earlier this week, Mehul Goyal spoke at Ray Summit alongside Anyscale's Marwan Sarieddine about using the power of Airflow and Ray for scalable AI deployments. Many organizations struggle to create a well-orchestrated AI infrastructure, using separate and disconnected platforms for data processing, model training, and inference. This slows down development and increases costs. There is a clear need for a unified system that can handle all aspects of AI development and deployment, regardless of the size of data or models. Astronomer and Anyscale partnered to provide a comprehensive joint solution to simplify the development and deployment of LLMs in production. Try our demo to learn more about how to streamline your AI operations by implementing an end-to-end ML lifecycle on your custom data, including automated LLM fine-tuning, LLM evaluation, LLM serving and LoRA deployments: https://bit.ly/47QcOAs