Orchestra

AWS Lambda users - we have something for you💡 Have you ever wanted to get visibility of all your Lambda processes that form core parts of your data pipelines? Well now you can! Orchestra now supports the triggering and monitoring of AWS Lambda Functions as part of end-to-end data pipeline flows. Combined with our integrations with EC2, ECS, Redshift and Snowflake this is a game-changer who is ready to move on from Step!!! Read more here 😎 https://lnkd.in/dy_bJdjc #dataengineering #aws #awslambda #orchestra #pipinstall

Join us live on November 14 for another episode of Coffee with Coalesce as we talk all things data orchestration with Orchestra Founder and CEO Hugo Lu.

Analytics Engineers! Here's how you can use dbt to run any arbitrary SQL https://lnkd.in/gh8d633J Seems powerful doesn't it? #dbt #analytics #orchestration #orchestra

Data People! Your weekly roundup just dropped: 🧠 Tutorial: dbt™️ incremental microbatch in 5 minutes (link) 🧠 dbt™️ Incremental Models Cookbook ; Merge vs. Insert Overwrite (BQ) 🧠 What actually is a data control plane? 🧠 The 1 question new data engineers never ask (but should) 🧠 10 Snowflake Admin tasks you should consider automating 🧠 CEO-Dashboard-Gettier problem 🕴Filigran raise $35m Series B 🕴 Interesting updates to QuestDB 📰 Data Platform Supermom raises $18m Series B 🎥What is SSH Tunneling 🎥Incremental models in dbt with Bigquery And also 50% OFF the Winter Data Conference with LEIT DATA inside 👇 https://lnkd.in/eRp2Pi56 #dataengineering #dataroundup #datablog #analytics

Analytics Engineers! Are you feeling bottlenecked? Do you want to be your own platform engineer? Now you can be 🔗 https://lnkd.in/e_Vx5Fna #dataengineering #analytics #dataplatformengineering #orchestra

Data Engineers! Pay attention - today we're paraphrasing "As you start building your first data pipelines, you’ll slowly realize you need to address a growing number of recurring issues. Maybe you implement a component or process that tracks what jobs are running, a scheduler, a set of generic scripts to run transforms and data ingestion, or even some form of UI. Before you know it, you’ve pieced together something that looks like Airflow. Something that goes beyond just being a set of data pipelines but starts looking like an orchestrator. Surprisingly (or maybe not), I’ve seen countless homegrown orchestration/data pipeline systems. Often, it feels like, given enough time, the team might build its own Airflow-esque solution. In fact, I’ve come across plenty of posts where people ask if they should just build an orchestrator in-house." These are wise words from Seattle Data Guy. This is why we built Orchestra - to give engineers the power of the end state (orchestration, observability, data quality monitoring, metadata management) from day 1. This means 🔥 No more fighting fires as you try to scale the data time 🤚 Less Ad-hoc requests and tickets; Data power users self-serve ⏲️ More time to spend driving value, modelling data - the fun stuff 💸 Saved time and money ⛱️ Holiday - data engineers! You never go on holiday without being stressed! Let's change that See more of Orchestra in action below. #dataengineering #dataorchestration #analyticsengineering

We are excited to share our latest Enterprise Architecture Blueprint for the MDS and Snowflake 🥳 In this detailed read, we dive into what a Data Architecture using MDS Services look like, and how this can evolve in an enterprise context including Snowflake. Read more here 🔗 https://lnkd.in/dq9A3EWF cc Stratos Consulting FivetranAirbyte Coalesce.io Snowflake Rivery

As a Data Engineer building a modular data stack to power BI use-cases, I didn't want to have the hassle of setting up and maintaining whole separate repo + infrastructure like Airflow to monitor all our processes. There was an added complication that we "abstract" base operations like running queries around frameworks like dbt. So we built something that would do both 💡 The best part is you can go from dbt Core locally to production in minutes with Orchestra ⚡ Deploy and run in Production rapidly 🔀 Include dbt in an end to end data flow with tools like Fivetran Airbyte Portable Power BI etc. 📊 Metadata gathered automatically so you can identify dbt model bottlenecks (timing + cost). Check it out below. #dataengineering #dbt #sql #analytics #analyticsengineering

Data Teams! Are you tired of how much content there is out there? Ever find it overwhelming? We read hundreds of pieces of content and curate these for you! Check out the latest roundup below. 🧠 You dont’ have a data problem Barbara Galiza 🧠 A fantastic comparison of Delta Live Tables in Databricks to Dynamic Tables in Snowflake René Luijk 🧠 Big Data Vendors embrace Apache Iceberg Alex Wiss-Wolferding 🧠 Incremental but Incomplete Tobias (Toby) Mao 🧠 The problem with Snowberg (link) 🧠 How to empower citizen analytics engineers with Coalesce and Orchestra 🧠 Introducing Orchestra dbt Core; Lightweight dbt Core transformations Josh Hall 🧠 dbt™ latest features and takeaways from Coalesce™ 2024: One dbt™ 🧠 Idbt™ latest features and takeaways from Coalesce 2024: iceberg  Hugo Lu Read more below. #dataengineering #dataroundup #analyticsengineering

dbt now supports iceberg - but what does that mean for you and where does Orchestra fit in? Find out below. #dataengineering #analyticsengineering #dbt #iceberg