Dagster Labs

New Case Study! In an industry notorious for paperwork and manual processes, Group 1001 is redefining what's possible with modern data orchestration. "The velocity we've achieved is insane. And it's not just about speed—it's about making sure everything we build is maintainable," says Gu Xie , Head of Data Engineering at Group 1001. 🚀 The results speak for themselves: - Deployed a custom RPA for 1035 insurance transfers in just 3 weeks - Processed thousands of transfers within hours of deployment - Delivered new analytics dashboards in just 2 days - Migrated legacy systems in 4 months with only 2 developers Read the full case study today!

At Dagster, we understand that your data platform is highly contextual, and you need a tool to accommodate this. Components provide a low-code YAML interface for your users, backed by tools that support software engineering best practices and give platform teams complete control. Our Founder and CTO, Nick Schrock, will be hosting a live webinar on April 16, 9 AM PT. Where he will share: 🛠️ Build maintainable, low-code data platforms. 💪 Empower self-serve workflows without sacrificing standards. ⚙️ Customize components to fit your stack. Register today! Link in the comments.

Attending Google Cloud Next in Las Vegas in a few weeks? Join us and our friends at Databricks, Striim, Hex, Dagster Labs, and Astronomer for our Serving Data + AI dinner on April 9th! Join us for an evening of dinner, drinks, and an insightful panel on building AI-ready systems with trusted data, featuring: 🚀 Joe Reis, Data Engineer & Architect and author of O'Reilly's Fundamentals of Data Engineering 🚀 Krishna L., Senior Data Architect - Data Architecture and Engineering at General Mills 🚀 Naveen Punjabi, Director, Analytics ISV Partnerships at Google Cloud Register here: https://lnkd.in/eN2ZCj3K #GoogleNext #GoogleCloudNext #dataquality #dataevents #AIevents

Hear from our Community! Emil Sundman is a Data Engineer at Nova by Bizware and is an experienced data professional across the entire data stack. Dagster's abstractions were built specifically with data work in mind. By being asset-aware, separating business logic from storage, and applying software engineering best practices to data engineering, it gives you the tools necessary to build powerful data platforms at scale. Thanks Emil!

🤔 Ever wonder how real data teams deploy Dagster in production? I just collected some insights from the community on their Dagster deployments, and wanted to share what I learned. The diversity of deployment options is striking. From simple Docker Compose setups to sophisticated Kubernetes deployments, Dagster adapts to your infrastructure needs rather than forcing you to adapt to it. One user runs Dagster on-prem with Podman quadlets, proving you don't need complex infrastructure to get production-ready deployments. Their architecture separates concerns beautifully with distinct networks for authentication, Dagster services, and code locations. "What I love is the UI and effortless way it runs each task in its own containers. It's also been super reliable," shared one happy user. This is exactly why we built Dagster - reliability by design isn't just marketing speak, it's baked into everything we do. Another team runs Dagster in AWS ECS with separate instances for the webserver, daemon, and user code. "Dagster is awesome and the team is moving fast!" For Azure users like the original poster, several community members shared their Azure Container Apps, VM, and App Service implementations. You don't need to be a Kubernetes expert to get started. The most valuable insight? Dagster works well regardless of your data volume - it's about managing complexity. When your data pipelines get intricate, having end-to-end lineage and visibility becomes crucial. As one user put it: "Our data is not massive, but it is highly complex and dagster helps keep track of how everything connects." #DataEngineering #DataOrchestration #Dagster #CloudDeployment #DataPipelines #ModernDataStack

Let's say you join a company as the first data hire. This position can be daunting, but having the opportunity to build something from the ground up is a unique and exciting experience. After chatting with many Data folks who have found themselves in this position, I collated their advice into some best practices to maximize success. Get your data. Don't worry about optimizing for performance at this stage. Your time is valuable, and oftentimes, the simplest solution is the best way forward, whether that is spinning up a read replica database, using a managed solution like Fivetran or Airbyte, or using an open-source ingestion/replication tool like dltHub or sling. If data is about anything, it's about the little moments we have with our stakeholders. You need to focus on quick wins at first to establish your credibility and solve problems for your stakeholders. This generally means setting up some basic reporting and just counting things. The relationships and queries you build in this stage will be foundational as your data platform scales up to handle more data and complexity. Tech debt is acceptable. In finance parlance, debt = leverage, and at first, you will create some tech debt in service of these quick wins. Generally, this payoff works; you can work on optimization and scalability when the time is right. You don't want to work yourself into a corner where you spend more time keeping the lights on instead of adding value. With that in mind, bringing software engineering best practices (testing, ci/cd, versioning, code-first) early can help mitigate this. In the early days, it was all about building a solid foundation. Delivering value early and often to stakeholders, not breaking the bank, and minimizing quality issues and downtime are all ways to maximize success. When thinking about tools, Dagster Labs is a great one to have in your foundational tool stack. Its rich integration ecosystem, software engineering best practices, and observability and alerting will all help ensure that your new data platform grows with you and your team.

I don't know about the SEO qualities of LinkedIn posts, but maybe this will reach the right sources. If you're trying to use a run config in #dagster to permanently reconfigure an asset (say, a .yaml file that includes all the tables you are interested in), you've probably run into how picky Dagster Labs implementation of their dagster.Config class is about typing. I'm sure there are very good reasons for this, but it can be frustrating. One way you can work around some of the issues raised by this, specifically the fact that dagster won't pick up on these changes if you configure the asset again, is to use dagster_graphql to refresh your entire project. It's shockingly easy to do this with an asset sensor, a very nice out of the box functionality, and aside from a few lag issues that I'm sure can be fine tuned fixed my issue immediately when I tried it today.

Knowing When to Graduate from Cron Jobs 🕒 Ever noticed how we start data pipelines? A simple cron job feels like enough. Write a script, schedule it, move on. But then reality hits. Your simple pipeline breaks at 2am. No error logs. No alerts. Just angry messages from the analytics team wondering why their dashboards are empty. 📊 Cron jobs are perfect when you have: • Simple, independent scripts • Non-critical workflows • No complex dependencies • Limited reporting needs But they become painful fast when you need: • Visibility into failures • Dependency management • Retries and error handling • Data quality checks • Cross-team collaboration The transition from "it works" to "it works reliably" separates casual data projects from professional data platforms. You don't need to immediately jump to a complex solution for every problem, but you do need to recognize when you're outgrowing your toolset. At Dagster, we see this journey every day. Teams start with a handful of cron jobs, then suddenly they're managing hundreds of critical data pipelines with no way to understand what's happening when things break. 🔥 "You don't need to associate your data with a single pipeline, but your data exists within the context of all which you build and what was built before you." This perspective shift is what makes modern orchestration so powerful. The beauty of proper orchestration isn't merely scheduling, it's reliability by design. You prevent data outages before they happen. You track lineage to understand impacts. You build data quality checks directly into your workflows. And perhaps most importantly? Your data engineers can have nice things too. Local development environments. Branch deployments. Clean Python-based frameworks instead of XML and YAML hell. When did you know you needed to graduate from cron? Was it after the third 3am alert? After building your fifth "cron job monitoring cron job" script? Share your stories below! I'd love to hear when that moment of clarity hit you. 💡 #DataEngineering #DataOrchestration #DataReliability #Dagster #ModernDataStack #DataPipelines #CronJobs

We're pleased to announce that Dagster has officially earned Snowflake Technology Industry Competency status! This milestone validates our position as a critical component in modern data ecosystems. This recognition confirms what our users already know: Dagster's orchestration capabilities pair exceptionally well with Snowflake's data platform. Together, they create a powerful foundation for organizations at any stage of data maturity. For startups building their first data platform or enterprises seeking to enhance existing infrastructure, the Dagster-Snowflake combination delivers comprehensive orchestration, proactive alerting, and performance optimization to drive your data operations forward. Ready to see what this partnership can do for your organization? Check out Dagster today!