Dagster Labs

Can I brag about working with a fantastic team Dagster Labs for a second? Working with a prospect, mentioned the docs are unclear, chatted with an engineer and a PR was up 20 min later. Another prospect, and our eng team shipped two separate features in 2 weeks. ❤️ Velocity is an incredible feeling PS we’re hiring

Airlift is one of our most exciting tools for enterprise use cases. Being able to peer into and migrate Airflow projects into Dagster makes it easy for data teams to have a unified vision of their entire data platform.

✍️ The New Docs are Here! ✍️ We are thrilled to announce that the new Docs experience is now available! This is part of an ongoing effort to continually improve our documentation experience, and we hope it makes your life at least a little easier every day. We heavily focused on creating a more user-friendly docs structure to make sure the information that you need is easy to find. In addition, keep an eye on our Tutorials section for regular additions that will help you level up your Dagster and data/AI engineering game! If you have feedback, we have a GitHub discussion and the docs-feedback channel in our community Slack. Links are in the comments. P.S. The new docs have dark mode!

Excited to congratulate a great Dagster user, Clay, on their $40M Series B expansion at a $1.25B valuation! ✨ As a happy Clay customer, we've seen firsthand at Dagster how their product helps our GTM team turn growth ideas into reality. Their unique approach to data enrichment and activation has been invaluable for our go-to-market efforts. Read more about their vision: https://lnkd.in/g3EHm8ub

Good old Azure Data Factory... When is it time to say goodbye? The truth is: ➡️ Many of the connectors you use inside Data Factory are nothing but a wrapper around Rest or JDBC (same goes for many other data integration tools). ➡️ Many data sources these days can be extracted via rest API (happens more and more) And the extra downsides of Azure Data Factory: ➡️ The developer experience sucks compared to for example Airflow, Dagster or Prefect. ➡️ Pretty bad CI/CD (multiple repos, needs black box npm packages) ➡️ Microsoft is not really adding any new useful features. ➡️ Mapping data flows has lagging Spark versions (most recent version is 2 years old). ➡️ Only really useful for the 'E' in ELT/ETL. And perhaps the worst part? Microsoft is doing nothing to address these issues inside Fabric. Instead they introduce a version of ADF inside Fabric that is even worse than the stand-alone version. How is it possible?

Had a great time with my friends at Hex talking all things data and why sometimes it's not enough to be right. https://lnkd.in/g3N66ES7

Want to start building AI applications but feeling overwhelmed? We've got you covered. RAG (Retrieval Augmented Generation) - a powerful approach that's perfect for your first AI project. By embedding your specific knowledge in a vector database, RAG lets you create AI applications that draw from your unique context to answer user questions with precision. Watch our Developer Advocate, Alexander Noonan, build a RAG system from scratch using Dagster. Our abstractions handle the complex parts of working with LLMs, so you can focus on building production-grade applications that deliver real value. Ready to turn your domain expertise into powerful AI solutions? Check out Alex's tutorial to see how Dagster makes it possible. Link in the comments. #ArtificialIntelligence #DataEngineering #RAG #LLM #Dagster

If I had a nickel for every time I heard this request from a customer or management, I would have enough money for a Big Mac with fries and soda, even in this economy. If you are in this position, here are two advice that will save your company millions of dollars in the long term. 1. Don't try to implement a real-time solution from the get-go; first, ask what your manager means with real-time. In very few limited cases, you would ever need real-time. You can pitch a lambda architecture and push the real-time part down the road. https://lnkd.in/gi-kFbwX 2. Use a proper Data Warehouse. By following best practices and designing a proper data model, you can achieve a cost-effective solution. My stack choice for this specific question would be Dagster Labs + Databricks for the Extraction and Load part from the ELT and then for Transformation Tobiko's SQLMesh. Finally, the data is useless unless presented to users. I'm biased, but I would use Cube for the Semantic Layer and let the analyst choose their preferred BI tool. I have never used Dagster and SQLMesh together. Do you know if they work well together? It looks like Reuven Gonzales made an integration (WIP). https://lnkd.in/gXJz6HxN

The first Dagster Labs Deep Dive of the year is in the books! Please see here the recording from this week's session: https://lnkd.in/gkFDrAV4

🚀 Ready to Shape the Future of AI? Join Scale AI's Platform Engineering Team! 🚀 At Scale, we're driving the next wave of AI innovation, building tools that power cutting-edge large language models (LLMs) and generative technologies. If you're a software engineer passionate about solving complex problems, building scalable platforms, and enabling groundbreaking AI solutions, this opportunity is for you! My team is looking for talented engineers to design and implement core systems using their skills in distributed systems, cloud infrastructure, and back-end engineering to help transform how humanity interacts with technology. 🔗 Check out the full job description and apply below: https://lnkd.in/gcRF_MpC Let’s build the future of AI together! 🌍✨ #AI #SoftwareEngineering #Careers #TechJobs