SDF Labs

SDF, the evolution of data transformation, is officially in Public Beta 🎉. We raised $9M from RTP Global, Two Sigma, and Founders Co-op From four founders obsessed with SQL to a dedicated team that feels more like family, we are building the next generation developer platform for data. What started at Meta as a solution to data at scale has grown into the best way to build a reliable and optimized data warehouse. SDF is free to use and built on an open source core. Install our CLI (https://lnkd.in/ga_sRzfn) and join our slack community! (https://lnkd.in/gpuCHKgF) Unlike other data transformation tools like DBT, SDF extracts SQL compilers from their clouds, understanding proprietary dialects of SQL (like Snowflake) so deeply that it can ultimately execute them. (See blog post in comments for more) A lot has been wrapped into SDF, and we’ve learned so much from our customers along the way. We’d like to send a huge shoutout to some incredible engineers at Patreon, Deel, ClassDojo, Cybersyn, Obie, Linqto and others at our partner F500 companies. SDF would not be here without you. Thank you.

DuckDB + dbt = SDF. A customer of ours recently described our product as a combination of DuckDB and dbt. It begs the question, is this accurate? Why this makes sense: SDF is a locally installable binary built entirely in Rust. Powered by Apache DataFusion, SDF contains a super fast in-memory analytical database. Unlike other databases, it analyzes multiple query logical plans at the same time, constructs a dependency graph, then executes all of them in order based on their dependencies. You can think of this part like a DuckDB local execution of an entire DAG. SDF also supports executing queries remotely against cloud warehouses like Snowflake or BigQuery, and materializing these queries as incremental models, snapshots, seeds, etc. (similar to dbt). SDF also supports Jinja preprocessing, environments for staging/prod, and other transformation layer features that build on the dbt experience. What this misses: SDF understands dialects like Trino SQL so well, it can execute them natively against its DB. This is unlike DuckDB where you're locked in to the DuckDB SQL dialect. Support for execution of proprietary dialects like Snowflake and BigQuery is coming soon. SDF works alongside dbt! In fact, this is a very common use case for many of our customers. As a compiler and runtime that supports Jinja, it simply looks at dbt as another "dialect" to understand. SDF has a built in caching layer that neither dbt nor DuckDB offer. It understands when either the code and/or data have changed, then incrementally rebuilds only what's needed. In this way, SDF models something like a build system in software. SDF also adds an entire type system on top of SQL. Think of this like Typescript for SQL. SDF classifiers are metadata that propagate via column-level lineage to downstream usages of columns. This enables governance scenarios and custom compile-time checks for data quality. What do you think? Do you agree with this equation? #dbt #duckdb #sdf #dataengineering

Join SDF Labs CEO Lukas Schulte and Dagster Labs Head of Data Engineering and Devrel Pedram Navid on August 22nd in the latest Dagster Deep Dive to learn how you can ensure consistent and trustworthy data across your pipelines. Sign up now to reserve your seat! https://bit.ly/4dpXKLR

Quality data is the foundation of reliable insights. Join us on August 22nd to discover how SDF Labs' advanced transformation techniques can elevate your data quality and reliability-- ensuring consistent and trustworthy data across your pipelines. Sign up now: https://bit.ly/4dpXKLR

Dagster Labs brings you fine-grained orchestration and deep observability. SDF Labs brings you advanced SQL transformations and static analysis. Together, you get the most efficient, scalable, and transparent #data pipeline management. Explore one of the best combos since PB&J here: https://bit.ly/4e14dgv #Dagster #SDF #DataEngineering

🚀 Announcing the SDF Labs & Dagster Labs Integration 🚀 Say goodbye to the friction between local development and cloud orchestration! SDF Labs and Dagster Labs are thrilled to announce a groundbreaking integration that bridges the gap between offline and online data processes. With SDF's multi-dialect SQL compiler and Dagster's powerful, asset-oriented orchestration platform, you're now empowered to build robust, seamless data pipelines with unparalleled efficiency and confidence. 👉 Experience shorter feedback loops, reduced cloud costs, and true column-level lineage—all within a unified toolchain. Ready to elevate your data engineering game? Learn more in our latest blog post about the integration. https://lnkd.in/g3wDQkQE #DataEngineering #SDF #Dagster #DataPipelines #Rust

I wrote a release blog post with Dagster Labs! Be sure to check it out! Fun Fact: We name every release after a pop song, so this release is 1.8, or "Call Me Maybe" (see: Carly Rae Jepsen). • For existing Dagster users, we've got a lot of cool stuff out that I'm sure you'll want to get your hands on --- including a new integration with SDF Labs! • For data professionals who aren’t using Dagster, explore how our new features can enhance your existing setups, particularly if you’re working with tools like dbt or Looker, by improving asset management and workflow automation. • For those who just like me -- well, you'll get to see me brag about something cool that I did at work. Enjoy!

Well what do we have here?? 😉 "The new integration allows for more comprehensive and versatile data pipelines by leveraging SDF's developer experience, model validation framework, hybrid execution engine, and metadata system." Congrats on the release Dagster Labs! We’re excited to share more very soon.

We’re thrilled to welcome Sam Brandt to the SDF Labs team as our incoming Head of Sales. Sam brings a deep expertise in data development, query federation, and heterogenous data systems from his previous role as Sales Director at Starburst. “Let’s bring a new standard to the transformation space, and raise the bar across the industry to achieve more impact with data!” Welcome, Sam!

Feature of the week - SDF Now Supports Custom Scripts! Almost anyone familiar with NPM, or Docker, Inc is going to be familiar with the RUN and scripts capabilities that let you create custom workflows running as shell subprocesses. Scripting is incredibly powerful letting you automate developer workflows. We’re happy to announce that SDF now supports pre-compile, post-compile, pre-run, and post-run scripting letting you create more advanced workflows. Custom Scripts help with: Ease of Use Running scripts is straightforward (sdf exec <script-name>, or as part of sdf compile, or sdf run), making them accessible for developers of all skill levels. Simplified Automation Scripts allow you to define scripts directly in your workspace.sdf.yml file, and run them natively with SDF making it easy to manage project-related tasks. Consistency Scripts ensure that everyone on the team uses the same commands and versions, leading to consistent development and build processes. Integration with CI/CD Scripts can be easily integrated into Continuous Integration/Continuous Deployment (CI/CD) pipelines, automating testing, building, and deployment processes. Read the docs & try them out for yourselves. Scripts are available in SDF Version 0.3.10 and onwards! https://lnkd.in/gEnH4R37