Tobiko

One of the most rewarding things about open source is when someone builds something amazing based off of your work. Nico Ritschel has created a demo (https://lnkd.in/gy5HhXET) of how SQLMesh's semantic layer can be used within a Postgres proxy. Meaning, you can query metrics through a Postgres cli and it will automatically rewrite, query, and return data! When we first started brain storming ideas for Tobiko, we thought about building out a semantics layer. But I quickly realized that semantics layers depend on robust transformation. Without good data, you cannot have good metrics. So instead of immediately jumping into metrics, we built out SQLMesh, trying to build the best transformation framework possible. During the process, I decided to build out a SQL based metrics system (https://lnkd.in/g_XzU69r) based on my experience working on Airbnb's Minerva (https://lnkd.in/gcGQiGXq). I managed to build a purely SQL based definition framework and a simple rewriter that can handle automatic joins in a couple weeks. Although I built that prototype and published it, we decided not to continue working on it because it wasn't the main focus of our company. But eventually, we'll come back to it and ship a real product. Nonetheless, it's fun to see a community member take our prototype and take it one step further. I hope to see this spark some discussion about what semantics layers built on top of SQLMesh could look like. #dataengineering #metrics

I’ll be at tech crunch disrupt today chatting about the future of the modern data stack! Let me know if you’ll be there! #dataengineering

SQLMesh is the first and only tool (that I'm aware of) that can do true data deployments. What does it mean to really deploy data? You need to ship code AND data. Right now, existing tools ONLY ship code which means that a bad 'merge' or 'deployment' will bring down production. Imagine using dbt or stored procedures. Developers will make changes to SQL and ship it to prod. If those changes introduce a bug midway in the process, the downstream models will now fall behind due to the broken code. Even if you "dry run" your code, it may not be enough to catch errors that only exist with actual data sets. Reverting/redeploying and rerunning jobs is now needed to recover the pipelines. This means that you've now suffered a data outage. Due to SQLMesh's virtual environments, all code changes can be fully evaluated and audited before being deployed to production. That means that datasets will never fall behind and stakeholders won't send angry emails about why their dashboards are missing data! This is possible because SQLMesh has a deep understanding of what SQL has been deployed to every environment. In CI / CD, when a deployment is triggered, SQLMesh will run all of the changed models and store the results in a physical layer. This is separate from the actual tables that are consumed by dashboards and external jobs. The production tables are actually just simple `SELECT *` views that act like pointers to the appropriate physical tables. Once all of the models have actually run and been tested, the production tables are then SWAPPED to these new tables by changing the view pointers. Zero down time deployments. No merging and praying. Instantaneous rollbacks when necessary. If this sounds too good to be true, try it for yourself. Over a hundred companies are now in production with SQLMesh. https://lnkd.in/dhQhpMFW

𝐓𝐡𝐞 𝐦𝐨𝐝𝐞𝐫𝐧 𝐝𝐚𝐭𝐚 𝐬𝐭𝐚𝐜𝐤 𝐭𝐨𝐨𝐥𝐬 𝐝𝐨𝐧'𝐭 𝐭𝐚𝐥𝐤 𝐭𝐨 𝐞𝐚𝐜𝐡 𝐨𝐭𝐡𝐞𝐫 𝐞𝐧𝐨𝐮𝐠𝐡! Modern data stack tools are fighting to push boundaries, and this fight sometimes gets them in conflict with each other. You may have heard of a recent disagreement between Tobiko and dbt Labs, competitors in the T layer, around Coalesce. In happier news, Tobiko team are working on a SQLMesh + dlt metadata handover that enables SQLMesh to generate scaffolds with built in incremental models for dlt pipelines. Why is this awesome and pushing boundaries? Because metadata handovers make open source tools highly interoperable. Open apis on those tools enable other vendors or developers to use them as de facto standards and build better pipelines. Check out the amazing work being done by Tobiko team on this project: dlt-SQLMesh generator: https://lnkd.in/eSNFBBcC

Streamline your data pipeline with the powerful combo of #SQLMesh and dltHub! This integration accelerates data workflows by enabling seamless metadata handovers, automating scaffolding, and supporting incremental processing. 💻 Boost efficiency, reduce errors, and unlock the full potential of your 𝒑𝒐𝒔𝒕𝒎𝒐𝒅𝒆𝒓𝒏 data stack. 😉 🔗 https://lnkd.in/gZ3SaV43 #DataEngineering #dlt #DataPipelines

Join Tobias (Toby) Mao and Alexey Grigorev from DataTalksClub on their Open-Source Spotlight series, where these two founders talk through the benefits of #SQLMesh With our #opensource #DataTransformation tool, you get: ♦ Column-level lineage ♦ Environment Management ♦ Instant Prod deployments and much more! https://lnkd.in/dPcxfQtz

As a data engineer, you should consider how changes can be done in a non-breaking way. A non-breaking change to a data model is something that won't have any down stream impact, like adding a column or re-ordering columns. Adding columns only impacts down stream models when they do SELECT * statements, which is one of the reasons why it's best practice to avoid them. On the other hand, a breaking change will have significant impact on down stream models and usually requires expensive back-fills. An example of a breaking change is modifying a WHERE statement which changes the cardinality of a table. If you're working at any significant scale where it's expensive and time consuming to back-fill many tables, consider whether or not a change can be done in a backwards compatible way and how expensive a breaking change would be. If it's not very expensive to make a breaking change, it can be easier to maintain since all models are kept up to date without any legacy, so there's always a trade-off. Even if it's not too costly to back-fill many tables, it can be time consuming communicating breaking changes to stakeholders or validating all data consumers are up to date. Arguably, this is even more challenging than the technical/compute costs of breaking changes. As a software engineer, it's commonplace to consider whether changing an API or a database model should be done in a breaking or non-breaking fashion. I believe this best practice should be adopted by data teams as well. That's why we designed SQLMesh to provide automatic detection of breaking and non-breaking changes by analyzing your SQL queries. This allows you to assess the impact of your changes at compile time and understand potential costs (both compute and organizational) before you finalize your changes. https://lnkd.in/gRZfyu_M #dataengineering

Want to learn about running your #dbt project with #SQLMesh? Check out our latest YouTube video to run through a workflow using the classic Jaffle Shop example project! https://lnkd.in/garV5jmE

#learn about #SQLMesh plans in our latest YouTube #video What do you want to learn next? 🤔 https://lnkd.in/eTq7f5Yv

Come see the Tobiko Data team at our Databricks #booth! We've got #transformation, #conversation and #schweet #schwag