DAGWorks Inc.

Happy Thursday. Here's what we shipped this past week: > #Hamilton release highlights: - some minor fixes and docs updates - 🤯: this was all driven by five different OS contributors! - We also published a walkthrough of our latest caching feature! > #Burr release highlights: - HaystackAction: we provide a bridge to use #Haystack with Burr! > Office Hours & Meet ups for Hamilton & Burr. - Link in the newsletter. > #MLOps World & #GenerativeAI World Summit 2024 - come to my workshop with Hugo Bowne-Anderson or get a discounted ticket. > Blog post: - Building Reliable AI: Annotating Data using Burr. Link in the newsletter > In the wild: - Instructor + Burr post by Thierry Jean & Jason Liu. - Burr at DataForAI event - sign up, link in the newsletter.

The other week we shipped a great general purpose caching feature for #hamilton. This is a feature for all you #dataengineer #datascience #machinelearning #aiengineer folks. The premise is that with Hamilton we can intelligently determine whether code or data has changed between executions, and then only run/re-run the parts of your dataflow/DAG that require it. This is very useful in development, but also has production applications! E.g. reducing your compute bill perhaps? Want to learn more, check out Thierry Jean's high level overview of the feature on our youtube channel:

Hey folks! Haven't done an update in a while. Two things I'm excited about: 🔥 You can now annotate data with the Burr UI 🔥 Label, review, and download evaluation datasets when you build with Burr. You can annotate data points https://lnkd.in/gsm5MuZs 🔢 Really exciting numbers for OS engagement 🔢 📈 750k+ downloads on pypi between Hamilton and Burr (+ > more than 100k/month) ⭐ 3000 stars on github ☕ 550 community members between discord and slack 📖 300+ subscribers for our blog (blog.dagworks.io) It's an honor to be running this OS community, and I'm super exciteda bout hte growth we've seen recently.

Learn how to generate flashcards (or anything really) from YouTube videos in my guest post on the Instructor blog with Jason Liu! https://lnkd.in/eG_FDGGc Instructor lets you get reliable structured outputs from LLMs. It truly changed how I approach LLMs and it's one of the first library I `pip install` when starting a project! No more prompting incantations required 🔮 After some hacking, I developed the following mental model: - Instructor helps you structure the reasoning of LLMs. - Burr helps you structure your application, creating an explicit flow between the user, the LLM, and the system (e.g., database, tools) With only two tools, you can quickly build a reliable application that solves a problem and delivers value. Next up, I want to show how convenient it is to use Instructor + LanceDB to ingest documents and extract structured metadata!

Can't wait to speak at Sword AI Summit! Honored to be included among these speakers. If you're in the area (Porto) on November 9th reach out! Happy to grab coffee + talk AI/ML/data.

Howdy. Here's my update this week! 1. Announcing Shreya Shankar as an advisor to DAGWorks Inc.! I'm super excited to work with Shreya, she's an awesome person, and has great insights into the ML & AI space. If you don't who she is, check out her blog (link in the newsletter). 2. #Hamilton release highlights: - tweaks to @pipe_input decorator - a new @hamilton_exclude decorator - #polars + #pandera fix 3. #Burr release highlights: - some annotation tweaks to recently released annotations workflow - user contributed docker files for deploying the Burr UI. Thanks Aditya K. & Matthew Rideout for the contributions. 4. Office Hours & Meet ups for Hamilton & Burr. - link in the newsletter for times and links 5. MLOps World & Generative AI World Summit 2024 - come to the workshop I'm giving, save some money with my discount code 6. In the wild: - Hamilton at PyConZA ; - Hamilton Meet-up Recording - Burr at #DataForAI meetup in SF this month

We all know how painful it is to launch a notebook / script / pipeline, wait a few minutes for execution, and have it crash midway. You then try to identify and fix the bug before retrying 🤞 In the process, you repeat expensive operations: - loading data from an external source - complex table joins - training a machine learning model - make paid LLM requests - embedding text - etc. Introduced in Hamilton 1.79.0, *caching* allows you to skip these expensive and redundant operations, resulting in speedups and resource savings. In short, it checks code and data changes to the DAG to execute only the necessary parts. If you're adding a node, only this one will be executed. If you're editing a node, only the dependent path is re-evaluated. Development is much faster! Also, it allows to safely shutdown your notebook, and resume where you left later on. Caching is complex and can quickly become useless if unreliable or opaque. To give users full visibility, this feature comes with structured logging, a new visualization, and many utilities to inspect the cache. Here's a famous quote from Phil Karlton: "There are only two hard things in Computer Science: cache invalidation and naming things" Hamilton was already tackling naming, now it's tackling both problems 😅 To learn more, see the release blog and the in-depth Google Colab tutorial (links in comments)

Database (vendor) agnostic pipelines are a current possibility. Why is Ibis + dlt + Hamilton a really cool combo? Using this combination results in db-agnostic portable pipelines. Since dlt schemas are db agnostic and ibis merges db dialects (and python), you are enabled to write code once and run it anywhere. Hamilton brings it all together serving as a way to transform and orchestrate the data. What's next? Plugging open compute engines could be a last step to make pipelines even more scalable and cost efficient while remaining portable. Check out this demo of dlt + ibis + hamilton:

Hamilton just keeps getting better, loving this! GS Strats/alumni and beyond: Hamilton's graph node caching is getting very close to lazy evals of SecDB's compute graph - worth checking out! #strat #secdb

Are you in town for #SFTechWeek on Monday? Are you looking for the best place to learn about cutting edge #AI from not 1, not 2, not 3, but 4 different companies? Do you want to see demos from seven (7) of the best up and coming AI tooling companies? Then you won't want to miss out on our SF Awesome AI Dev Tools October event by OSS4AI Come see demos from: - Tim Gilboy - Stefan Krawczyk - Wes Nishio - Uli Barkai - Hemnaa Subburaj - Olaoluwa Ogundeji - Yash Khandelwal As well as talks from: - Shuveb Hussain - Ana Robakidze - John Gilhuly - Sushobhan Ghosh RSVP in the comments - hope to see you all tomorrow!