Coiled

As we started the building out the LLM capabilities for finding activating antibodies for GPCRs at Abalone Bio we needed to go from local notebooks to code running on GPU clusters quickly, cost efficiently, and ideally without locking us into a cloud vendor. After evaluating vendors across the spectrum we chose Coiled as it seemed to check all the boxes. I am glad we did as they have proven to be amazing not just in the technical product they deliver but also in the high-touch, human support they provide. If you want to spend time shipping code and models and not thinking about k8s (eek!) all while working with great people I can easily recommend Coiled!

Using Hugging Face models and datasets is powerful for machine learning, but scaling tasks like model inference on large datasets can be challenging. Dask handles out-of-core computing, breaking up datasets into manageable chunks so that even large-scale tasks can run smoothly. In this example, we processed the FineWeb dataset (~715 GB in memory) using the 🤗 HF FineWeb-Edu classifier. Locally, processing 100 rows with pandas took ~10 seconds, but scaling up to 211M rows was possible with Dask on multi-GPU clusters deployed with Coiled. Results: - Handled large-scale text classification, filtering, and saved results to Hugging Face storage - Optimized GPU utilization to efficiently use expensive hardware This example could be adapted for other workflows like: - Genomic data filtering - Large-scale content extraction - Multimodal AI (audio, image, text) Had a lot of fun learning about Hugging Face and putting together this example with Quentin Lhoest, James Bourbeau, and Daniel van Strien Blog post: https://lnkd.in/gcV348fA

⭐️ New video release 📺: Pandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars Watch how #Dask DataFrame 2.0's improved performance and new features compare to #Spark, #DuckDB, and #Polars, offering a faster and more robust system for big data processing. 📺 Watch the video on YouTube: https://lnkd.in/em9c2Qba Florian Jetter and Patrick Höfler discussed the significant enhancements to Dask, a Python library for distributed computing that integrates well with pandas. Historically, Dask was user-friendly but lacked robust performance. The re-implementation of the DataFrame API has addressed these concerns, making Dask faster and more efficient. Patrick Hoefler, a pandas core team member and Dask maintainer at Coiled, highlighted the improvements in Dask, including a new shuffle algorithm, a logical query planning layer, and a reduced memory footprint. These changes have led to a better user experience and a more robust system overall, especially when compared to tools like Spark, DuckDB, and Polars. The speakers emphasized the seamless integration of Dask with pandas and other PyData stack libraries, making it a compelling option for big data applications. They compared Dask's performance against other tools using TPC-H benchmarks. They also discussed future developments, including extending the logical query planning layer to frameworks like Dask Array and XArray.

In the latest updates from Dask + Coiled: - We need your help building a large-scale geospatial benchmark suite - Better automatic shutdown for Dask clusters - Join us in-person at Open Data Science Conference (ODSC) later this month

Come eat some free food with me in SF next week! I'll be talking about some of the latest performance improvements in Dask DataFrame and showing how you can use Dask + Streamlit to build ETL pipelines. Thanks Aba Micah for organizing and Streamlit for hosting.

Join us Wednesday afternoon for talks on: - Building large, scalable, ETL pipelines with Dask from Patrick Höfler https://lnkd.in/eWz6J4tw - Geoscience at massive scale from Hendrik Makait https://lnkd.in/eEBKCWMg

Come join me and the Coiled team in Amsterdam next week to talk about all your needs for running Python in the cloud with only a few lines of code. From AI/ML scripts needing GPUs, to thousands of workers running a distributed Dask workload, to writing ETL pipelines with DataFrames, to processing Petabyte workloads in minutes not days. Coiled can do it all - come talk to us to see how!

Some thoughts on usage based pricing, and how and why we're slowly shifting away from it https://lnkd.in/gPPv7ccS

Many data workloads are bursty, and you may need to scale up quickly for periods of compute-intensive work. Coiled + Dask support adaptive scaling, or autoscaling, so your cluster scales dynamically as your workload changes. Thanks to improvements in adaptive scaling and cross-zone AWS clusters, it’s now much easier to get clusters with 1000s of virtual machines on AWS.

Latest updates from Dask and Coiled: - It's now easier to get large clusters - Faster software environment builds - Coiled + Earthmover's Arraylake for large geoscience workflows