DVC.ai

𝐄𝐧𝐝 𝐭𝐨 𝐄𝐧𝐝 𝐌𝐋 𝐏𝐢𝐩𝐞𝐥𝐢𝐧𝐞 𝐮𝐬𝐢𝐧𝐠 𝐃𝐕𝐂 & 𝐀𝐖𝐒 𝐒𝟑 In this video, you will learn how to build an end-to-end machine learning pipeline. The pipeline will include all the components starting from data injection to model evaluation. You will also use DVC to reproduce this machine learning pipeline and for experiment tracking. Further, you will work on the AWS cloud computing platform and will use services like IM User and S3 bucket for data versioning. Here is a quick recap of what you will learn in this video: 🔹Construct a complete ML pipeline from data injection to model evaluation 🔹Leverage DVC for reproducibility and experiment tracking. 🔹Harness AWS cloud services (IAM, S3) for efficient data versioning Link: https://lnkd.in/dCYpEWDa Follow DVC.ai

🚀 Excited to have presented my research on Data Version Control (DVC) at the Experts Meetup of the Cloud Centre of Excellence, Vodafone Germany (Düsseldorf campus)! 🚀 My paper covered versioning strategies for Data Science & ML projects, emphasizing reproducible and collaborative research methods. The session included live demos and hands-on exercises to showcase key concepts and best practices. Big thanks to the organizers Thomas Pippig, Sven Schuster, Tobias Rittich, Fabian Heck, Jhealyn Samson & Christian Albers for an engaging program with insightful presentations. Appreciation to the DVC team Jenifer De Figueiredo, Dmitry Petrov & Ivan Shcheklein for their fantastic tool improving data and ML model management. Special thanks to Elle O'Brien for her invaluable DVC tutorials! Interested in learning more? Check out my GitHub repo with the live demo code and a comprehensive DVC tutorial: https://lnkd.in/eYHqc-BJ Explore my blog for research papers, open-source code, and AI/ML projects - https://lnkd.in/eTgvFuTD #VersionControl #Git #DataVersionControl #AI #MachineLearning #MLOps Image - © Vodafone

This article by Alph2phi discusses the challenges of managing data and models in machine learning projects and how DVC can help address these issues. The article highlights several key challenges in managing machine learning projects from reproducibility, collaboration, and scalability. The article explains how DVC can help overcome these challenges: 𝐕𝐞𝐫𝐬𝐢𝐨𝐧𝐢𝐧𝐠: DVC allows versioning datasets and models, similar to how Git manages code. This ensures reproducibility and facilitates collaboration. 𝐏𝐢𝐩𝐞𝐥𝐢𝐧𝐞𝐬: DVC pipelines define dependencies between stages of a machine learning workflow, making it easier to reproduce and share the full process. 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 𝐭𝐫𝐚𝐜𝐤𝐢𝐧𝐠: DVC tracks metrics for each experiment, enabling comparison of different runs and model performance over time. Link: https://lnkd.in/dJWdZUx8 The article highlights the key benefits of using DVC as being: 🔹Improved reproducibility of experiments and results 🔹Enhanced collaboration among team members 🔹Scalability for handling large datasets and models 🔹Better organization and transparency of machine learning projects.

Are we on the cusp of a quiet (or not so quiet) revolution in enterprise data processing? 🚀 Daniel Kharitonov just published a deep dive into the emerging "post-modern" data stack and how it's reshaping the way we handle data. It covers: 🔹 The limitations of the current "modern" data stack 🔹 How AI and foundational models are driving change 🔹 What the future of data processing might look like Key takeaways: ➡️ Direct interaction with unstructured data in cloud storage ➡️ Shift from SQL to Python for data manipulation ➡️ Foundational AI models replacing traditional ML approaches Is your organization ready for this paradigm shift? Read the full article to stay ahead of the curve. Link in comments! #AI #techtrends #futureoftech #datascience

Aditya Wardianto outlines a step-by-step workshop process focused on DVC. His article emphasizes the importance of versioning datasets and models to ensure reproducibility and collaboration in data science workflows. 𝗞𝗲𝘆 𝗮𝘀𝗽𝗲𝗰𝘁𝘀 𝗶𝗻𝗰𝗹𝘂𝗱𝗲: 𝗢𝘃𝗲𝗿𝘃𝗶𝗲𝘄 𝗼𝗳 𝗗𝗩𝗖: The workshop introduces DVC, explaining its functionality in tracking data and model changes, akin to Git for code. 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Participants engage in hands-on exercises, learning how to implement DVC in their projects, including key commands for data management. 𝗔𝗱𝘃𝗮𝗻𝘁𝗮𝗴𝗲𝘀 𝗼𝗳 𝗗𝗩𝗖: The article emphasizes the benefits of using DVC, such as better project organization, improved team collaboration, and a comprehensive history of changes. Overall, the workshop aims to empower readers with the knowledge and skills to utilize DVC in their machine learning workflows effectively. Link: https://lnkd.in/dEEv6nRD Follow DVC.ai

🙋🏽♂️ Looking for a fun project to try out? Tibor Mach shows you how in this video tutorial on Scalable PDF Processing with 𝗗𝗮𝘁𝗮𝗖𝗵𝗮𝗶𝗻 and unstructured.io ! In this project you'll learn how to: - Extract and parse text from documents - Create vector embeddings for downstream tasks - Scale up document processing effortlessly - Version and persist datasets for reproducibility The best part? You can accomplish all this in less than 70 lines of code! Perfect for data scientists and ML engineers working with unstructured data. Key tools: - unstructured.io for document processing - DataChain for scalable data handling and versioning Give it a try and unlock insights from your document collections! 📚💻 Link to the video and blog post in the comments! #DataScience #MachineLearning #WeekendProject #PDFProcessing

ICYMI 𝗗𝗮𝘁𝗮𝗖𝗼𝗺𝗽-𝗟𝗠: 𝗜𝗻 𝘀𝗲𝗮𝗿𝗰𝗵 𝗼𝗳 𝘁𝗵𝗲 𝗻𝗲𝘅𝘁 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗼𝗳 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝘀𝗲𝘁𝘀 𝗳𝗼𝗿 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 was released in June. The DataComp competition was developed to create a benchmark where the models are fixed and the objective is to create the best possible dataset. The competition has expanded to include LLMs. The research by Jeffrey Li, et. al. representing a large number of Universities and Industry professionals alike shows how 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗱𝗮𝘁𝗮𝘀𝗲𝘁 𝗰𝘂𝗿𝗮𝘁𝗶𝗼𝗻 𝗰𝗮𝗻 𝗵𝗲𝗹𝗽 𝗱𝗲𝗹𝗶𝘃𝗲𝗿 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗮𝘁 𝗮 𝗹𝗼𝘄𝗲𝗿 𝗰𝗼𝘀𝘁 𝘁𝗵𝗮𝗻 𝗰𝗼𝗻𝘁𝗶𝗻𝘂𝗲𝗱 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗼𝗳 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮. Currently controlled comparisons are a key challenge to this research. Apples-to-apples comparisons are not possible with models with different architectures, learning rate schedules, or different compute usage rates. In addition, the data used to train the models is opaque even for open-weight models. To reconcile these issues the researchers: 📌 created a 240 trillion token corpus (DCLM-POOL) available for researchers 📌 ran experiments across multiple compute scales (400M to 7B parameters) 📌 created a new state-of-the-art open-source dataset: DCLM-BASELINE 🏆 Results: A 7B parameter model trained on DCLM-BASELINE achieves 64% accuracy on MMLU, rivaling larger models 𝘄𝗵𝗶𝗹𝗲 𝘂𝘀𝗶𝗻𝗴 𝗹𝗲𝘀𝘀 𝗰𝗼𝗺𝗽𝘂𝘁𝗲. This work opens up new possibilities for more efficient and effective language model training. It's a must-read for anyone interested in the future of AI and natural language processing! 🔗 Read the full paper (attached) and find a link to the paper in the comments. What are your thoughts on this development? Let's discuss this in the comments! 👇 #AI #MachineLearning #NLP #DataScience #ResearchBreakthrough

📰 Out on The New Stack - Dmitry Petrov's latest article: The AI Hype Cycle: From Excitement to Engineering See the full article at the link in the comments, but here is the gist: The initial buzz around generative AI is giving way to a more mature, engineering-focused approach. This shift mirrors historical patterns we've seen with other transformative technologies like the dot-com boom and cloud computing. Key takeaways: 1. Despite widespread adoption (92% of Fortune 500 companies using OpenAI tech), AI often remains in experimental rather than production environments. 2. We're moving from the "shiny new toy" phase to addressing critical challenges in data quality, scalability, and practical implementation. 3. This transition is crucial for sustainable growth and real-world applications of AI. 4. Focusing on engineering and data curation will lead to more stable, predictable, and valuable AI systems. 5. Expect the development of better tools and frameworks to support AI workflows, especially for handling unstructured data. 6. The alignment of AI and data management practices will drive more widespread adoption across industries. While this "deflation" of the AI bubble may seem like a setback, it's a necessary step towards creating truly transformative AI applications. By tackling the core engineering challenges, we're paving the way for AI to deliver on its immense potential in healthcare, finance, and beyond. The AI industry is maturing. Are you ready to move beyond the hype and focus on the engineering that will make AI truly revolutionary? What else do you think will get AI on a path to maturity? Comment below! 👇🏽 #ArtificialIntelligence #AIEngineering #TechTrends #DataScience

This comprehensive article by Bex Tuychiev on DataCamp teaches how to use DVC for large datasets alongside Git to manage data science and machine learning projects. 𝐊𝐞𝐲 𝐚𝐬𝐩𝐞𝐜𝐭𝐬 𝐢𝐧𝐜𝐥𝐮𝐝𝐞: 𝐎𝐯𝐞𝐫𝐯𝐢𝐞𝐰 𝐨𝐟 𝐃𝐕𝐂: It explains DVC as an open-source tool for data and models, similar to how Git manages code and how it supports large file types like datasets and deep learning models unsuited for Git. DVC uses remote storage (e.g. cloud, SSH) to store data and models while tracking dependencies in a human-readable format. 𝐊𝐞𝐲 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬 𝐝𝐢𝐬𝐜𝐮𝐬𝐬𝐞𝐝: 🔹Reproducibility. 🔹Collaboration. 🔹Metrics tracking, and 🔹Pipeline management. Link: https://lnkd.in/djAtDKXA Follow DVC.ai #machinelearning #mlops #ai #datascience

Dmitry Petrov is looking forward to contributing to the Panel Discussion on How Open Source is Fostering Innovation in AI at #InnovateWeek UW–Madison Data Science Institute tomorrow with Paige Bailey, William Falcon, and Nhi Lê See more info in the link!