Neural Magic

For the past 6 months, we at Neural Magic have been hosting bi-weekly #vLLM office hours in collaboration with the vLLM project, its committers, and the wider open source community. If you haven’t attended yet, join us soon! Here's what's coming up for the rest of 2024: 🔹 Oct. 30: SOTA Tool-Calling Implementation in vLLM Explore advanced tools and OpenAI-style functions for vLLM with Kyle Mistele. We’ll cover compatibility, standardization, and streaming. 🔹 Nov. 14: The Impact of Disaggregated Prefill and KV Cache Storage in vLLM Dive into vLLM’s new storage architecture and optimizations for performance, scalability, and efficiency with vLLM Committer, Kuntai Du. 🔹Dec. 5: Deep Dive into Machete, a Mixed-Input GEMM Kernel Optimized for NVIDIA Hopper GPUs We’ll cover how Machete's memory-bound optimizations and pre-shuffling deliver 42% faster throughput on large models. Lucas Wilkinson recently wrote a blog on this topic—comment for the link! 🔹Dec. 19: vLLM Project Update: 2024 Retrospective and 2025 Roadmap Wrap up the year with us as our office hours host, Michael Goin, reflects on 2024’s milestones and shares an early look at 2025. 👉 Register for all sessions (and explore previous recordings!): https://lnkd.in/euF8m73q

Our latest vLLM office hours recording is ready! We delved into Mistral AI's architecture choices with Patrick von Platen and shared how to efficiently deploy their models using vLLM. We also explored key updates in vLLM v0.6.3, including: ▶️ Experimental fullgraph torch.compile ▶️ Feature Compatibility Matrix ▶️ Machete w4a16 kernel for Hopper GPUs ▶️ VLM support: GLM-4V, Molmo, NVLM-D ▶️ Tool-Use support: Llama 3.1+3.2, InternLM2.5 ▶️ Reward LM support: Qwen2.5-Math-RM-72B 📺 Watch the recording: https://lnkd.in/esMrPCS2 📄 View the slides: https://lnkd.in/eGqxT-KJ

How does quantization impact the performance of LLMs? Only minimal! 🤯 A new study ran 500,000 different evaluations on Meta Llama using different quantization strategies. The impact is <1%, but the benefits are up to 2.4 faster inference and 3.5 model size reduction! 🔥 TL;DR; 💯 Quantized models achieve 99% accuracy recovery compared to full-precision 🚀 Up to 2.4x speedup and 3.5x model size reduction with quantization. 📊 Tested Llama 3.1 8B, 70B, and 405B models on OpenLLM Leaderboard, ArenaHard, HumanEval, and text similarity metrics. 🥇W8A8-FP8 dynamic yields the best results 🤗 Quantized models available on Hugging Face. Blog: https://lnkd.in/d86-AiGG Kudos to Neural Magic for their work on Quantization and comprehensive testing of models! 🤗

Our team continues to deliver highly optimized models right after their release! We’ve just compressed NVIDIA’s Llama-3.1-Nemotron-70B-Instruct model, which is making waves for outperforming GPT-4o on the ArenaHard benchmark. 🚀 Post-compression, at FP8 precision, it achieves 99.4%+ accuracy recovery on both the ArenaHard and OpenLLM Leaderboards (v1 & v2), while being 2x smaller and 2x faster. #AI #ModelCompression #LLMs #NVIDIA #MachineLearning

We've just added more sessions to our bi-weekly vLLM office hours! Whether you're scaling AI deployments or optimizing model performance, these sessions offer valuable insights. 🧠 Topics & Dates: 🗓️ Oct. 17: Deep Dive into Mistral AI on vLLM 🗓️ Oct. 30: SOTA Tool-Calling Implementation in vLLM 🗓️ Nov. 14: The Impact of Disaggregated Prefill and KV Cache Storage in vLLM 🗓️ Dec. 5: Machete Kernel – Performance Optimization for H100 GPUs 🗓️ Dec. 19: Year-End Review – 2024 vLLM Achievements & 2025 Roadmap 🔗 Find and register for all upcoming sessions here: https://lnkd.in/euF8m73q

Join us tomorrow for bi-weekly vLLM office hours to hear about Mistral AI model architectures and how to deploy Mistral's models efficiently with vLLM!

It was a privilege to sit down with Chris Brandt from FUTR.tv and explore the state of AI today. We covered everything from the promise of smaller, specialized models to the real risks enterprises face when adopting AI. Key points in the podcast: 1️⃣  Why is AI replacing creative fields and not mundane tasks 2️⃣ The pitfalls of larger, general models and the importance of scaling smaller models 3️⃣ The power of open-source AI 4️⃣ Smarter algorithms for the future Watch the full talk on YouTube: https://lnkd.in/edh_MrPM Or listen to the podcast: https://lnkd.in/e8PkccSG What’s your take on these trends? Are enterprises ready for the shift to AI? How will creativity and automation coexist? Let’s discuss in the comments!

🚀 Introducing Machete: A New Mixed-Input GEMM Kernel for NVIDIA Hopper GPUs 🚀 We’re excited to unveil Machete, a major step forward in high-performance LLM inference. By focusing on w4a16 mixed-input quantization, Machete reduces memory usage by ~4x, making deployments significantly more efficient in memory-bound regimes. While compute-bound performance remains in line with FP16, Machete truly excels in optimizing memory bandwidth for GPTQ-style models. 🧠 Key highlights of Machete: - Built on CUTLASS 3.x, utilizing wgmma tensor core instructions, overcoming limitations in compute-bound scenarios. - Weight pre-shuffling for faster shared memory loads and reduced bottlenecks in large-scale LLMs. - 128-bit shared memory loads for high throughput and further reduced latency. Optimized upconversion routines to maximize tensor core utilization by converting 4-bit elements to 16-bit efficiently. With Machete, we’ve achieved a 29% faster input and 32% faster output token throughput on Llama 3.1 70B, with a TTFT of <250ms on a single H100 GPU. And that’s not all... On a 4xH100 setup, Machete delivers a 42% throughput speedup on Llama 3.1 405B—with more optimizations on the way, including support for w4a8 FP8, AWQ, QQQ, and low-batch-size performance. 🎉 A huge shoutout to Lucas Wilkinson for leading the development of Machete and the team at NVIDIA AI for continual support! Special thanks to 3Blue1Brown and the Manim community for the amazing animations that helped visualize these optimizations. Read the full blog here: https://lnkd.in/ggKYbmKR #AI #LLMs #GPUs #NVIDIA #vLLM #DeepLearning #MachineLearning

Our CTO, Mark Kurtz, was recently featured on The Feed podcast, sharing insights on the current state of AI and what’s happening behind the scenes. This episode offers valuable perspectives on AI development and deployment. Check it out here: https://lnkd.in/egyqvfJp

Lily (Xiaoxuan) Liu, vLLM Committer and PhD student at UC Berkeley, joined our recent #vLLM office hours to share valuable insights into speculative decoding - what it is, how it performs in vLLM, and its applications. In addition to this deep dive with Lily, we covered the latest features in vLLM v0.6.2, including: - Llama 3.2 Vision support - MQLLMEngine for API Server - Beam search externalization Watch the full session: https://lnkd.in/eBPv8kNY View the slides: https://lnkd.in/eu-JWpWp Join us for our upcoming office hours, including a deep dive into Mistral AI on vLLM with Patrick von Platen on October 17th and SOTA tool-calling implementation in vLLM with Kyle Mistele on October 30th. Explore and register here: https://lnkd.in/euF8m73q