Neural Magic (Acquired by Red Hat)

Check out our latest work on LLM compression and efficient training & deployment!

[vLLM Office Hours #21] vLLM Production Stack Deep Dive Join us for an overview of the components in the vLLM Production Stack (https://lnkd.in/gsSnNb9K) and practical guidance on deploying it effectively. We’ll dive into the technical details, including an in-depth look at the prefix-aware router and its role in optimizing request routing, as well as KV cache offloading and its impact on performance and scalability.

Friends from the East Coast! Join us on Tuesday, March 11 in Boston for the first ever East Coast vLLM Meetup. You will meet vLLM contributors from Neural Magic (Acquired by Red Hat), Red Hat, Google, and more. Come share how you are using vLLM and see what's on the roadmap!

📚 Our very own Yihua Cheng will be presenting in the vLLM office hour this 𝗧𝗵𝘂𝗿𝘀𝗱𝗮𝘆 with Neural Magic (Acquired by Red Hat)!  ⏰ It is time to learn more about 𝗹𝗮𝗿𝗴𝗲-𝘀𝗰𝗮𝗹𝗲 𝗟𝗟𝗠 𝘀𝗲𝗿𝘃𝗶𝗻𝗴 with 𝘃𝗟𝗟𝗠 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗦𝘁𝗮𝗰𝗸! We will be discussing in depth about the design and functionality of the vLLM Production-Stack, an open-source framework to serve LLM models 𝗮𝘁 𝘀𝗰𝗮𝗹𝗲 with 𝗹𝗼𝘄 𝗰𝗼𝘀𝘁 and 𝗲𝗮𝘀𝘆 𝗺𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁. We will also share updates on the 𝗹𝗮𝘁𝗲𝘀𝘁 𝘃𝗟𝗟𝗠 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻.  Mark you calendar for 𝗠𝗮𝗿𝗰𝗵 𝟲, 𝟮:𝟬𝟬𝗣𝗠 𝗘𝗧 / 𝟭𝟭:𝟬𝟬𝗔𝗠 𝗣𝗧 and register at this link: https://lnkd.in/euF8m73q Comment below for anything else you want us to talk about in the office hour! 🔗 Code: https://lnkd.in/gsSnNb9K 📝 Blog: https://lnkd.in/gdXdRhEj 📚 Tutorials: https://lnkd.in/gWz7gW6T 🎥 30-sec demo: youtu.be/RLk8zbQ-eqM #LLM #vLLM #opensource #K8s #AI #AIInfra #Systems

🚀 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗲𝗱 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗥𝟭 𝗟𝗟𝗠𝘀 – 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗳𝗼𝗿 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗲𝗱 𝗳𝗼𝗿 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁! While many companies are still reasoning through the implications of the open-source DeepSeek R1 launch, we've been heads-down refining and compressing these models for deployment to maximize efficiency with quantization while maintaining accuracy. To our knowledge, this is the first comprehensive exploration of quantization for reasoning LLMs, spanning 𝘀𝗶𝘅 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗺𝗼𝗱𝗲𝗹𝘀, 𝗵𝘂𝗻𝗱𝗿𝗲𝗱𝘀 𝗼𝗳 𝘁𝗵𝗼𝘂𝘀𝗮𝗻𝗱𝘀 𝗼𝗳 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗲𝗱 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝘀, and 𝘁𝗵𝗼𝘂𝘀𝗮𝗻𝗱𝘀 𝗼𝗳 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝗲𝗱 𝘀𝗰𝗲𝗻𝗮𝗿𝗶𝗼𝘀. Our findings offer key insights into model behavior, trade-offs, and performance improvements across different quantization techniques. 📊 Specifically, we found: • 𝗟𝗮𝗿𝗴𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀 𝗮𝗿𝗲 𝗲𝗮𝘀𝗶𝗲𝗿 𝘁𝗼 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗲𝗱: 7B+ models retained full accuracy at 8-bit (FP8, INT8) and 97% at 4-bit (INT4) • 𝗦𝗺𝗮𝗹𝗹𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀 𝗻𝗲𝗲𝗱 𝗺𝗼𝗿𝗲 𝘁𝘂𝗻𝗶𝗻𝗴: After more thorough hyperparameter tuning, the 1.5B model recovered 97% at 8-bit and 94% at 4-bit • 𝗖𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝘁 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗯𝗼𝗼𝘀𝘁𝘀: Speedups averaged around 1.5X with some up to 4X, depending on the model size, hardware, and inference scenario 🔍 Some interesting insights: • 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀 𝗵𝗮𝘃𝗲 𝗵𝗶𝗴𝗵 𝘃𝗮𝗿𝗶𝗮𝗻𝗰𝗲: AIME had up to a 7-point standard deviation in pass@1, and to address this generally, we ran 20 different random seeds for each eval • 𝗧𝗼𝗸𝗲𝗻𝗶𝘇𝗲𝗿 𝗶𝘀𝘀𝘂𝗲𝘀 𝗱𝗲𝗹𝗮𝘆𝗲𝗱 𝗿𝗲𝘀𝘂𝗹𝘁𝘀: The DeepSeek tokenizer on Hugging Face was missing the <think> token in chat templates, leading to degraded accuracy even in baseline models If you're interested in diving in more, check out our latest Red Hat research blog (our first piece not pushed solely through Neural Magic (Acquired by Red Hat)!): https://lnkd.in/eENPT8xz Or get hands-on with the Hugging Face collection and start deploying now: https://lnkd.in/eQu_bhMR Let me know what you think and what you want to see next!

Accurately quantized DeepSeek-R1 is here!

[vLLM Office Hours #20] DeepSeek and vLLM DeepSeek is dropping a lot of exciting goodies this week during their Open Source Week, and we’re thrilled to spotlight them at our bi-weekly vLLM Office Hours! We'll dive into "DeepSeek on vLLM: New Features, Optimizations, and More," making it a hands-on learning opportunity for anyone curious about DeepSeek’s innovations and how they work with vLLM. DeepSeek's advancements aren’t just cool tech—they’re reshaping how we build and deploy AI. DeepSeek’s focus on efficiency means you can tackle bigger problems with fewer resources. Plus, with vLLM’s seamless integration, you get these benefits without the headache. This week’s Office Hours is your chance to learn how DeepSeek and vLLM team up. We’ll unpack DeepSeek's features, demo their vLLM integration, and brainstorm what’s next—together.

Make sure to join the first vLLM East Coast meetup in Cambridge, MA! Great opportunity to learn more about production-grade inference serving with vLLM. We are excited to share project updates! The best feature requests are made in person https://lu.ma/7mu4k4xx vLLM Red Hat Neural Magic (Acquired by Red Hat)

It has been an amazing week for open source AI infrastructure as the DeepSeek team releases key components of their infrastructure stack that supports the complex and innovative V3/R1 architecture. https://lnkd.in/eWV7wQgr So far, we have seen: - *FlashMLA*: an efficient MLA decoding kernel for Hopper GPUs, which helps to accelerate attention (the bottleneck for long context "reasoning-style" workloads) - *DeepEP*: the first open-source EP communication library for MoE model training and inference, which helps to enable more complex parallelism schemes for serving the 600B+ parameter model with 256 experts - *DeepGEMM*: an FP8 matmul library that supports both dense and MoE layers, helping to accelerate the first foundation model trained in Fp8 - *EPLB*: an expert-parallel load balancer for V3/R1, enabling more complex expert parallel deployments for at scale workloads At 2pm ET during the Neural Magic (Acquired by Red Hat) Office Hours, Lucas Wilkinson, Tyler Michael Smith, Michael Goin, Simon Mo, and I will dive into these items and cover our progress integrating them into vLLM! Stop by to ask questions! Link to signup: https://lnkd.in/ePqDYgpT

Multimodal Quantization with LLM Compressor! Compression isn't just for text models anymore. With the latest v0.4.0 release of LLM Compressor, we now support multi-modal models, including vision-language (Llama 3.2 Vision) and audio (Whisper Large V2) models! Underlying this enablement are the same productized, state-of-the-art compression algorithms powering smaller, faster models without sacrificing accuracy. Key highlights: 🔹 Vision & audio data pipeline support added. 🔹 Model tracing abilities for vision & audio models for performant GPTQ compression. 🔹 Verified implementations showing >99% accuracy recovery. 🔹 Open-source, fully customizable, and natively supported through vLLM for seamless deployments. Want to try it out? Dive into the open-source examples: https://lnkd.in/eSQgHS2m Want to learn more? Read through our blog post: https://lnkd.in/e8MNy8kN What would you like us to work on next? Let me know in the comments!