Baseten’s Post

We’ve recently contributed FP8 support to vLLM in collaboration with Neural Magic -- with this feature, you can see up to a 1.8x reduction in inter-token latency, with >99% accuracy preservation! A common concern with FP8 is whether users will experience accuracy degradation. To address this, Neural Magic has produced many checkpoints for key models with >99% accuracy preservation across a wide range of benchmarks (https://lnkd.in/gTimN5dZ), including: - Llama3-70b - Mixtral 8x7b - Llama3-8b You can easily try this out on vLLM, and read more about the feature here -- https://lnkd.in/gzKJqerB

When it comes to efficiently serving LLMs, we often hear about quantization — INT8 quantization in particular. Turns out FP8 often has advantages over INT8 in two dimensions: a) model output quality, and b) inference performance. https://lnkd.in/gVMeMYm7

It's time for the next Hugging Face accelerate release, and there is a LOT of ground to cover! From new optimizer supports to FP8 fixes to DataLoader improvements and more, let's dig in: * We've added support for the schedule-free optimizer released by Meta earlier this month, as well as the new LOMO optimizer! For schedule-free, no new changes are needed. For LOMO, just pass in the learning rate during backward() * We've reduced the vRAM needed when using FP8/TransformersEngine by autocasting the model in BF16 (like usual when doing BF16 MP) to keep gradients low, before it was in full FP32. Note that the original model weights will still be in FP32 for stability reasons like normal * We have a fantastic new piece of documentation has been added helping lay out the core differences between DeepSpeed and FSDP in a quick and easy to digest manor, including a key confusion answered towards why we autocast during Mixed Precision. See the comments to find that link * A new slew of distributed examples have been added thanks to Marc Sun. These don't use PiPPy or big model inference, just raw DDP to show you how to leverage accelerate when in an eval scenario. Check them out in the examples/inference folder on the repo! * We've added support for MoE models in DeepSpeed and let you pass in `auto` for gradient clipping in your config * Prepared DataLoaders can now leverage a new `non_blocking` argument to help increase speed when training by reducing cudaStreamSynchronize calls. Just set `non_blocking=True` in your `DataLoaderConfiguration` (and for the best results, use pin_memory=True) * And PLENTY of bug fixes! Give it a whirl today, `pip install accelerate -U`

This is a fantastic (if not obvious) result. Using LLMs together in an ensemble outperforms single models. Paper below: https://lnkd.in/ghSWhyGn

Tech revolution 🚀 unleashed 🌟 with NVIDIA's LLM optimization 💡 & FP8 quantization 📈. Embrace #AI, and boost productivity. Post by: NVIDIA AI Follow me for the latest AI advancements and tools that power productivity and stay ahead in the smart technology curve. #AI #NVIDIA #LLM #Mistral #TensorRT #GPU #TechRevolution

High performance #generativeAI checklist: ✅LLM…………........……#Mistral ✅Optimization………#TensorRT-LLM ✅Quantization……..…FP8 A GPU-backed notebook awaits your input #LLMs. ✨ Click to quantize & code: https://lnkd.in/gU4uSGKm

I am glad to share my latest blog post on Quantization. In the era of extravagance, where models casually cross 100B parameters, devouring over 500GB of GPU memory and costing millions of dollars for a single training session, quantization comes in as a prudent accountant. It ensures that models refrain from indulging in excessive memory consumption while minimizing any loss in model quality. In this blog post, we aim to demystify this potent mathematical framework using intuitive explanations, relatable examples, and accessible language. We will also delve into the fancy jargon and the ugly math that come along with quantization, just deeply enough to allow readers to navigate research papers and documentation on quantization libraries. The objective is to make these esoteric concepts more approachable and less daunting. So buckle up as we embark on this journey, as we learn how to take mammoth ML models, and prune them down to preserve only the essential. https://lnkd.in/eUrNEHVu #quantization #hpc #machinelearning #int8

Knowing float point precision is essential in understanding how quantization works in ML and why you shouldn't quantize the model hastily. The more the precision the bigger the memory needed to store them and fine-tune them so reducing precision without losing information becomes challenging. There are various techniques for quantizing ML models, but understanding why you need quantization and how much information you lose when you go from FP32 to FP16 is essential. For example in this image below 7.567856 is represented as a signed 32-bit but when quantized to 16 bits the information available is just 7.566, meaning we lot the rest of the FP values. Doing this for billions of weights can lead to severe performance degradation.

Since most people like to use vLLM, this might be an incredible speed-up! Notes from the blog > Fireworks FP16 Mixtral model implementation is superior to the one from vLLM > Fireworks FP8 implementation significantly improves over the already quite efficient Fireworks FP16 implementation. > Because FP8 shrinks model size 2x, it allows for more efficient deployment. > Combined with memory bandwidth and FLOPs speed-ups this results in 2x improvement of the effective requests/second. Overall, Fireworks FireAttention FP8 implementation provides the best tradeoff for LLM serving on the accuracy/performance trade-off curve. #llms #mistral

In the machine learning jargon FP32 is called full precision (4 bytes), while BF16 and FP16 are referred to as half-precision (2 bytes). On top of that, the int8 (INT8) data type consists of an 8-bit representation that can store 2^8 different values (between [0, 255] or [-128, 127] for signed integers).