Pascal Biese’s Post

Just read a mind-blowing blog from Fireworks AI on LLM quantization. Here are my key takeaways: 1. Quantization often doesn't reduce output quality 2. It's an easy performance boost 3. Has minimal impact on MMLU scores 4. How to properly evaluate quantized LLMs: KL divergence > task-based metrics Expose logprobs for community analysis 5. Fireworks AI: Great serving option Rumored to use AMD GPUs (cost-effective, more memory, and therefore larger context windows.) I'm pretty sure this is why Fireworks' pricing blows everyone else's out of the water. Based on this, I think most models served in production should be quantized in some way, but there doesn't yet seem to be a one-size-fits-all quantization technique yet. Full blog here: https://lnkd.in/gP5RGimB

We’ve seen minimal to no impact in quality and obvious improvements in performance and cost for inference when running quantized models including Llama 3.1 at InternPro.ai Then consider the crazy performance and memory you have available when using clusters from TensorWave with the inference engine from MK1 and running fine-tuned model using Predibase you have the perfect solution. Price, performance and privacy for your data. There is no reason to use OpenAI… Although after reading this we’ll be exploring using Fireworks AI 🚀

Quantized Models are Production Ready 📊✅

Now here’s a revolutionary idea …. Watch this snippet from MadMoney “SQL processing with GPUs” Kinetica is the GPUdatabase for analytics and generative AI with real-time RAG. GPU ready today, battle tested (in gov), HA all here now with ansi compliant SQL, vector, graph and NL2SQL processing working on Nvidia’s RAPIDS, NeMo and NIM.

There’s a new Raspberry Pi AI Kit for only $70. This extremely cheap hardware lets you run neural networks for object detection, semantic and instance segmentation, pose estimation, and more. The main value prop here is that the camera comes pre-integrated with the AI framework so you don’t have to write any drivers. Some specs: - 13 tera-operations per second (TOPS) of inference performance - Single-lane PCIe 3.0 connection running at 8Gbps - Full integration with the Raspberry Pi image software subsystem - Compatibility with first-party or third-party cameras - Efficient scheduling of the accelerator hardware: run multiple neural networks on a single camera, or single/multiple neural networks with two cameras concurrently

New Image AI model was just released, results are very impressive Looks to be on par with state of the art - mid journey v6 Open weights, non commercial licence. Released by Black Forest Labs - Flex.1 You can download it here - https://lnkd.in/gbJmPUkN You will need a enterprise GPU to run this at any reasonable speed, but a 12gb consumer grade GPU will run the q8 version, at around 1min per image based on what I have read. The image is from here --> https://lnkd.in/g99jC26H Many more example images in the above link

Nvidia just publised a curated collection of utilities to build QA-RAG like solutions, including datasets and models weights. It is particularly important to point out the multi-turn encoder, that highlights the importance of short term memory during information retrieval step. Instead of using your LLM (or a smaller version) to reformulate your query, simply ingest your whole short-term memory in the encoder. Not only does it drastically reduce latency, but it also seems to be a much more natural approach. Kudos to NVIDIA AI team! https://lnkd.in/dRj3eVMU

Quantization is a powerful technique that can significantly improve machine learning model speed and throughput. Check out this blog to explore two different quantization setups, and the benefits of using both👇

Quantization is a powerful technique that can significantly improve machine learning model speed and throughput. Check out this blog to explore two different quantization setups, and the benefits of using both👇

Quantization is a powerful technique that can significantly improve machine learning model speed and throughput. Check out this blog to explore two different quantization setups, and the benefits of using both👇