DDN’s Post

#LLMSys For LLM serving, a homogeneous setting may not be cost-effective. The paper "Efficient and Economic Large Language Model Inference with Attention Offloading" (https://lnkd.in/ed3aRDu2) shows that combining two different GPUs and separating attention/linear calculations (as they have different memory/compute requirement) actually achieves higher throughput per dollar. (I also wondered about serving a language model by combining a 3090 and a much cheaper P40 at home😺)

Microsoft released a groundbreaking paper proposing a technique that achieves performance and perplexity on par with full FP16 models of the same size, but using significantly fewer resources. This approach enables fitting a 120-billion parameter model on a single consumer GPU with only 24GB of VRAM. This development has the potential to democratize access to powerful language models for a wider range of users. https://lnkd.in/gRZfSRm4

Retrieval Augmented Generation: a simple and cost-effective approach to introducing new data to a Large Language Model. RAG can be used to extend the already powerful capabilities of LLMs to specific domains or an organization's internal knowledge base, without the need to retrain the model. Check out this blog by NVIDIA: https://lnkd.in/egEzGvrN [Demo] Q&A with RAG (using an open source LLM and Embedding model from Hugging Face hub): https://lnkd.in/e8tpEqCQ

Learn about training open source large language models from a great deeplearning.ai & predibase talk: https://lnkd.in/eeXm7xX5 (best overview of lora and qlora I've seen + a noteboook)

FastGen: Cutting GPU💻 Memory Costs Without Compromising on LLM Quality

Make Any* LLM fit Any GPU in 10 Lines of Code https://bit.ly/41x4qms

I'm thrilled to spotlight this customer story: Perplexity and NVIDIA's groundbreaking efforts on AWS to accelerate Large Language Model (LLM) inference. 🔹 **Objective**: Perplexity introduces `pplx-api`, an efficient API tool leveraging NVIDIA GPUs, including the powerful NVIDIA® TensorRT™-LLM, designed for lightning-fast LLM inferences. This innovation is a game-changer for developers eager to integrate cutting-edge LLMs into their projects seamlessly. 🔹 **Achievements**: - **Fast and Efficient**: Perplexity's `pplx-api` delivers unprecedented speed, efficiently handling high traffic while minimizing costs. - **Infrastructure Optimization**: Running on Amazon EC2 P4d instances powered by NVIDIA A100 Tensor Core GPUs, with plans to shift to even more powerful P5 instances with NVIDIA H100 GPUs, `pplx-api` is set to redefine performance metrics. - **Cost Efficiency**: A remarkable cost reduction, saving $600,000 annually by optimizing LLM inference deployment, showcasing a 4X reduction by integrating pplx-api. 🔹 **NVIDIA Inception Program**: Perplexity's participation in this program underscores their commitment to leveraging NVIDIA's cutting-edge technology, fostering rapid growth and innovation. 💡 **Exploring the Future**: As AI continues to evolve, partnerships like Perplexity and NVIDIA on AWS will be pivotal in harnessing the full potential of generative AI and LLMs, paving the way for more efficient, scalable, and cost-effective solutions in the cloud.

New Project Update: Fine Tuning 2B Instruct Gemma model on Indian History Dataset Using PEFT Hardware Used : Free Single P100 GPU by Kaggle The process is lightning-fast, taking less than 30 minutes depending on the dataset's size. This project showcases the potential of Small language models and PEFT methods. With readily available Free GPUs online, there's immense future scope for fine-tuning domain-specific data in minutes. https://lnkd.in/dP9tzWDk

Microsoft's FP6-LLM introduces a new method for optimizing large language models (LLMs) using six-bit quantization (FP6). This approach reduces the size of LLMs while maintaining their quality. The innovation includes TC-FPx, a GPU kernel design that supports various quantization widths, improving the balance between inference cost and model quality. The FP6-LLM method notably enables running complex models like LLaMA-70b on a single GPU with up to 2.65 times more efficiency compared to previous methods. Paper page here : https://lnkd.in/eD-zmPEd

Text-to-Text Transfer Transformer, T5 for short, is a special variation of transformers developed by Google that treats NLP tasks as text-to-text problems. This enables a unified and highly adaptable approach to a diversity of NLP tasks. In this artical, I am diving deeply into this model, highlighting: ❇ T5 Architecture and applications ❇ T5 fine-tuning using PyTorch ❇ Setting up training environment including GPU ❇ Containerizing the training pipeline with Docker ❇ Saving and loading the finetuned model ❇ Performing inference and evaluation of the model ✴ Despite the T5 model being relatively older compared to the latest advancements in large language models, the principles and techniques demonstrated here remain highly relevant and applicable to various modern architectures. #T5 #FineTuning #GPU #NLP #DataScience #Docker