Andy Le’s Post

New breakthrough from Microsoft: 1-bit LLMs. New models that use ternary values (-1, 0, 1) instead of 16-bit. This makes them 2.7x faster, use 3.5x less GPU memory, and 71x less energy. Bitnet also matches or outperformed traditional models like LLaMA 3B. https://lnkd.in/gGThq842

How to Build llama.cpp on MacOS and run large language models https://lnkd.in/gqgcUAnQ

How to Build llama.cpp on MacOS and run large language models https://lnkd.in/g8WeXHP6

#LLMSys For LLM serving, a homogeneous setting may not be cost-effective. The paper "Efficient and Economic Large Language Model Inference with Attention Offloading" (https://lnkd.in/ed3aRDu2) shows that combining two different GPUs and separating attention/linear calculations (as they have different memory/compute requirement) actually achieves higher throughput per dollar. (I also wondered about serving a language model by combining a 3090 and a much cheaper P40 at home😺)

Text-to-Text Transfer Transformer, T5 for short, is a special variation of transformers developed by Google that treats NLP tasks as text-to-text problems. This enables a unified and highly adaptable approach to a diversity of NLP tasks. In this artical, I am diving deeply into this model, highlighting: ❇ T5 Architecture and applications ❇ T5 fine-tuning using PyTorch ❇ Setting up training environment including GPU ❇ Containerizing the training pipeline with Docker ❇ Saving and loading the finetuned model ❇ Performing inference and evaluation of the model ✴ Despite the T5 model being relatively older compared to the latest advancements in large language models, the principles and techniques demonstrated here remain highly relevant and applicable to various modern architectures. #T5 #FineTuning #GPU #NLP #DataScience #Docker

Microsoft's introduction of 1-bit LLMs. These models use a novel approach where each weight is represented by only 1.58 bits, as opposed to traditional LLMs which use 16-bit floating-point values. The reduction in bit usage improves performance and cost-effectiveness while also demonstrating the potential of dedicated hardware optimized for 1-bit LLM. Thanks Krish Naik for the video. Krish Naik https://lnkd.in/evxt6kjj

Microsoft Open-Sources bitnet.cpp: A Super-Efficient 1-bit LLM Inference Framework that Runs Directly on CPUs The rapid growth of large language models

Microsoft Open-Sources bitnet.cpp: A Super-Efficient 1-bit LLM Inference Framework that Runs Directly on CPUs The rapid growth of large language models

One more open source LLM - DBRx - A New State-of-the-Art Open LLM ! 🌟 https://lnkd.in/gVNuCiJi Trained on 3072 NVIDIA H100s for 90 days - Demand for H100s going High!!! Model parameters: 132b Active parameters: 32b 💡 Despite having 132 billion model parameters, DBRx utilises MOE to efficiently utilise resources, with only 4 experts out of 16 experts used in inference, resulting in active parameters being just 32 billion! 🤯 #DBRx

Microsoft Open-Sources bitnet.cpp: A Super-Efficient 1-bit LLM Inference Framework that Runs Directly on CPUs The rapid growth of large language models