Yosef Worku Alemneh’s Post

I'm thrilled to spotlight this customer story: Perplexity and NVIDIA's groundbreaking efforts on AWS to accelerate Large Language Model (LLM) inference. 🔹 **Objective**: Perplexity introduces `pplx-api`, an efficient API tool leveraging NVIDIA GPUs, including the powerful NVIDIA® TensorRT™-LLM, designed for lightning-fast LLM inferences. This innovation is a game-changer for developers eager to integrate cutting-edge LLMs into their projects seamlessly. 🔹 **Achievements**: - **Fast and Efficient**: Perplexity's `pplx-api` delivers unprecedented speed, efficiently handling high traffic while minimizing costs. - **Infrastructure Optimization**: Running on Amazon EC2 P4d instances powered by NVIDIA A100 Tensor Core GPUs, with plans to shift to even more powerful P5 instances with NVIDIA H100 GPUs, `pplx-api` is set to redefine performance metrics. - **Cost Efficiency**: A remarkable cost reduction, saving $600,000 annually by optimizing LLM inference deployment, showcasing a 4X reduction by integrating pplx-api. 🔹 **NVIDIA Inception Program**: Perplexity's participation in this program underscores their commitment to leveraging NVIDIA's cutting-edge technology, fostering rapid growth and innovation. 💡 **Exploring the Future**: As AI continues to evolve, partnerships like Perplexity and NVIDIA on AWS will be pivotal in harnessing the full potential of generative AI and LLMs, paving the way for more efficient, scalable, and cost-effective solutions in the cloud.

🧠 Now you can run LLaMA 3 70B on a Single 4GB GPU with AirLLM and Layered Inference 🔥 Layer-wise inference is the "divide and conquer" approach to efficiently run large language models (LLMs) on limited GPU memory. AirLLM has implemented this technique without using quantization, distillation, or pruning. Key points to note: 📌 LLMs are large due to their numerous identical transformer layers (e.g., 80 layers in a 70B model). 📌 During inference, each layer is independent and relies only on the output of the previous layer. 📌 AirLLM loads one layer at a time from disk, executes it, and frees the memory after, requiring only 1/80 of the full model's memory (around 1.6GB). 📌 Flash Attention optimizes CUDA memory access for multi-fold speedups. 📌 Model files are sharded by layers, and HuggingFace Accelerate's meta device

Elon Musk's Groq Surprises with Strong Cost - Speed Ratio Groq was founded by Musk to compete against OPENAI and others on the strongly growing LLM market. Now, a study was conducted to compare language models based on infecence cost and speed, focusing on text generation about Nvidia's business from its 10-K report. The methodology included 10 trials per model, limiting outputs to 1000 tokens, aiming to find cost-effective and speedy solutions. Groq surprisingly emerged as an excellent performer. And while one may argue that different test conditions may result in a different results, it is a hint that Groq may be a model to be taken serious moving forward. This efficiency makes Groq a potential candidate e. g. into RAG pipelines, highlighting its ability to enhance text generation tasks while maintaining low operational costs. #AI #TechEfficiency #Groq

Running CNN on memory-constrained SoC is hard Stuck with Feature Maps activations exceeding your embedded SRAM size ? Slice Features Maps and go past memory wall with tinyRAPTOR NPU Capture more value by running bigger AI models, yet on tiny devices.

Nvidia just publised a curated collection of utilities to build QA-RAG like solutions, including datasets and models weights. It is particularly important to point out the multi-turn encoder, that highlights the importance of short term memory during information retrieval step. Instead of using your LLM (or a smaller version) to reformulate your query, simply ingest your whole short-term memory in the encoder. Not only does it drastically reduce latency, but it also seems to be a much more natural approach. Kudos to NVIDIA AI team! https://lnkd.in/dRj3eVMU

Bringing Natural Language Search to the world of PDF extract automation! Meet Monocle 🧐, a powerful, open-source tool backed by Large Language Model (LLM) for advanced binary code analysis. Blessed with the ability to decompile binaries in light-speed timing 🚀, it swiftly nails down potentially vulnerable code sections tailored to input search criteria. This monumental technology stands to reshape our code-risk identificationn framework.🔍 Mind though, mastering Monocle's full power calls for muscular gear with at least 16GB RAM and robust Nvidia GPU inside.💻 Sure, it can tango with something modest, but expect pace cuts. A shining symbol of relentless innovation in our intelligent automation sphere. Together, let's dive deeper into this immersive alteration tsunami, pushing boundaries for a more secure, intuitive future.🚀💡🌐 Peek deeper into Monocle story here: https://lnkd.in/dVKv_HG4 #IntelligentAutomation #Innovation #Monocle

GroundingDINO: Bridging Language and Vision for Open-Set Object Detection 🔍 #GroundingDINO – a groundbreaking project that marries language and vision to achieve open-set object detection. 💡 Whether it's category names or descriptive expressions, GroundingDINO can detect arbitrary objects with astonishing accuracy. 👉 Check out our latest blog post where we delve into the fascinating world of GroundingDINO and its application in open-set concept generalization: https://lnkd.in/gMatsk_A In this tutorial, we cover step-by-step: ✅ Setting up the environment with GPU support ✅ Installing necessary libraries and downloading pre-trained model weights ✅ Loading the GroundingDINO model and processing images ✅ Demonstrating object detection with human language inputs 👩💻 Dive into the tutorial and unlock the potential of GroundingDINO in object detection! Don't miss out on this game-changing technology! #ObjectDetection #GroundingDINO #MachineLearning #AI #ComputerVision #OpenSetDetection #Innovation #Tech #LinkedInLearning

If you are like me and are starting to fine-tune your own models, you will find QLoRA a breath of fresh air. This blog post provides resources for using 4-bit models and QLoRA, including instructions for loading and fine-tuning models in 4-bit precision. I was able to get it to run on Google Colab once I upgraded to a GPU A100. https://lnkd.in/eTMzhNYh

I just wrote this blog with Cheng Su about The Milvus Project Zilliz and Anyscale. If you have a small laptop 💻 (Apple M2 16GB RAM) typical 48K embeddings don’t fit in memory. This causes #Pandas to go really slowly! ⏳ ⏱️ 🚀 If you want quick iterations in Minutes! on your laptop, instead of waiting Hours! with Pandas, Ray Data / Anyscale + Milvus / Zilliz is the way to go!

Speeding up Mixtral by 4x with FireAttention 🔥 FireAttention is a custom CUDA kernel from Fireworks AI that is optimized for Multi-Query Attention models. Benchmarked on the popular Mixtral-8x7B, their new FP8 variant is leading to ~4x faster inference compared to vLLM FP16. In 2024, we will probably see more kernel and even chip level advancements that typically are taking longer to realize than the innovations we'e seen last year. The caveat for the Open Source community is that lower level soft- and hardware are also more defensible and will thus often remain proprietary. Nevertheless, I'm excited to see what this year will have in store for us. [Blog] https://lnkd.in/e4f__A7U