novita.ai’s Post

The Abacus.ai model "Smaug-72B" is the new king of open-source LLM's! According to the latest rankings from the Hugging Face Open LLM leaderboard, our model outperforms both GPT-3.5 and Mistral Medium. Want to learn more? You can reach me at nate@abacus.ai

GPT-4 Turbo is a large multimodal model that accepts both image and text inputs and generates text outputs https://lnkd.in/gTEjHkjg

🔍 Technical Deep Dive: Building GPT-2 from Scratch 🔍 Thrilled to announce that I've completed a personal project where I built and trained a GPT-2 (124M) model from scratch, inspired by Andrej Karpathy's comprehensive tutorial. Here's what went down: 🧠 Architecture: Implemented GPT-2 architecture, focusing on attention mechanisms and transformer blocks. 📊 Dataset: Trained on a curated dataset from the Jeopardy game show, making the model endlessly generate the question, prize money, and answer. 🛠️ Challenges and Optimizations: Overcame hurdles in data preprocessing and iteratively explored all the optimizations Andrej Karpathy discusses in his video, leading to a 3x increase in training efficiency after implementing all the optimizations. This project not only deepened my understanding of NLP and transformers but also honed my coding and problem-solving skills. Massive thank you to Andrej Karpathy for providing such an amazing tutorial. 📹 Video Link: https://lnkd.in/gZxZuqJF 📂 Colab Notebook: https://lnkd.in/gsZ2TK3F #MachineLearning #AI #DeepLearning #NLP #GPT2

🎥 Recently completed an amazing video by Andrej karpathy and learned about self attention, masking techniques, optimized matrix multiplication, etc etc.... in the context of "Transformers - Attention Is All You Need." 🌟🌟 If you're passionate about AI and deep learning, you've got to check out this amazing YouTube video. It walks you through a complete implementation of a Transformers, offering clear explanations every step of the way. 🔗 Watch the full video here: https://lnkd.in/gNvrkBAM Whether you're new to Transformers or looking to deepen your understanding, this video is a must-watch. Share your thoughts in the comments. #AI #DeepLearning #Transformers #SelfAttention #Selfattention #encoderdecoder #genai #nlp #pytorch #aiml #machinelearning

Google's new paper about infinite context: Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention https://lnkd.in/gKbdRxaC #inifinitecontext #context #AI #LLM #Google

🎓 Learning ML and LLM from Andrej Karpathy 🎓 I recently watched an insightful video by Andrej Karpathy, “Let’s Reproduce GPT-2,” where he builds a GPT-2 model from scratch in PyTorch. The video is so underrated and it covers so many concepts and techniques in ML, just to list a few: - 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧: Using `torch.compile`, mixed precision, and kernel fusion. - 𝐀𝐝𝐯𝐚𝐧𝐜𝐞𝐝 𝐌𝐞𝐭𝐡𝐨𝐝𝐬: Flash attention, gradient accumulation, and distributed data parallel (DDP). - 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧: Loading datasets, training loops, system concepts, and evaluating performance. The dataset used, 𝐅𝐢𝐧𝐞𝐖𝐞𝐛 by Huggingface, very recently developed, demonstrated how few quality data can produce equal performing results. The video is 4hrs+, but it's worth every minute after watching. It's a condensed show of how you can apply theories and understanding to practices. Link to the video: https://lnkd.in/gzap7Jnu FineWeb Paper: https://lnkd.in/gSa9iEUd Additionally, Karpathy is prepping for an LLM class called 𝐋𝐋𝐌𝟏𝟎𝟏𝐧: https://lnkd.in/gRCqVMrh. Follow closely so you don't miss this! #MachineLearning #DeepLearning #GPT2 #AI #PyTorch #ModelTraining #Optimization #DataScience #AndrejKarpathy

Mistral Large vs GPT-4 vs Gemini Advanced performance comparison https://flip.it/BwhpOF

Just finished "GPT-4 Turbo: The New GPT Model and What You Need to Know" by Jonathan Fernandes! Check it out: https://lnkd.in/gku6PXe5 #gpt4

An astounding and stimulating article on Large Language Models (LLM). Regardless of your familiarisation with this field, I recommend a read if you have heard of "Artificial Intelligence". By the way, this review could have been the prologue of the popular movie trilogy "The Matrix". #artificialintelligence #aimodels #dataanalytics #deeplearning #technology #largelanguagemodels #neuralnetworks #statistics