Rutuja Kokate’s Post

Evaluate anything you want | Creating advanced evaluators with LLMs 1. The Importance of Evaluating Language Models 1.1. Understanding the capabilities and limitations of language models is crucial for aligning with business objectives. 1.2. Standard metrics such as perplexity, BLEU scores, and sentence distance often fail to capture the subtle nuances in real-world applications. 1.3. LLMs use 'black box' metrics to assess the quality of generated text using large language models themselves. 2. Building and Running Evaluation Examples 2.1. Build chatbots and RAG using the LangChain framework, as it incorporates simple evaluation functionalities. 2.2. Demonstrate how to create custom evaluators through detailed code examples. 2.3. Discuss the implementation of translation quality assessment and context relevance evaluation. 3. Strategies for Creating Assessment Prompts 3.1. Establishing assessment criteria and defining a numerical scoring scale are key to success. 3.2. Require reasoning behind scores to gain deeper insight into the assessment logic. 3.3. Provide queries and context for reference, and demand a strict response format for ease of parsing. 4. Implementation and Optimization of Evaluation Chains 4.1. Implement a basic evaluation chain class for parsing output scores and reasons. 4.2. Consider the randomness of evaluations by running asynchronously and averaging scores. 4.3. Integrate frameworks to leverage their maximum benefits (optional). 5. Case Studies in Practical Assessment 5.1. Assessment of translation chains from English to French, identifying and reasonably explaining issues. 5.2. Contextual relevance assessment effectively identifies information unrelated to the query. 5.3. Results visualization and experimental tracking through the Langsmith platform. 6. Conclusions and Practical Applications 6.1. Customizing control model performance allows companies to build AI systems that align with their unique business objectives. 6.2. Encourage experimentation and the creation of custom evaluators for specific use cases. 6.3. All code is available on GitHub, facilitating practical application and further development. #LLMPerformance #CustomEvaluators #LangChainTech #TranslationQuality #RealTimeFeedback

With Multilingual Image Generation & Text Prompt, I've been exploring the fascinating intersection of language and art. Using stable diffusion models and Google Translate, I can generate stunning images from text prompts in various languages! 🎨🌍 https://lnkd.in/gHeXJtgm #AI #ImageGeneration #LanguageArt

With the holidays upon many of us, I wanted to add some holiday reading materials for those using LLMs. The article below (by Natalia Kuzminykh here at Winder.AI) is about Calculating Large Language Model Token Counts. This article is a practical guide that aims to demystify the concept of token counts in LLMs, detailing how they influence model performance and interaction. Starting with the basics, it defines a token and how it’s used for text processing. The paper then moves to the tokenization process, examining different models and the tools used for token counting. Finally, it looks at both manual and automated methods for calculating tokens, highlighting their roles in larger datasets. https://lnkd.in/e8Us_p-y Happy Reading! #LLM, #ContextWindows, #TokenCounts, #LanguageModels, #AI

This Bengali Sentence Error Correction Model is trained on 1.3 M Bengali incorrect-correct sentence pairs and archives a score of BLEU: 0.443, CER: 0.159, WER: 0.406, and Meteor: 0.655 when tested on unseen data. For a closer look at the model in action, check out the live demo here: https://lnkd.in/gSsh7u6x The full training process and methodology can be reviewed on GitHub: https://lnkd.in/gzeDC_TK The model was trained on Google Colab for a single epoch, spanning 6500 steps with a batch size of 128. I believe that expanding the dataset and extending the training duration could significantly enhance the model’s performance and utility. That would be the next step for this model. Try this model and let me know your experience. https://lnkd.in/gb6iPqwQ #BengaliNLP #MachineLearning #NaturalLanguageProcessing #Finetune #LLM #DataScience #AI #DeepLearning #HuggingFace #ErrorCorrectionModel

📅 July 10, 2024 AIBuzzWorld Daily Newsletter! Dive into the fascinating world of Artificial Intelligence and be the first to learn about the latest AI news: 1. **LLMs Struggle with Book-Length Text**      • Current long-context LLMs struggle to understand and reason over book-length texts.   • Researchers created NOCHA, a dataset to test LLM's comprehension of lengthy narratives.   • Even advanced LLMs like GPT-4o achieved only 55.8% accuracy.   • Readmore: https://lnkd.in/g2tFiH5g 2. **Quora’s Poe Introduces Artifacts-Like Feature**      • Quora’s Poe now has a feature to create custom web apps within the chat.   • This feature works well with LLMs that excel at coding.   • Users can create interactive experiences like games and data visualizations.   • Readmore: https://lnkd.in/gSJffbB9 3. **New Features in Ollama 0.2**      • Ollama 0.2 now supports multiple chat sessions and running various models simultaneously.   • Users can load different models for tasks like RAG and running agents.   • This update improves efficiency and multitasking.   • Readmore: https://lnkd.in/g-BCN4zH 4. **Wheebot for Quick Landing Page Creation**      • Wheebot allows users to create and edit landing pages via WhatsApp.   • Users can describe their requirements in plain English.   • Wheebot generates and updates sites instantly through encrypted chats.   • Readmore: https://lnkd.in/gsRtzGrd #AI #ArtificialIntelligence #MachineLearning #LLM #Microsoft #Quora #ClaudeAI #Wheebot #TechNews #Innovation #AIUpdates #FutureTech

📅 July 10, 2024 AIBuzzWorld Daily Newsletter! Dive into the fascinating world of Artificial Intelligence and be the first to learn about the latest AI news: 1. **LLMs Struggle with Book-Length Text**      • Current long-context LLMs struggle to understand and reason over book-length texts.   • Researchers created NOCHA, a dataset to test LLM's comprehension of lengthy narratives.   • Even advanced LLMs like GPT-4o achieved only 55.8% accuracy.   • Readmore: https://lnkd.in/gUCjmA_W 2. **Quora’s Poe Introduces Artifacts-Like Feature**      • Quora’s Poe now has a feature to create custom web apps within the chat.   • This feature works well with LLMs that excel at coding.   • Users can create interactive experiences like games and data visualizations.   • Readmore: https://lnkd.in/gAn9g3sp 3. **New Features in Ollama 0.2**      • Ollama 0.2 now supports multiple chat sessions and running various models simultaneously.   • Users can load different models for tasks like RAG and running agents.   • This update improves efficiency and multitasking.   • Readmore: https://lnkd.in/gfuP7yMs 4. **Wheebot for Quick Landing Page Creation**      • Wheebot allows users to create and edit landing pages via WhatsApp.   • Users can describe their requirements in plain English.   • Wheebot generates and updates sites instantly through encrypted chats.   • Readmore: https://lnkd.in/gKA8Tj-N #AI #ArtificialIntelligence #MachineLearning #LLM #Microsoft #Quora #ClaudeAI #Wheebot #TechNews #Innovation #AIUpdates #FutureTech

𝐆𝐞𝐦𝐢𝐧𝐢 1.5 𝐢𝐬 𝐨𝐮𝐭! 𝐀𝐮𝐭𝐡𝐨𝐫𝐬 𝐩𝐫𝐨𝐩𝐨𝐬𝐞 𝐧𝐞𝐚𝐫-𝐩𝐞𝐫𝐟𝐞𝐜𝐭 𝐫𝐞𝐜𝐚𝐥𝐥 𝐨𝐧 𝐥𝐨𝐧𝐠-𝐜𝐨𝐧𝐭𝐞𝐱𝐭 𝐫𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐭𝐚𝐬𝐤𝐬 𝐚𝐜𝐫𝐨𝐬𝐬 𝐦𝐨𝐝𝐚𝐥𝐢𝐭𝐢𝐞𝐬, 𝐜𝐨𝐧𝐭𝐢𝐧𝐮𝐞𝐝 𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐦𝐞𝐧𝐭 𝐢𝐧 𝐧𝐞𝐱𝐭-𝐭𝐨𝐤𝐞𝐧 𝐩𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐧𝐞𝐚𝐫-𝐩𝐞𝐫𝐟𝐞𝐜𝐭 𝐫𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 (>99%) 𝐮𝐩 𝐭𝐨 𝐚𝐭 𝐥𝐞𝐚𝐬𝐭 10𝐌 𝐭𝐨𝐤𝐞𝐧𝐬, 𝐚 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐥𝐞𝐚𝐩 𝐨𝐯𝐞𝐫 𝐞𝐱𝐢𝐬𝐭𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥𝐬. 𝐑𝐞𝐩𝐨𝐫𝐭 𝐩𝐚𝐩𝐞𝐫 𝘎𝘦𝘮𝘪𝘯𝘪 1.5: 𝘜𝘯𝘭𝘰𝘤𝘬𝘪𝘯𝘨 𝘮𝘶𝘭𝘵𝘪𝘮𝘰𝘥𝘢𝘭 𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥𝘪𝘯𝘨 𝘢𝘤𝘳𝘰𝘴𝘴 𝘮𝘪𝘭𝘭𝘪𝘰𝘯𝘴 𝘰𝘧 𝘵𝘰𝘬𝘦𝘯𝘴 𝘰𝘧 𝘤𝘰𝘯𝘵𝘦𝘹𝘵 (𝘣𝘺 𝘎𝘦𝘮𝘪𝘯𝘪 𝘛𝘦𝘢𝘮, 𝘎𝘰𝘰𝘨𝘭𝘦) https://lnkd.in/eNacYzEF 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 "In this report, we present the latest model of the Gemini family, 𝘎𝘦𝘮𝘪𝘯𝘪 1.5 𝘗𝘳𝘰, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. 𝘎𝘦𝘮𝘪𝘯𝘪 1.5 𝘗𝘳𝘰 𝘢𝘤𝘩𝘪𝘦𝘷𝘦𝘴 𝘯𝘦𝘢𝘳-𝘱𝘦𝘳𝘧𝘦𝘤𝘵 𝘳𝘦𝘤𝘢𝘭𝘭 𝘰𝘯 𝘭𝘰𝘯𝘨-𝘤𝘰𝘯𝘵𝘦𝘹𝘵 𝘳𝘦𝘵𝘳𝘪𝘦𝘷𝘢𝘭 𝘵𝘢𝘴𝘬𝘴 𝘢𝘤𝘳𝘰𝘴𝘴 𝘮𝘰𝘥𝘢𝘭𝘪𝘵𝘪𝘦𝘴, 𝘪𝘮𝘱𝘳𝘰𝘷𝘦𝘴 𝘵𝘩𝘦 𝘴𝘵𝘢𝘵𝘦-𝘰𝘧-𝘵𝘩𝘦-𝘢𝘳𝘵 𝘪𝘯 𝘭𝘰𝘯𝘨-𝘥𝘰𝘤𝘶𝘮𝘦𝘯𝘵 𝘘𝘈, 𝘭𝘰𝘯𝘨-𝘷𝘪𝘥𝘦𝘰 𝘘𝘈 𝘢𝘯𝘥 𝘭𝘰𝘯𝘨-𝘤𝘰𝘯𝘵𝘦𝘹𝘵 𝘈𝘚𝘙, 𝘢𝘯𝘥 𝘮𝘢𝘵𝘤𝘩𝘦𝘴 𝘰𝘳 𝘴𝘶𝘳𝘱𝘢𝘴𝘴𝘦𝘴 𝘎𝘦𝘮𝘪𝘯𝘪 1.0 𝘜𝘭𝘵𝘳𝘢’𝘴 𝘴𝘵𝘢𝘵𝘦-𝘰𝘧-𝘵𝘩𝘦-𝘢𝘳𝘵 𝘱𝘦𝘳𝘧𝘰𝘳𝘮𝘢𝘯𝘤𝘦 𝘢𝘤𝘳𝘰𝘴𝘴 𝘢 𝘣𝘳𝘰𝘢𝘥 𝘴𝘦𝘵 𝘰𝘧 𝘣𝘦𝘯𝘤𝘩𝘮𝘢𝘳𝘬𝘴. Studying the limits of Gemini 1.5 Pro’s long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to 𝐚𝐭 𝐥𝐞𝐚𝐬𝐭 10𝐌 𝐭𝐨𝐤𝐞𝐧𝐬, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content."

In the future, how will we get the best search results on a company’s website? Combine a large language model (which understands natural language) + a user’s search history (including what pages they’ve viewed) + multi-query. Read more here: https://lnkd.in/g3zTiQwk #search #multiquery #llm #generativeai

🚀 Tokenizers: The Unsung Heroes of LLMs! 🤖✨ If you’ve ever wondered how language models (like GPT) understand your input, the answer lies in tokenizers. Tokenizers break down your input prompt into smaller chunks called tokens, which help the model "read" and respond in the best way possible. Here's a quick overview of the different types of tokenizers and how they work: Word-Level Tokenizers 📝: They break inputs into individual words. 📌 Simple, but they can have a limited vocabulary and can be inefficient for similar words (e.g., "running" vs. "ran"). Sub-Word Tokenizers 🔍: They split words into smaller parts (like "run" + "ning"). 💡 Efficient and great for handling new words without bloating the vocabulary. Character-Level Tokenizers 🔡: They break inputs down to individual characters. ⚠️ Infinite vocabulary, but not very efficient since it eats up the context window quickly. Byte-Level Tokenizers 🔗: These tokenizers work at the byte level, allowing the model to handle any character or special symbol. 🌍 Extremely versatile—can handle different languages, emojis, and even misspellings without missing a beat! ✨ Bottom Line: Tokenizers determine how language models understand and respond to your input. Choosing the right tokenizer helps in making LLMs smarter and more efficient! 💡 Which tokenizer do you think is the coolest? 😎 Drop your thoughts below! ⬇️ #llm #handsonlargelanguagemodels Credits: Jay Alammar

In the rapidly evolving landscape of Large Language Models (#LLMs), the art of prompt engineering emerges as a critical skill set for optimizing AI performance. It's not merely about asking questions but intricately guiding the AI to produce desired outcomes. For data science and AI development professionals, understanding how to craft, evaluate, and manage prompts is essential. This encompasses structuring inputs that align with specific tasks and refining and tuning these prompts through manual and automated methods to enhance model responsiveness. Moreover, navigating challenges such as prompt overfitting, ambiguity, and maintaining contextual relevance requires a blend of technical expertise, creativity, and strategic thinking. As we explore advanced prompt tuning techniques and integrate metrics for tuning success, we are charting a future where prompt engineering is both an art and a science, pivotal in harnessing the full potential of LLMs for diverse applications. #LLMInnovations #LanguageModelMagic #AIIntelligence #NextGenAI #MachineLearningMastery #PromptEngineering #FutureOfCoding #TextGenerationTech #AIResearchRevealed #GPTGenius