Feedback Intelligence

As we step into 2024, we are excited to share a new case study authored by Chinar Movsisyan and Erik Harutyunyan on Emotion recognition application. In this post, you will learn about: 🔄 A Comparative Look at Active Learning versus Manot's Actionable Insights 📚 Manot’s comprehensive guidance on Identifying Edge Cases 📈 Experiment Results on SOTA DAN Architecture Your feedback is super helpful for us. Take a moment to read, share, and leave your comments.  #emotionrecognition #activelearning #modelevaluation #actionableinsights

Large Language Model (LLM) applications require continuous improvement to deliver exceptional user experiences. However, AI teams often struggle to gather actionable insights from real-world usage. Chinar Movsisyan, founder and CEO of Feedback Intelligence will address this challenge head-on. Key takeaways: - Overcoming common obstacles in optimizing LLM applications - Harnessing implicit feedback from user behavior for measurable improvements - Accelerating the development of smarter, more responsive LLM applications Why it matters: As LLM-powered applications become increasingly prevalent, fine-tuning and optimising these systems is crucial. This talk offers a unique perspective on harnessing real-world usage data to create smarter, more responsive AI applications. Whether you're an AI practitioner, product manager, or simply curious about the future of LLM technology, this session promises valuable insights for anyone working with or interested in AI. Chinar brings over 8 years of deep learning experience, from research labs to venture-backed startups. Her expertise in mission-critical AI applications across healthcare, drones, and satellites makes her uniquely qualified to address the complexities of LLM optimization. https://lnkd.in/dBWq9t6V #AIOptimization #MachineLearning MLOps World Generative AI World

When building LLM apps, dev and data science teams focus on one set of metrics, while PMs, POs, and business teams focus on another aka “dev metrics” and “business metrics.” Dev and DS teams typically zero in on technical metrics like accuracy, precision, recall, or F1 scores. These are crucial to making sure the AI system works well technically, but they don’t always tell you how it impacts the business. On the flip side, business teams care about things like customer satisfaction, user intent, correctness, or helpfulness - metrics that tie directly to business goals. One big challenge? Translating dev metrics into business metrics - and the other way around. Take this example: Imagine a chatbot used by agents at an insurance company. On paper, it’s performing great - high accuracy, F1 scores, etc. - all things the dev team tracks. But when agents use it, they’re getting a lot of unexpected outputs on the specific questions they’re asking. This highlights the real challenge: bridging that gap between dev and business metrics 🌉 Figuring out how to connect them is key to moving from just a PoC to a full-scale AI solution that solves actual business problems.  I’d love to hear how you approach this!

After pivoting to LLMs, I've spoken to hundreds of practitioners and companies to understand their LLM app development processes and pinpoint where they struggle most. Here’s the TL;DR: When AI teams start building LLM applications for a specific use case, they typically begin with a small data set (100-300 data points) to evaluate the app (whether it’s a chatbot, conversational AI, or an agent) using familiar metrics like correctness and relevance. Often, they rely on open-source libraries for this (e.g., Ragas). After some analysis, they make a decision to put the app into production, asking users to provide feedback—essentially to let them know when things go wrong. But here’s where things fall short: This strategy doesn’t yield significant ROI. Users may leave a rating or give a thumbs-up/down, but they rarely read lengthy responses in detail, and they often lack any real incentive to give explicit feedback. The takeaway? Instead of hoping for explicit feedback, they start to focus on implicit feedback, capturing user preferences based on behavior rather than direct input. By designing a robust implicit feedback function for LLM applications, they gain deeper insights into tasks, user needs, and ethical considerations. This approach allows for more accurate optimization by continuously capturing app usage patterns, breakpoints, and evolving user needs. It’s like traditional application testing followed by ongoing monitoring and maintenance—distinct phases, each with its own set of stakeholders. It’s interesting to see the industry moving towards investing in implicit feedback functions at the enterprise level 🔄

User feedback is always important. Tools like Amplitude and Mixpanel are excellent for capturing user experiences and improving products, ultimately boosting satisfaction and helping businesses stay user-centric. When it comes to AI, improving models is all about learning, and RLHF plays a big role in optimizing performance over time. With LLMs, traditional deterministic evaluations don’t work as well, and RLHF is typically applied only at the foundation model level. A good example is ChatGPT: its performance improved gradually as they collected data from real usage aka interactions. While users can give explicit feedback, most people don’t tend to do it. That’s why ChatGPT sometimes offers two response options to nudge users into providing feedback, which then helps fine-tune the model. However, traditional RLHF doesn’t apply well to the application layer of LLMs, such as in RAG, agents, or prompt engineering. But there’s potential to rethink this. Combining RLHF with tools like Mixpanel or Amplitude could create a more effective approach to improving LLM applications. Placing users at the center of evaluation and optimization is essential — they’re the ones who know their tasks and needs best.

Agents are being built to automate various tasks, and LangChain is the go-to tool. But once your agent is in production, the real challenge begins—automatically tracking which tools are invoked for user queries and flagging when similar queries trigger different tools… 😵💫 and the list goes on. Observing the step-by-step process your agent takes is crucial for refining its performance and optimizing through real-world usage! Mels has created a step-by-step guide on how to build an agent using Langchain and optimize it with Feedback Intelligence 🛠️ 💡 Interested? Let’s chat!

If you're planning to build a RAG system this weekend and need to improve the performance continously, we have put together step-by-step guidance on 🤖 implementing a RAG model from zero using GPT-4o as a base model ✂️ splitting the GDPR open-source dataset into chunks ⚙️ vectorizing and storing these chunks in ChromaDB 🚀 launching the chatbot and 🧠 automatically improving the performance using the usage aka optimizing the chatbot using human feedback Would love your thoughts!

🔄 Augmenting evaluation data for RAG is crucial to improving the quality in production — but how are most teams doing this today? Often, teams rely on static datasets: a few sample queries, pre-defined responses, and limited context. But is this enough to ensure real-world RAG #effectiveness? Not really. The key is smart generation to evolve evaluation data alongside the RAG application. 🔍 How does it work? 1. Start with a base set of queries and responses for a specific context 2. Use that data to evaluate the RAG app — is it accurate, relevant, and diverse? 3. Automatically generate alternative responses using prompt engineering or LLMs then reverse the corresponding queries 4. Reintroduce the new data into the evaluation cycle to further challenge the RAG 5. Continuously optimize based on evolving triplet of queries, responses, and context 💡This approach challenges static and limited evaluation methods by introducing dynamic, ever-changing queries and responses that better reflect the complexities of user interactions. Have you tried augmenting your evaluation data dynamically? I'd love to hear how you approached it!

This space is more critical than ever - we've got ~300 innovative startups trying to enable safer, more trustworthy AI systems. From security to hallucinations to risk to fairness, solutions exist and they're on this map. Proud of Ethical AI Database (EAIDB) for another good one!

Explicit vs. Implicit Feedback When running RAG and prompt-engineered solutions in production, gathering both explicit and implicit feedback is key to keeping them accurate and effective ... but making the feedback actionable is much needed! I’ve put together an article on feedback collection methods and how to make them actionable for continuous optimization ✨ If you have a RAG or prompt-engineered solution (chatbot, conv Ai, agents) in production, let's chat! #rag #feedback #optimization