Athina AI (YC W23)

We're hiring! We're building an IDE that lets engineering teams develop AI-powered products 10x faster. If you're a highly-driven, talented individual interesting in shaping the future of AI-powered software, then apply to work at Athina. Location: San Francisco

🚀 How GoDaddy Scaled their LLM Application From Prototype to Production S/o to Richard Clayton for sharing their journey about scaling AI assistants from prototype to production. Here’s a breakdown of their insights: 1. Prompt Size Issues 📜 Challenge: As more topics and questions were added, the prompt grew to over 1500 tokens, causing high costs, token limit issues, and reduced accuracy. Solution: They moved to a multi-prompt architecture, using a Controller-Delegate model. Task-oriented prompts at key transition points reduced token usage and improved accuracy! 2. Structured Outputs 🧩 Challenge: LLMs struggled to produce structured outputs like JSON, which is essential for certain tasks. Plain-text responses weren’t sufficient for AI analysis needs. Solution: They decreased prompt temperature and used more advanced models for structured outputs. In parallel, they employed two prompts to ensure structured and communicative responses simultaneously. 3. Prompt Portability 🔄 Challenge: They discovered that prompts aren’t universally portable across models or even model versions. Solution: By tuning prompts for each model, they achieved better consistency. Continuous testing and adjustments kept performance high! 4. Guardrails are Essential 🚫 Challenge: LLMs sometimes failed to transfer users to human agents, leaving customers stuck. Solution: They implemented deterministic methods (like stop phrases and chat limits) to ensure users could always reach a human when needed. Also added PII and content filters for safe interactions. 5. High Latency and Reliability Issues ⏱️ Challenge: Latency and reliability took a hit with longer token sizes, and around 1% of chat completions failed. Solution: They integrated streaming APIs to enhance user experience and implemented retry logic, balancing cost and latency. 6. Memory Management 🧠 Challenge: Maintaining LLM context over long conversations proved costly and sometimes led to unexpected model behavior. Solution: They retained recent conversation content and summarized earlier interactions. Started exploring multi-agent architectures to manage memory across delegate prompts, enhancing focus on core interactions. 7. Optimizing RAG Architecture 🗂️ Challenge: Early retrieval in RAG processes led to distractions, as irrelevant documents were pulled in too soon. Solution: By using specialized RAG prompts and tool-calling, they allowed the model to gather relevant context at the right time, significantly improving the quality of recommendations. Link to article in comments 👇

Looking for a comprehensive guide on AI Agents? 🔍 📚 Check this new blog that our expert, Haziqa Sajid published that covers everything from what AI agents are, how they work, and even dives into its technical implementation! 🚀 🔥 Some of the technical insights were really interesting and went right into building your own AI agent. Here’s a snapshot of what you can expect: 1️⃣ An overview of AI agents 🛠 2️⃣ Benefits like better efficiency, cost savings, high availability, and effortless scalability 🚀 3️⃣ Real-life applications 📖 4️⃣ Introduction to LLM agents, including concepts like reasoning and acting 🧠✨ 5️⃣ A deep dive into Planning, Memory, and Tools for LLM agents, with frameworks like ReAct, CoT, and Tree of Thoughts 🗃 6️⃣ Hands-on guide to building task automation with LangGraph 🤖 If you're interested in AI and automation, this guide is a must-read! 😊

🚀 PEFT methods like Low-Rank Adaptation (LoRA) is one of the preferred choices for Fine tuning LLMs because of its capability to reduce the number of trainable parameters without increasing inference costs and adapter modularity. ✨ An extended approach DoRA (Weight-Decomposed Low-Rank Adaptation) improves LoRA's learning capacity and stability while retaining efficiency. By decomposing pre-trained weights into magnitude and direction components, DoRA enables targeted updates, minimizing trainable parameters and mimicking FT's learning capabilities. We’ve just published a comprehensive article to dive deeper into this technique and its benefits for your projects! 📚✨ Here’s what our post covers: 1️⃣ A quick overview of PEFT techniques and their types: 🟢 Adapter-based Methods 🟢 Prompt-based Methods 🟢 Low-Rank Adaptation (LoRA) 2️⃣ Patterns of LoRA vs. Full Fine-Tuning 3️⃣ Introduction to DoRA 4️⃣ Implementation details and effectiveness evaluation Link to complete article in comments 👇

When your team is obsessed, shipping features fast for customers every day is an Atomic Habit. 1.01^365 Over time, it starts to compound into a great product. Another week, another shiplog! 🚢 👀 Observe: 1. 🎯 Multi-select filters on the Dashboard, Analytics, and Compare pages ... So you can find exactly what you're looking for with more precision. 2. 🟢🔴 Status codes and error tracking in LLM requests ... So you can trace failures and successes more granularly. 3. 💬 Chat history in custom prompt evaluations ... So your LLM evals always have the full context. +++ many minor upgrades 👨💻 Develop: 1. 🏗️ Support for Structured Outputs, JSON mode, and Tool Calling ... So your responses are more predictable and actionable. 2. ⌛ Benchmark latency and token usage in Dynamic Columns and Experiments ... So you can measure everything that matters. 3. 💬 OpenAI Assistant as a dynamic column ... So you can bulk test OpenAI Assistants seamlessly. +++ many minor upgrades +++ Countless bugs were killed in the making of this week's release at Athina. 🐛

🚀 How Faire Scaled to 70M Predictions/Day with Fine-Tuned LLMs 🔍 Faire recently shared an impressive case study on how they fine-tuned LLMs to measure the performance of their search algorithms. Here’s a breakdown: ⚡️ 70M search relevance predictions per day! Faire connects hundreds of thousands of independent brands and retailers globally, and search is critical to help retailers discover the right products. But evaluating search relevance manually is hard to scale. 😨 Search relevance explained: Search queries can be vague, leading to multiple interpretations. For example, the query “bat” could mean either a baseball bat or a flying mammal 🦇⚾️. To solve this, Faire developed guidelines to categorize results as: 1️⃣ Exact (E) - Perfect match 2️⃣ Substitute (S) - Close enough 3️⃣ Complement (C) - Works with the desired item 4️⃣ Irrelevant (I) - Not a fit ❌ 🧑💻 Leveraging LLMs for Relevance Labeling Initially, human annotators labeled query-product pairs, but this approach was costly and slow. Using LLMs like GPT helped scale, but costs rose as their search system evolved. That’s when they turned to fine-tuning smaller models (Llama2 & Llama3). With techniques like PEFT and LoRA adapters, they reduced the training overhead to just 4% of the base model’s parameters. 💡 Results: ✅ Llama3-8b improved relevance by 28% ✅ Fine-tuned models outperformed basic prompting by 2X ✅ Scaling the labeled dataset further improved performance 🏗 Self-Hosted Inference Impact: They scaled their data labeling systems to 70M predictions per day using 16 A100 GPUs, 8-bit quantization, and efficient horizontal scaling. This approach drastically improved speed and reduced costs, making their search system more effective and scalable! 🔥 You can read the full case study using the link in the comments below 👇

9 open-source libraries that LLM developers should know about 👇 1. LiteLLM (YC W23): Python SDK to call 100+ LLM APIs in OpenAI format. 2. Rebuff: Designed to protect AI applications from prompt injection (PI) attacks through a multi-layered defense. 3. PurpleLlama by Meta: Set of tools to assess and improve LLM security. 4. BAML (Boundary (YC W23): Domain-specific language to write and test LLM functions. 5. Marvin (Prefect): Combine any inputs, instructions, and output types to create custom AI-powered behaviors without source code. 6. assistant-ui: A set of React components to build a ChatGPT-like AI chat UI, with integrations for Langchain, Vercel AI SDK, and more. 7. Promptfoo: A tool for testing and red-teaming LLM apps. 8. vLLM: A fast and easy-to-use library for LLM inference and serving. 9. Ollama: A lightweight, extensible framework for building and running language models on the local machine. #llm #languagemodel #GenAI #OSS #opensource

Thanks for the feature Athina AI (YC W23) Check them out in case you haven’t. They are revolutionzing the LLM B2B space.

✨ Weekly Roundup! ✨ Excited to spotlight 5 incredible authors sharing their knowledge with the AI/ML community! 🌐 Dive into these must-read articles to explore the latest in AI Agents, model optimization, fine-tuning, RAG, and much more. 🚀💡 Sandi Besen https://lnkd.in/dmMtD_sw 1️⃣ The Landscape of Emerging AI Agent Architectures A deep dive into the evolving AI agent architectures that drive complex decision-making systems. 2️⃣ Key Insights for Teaching AI Agents to Remember Discover techniques to build robust memory capabilities in AI agents using insights from Autogen's "Teachable Agents." Paras Madan https://lnkd.in/d3uCPeez 1️⃣ Advanced RAG for Databases without Exposing Data Learn privacy-focused strategies to access databases securely with LangChain, OpenAI, and SQLAlchemy. 2️⃣ Building a Multi PDF RAG Chatbot: Langchain, Streamlit & Code Build a Streamlit web app with Multi-RAG to interact with PDFs through an AI-powered chatbot. MANPREET SINGH https://lnkd.in/dy8d447g 1️⃣ Analyzing Hotel Reviews with LLM and NLP A project that demonstrates how to extract meaningful insights from hotel reviews using LLMs. 2️⃣ How to Use AI Agents for Research and Writing Boost your productivity by integrating AI agents into your research and writing workflow. Fabio Yañez Romero https://lnkd.in/drZXssJr 1️⃣ Token Masking Strategies for LLMs An exploration of token masking techniques that enhance AI model training and efficiency. 2️⃣ Get the Most Out of Llama 3.1 Maximize LLaMA 3.1’s performance with practical tips and best practices. Eivind Kjosbakken https://lnkd.in/dpv6vrh3 1️⃣ How to Build a RAG System for Powerful Data Access A hands-on guide to implementing Retrieval-Augmented Generation (RAG) for smarter data interactions. 2️⃣ Creating Powerful Embeddings for Your AI Master the art of generating efficient data embeddings to supercharge your AI models. 🔗 Check out their articles and let us know what topics you'd like to see next! ⚡️

Athina AI (YC W23) is now SOC 2 Type II compliant! 🚀 At Athina, SOC 2 Type II compliance represents our dedication to upholding the highest standards of security and data safety, both for our current and future clients. If you'd like to learn more, feel free to DM me!