Patronus AI’s Post

Llama Guard is Off Duty 😲 We benchmarked popular toxicity datasets spanning languages like Portuguese, Ukrainian, and Turkish, and found that Llama Guard has a very high false negative rate for toxic content! We found that base models like Llama 3.1 do all the heavy lifting on toxicity filtering, and that the joint usage of Llama Guard might be redundant. 🤔 At Patronus AI, we rigorously benchmark all things AI to help engineers trust what they use. Reach out to contact@patronus.ai to learn more! Llama Guard might be off duty today, but you don't have to be 🎯 — Read more in our blog post here: https://lnkd.in/eayCX4ct

Introducing Patronus AI + Portkey 🚀 Portkey is the leading open source AI gateway. It’s blazing fast and supports over 200+ LLMs. Developers around the world use Portkey to operationally manage their AI products more easily. There are lots of challenges to building and deploying LLM products to production: lots of LLMs to choose from, various frameworks to integrate, and costs are hard to track. But the biggest challenge of all is the lack of highly reliable LLM guardrails. Enter Patronus AI + Portkey 🚀 You can now use 10+ Patronus evaluators in Portkey, including Lynx, the best hallucination evaluator. ✨ Read the Portkey docs on how to get started: https://lnkd.in/gX8iYPjR Read our blog post: https://lnkd.in/gDvMRjE3 Check out Portkey on Github: https://lnkd.in/dqicscMy

Today, we are excited to release Lynx v1.1, a smaller, state of the art RAG hallucination detection model 🚀 Even though companies use RAG to reduce hallucinations, LLMs can still produce unsupported or contradictory information. Since we released Lynx v1.0 a few weeks ago, thousands of developers have used it in all kinds of real world applications. Lynx v1.1 is the best performing RAG hallucination detection model of its size, enabling real-time hallucination detection in AI applications ✨ - Beats Claude-3.5-Sonnet on HaluBench by 3% - Outperforms GPT-4o on medical questions and answers by 6.8%  - 1.4% higher accuracy than Lynx v1.0 on HaluBench - Outperforms all open source models on LLM-as-judge tasks - Open source, open weights and open data Use Lynx 1.1 with any of our Day 1 integration partners like NVIDIA, MongoDB, and Nomic AI 🚀 Check out the Hugging Face Spaces demo: https://lnkd.in/gcjVmeNG Download Lynx v1.1 on Hugging Face:  https://lnkd.in/gvMbRddM Download Lynx v1.1 (Quantized) on Hugging Face: https://lnkd.in/g3xAFJZh Read the arXiv paper: https://lnkd.in/eznVjrWA Read the blog: https://lnkd.in/eYaP5Zpe

We are thrilled to have won VentureBeat's AI Innovation Award for Best Enterprise Implementation of Generative AI (Finance)! It's an honor to be recognized alongside companies like OpenAI, Microsoft, and Hugging Face 🚀 FinanceBench is the industry’s first benchmark for LLM performance on financial questions. It's a large-scale set of 10k question and answer pairs based on public filings like SEC 10Ks. Since its launch, it has been used by financial institutions, universities, regulatory groups, and leading AI companies around the world. Read about the VentureBeat Transform 2024 award winners: https://lnkd.in/eM-yxxU8 Download the FinanceBench sample on Hugging Face: https://lnkd.in/emBP3DGu

Forbes just covered our latest Lynx launch! 🚀 Yesterday, we open sourced Lynx, the leading hallucination detection model. It beats GPT-4o on RAG hallucination detection tasks, and excels in practical domains like medicine and finance. Lynx is especially unique because it not only produces a score, but it can also reason about the score like a human grader. This makes RAG outputs more explainable and interpretable. "CEO Anand Kannappan thinks of [Lynx] as a 'coach' for other AI models that can guide them to be more accurate." Awesome to see this piece by Rashi Shrivastava! Read the story: https://lnkd.in/e6kM7w-D Download Lynx-8B on Hugging Face: https://lnkd.in/eZ672uRH Read the arXiv paper: https://lnkd.in/eznVjrWA Read our blog: https://lnkd.in/eYaP5Zpe