LiteLLM (YC W23)

Thrilled to launch native support for Cohere Rerank API on LiteLLM (YC W23) Python SDK and LiteLLM (Proxy) Gateway 👉 Start here: https://lnkd.in/dQVPRfZW 🔥 Support for using Together AI Rerank API Live now https://lnkd.in/df2SbMz8 ✨ Support for ALL Vertex AI21 models live here https://lnkd.in/d-g_nquc (thanks Jian Guo) ✅ Support for using /rerank api on LiteLLM Proxy (thanks Andres Aldana) 📖 Doc add link on using LiteLLM Proxy with Langfuse: https://lnkd.in/dc3NZGeF (thanks Jian Guo)

⚡️ Launching support for calling Fine Tuned VertexAI LLMs on LiteLLM (YC W23) 1.44.6 https://lnkd.in/dPMb6bwW (h/t Vivek Aditya) 🔥 Support for regenerating Virtual Keys on LiteLLM Proxy (h/t Dmitry Yakushechkin) ✨ Support for regenerating Virtual Keys on Admin UI ✨ Pr for using google ai studio fine tuned models with LiteLLM https://lnkd.in/dZ-6xMtP 📖 Doc - show how to mount custom guardrail on litellm gateway https://lnkd.in/dphZi5rU 📖 Doc - using LiteLLM Gateway with LiteLLM python SDK: https://lnkd.in/dqZ9uGxF 📖 Doc fix - using Qdrant with LiteLLM SDK (h/t Kameshwara Pavan Kumar Mantha)

Launching support for Vertex Text To Speech API on LiteLLM (YC W23) 1.44.5 🎤 Start here: https://lnkd.in/dfZYmRPQ 🛡️ Support for using Custom Guardrails on LiteLLM Gateway here: https://lnkd.in/dphZi5rU 📈 Prometheus metric to track LLM API latency vs Latency from LiteLLM https://lnkd.in/dFPyZ4XW 🔥 Admin UI - view user id that created a virtual key https://lnkd.in/d6dWwSQa 🛠️ Support for using base64 encoded images for multimodal vertex ai embeddings: https://lnkd.in/ddYDtcUW

🔥 We're launching support for using Bedrock Guardrails on LiteLLM (YC W23) Gateway - use Bedrock guardrails with 100+ LLMs supported by LiteLLM 👉 Start here: https://lnkd.in/daNCVfMD 🔐 Support for using tenant_id, client_id (Entrata ID) auth for Azure OpenAI https://lnkd.in/dk2675bV ⚡️ [Feat-Proxy] Prometheus Metrics to Track request latency, track llm api latency by 📖 Add example curl on /chat, /completions and /embeddings doc string (h/t Steffen Ridderbusch) 📖 Fix doc string for /user/delete (h/t Steffen Ridderbusch)

🔥 Launching Qdrant Semantic Caching Support on LiteLLM (YC W23) Gateway (Proxy) and Python SDK - use Qdrant semantic cache to reduce LLM costs and LiteLLM to call 100+ LLMs in OpenAI format 📖 Using LiteLLM Gateway with Qdrant: https://lnkd.in/gjf-wAaq 📖 Using LiteLLM Python SDK with Qdrant: https://lnkd.in/gjf-wAaq 🔥 Support for VertexAI multimodal embeddings (h/t Claudio Urbina Lara) ✅ Support for using Vertex AI Python SDK with LiteLLM Gateway https://lnkd.in/dtwkkV_j (h/t Claudio Urbina Lara) 📖 Doc - deploying LiteLLM on Google Cloud Run: https://lnkd.in/dJzYiCUk

🛡️ Excited to launch 2 Free LLM Guardrail integrations on LiteLLM (YC W23) Gateway (v1.44.0+) with Aporia and Lakera, start here: https://lnkd.in/g3bUyZM7 ✨ Feat - control Guardrails per LiteLLM Proxy API Key https://lnkd.in/gxe52483 (h/t Nitai Itzhacky for this request) 🐳 Fix using LiteLLM Docker container in read only file systems (h/t Brian H. and Filipe Andujar) - live on 1.44.0+ ✅ [Fix] Router - don't recursively use the same fallback [Fix Router] - Don't cooldown Default Provider deployment  🔥 [Fix Router] Don't retry errors when healthy_deployments=

Launching support for setting up Aporia Guardrails on LiteLLM (YC W23) OSS - Use Aporia for PII detection, Block Topics, detect profanity - Start here: https://lnkd.in/dXZBYZxq (h/t Nitai Itzhacky) 👉 Use Aporia guardrails as a pre_call, post_call, during_call hook 👉 Support for returning guardrails used in LiteLLM response headers 💵 Fix for returning spend reports in weekly slack alerts (h/t Matthias Lau) 🔐 Support for defining guardrails per Proxy API Key https://lnkd.in/d9ZNBeXm

Excited to launch the ability to set Model Rate Limits per Virtual Key on LiteLLM (YC W23) Gateway 1.43.18 (thanks Jian Sheng Low) https://lnkd.in/d9Pv29u5 🔐 Support for LLM Access groups for Teams on LiteLLM Gateway (thanks Mandar Phadate) https://lnkd.in/dvsYz8Jn 👉 Return x-litellm-key-remaining-tokens and x-litellm-key-remaining-requests in response headers 👉 Prometheus metrics for x-litellm-key-remaining-tokens and x-litellm-key-remaining-requests in response headers 📖 Updated docs for using Virtual Keys with LiteLLM Gateway: https://lnkd.in/dndDkqES

Excited to launch OAuth 2.0 on LiteLLM (YC W23) Gateway 🔐 use OAuth 2.0 to authenticate /chat/completions, /completions, /embeddings requests (h/t Nivid Dholakia) -> Start here: https://lnkd.in/dcnjqt2k 🪨 Support for returning trace for Bedrock Guardrails requests (h/t Ivane Kelaptrishvili) 📖 Docs - add example on setting up LiteLLM Gateway with Sagemaker 🛠️ Support for setting temperature=0 for Sagemaker requests on LiteLLM Gateway 🔐 PR - setting tpm/rpm limits per model for a given API Key https://lnkd.in/d-t_4pR9

Excited to launch support for Async Sagemaker calls on LiteLLM (YC W23) Gateway 1.43.14 - start here: https://lnkd.in/gs23XE8R (h/t Mike Arov) 🪢 [Fix-Proxy + Langfuse] Always log cache_key on hits/misses (h/t Kshitij Grovor) ✅ Fix-Proxy - Allow passing model_id for /image/generation, /audio/transcription, all endpoints 🛠️ Fix - using Prompt Caching with LiteLLM Proxy https://lnkd.in/dw2xCbJR 🛠️ Fix for Sagemaker streaming when "index" does not exist in chunk, and when raw chunks are not JSON