Decart

Our case study with Cerebrium is now live on our website! Together with Decart AI, Cerebrium set out to see if it was possible to get Llama 2 70B down to $0.50 per million tokens, while also keeping latency low. Achieving this feat is only possible with highly performant and cost-effective infrastructure. Cerebrium's serverless GPU infrastructure platform allows companies to scale from 0 to 10,000 requests in seconds, which translates to large cost savings compared to other platforms. Decart created an #LLM inference engine from scratch using NVIDIA's #CUDA and C++, and by leveraging NVIDIA #H100 GPUs and new versions of #Cutlass, they were able to achieve the same cost per token for Llama 2 70B as they achieved on #A100 GPUs. Take a look at our blog post to see how Decart and Cerebrium used CoreWeave infrastructure with NVIDIA hardware and software to increase throughput and decrease latency across the board. #Falcon180B and #Llama2 70B benchmarks are included in the blog as well! https://hubs.la/Q029BMrw0 #LLM #NVIDIA #H100 #GPU

We are proud to announce our new partnership with Decart AI, allowing our customers to process 1 million tokens of Llama 2 70B for just $0.50! Decart built a proprietary LLM inference engine from scratch in C++ and CUDA to outperform other existing engines (vLLM, TGI, etc.). The engine is hosted on the Cerebrium platform, making it accessible to developers with <5s cold start times. We are hoping this partnership will create a better user experience with lower latencies while at the same time making it more affordable for companies to run these models at scale. Try out the API here: https://lnkd.in/en9yiyqZ Read the official Blog post here: https://lnkd.in/eXzC4htK #generativeai #openai #llama2 #nvidia Currently supported models: Llama 2 70B, Mistral 7B and Zephyr 7B.