Really excited to share some of our learnings integrating Amazon Web Services (AWS) Inferentia instances into our evals stack. I know a lot of our customers have trouble obtaining GPUs for quick eval iteration cycles in their development process so we've worked to remove that friction. #AWS #GenerativeAI #LLMs #Evals
We’re excited to share our open-source framework that lets you score different generative language models across various evaluation tasks and benchmarks - used by leaderboards such as Hugging Face. While working with the Amazon Web Services (AWS) team to train our models on AWS Tranium, we realized we were restricted to both VRAM and the availability of GPU instances when it came to the mainstream tool for LLM evaluation. Our open source solution overcomes these challenges—integrating AWS Neuron, the library behind AWS Inferentia and Trainium, into lm-evaluation-harness. Take a look at: ✅ How We Broke Down Our Tests ✅ The Challenges We Encountered ✅ An Example of Using the Testing Harness on AWS Inferentia A huge thank you Michael Feil and Jim Burtoft for the partnership and collaboration, giving back to the developer community. #Gradient #GradientAI #AWS #LLM #OpenSource #LLMEvaluation
