We’re excited to share our open-source framework that lets you score different generative language models across various evaluation tasks and benchmarks - used by leaderboards such as Hugging Face. While working with the Amazon Web Services (AWS) team to train our models on AWS Tranium, we realized we were restricted to both VRAM and the availability of GPU instances when it came to the mainstream tool for LLM evaluation. Our open source solution overcomes these challenges—integrating AWS Neuron, the library behind AWS Inferentia and Trainium, into lm-evaluation-harness. Take a look at: ✅ How We Broke Down Our Tests ✅ The Challenges We Encountered ✅ An Example of Using the Testing Harness on AWS Inferentia A huge thank you Michael Feil and Jim Burtoft for the partnership and collaboration, giving back to the developer community. #Gradient #GradientAI #AWS #LLM #OpenSource #LLMEvaluation
This is great - awesome work Michael Feil and Jim Burtoft
Michael Feil thank you for sharing the code and the blog post!
Great work!! Michael Feil 👑
Fractional Head Of Data | Reach Out For Data Infra And Strategy Consults
4moThis is awesome! Congrats Michael Feil and Jim Burtoft