Yesterday I participated in the NVIDIA + AI21 Labs' Dev Day and... I won 1st place in the Jamba Challenge mini-hackathon 😁 It was a great experience working on AI solutions with many talented people.
The Mabma-based architecture of Jamba models, based on SSM, offers interesting advantages over transformers - reducing inference time by 3x compared to transformer architectures and significantly improving the ability to answer complex queries in long context lengths.
However, the challenge remains in aligning these models to accurately follow specific instructions, an area where they still struggle.
In the session led by Roi Cohen, we explored the Jamba architecture in detail to understand how its components offer advantages over transformer-based models. Roi recommended the blog "Mamba: The Easy Way", by Jack Cook (https://lnkd.in/dFW8VM7f), which explains the math behind the Mamba blocks.
During the mini-hackathon, Roi and Ori Shapira challenged us with creating an LLM flow that can answer many types of complex questions on realistic data (financial in this challenge) - this task is highly non trivial and requires deep dive into how to tackle these kind of problems. Participants presented a range of interesting approaches, from chunking the context and guiding the model to the right section of text to using a planning framework that breaks down queries into sequential steps.
Thank you Hila Weisman-Zohar, Tsachi Shushan and Amit Mandelbaum, for the enjoyable social time we shared at the conference!
And many thanks to AI21 Labs for hosting such a well-organized event Olivia Gorvy! The prize for winning 1st place in the Jamba Challenge included high-quality headphones and $3K in credits to use with AI21 LLM models. I’m excited to bring these credits to Embie Clinic, where I work, so we can put them to good use in improving fertility care worldwide.
I'd also like to take this opportunity to thank Embie Clinic and our CTO Dana Averbuch, who always encourages me to grow, learn, and expand my professional skills, understanding how this personal development benefits both me and the company. Dana Averbuch - I truly truly appreciate it ❤️.