Anthony Rathé’s Post

Excited to share our latest work on ChocoLlama - 6 Dutch language models adapted from Llama 2 and Llama 3! We spend a lot of effort on finding good-quality data (32B tokens) and I'm particularly proud of our thorough evaluation showing how different approaches affect Dutch language performance. Check out our full technical analysis and models on HuggingFace: 🤗 https://lnkd.in/eZCtsqtw I am very grateful to have collaborated with such an excellent team.

As proud Dutch speakers, it’s unfortunate to see our language often overlooked in the development of Large Language Models (LLMs).    To change this, Anthony Rathé and I built ChocoLlama, a family of 6 Dutch LLMs based on Meta’s Llama. We’re excited to release all models on Hugging Face (https://lnkd.in/eGuBnKnV), along with our paper (https://lnkd.in/etZAb5DK).   Technical TL;DR: - We train Llama-2 on all Dutch data we could find (32B tokens). We find that LoRA scales well for language adaptation and modifying the tokenizer boosts performance.  - In the meantime, the new Llama-3 turned out to be better in Dutch than our Llama-2 adaptations– showing how hard it is to keep up in this field! - Repeating our method on Llama-3 yields limited returns, suggesting that more multilingual models may benefit more from posttraining than pretraining.    Big thanks to our co-authors (François REMY, Pieter Delobelle, Jens-Joris Decorte, Thomas Demeester) advisors (Els Lefever, Veronique Hoste, Cédric G., Karel D'Oosterlinck, @Loic Delanghe , Pranaydeep Singh and Bram Vanroy) and industry partners who helped us expand our Dutch data (TechWolf, Jeroen Van Hautte 🐺, Bizzy, Hendrik Keeris and ML6 | Your partner in AI, Michiel De Koninck). And a special thanks to the "Vlaams Supercomputer Centrum" for extending us the grant making this project possible. It's amazing to see initiatives like this being supported in Flanders.

Exciting times for AI in Norway! Last week we saw the national launch of NorwAI’s next-gen Norwegian language models, NorLLM. Our CDTO Sven Størmer Thaulow and Jon Atle Gulla hosted the event at Norwegian University of Science and Technology (NTNU), with Norway’s Minister of Trade and Industry, Cecilie T. Myrseth, highlighting the models' role in safeguarding copyright and privacy. Språkrådet’s Senior Advisor, Kristine Eide, and other business leaders also shared their thoughts on the importance and potential of NorLLMs. But why is a Norwegian language model needed when we have ChatGPT? Sven shares three main reasons in our previous Schibsted Future Report: 👉 A model trained primarily on content in the Norwegian language will likely also be better in Norwegian. 👉 To have control over our own infrastructure.  👉 To be consistent with Norwegian culture. Read more about the project and how Sven discusses the importance of having a Norwegian LLM here: https://lnkd.in/d8mKfUCS #AI #artificialintelligence #machinelearning #ml #llm #schibstedfuturereport #largelanguagemodels

Earlier this year, Silo AI and TurkuNLP released Poro 34B, a best-in-class open foundation model for Finnish, English and code. Now we are releasing Poro 34B chat – a chat tuned version of our Poro 34B model. Foundation models are a cornerstone of the AI infrastructure needed to build AI-powered products, services and businesses. While these models are versatile in nature, they require fine-tuning or special prompting when deployed into production. A common challenge for models designed for low-resource languages is that most of the instruction datasets that are publicly available are written in English. To chat tune a model to follow instructions in a low-resource language, we put forward an approach that uses the base model’s translation capability to generate instruction-tuning data. This results in a model that produces high-quality responses in the same language as in which the prompts are written. Poro 34B chat is evidence of this approach, and an important milestone on the journey towards providing models for all official EU languages. Check out our blog, linked in the comments, for more on Poro 34B chat, including download link for HuggingFace. We look forward to seeing how you will put the model into use!

Ambitious Danish language models are on the way! 🇩🇰💬 The Danish Ministry of Digital Affairs has allocated DKK 30.7 million to support the development of Danish language models through a groundbreaking project: Danish Foundation Models (DFM). This initiative brings together partners from Københavns Universitet - University of Copenhagen, Syddansk Universitet - University of Southern Denmark, Aarhus Universitet, and the Alexandra Institute to advance the field of language models and language technology in Denmark. As part of the DFM consortium, DIKU will play a crucial role in ensuring that the Danish language models meet the highest standards for data integrity, transparency, and safe AI development. "I’m excited that DIKU will contribute to developing Large Language Models that conform with the EU AI Act and GDPR regulation. I see a great deal work ahead in the areas of data curation, model development, and culturally relevant evaluation. It is important for the DFM consortium to focus on developing fully documented open-source models, as this can drive local innovation and development in Denmark", explains Associate Professor Desmond Elliott, co-PI of the project. DFM aims to establish a robust R&D platform for training, fine-tuning, evaluating, and maintaining language models tailored to Danish-language needs. In the long term, this will ensure that Danish investments in AI are directed towards solutions that meet critical and specific needs in Danish society and promote sustainable development, ensuring a fair digital economy. Read more about the DFM project on our website: https://lnkd.in/dWqugf2D

The boundaries between machine translation (MT) and large language models (LLMs) are disappearing—quickly. Consider just a few announcements from the past couple of weeks: • DeepL's next-generation language model is powered by a new LLM infrastructure (https://lnkd.in/guBsxpFe). • Unbabel's new service, Widn.Ai, focuses on machine translation but is driven by Unbabel's Tower LLM (https://lnkd.in/gtiQs-b9). • Translated's new platform, Lara, is a large language model designed for high-quality machine translation (https://lnkd.in/gMVbMf7t). At Blackbird, our app category filtering options currently separate "MT/MTQE" and "Artificial Intelligence" into distinct categories. However, as the line between these two areas continues to blur, we are considering merging them under a unified "Language AI" label. Are MT and LLMs converging into a single category? What’s your take?

I recommend reading the report on the development of genAI in Poland. I was involved in writing it with Kuba Piwowar and Alek Tarkowski, we talked to leaders and experts in the field. My favorite conclusion is why it is worthwhile to build Polish LLMs: because the process creates know-how in the industry and allows institutions to store and process their data locally.

📝 Introducing our latest report – “AI speaks Polish. The ecosystem of open language models in Poland”! It presents research and conclusions of Alek Tarkowski (Open Future Foundation), Kuba Piwowar (Fundacja Centrum Cyfrowe), and Michał Owczarek (Uniwersytet SWPS). The goal of the report is to provide a case study of Poland’s ecosystem for creating open AI models for the Polish language 🇵🇱 Small language models are filling the gap left by large commercial models, which are not adapted to the Polish language or cultural nuances. The work on these models serves as an example of effectively creating alternatives to dominant entities. The report focuses on two key projects: building the SpeakLeash | Spichlerz language corpus and using it to create the Bielik model, as well as the activities of the #PLLuM consortium (Polish Large Language Model). Based on interviews with the creators of Polish models, the authors outline the development processes and the challenges they presented, and summarise the lessons learned from the achievements so far. 👉 See the full report on our website – in Polish now, and in English next week! 🔗 https://lnkd.in/d7JTr4G8 __ Image: Portrait of Adam Mickiewicz, Austrian National Library [Public Domain, via Europeana.eu]

Don't underestimate the power of a single space in your prompts—it might just make your AI chatbot speak Spanish back to you. Today, I built a chatbot and shared it with my team. Eager to test it out, one of my colleagues jumped in, and the very first response he got was entirely in Spanish. This wasn't a fine-tuned, multilingual model; it was just GPT-4o. What happened here? Here's my best guess: The issue likely stemmed from my prompt. I ended it with an extra space, and that small slip-up completely threw GPT-4o off. But why would an extra space cause such a glitch? It comes down to the tokenizer. The tokenizer splits your text into tokens, and its method isn't always aligned with how we as humans understand text. That extra space altered how the text was tokenized, leading the model to respond in an unexpected language. The prompt now likely included a rare token that's mostly present in Spanish language texts! Lesson learned: always test your prompts thoroughly and never underestimate the tokenizer's impact.

🌍 How can technology truly serve local communities?, The Distributed AI Research Institute (DAIR) believes that tools created by and for communities are far more impactful than those imposed from outside. Their latest blog shares examples, like how Māori researchers ensure their language is used to benefit their own people first. Read our post reflecting on these ideas and join the conversation on how DAIR is rethinking AI to prioritize people, culture, and community needs. https://lnkd.in/d3bMhkAR