Today, we’re releasing a new method that improves the way everyone trains LLMs. There's a significant bug that causes loss miscalculations during training. Our Gradient Accumulation fix corrects the issue, reducing L2 norm error by >10x. Blog details: https://lnkd.in/gXXmK_zz
Unsloth AI
Technology, Information and Internet
Sans Fransisco, California 3,921 followers
Making AI accessible for everyone! 🦥
About us
Easily finetune & train LLMs. Get faster with unsloth.
- Website
-
https://unsloth.ai
External link for Unsloth AI
- Industry
- Technology, Information and Internet
- Company size
- 2-10 employees
- Headquarters
- Sans Fransisco, California
- Type
- Privately Held
- Founded
- 2023
- Specialties
- artificial intelligence, ai, llms, language models, and finetuning
Locations
-
Primary
Sans Fransisco, California 94107, US
Employees at Unsloth AI
Updates
-
Unsloth AI reposted this
Did you know 🦥Unsloth AI has docs now? If you have ideas on improving it, I'm all ears! There's 30 free Colab & Kaggle notebooks for faster finetuning & inference listed on 1 page! If you're looking to finetune Llama and export to Ollama for local inference, we have a specific page for that in the docs as well - page here: https://lnkd.in/gx-YsHB5 There's a page for Reward modelling via DPO, ORPO and KTO here: https://lnkd.in/gpr7bTua, how to debug errors in Unsloth and more! All our docs here: https://docs.unsloth.ai/ The docs are a companion to our official Github Wiki here: https://lnkd.in/gpvMYtbY
-
Unsloth AI reposted this
sophia wrote an awesome blog post about how to fine-tune an autocomplete model on your development data using Unsloth AI 🦥 https://lnkd.in/d5NUwHH7
A custom autocomplete model in 30 minutes using Unsloth (Community post)
blog.continue.dev
-
Unsloth AI reposted this
I'm super excited to announce that Unsloth AI is part of Y Combinator S24! We’re so pumped to bring fine-tuning to an even larger audience and showcase how easy it can be to build your own custom ChatGPT! We’ll be focusing on improving accuracy, reducing hallucinations and improving the efficiency of models, and of course giving back to the open source community! We’ve got lots coming - so stay tuned! I'm extremely grateful to all the support everyone has given us, and I want to extend a huge thanks to everyone - thank you! We have a YC company page here: https://lnkd.in/gPvYVJTM and we just published our roadmap at unsloth.ai/roadmap-yc! If you don't know about Unsloth - we make finetuning of language models 2x faster and use 70% less memory, and we have free finetuning Colab and Kaggle notebooks at https://lnkd.in/gyaDBTxK. We also upload 4bit models to huggingface.co/unsloth for 4x faster downloads. Our website https://unsloth.ai/ has more details as well!
-
We’re excited to share that Unsloth is now backed by Y Combinator! Building on our foundation in open-source fine-tuning, we’re creating the all-in-one solution so you can focus on making the models you've always dreamed of, without the complexity. With a focus on accuracy, speed and accessibility, we use math algorithms, low-level languages (Triton, CUDA) to innovate the LLM ecosystem through software, not hardware. We are hiring! Join our wait list: https://lnkd.in/gV_QFGWS Read our Roadmap: unsloth.ai/roadmap-yc Star us on GitHub: https://lnkd.in/dcqhW9Vv
-
Unsloth AI reposted this
I'm sharing a Kaggle notebook to finetune Llama 3.1 2x faster for conversational style finetunes using the FineTome-100K dataset! Kaggle provides 30 hours of Tesla T4 GPUs for free per week, and is a fantastic resource! Had to change our installation instructions to make the latest Kaggle notebooks work. Kaggle notebook link: https://lnkd.in/gfCCTDVU The FineTime-100K dataset was curated by Maxime Labonne, and by training only on completions / the assistant prompt, you can increase accuracies by 10% or more with 🦥Unsloth AI! There's been a revamp of our notebooks since Colab and Kaggle both updated libraries to the latest versions - head over to https://lnkd.in/gyaDBTxK and huggingface.co/unsloth for all the latest notebooks!
Kaggle Llama 3.1 8b Conversational Unsloth
kaggle.com
-
We will be doing a live tutorial on continued pretraining & supervised fine-tuning with the lovely community at AI Makerspace! Hope to see you all there on Sept 4! Event: https://lu.ma/xd0zzk0h Thanks Sarah deSouza, Chris Lusk🕵️♂️ & 👨🏫🤖 "Dr. Greg" Loughnane for inviting us! 😀
-
Unsloth AI reposted this
This week, Microsoft released 3 new Phi-3.5 models with Mini, Vision, and MoE. Did you know they can be converted to the Meta Llama architecture? 👀 Phi is very similar to Lama 3, and the Unsloth AI team was able to convert the weights to the Lama architecture without performance loss. Now, you can use all of your existing Lama-optimized tooling and scripts, such as Hugging Face TGI, Transformers, llama.cpp, or vLLM. Llamafied Phi-3: https://lnkd.in/e_aZKMhj Phi-3.5 is released under MIT and, on paper, matches Llama 3.1 8B while being only ~4B big.
-
Unsloth AI reposted this
Looking to fine-tune GPT4o? Why not try the open source alternative with Phi 3.5 distilled from GPT4? Finetuning Phi 3.5 is 2x faster and uses 50% less memory with Unsloth AI! Free Colab notebook to finetune Phi 3.5: https://lnkd.in/gr7kvm7E I also llama-fied Phi 3.5 and uploaded 4bit quants to huggingface.co/unsloth for more accurate finetunes!
Google Colab
colab.research.google.com
-
We made a step-by-step tutorial on how to fine-tune Llama 3 with Google Colab & deploy it to Ollama 🦙 In this beginner's guide with screenshots, we explain: - How to Fine-tune Llama 3 for free on Colab using Unsloth 🦥 - The basics & reasons behind fine-tuning - Tips & tricks of structuring a dataset - The creation & best practices for chat templates - How to use Ollama & deploy your custom model ✏️ Step-by-Step Tutorial: https://lnkd.in/gUrm3xi2 📒 Google Colab notebook: https://lnkd.in/g4tE6aUY ⭐ Star our GitHub: https://lnkd.in/dcqhW9Vv