Small language models - Llama Tutorial

- [Instructor] Small language models are an emerging new technique that is pretty interesting because one of the focuses is this higher quality training data mindset. So the idea here is that you can actually do some of the same things that are happening with large language models, but if you focus instead on curated high quality training sets, you could potentially outperform these large language models and even do specialized training that can make it even better than a particular large language model at a task. And some of the advantages are that they have compact size, so this could make it so that inference becomes much easier because you don't need these large GPUs. Also, the training is going to be a lot more efficient because you're not dealing with, you know, essentially unlimited data. The resources to do inference and the resources to do training are much reduced. And then as well, there's an easier deployment. In fact, one of the things that's really an emerging capability potentially in the future of small language models is putting these on devices. We haven't really seen a ton of this, but for example, there's lots of examples of hardware that can support machine learning models, but we haven't really seen this yet with the small language models. But I believe we will see this pretty soon, where you'll take a small language model that has been customized for a particular piece of hardware, put that onto a drone or put that onto a phone or some other device, and then you can get these really specialized tasks that the small language model could do. So it's really an emerging area of research where we're going to see Edge-based devices play a large role in the future. And so it's important to know a little bit about small language models and some of the advantages they have over large language models.