Data Science Central’s Post

Better LLMs with Shorter Embeddings: Part 3

Better LLMs with Shorter Embeddings: Part 3 https://lnkd.in/gaSykQCg Variable Length Embeddings and fast ANN-like search (approximated nearest neighbors) for better, lighter and less expensive LLMs

Wrote down some thoughts about upcoming LLMs with (really) big context size. https://lnkd.in/eNWR7tK8

Your LLM application does not always need GPT-4o. It's better to use cost-effective and faster models (e.g. Mistral 8x7B) for some queries. RouteLLM proposes efficient router models that dynamically select between a stronger and weaker LLM during inference to balance cost and response quality. The paper proposes 4 different routing techniques - 1. Similarity-weighted (SW) ranking - performs a "weighted Elo calculation" based on similarity 2. Matrix factorization - learns a scoring function for how well a model can answer a prompt 3. BERT classifier - classifier that predicts which model can provide a better response 4. Causal LLM classifier - also a classifier Here's a sample code using Matrix Factorization with 50% strong-model calls (GPT-4o)

Quick sort algorithm 📊

There are so many things going on in LLM now that you can find experiments in almost every direction. There was recently a SOLAR model released where the authors increased the size of a trained transformer by copying some blocks, and the model got better. In parallel, works in the opposite direction are being released, where cutting out layers completely preserves quality and allows to prune the models in this way. ✅ A recent paper The Unreasonable Ineffectiveness of the Deeper Layers is an example of such work. The authors look at the distance between input and output of l and l+n layers, and if it is small, delete those layers. The intuition here is that a small distance means that the embedding has not changed much over the past transformations. In practice, it turns out that the layers-candidates for pruning lie closer to the end of the model, which seems logical: the model changes the embeddings a lot at first and just adjusts them over time for the final prediction. ✅ After the layers are pruned, a healing procedure is performed - QLoRA tuning on the C4 dataset (hundreds of millions of tokens). This fine-tuning allows throwing out even more layers without loss of quality. From measurements - MMLU and BoolQ, in both tasks the authors were able to throw away ~30% of LLaMA 2 70B layers, preserving accuracy. Now we need to merge the directions: take a 130B model, prune to 70B, and then expand again to the original size, getting a better model 🧠 #llm #deep_learning #transformers #pruning

Some days ago Google released Gemma 💎 📑 Report: https://lnkd.in/dQkmCwKJ The Gemma paper advances open-source ML models, outperforming LLaMA2 in coding and reasoning, thanks to its extensive training on 6T tokens. It nearly matches Mistral 7B's performance, notably excelling in safety metrics. Gemma offers two configurations, 2B and 7B, in pre-trained and instruction-tuned formats, highlighting innovations like rotary positional embeddings and GeGLU activations (not much information on this side). 📗 Finetuning Notebook: https://lnkd.in/dDmzm5TQ Note: if you finetune the instruction-tuned model make sure to use the same syntax: "<start_of_turn>user ..."

I spent some time this week using this new text embedding model that was released earlier this month, and I am happy with the results (for stuff I tested it with, of course!). I can't run it on a local mac machine anymore (for training under 5 epochs), like I did with other sentence transformer models I was using (it crashed within the first epoch). But performance wise, it was good. I will definitely explore further. #embeddings #sentencetransformers #nlproc https://lnkd.in/gzV9ScNQ

High-Performance LLM Inference Server with Constrained Grammar https://lnkd.in/eYw8bxDG For anyone who wants to play around with **Constrained Grammar** without the hassle of Llama.cpp 😋 The fork implements this feature with the latest vLLM v0.2.7. It's all just sparsely tested, so feedback is very welcome! 🙏 ### Some Context: 1. Practical Techniques to constraint LLM output in JSON format https://lnkd.in/eK76Vnwh 2. Usage Guide with vLLM's OpenAI-compatible API endpoints https://lnkd.in/e_ZzBsnb 3. Extended Backus-Naur-Form (EBNF) Syntax https://lnkd.in/ebYEP_qZ ### Motivation As stated by the project itself, Llama.cpp Server does not target to be an LLM inference backend for production. This comes with some pain points for use cases with a lot of calls and high input and generation throughput: 1. Slow inference 2. It is fragile as it stales or crashes when prompted with wrong grammar or calling without an open slot... 3. No convenient Model Directly integration vLLM does not come with the above, but lacked an implementation for Constrained Grammar.