Atria AI’s Post

Optimization by Prompting (OPRO) is an approach that utilizes large language models (LLMs) as optimizers by describing optimization problems in natural language and instructing the LLM to iteratively generate solutions. This method adapts to different tasks by modifying the problem description in the prompt and allows for customization based on desired solution properties. Case studies on linear regression and the traveling salesman problem demonstrate that LLMs can achieve competitive results through prompting alone. OPRO also explores prompt optimization, focusing on maximizing task accuracy by refining the prompt format, which LLMs are sensitive to. By using a meta-prompt that includes past prompts and their scores, OPRO iteratively generates new prompts to improve task accuracy. Experiments across various LLMs show consistent improvements in task performance through optimization, with OPRO-optimized prompts outperforming human-designed ones in several benchmarks. Key challenges addressed include the trade-off between exploration and exploitation, optimization stability, and managing prompt space constraints. OPRO is further evaluated on different models and tasks, demonstrating its effectiveness in both mathematical and prompt optimization contexts. https://lnkd.in/gKBF9grE

How confident are you that the prompt you are using is the best prompt for your task? This paper from DeepMind finds the best prompt for you and explores the fascinating world of language models and their potential use as optimisers. The optimisation task is written in natural language. The researchers discovered that LLMs can outperform humans when it comes to optimization tasks. The study showcases how LLMs can refine and improve prompts to create better working prompts. https://lnkd.in/d7nabbYk Check out the original paper for more insights into the exciting possibilities of LLMs as optimisers.

Happy to share our latest work "MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translation" published in the Table Representation Workshop, NeurIPS 2024. Text-2-SQL translation is an important problem to enable agentic workflows for automating database tasks. Recent advances in the space of text-to-SQL utilize closed proprietary models like GPT-4 that present challenges in accessibility, privacy, and latency. We address these issues by developing small, efficient open models (under 10 billion parameters) for text-to-SQL translation and obtain state-of-the-art results compared to other open models while remaining competitive with larger proprietary models at a much lower cost. Joint work with my colleagues at Layer 6 AI. Ilan Gofman, Zhaoyan Liu, Paul Wu, Noël Vouitsis, Guangwei Yu, Jesse Cresswell, Rasa Hosseinzadeh. Paper link: https://lnkd.in/ghi5mKwB

Does the order in which we present information influence our ability to reason and make decisions? This question forms the core of a exploration into the workings of large language models (LLMs) and their reasoning capabilities. Let's consider a simple example to illustrate how the order of information can affect the performance of language models like GPT-4-turbo in reasoning tasks. Imagine we have a reasoning problem based on the following premises: If it rains, the ground will be wet. (Premise A) It is raining. (Premise B) The logical conclusion from these premises is that the ground will be wet. Now, if we present these premises to the language model in the order they're listed above (A then B), it aligns with the logical steps needed to reach the conclusion. The model sees that it's raining (Premise B) and, based on Premise A, it can easily conclude that the ground must be wet. However, if we mix up the order and present Premise B before Premise A, like this: It is raining. (Premise B) If it rains, the ground will be wet. (Premise A) The model might still reach the correct conclusion, but this rearranged order could potentially make it harder for the model to process the information as smoothly as before. This is because it first gets the information that it's raining, but without the immediate context of what happens when it rains, which comes from Premise A. When the information is presented in an order that logically builds up to the conclusion, it's easier for the model to follow along and make the right deduction. This behavior highlights a potential limitation in their reasoning capabilities and suggests that their performance might be influenced by the structure and presentation of information, rather than just its logical content. https://lnkd.in/ghnsDRbX

AI researchers have been working on overcoming limitations of the "regular" retrieval augmented generation (RAG) pattern with a new pattern called Retrieval Augmented Fine Tuning (RAFT) that instructs large language models (LLMs) to ignore specific documents for better generated results https://lnkd.in/dtg25FC7. #airesearch #rag #largelanguagemodel #raft #artificialintelligence #generativeai

pub.towardsai.net: The article introduces LoRA, a parameter-efficient method for fine-tuning models to solve downstream tasks, with a focus on text generation using large-language models (LLMs). It explains the concept of fine-tuning, reasons for doing it, and practical considerations. The method involves updating model parameters to optimize an objective function that represents the task, using a low-rank adaptation matrix to efficiently fine-tune the model. The article also discusses which parameters to update when fine-tuning and presents key experimental results. It concludes by emphasizing the importance of parameter-efficient fine-tuning methods in the large language models regime and encourages reaching out for further information.

Conducting RAG in unstructured data like tables is difficult because tables are not natural language and naive chunking strategies may split tables and it pose challenges for semantic similarity search. This issue can be solved by using unstructured file parsing and multi-vector retriever for RAG. The generation of summaries of table elements suited to natural language retrieval is a game-changer in handling this challenge. Looking forward to diving deeper into this innovative approach! #RAG #UnstructuredData #Innovation Code: https://lnkd.in/gwRnz8na

https://lnkd.in/exiBW9Xv Enlightening work that shows how language models trained on next word prediction can build internal representations that contain semblance of semantic meaning. Quoting Alistair Isaac - “Words bear natural meaning about other words, and, though it is not equivalent to the conventional meaning they bear about the world, this natural meaning nevertheless determines some of their paradigmatically semantic features.” Isaac discusses a theory of semantic vectors, rooted in Shannon’s information theory, that attribute semantic meaning to objects based on their correlations across a distribution of objects occurring in a natural environment. Highly recommend this reading to gain a better perspective: https://lnkd.in/eN3s6wnT

In the future, how will we get the best search results on a company’s website? Combine a large language model (which understands natural language) + a user’s search history (including what pages they’ve viewed) + multi-query. Read more here: https://lnkd.in/g3zTiQwk #search #multiquery #llm #generativeai

Reflections on Evaluating Retrieval-Augmented Language Models Reading about the evaluation of Retrieval-Augmented Language Models (RAG) through automated exams was enlightening. The use of Item Response Theory (IRT) to quantify model performance is innovative, yet I worry about its reliance on initially generated questions and potential biases. Integrating online data and user preferences could offer stronger, real-world insights. Including metrics like click-through rates and user satisfaction scores would align evaluations more closely with human preferences. This hybrid approach could bridge the gap between automated assessments and practical effectiveness, providing a more holistic understanding of RAG systems' real-world performance. https://lnkd.in/e8mkMVv5