Federica Vezzani’s Post

In this episode, we discuss Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? by Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig. The paper explores the effects of integrating new factual information into large language models (LLMs) during the fine-tuning phase, particularly focusing on how this affects their ability to retain and utilize pre-existing knowledge. It was found that LLMs struggle to learn new facts during fine-tuning, indicating a slower learning curve for new information compared to familiar content from their training data. Additionally, the study reveals that as LLMs incorporate new facts, they are more prone to generating factually incorrect or "hallucinated" responses, suggesting a trade-off between knowledge integration and accuracy.

An article, shared with me by Krassimir Boyanov of KBWEB Consult, links to an article from Science ABC. As is usual with such articles, the title is breathless: “How Tongue Prints Are Going To Revolutionize Identification Methods.” https://lnkd.in/g9cRiQnx

Explore KAN: Kolmogorov–Arnold Networks Proposes an alternative to MLP that outperforms in terms of accuracy and interpretability arxiv.org/abs/2404.19756

What an informative talk by Liang Zhang talking about all the research Anote has been doing about text classification. Watch the talk here: https://lnkd.in/exrr5cKh

Check out my blog post on How Crypta Ensures Literature Data Completeness! https://wix.to/MisTgOZ #newblogpost

*** Grammar as a Biometric for Authorship Verification *** If you are a forensic linguist and are interested in the topic of "Authorship Verification" (AV) --> Valerio Gherardi summarizes some key aspects of our latest AV approach called *LambdaG*. Enjoy reading! Paper: https://lnkd.in/eAq4SpVJ Overview of the AV field (including code): https://lnkd.in/e-qsJrUP #AuthorshipVerification #TextForensics #WritingStyleClassification

This paper proposes a novel mechanism for concatenating outer polar codes with inner repetition codes which we refer to as polar coded repetition. More specifically, the authors propose to transmit a slightly modified polar codeword by deviating from Arıkan’s standard 2×2  Kernel in a certain number of polarization recursions at each repetition block. ----Fariba Abbasi, @Hessam Mahdavifar, Emanuele Viterbo More details can be found at this link: https://lnkd.in/gVR7AtCb

New preprint on the ArXiv today! When are the correlations shared by several parties *genuinely* nonlocal? My coauthors and I explored this question, using GHZ and caterpillar states 🐛 (a family of quantum graph states that includes cluster states) to show that correlations that are genuinely multipartite nonlocal when we allow only for local operations and shared randomness (LOSR) do not necessarily retain this quality in our new model, where we allow for local operations and communication with nearest neighbours (Local-Operations-and-Neighbour-Communication, or LONC). 📞

Check out our latest work, featured on the Google Research blog, on our recent NAACL paper "Pre-translation vs. Direct Inference for Multilingual Applications."

In this episode, we discuss Transformers need glasses! Information over-squashing in language tasks by Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João G. M. Araújo, Alex Vitvitskyi, Razvan Pascanu, Petar Veličković. The paper explores how information propagates in decoder-only Transformers, revealing a phenomenon where different input sequences can result in nearly identical final token representations. This issue, worsened by low-precision floating-point formats, impairs the model’s ability to distinguish between these sequences, leading to errors in specific tasks. The authors provide theoretical and empirical evidence of this problem and suggest simple solutions to mitigate it.