FAR.AI

Leading scientists from China and the West gathered at Venice for the third in a series of International Dialogues on AI Safety, urging swift action to prevent catastrophic AI risks that could emerge at any time. Congratulations to the Safe AI Forum team for a successful event, convened by AI pioneers Professors Stuart Russell, Andrew Yao, Yoshua Bengio, and Ya-Qin Zhang. FAR.AI is proud to support this important effort as a fiscal sponsor. 👉 Read the full statement at http://idais.ai 📖 Blog post: https://lnkd.in/gnvDfR_8 📰 NYT coverage: https://lnkd.in/eEp8XqbH ✨ Follow us for the latest on AI Safety insights!

Day 2 of the Bay Area Alignment Workshop in Santa Cruz was packed with learnings and rich discussions! Thank you to all our incredible speakers and session leaders: - Industry Lab Safety Approaches: Dave Orr, Sam Bowman, Boaz Barak - Interpretability: Atticus Geiger, Andy Zou - Robustness: Stephen Casper, @Alex Wei, Adam Gleave - Oversight: @Julian Michael, Micah Carroll - SB 1047 Post Mortem with Ari Kagan & Thomas Woodside, hosted by Shakeel Hashim And a heartfelt thank you to all our attendees for your engagement and contributions throughout the workshop. Together, we've sparked valuable dialogues in AI alignment and safety. Follow us to be notified when new recordings from this Bay Area Alignment Workshop are published! Meanwhile, watch past sessions on our YouTube channel to learn from AI safety experts across academia, industry, and governance. https://lnkd.in/g_KkNbjn

Neural network mind control? Our research shows RNNs playing Sokoban form internal plans early on. Linear probes, can read and alter these plans as they form. With model surgery, we make the network solve puzzles far larger than it was trained for. Why does understanding planning matter? It’s key to AI alignment. Research on goal misgeneralization and mesa-optimizers shows that AIs can develop unintended goals. By studying planning, we can better guide AI to avoid optimizing for the wrong outcome. Previously, we found that RNNs "pace" in cycles to solve harder Sokoban levels, often using extra compute time in the first few steps—over 50% of these cycles happen within the first 5 moves. Giving the network thinking time with NOOPs largely eliminates the pacing behavior! By training linear probes on the RNN's hidden states, we can accurately predict future moves, demonstrating early planning capabilities (F1 scores: 72.3% for agent directions, 86.4% for box directions). Our non-causal linear probes are based on Thomas Bush et al's (2024) concurrent work on a very similar network. Intervening with these probes lets us adjust the agent's plan. Interestingly, box plans often take precedence—agent probe interventions only work when box probes are insufficient to explain the behavior. The final recurrent layer learned to allocate each possible action to a distinct channel, meaning the MLP layer doesn’t need to work hard—it simply reads these action channels and outputs the next move. While the RNN can process images of any size, the final MLP layer flattens the 3D hidden state, limiting the network to a fixed grid size. So, we asked: can the network generalize to larger puzzles by using these learned action channels? YES! With probes, RNN generalizes to out-of-distribution puzzles, solving 2-3x larger grids with 10+ boxes, despite training on 10x10 grids with 4 boxes. This level took 128 thinking steps, and lasted for 600+ steps! You can try #30 at https://lnkd.in/g9tD8gQe Many levels don’t require thinking steps to solve, for example Sokoban Jr. 2’s level 17 https://lnkd.in/gMVGmS9s We also trained top-K Sparse Autoencoders (SAEs) on the last layer’s hidden state. Surprisingly, all the interpretable features SAEs found were already present as individual channels, which were more monosemantic than the SAEs. A negative result for SAEs, or just an edge case? This is just the start. Our future work will focus on understanding how the RNN computes the best plan and whether these interpretability techniques generalize to other domains. Explore our open-source code, models, probes, and SAEs: 💻 Code: https://lnkd.in/g42pNjvF 📝 Blog: https://lnkd.in/g_5SpGRc 📄 Full paper: https://lnkd.in/gkVesDEd 👥 Research by Mohammad Taufeeque, Philip Quirke, Max Li, Chris Cundy, Aaron Tucker, Adam Gleave, Adrià Garriga-Alonso

Kicked off Day 1 of the Bay Area Alignment Workshop in Santa Cruz with amazing energy and a lineup of insightful talks! Huge thanks to our fantastic speakers: - Anca Dragan for her deep dive into Threat Models - Elizabeth (Beth) Barnes & Buck Shlegeris for unpacking Monitoring & Assurance - Hamza Tariq Chaudhry, Siméon Campos, Nitashan Rajkumar, Kwan Yee Ng 吴君仪, and Sella Nevo for adding their insights on Governance & Security Special shout-out to our discussion leaders: - Threat Models: Anca Dragan & Richard Ngo - Governance: Gillian K. Hadfield - Safety Portfolios: Anca Dragan & Evan Hubinger We’re looking forward to more thought-provoking sessions and conversations!

🚀 Are larger models more robust? We explored the impact of scaling model sizes on robustness in large language models (LLMs). Scaling up LLMs has made them increasingly powerful, but not always robust. Will continuing to scale up LLMs also improve robustness? Here’s what we found: 🔍 What we did: We tested 10 fine-tuned Pythia models ranging from 7.6M to 12B parameters on 6 binary classification tasks, like classifying emails as spam or not spam or classifying responses as harmful or not harmful. We used GCG and a RandomToken baseline for adversarial attacks. 📊What we found: 1️⃣Larger models are generally more robust, but this isn’t guaranteed. While bigger models can lower attack success rates, the trend is inconsistent and sometimes scaling hurts robustness. 2️⃣Adversarial training is significantly more efficient than scaling models, requiring far less compute to achieve the same level of robustness. 3️⃣Larger models generalize well when defending against different types of attacks, or stronger attacks than the ones they were trained on, but even if defenders double adversarial training compute, attackers need less than 2x to get the same attack success rate. 🎯  Key Takeaway: The offense advantage doesn't lessen as models get bigger, and might even increase. We need algorithmic breakthroughs in the science of robustness––not just bigger models. For more information: 📝 Check out the blog post: https://lnkd.in/g8CY4FcG   📄 Read the full paper: https://lnkd.in/e9YtUNiT

"The main challenge we point out … is that you can search for features that satisfy these consistency properties, but many features satisfy them, not just what we think of as truth." At the Vienna Alignment Workshop hosted by FAR.AI, Vikrant Varma explored the difficulties of extracting truthful information from large language models without labeled data. He stressed the need for advanced methods to distinguish truth from distracting features in unsupervised discovery. Key Highlights: - Overcoming challenges in unsupervised knowledge discovery - Addressing the risks of extracting distracting features over truth - Developing advanced methods beyond consistency checks - Identifying truth through structural differences in AI models 🎥 Watch the full recording and continue the discussion: https://lnkd.in/eQxvcPkd 🚀 Help build trustworthy, beneficial AI—explore careers at https://far.ai/jobs/.

"Sometimes, as models get bigger, they get worse [where the answers] are unexpected. The large average is hiding lots of details about individual tasks" - Ian McKenzie As a Research Engineer at FAR.AI, Ian McKenzie explores how AI scaling impacts safety. In a recent conversation with Heather Gorham, Ian dives into inverse scaling—a nuanced concept where bigger AI models don’t always produce better outcomes. In this episode of Applied Context, Ian shares insights on: 🔥 The impact of scaling laws on modern AI systems 🛠️ How a crowdsourced contest helped identify inverse scaling patterns 💡 Why larger models can struggle with unexpected patterns 🚨 The intriguing U-shaped scaling phenomenon—where models get worse before they improve Ian’s research focuses on understanding these complexities and ensuring that as AI systems grow, they do so safely and effectively. This interview provides a glimpse into his approach to solving some of AI's most important challenges. If you’re interested in AI safety or scaling challenges, this is a valuable watch. 👉 Full interview: https://lnkd.in/gT-EHhZS

"... we find that you need very, very few samples to unlock these models … we also find that unlocking generalizes super well."At the Vienna Alignment Workshop hosted by FAR.AI, Dmitrii Krasheninnikov showed how fine-tuning effectively unlocks AI capabilities, suggesting that AI sandbagging may be less problematic with high-quality demonstrations. Key Highlights: - Fine-tuning's role in unlocking AI capabilities - Testing capabilities with password-locked models - Leveraging high-quality demonstrations for better AI training 🎥 Watch the full recording and continue the discussion: https://lnkd.in/eR5D5DNM 🚀 Help build trustworthy, beneficial AI—explore careers at https://far.ai/jobs/.

"We have techniques like RLHF that are really, really good at … making models behave quite nicely in most typical situations. But what they do not seem to be good at is removing these latent capabilities very deeply.” Stephen Casper tackled the challenge of latent harmful capabilities in AI models at the Vienna Alignment Workshop hosted by FAR.AI. He proposed applying safety engineering principles to design and test systems that can endure stronger stresses than those expected in deployment. Key Highlights: - Addressing latent harmful capabilities in AI models - Applying safety engineering principles to AI - Ensuring AI robustness and safety through generalized adversarial training - Mitigating risks with model manipulation and latent space attacks 🎥 Watch the full recording and continue the discussion: https://lnkd.in/e-dZF7Fc 🚀 Help build trustworthy, beneficial AI—explore careers at https://far.ai/jobs/.

"The socio-technical approach in AI alignment...that emphasizes the importance of the dynamics in human-AI systems. But I think this may not be enough." At the Vienna Alignment Workshop hosted by FAR.AI, @Zhaowei Zhang proposed a structured framework for AI alignment, integrating real-time control, stakeholder alignment, and broader regulatory oversight to achieve clear and interpretable AI systems. Key Highlights: - Three-layer paradigm for AI alignment - Importance of real-time control in AI systems - Aligning AI with stakeholder goals and values - Integrating macroscopic regulation for broader oversight 🎥 Watch the full recording and continue the discussion: https://lnkd.in/eugwcBta 🚀 Help build trustworthy, beneficial AI—explore careers at https://far.ai/jobs/.