AI Safety Institute

The UK-US agreement on AI safety is a significant moment for the AI Safety Institute and for the development of global safety standards on AI.  Here’s what it involves: The US and UK AI Safety Institutes will jointly test advanced AI models. We will also share research insights, share model access, and enable expert secondments between the Institutes. This will allow us to develop a shared framework for testing advanced AI and international best practices for other countries to follow. By working together, the UK and US can minimise the risks of AI and harness its potential to help everyone live happier, healthier and more productive lives. Find out more: https://lnkd.in/eX4bsy6G

Our new blog, "Early lessons from evaluating frontier AI systems", explores how we're assessing the safety of advanced AI models. We're using a combination of automated evaluations, expert red teaming, and human uplift studies to identify potential risks. Key considerations for effective testing include: ➡️ Choosing the right systems - focusing on those with the highest potential for risk ➡️ Timing evaluations - both before and after deployment ➡️ Prioritising testing areas - such as misuse, societal impacts, and agent capabilities We're also developing robust testing methods, including: ➡️ Risk modelling - to understand potential harm pathways ➡️ Capability thresholds - to identify critical risks ➡️ Predictive evaluations - to anticipate future risks Evaluating AI safety is complex and the science is still new. We're committed to developing effective methods and sharing our insights. Read our full report for more details:  https://lnkd.in/e_FdiKN3

Our Systemic AI Safety Fast Grants scheme is open for applications. Working closely with UK Research and Innovation and The Alan Turing Institute, we’re supporting projects that advance this new area of research. Moving beyond the safety of models themselves, systemic safety focuses on the broader resilience of society and critical infrastructure to AI-related hazards. We expect to offer around 20 grants up to £200,000 each in this phase, and particularly welcome projects bringing together researchers, industry and civil society. Find out more information and full guidance on how to apply: https://lnkd.in/euCYmY8S

Many new LLMs now possess agentic capabilities - they can use external tools and complete tasks for users, instead of just interacting as chatbots—this creates a new set of risks which require new evaluations and mitigations.   We've published new research in collaboration with Gray Swan AI measuring the harmfulness of LLM agents.   The paper highlights the need for LLM safety evaluations that focus on unique harms from AI agents with access to external tools.   AgentHarm, a novel dataset, is designed to measure unique agent harms. It's easy to run, comprehensive & reliable.   Read the full paper: https://lnkd.in/eHYM3P_k View the dataset: https://lnkd.in/ey-geJM4

Should AI systems behave like people? AI systems that can interact with us naturally are getting better. Humanlike AI could result in more engaging systems, but also poses safety risks and raises ethical questions.  Our new study asks the UK public what they think about humanlike AI. We asked the UK public for their thoughts on a range of topics, including:  ➔ Should a person know when they’re talking to an AI system?   ➔ Should chatbots express emotions like joy, sadness or loneliness?   ➔ Is there a world where humans and AI systems form meaningful relationships?  Here’s a snapshot of our findings:  ➔ Most people think AI should reveal itself not to be human, but didn’t mind realistic conversation ➔ Most did not want AI to express emotions, unless they were idioms (like “I’m happy to help”) ➔ Most people did not think that people could or should form personal relationships with AI systems Understanding the public’s view on humanlike AI helps to ensure that what counts as “safe” AI behaviour isn’t decided by researchers or policymakers alone.  This is key as we work with the wider AI community to minimise potential harm to the public from AI. https://lnkd.in/eWKUZjVw

We are announcing new grants for research into systemic AI safety.   Initially backed by up to £8.5 million, this programme will fund researchers to advance the science underpinning AI safety.   The world needs to think carefully about how to adapt our infrastructure and systems for a new world in which AI is embedded in everything we do. This programme is designed to generate a huge body of ideas for how to tackle this problem, and to help make sure great ideas can be put into practice. Read more: https://lnkd.in/eHHiFCbG

We're opening an office in San Francisco! This will enable us to hire more top talent, collaborate closely with the US AI Safety Institute, and engage even more with the wider AI research community. In London, we have built a leading research team in government, attracting senior alumni from OpenAI, Google DeepMind, and Oxford. We're excited to keep building this team globally now and to drive international coordination around AI safety. Find out more: https://lnkd.in/eButCCAi

We have published our latest progress report.    Read more below on our fast-paced work delivering on our mission to evaluate AI models and advance the science on AI risk.    https://lnkd.in/giphDpGe

This has been a superb week for investment in the UK – a huge vote of confidence in our plan. Today CoreWeave – a US AI start up valued at $19 billion – announced a $1 billion investment in data centres in the UK. On Tuesday the biggest investment in a UK AI start-up in history was announced. Over $1 billion into autonomous vehicle start-up Wayve.  On top of that CoreWeave is also establishing its European headquarters in London. And earlier this week top US company Scale AI announced it was doing the same.   They are not the only ones. Microsoft recently announced their AI hub in London. OpenAI, Anthropic, Palantir and Cohere have all chosen to locate their European headquarters here.  We will keep building on this success. This Government is unashamedly optimistic about the power of technology.   The UK is at the cutting edge of applying AI to drive exciting scientific advances.  Work is already underway on an AI model that looks at a single picture of your eyes to predict heart disease, strokes or Parkinson's. When the pioneers say AI could cure cancer, we believe them. Too often regulation can stifle those innovators. We cannot let that happen. Not with potentially the most transformative technology of our time. That’s why we don’t support calls for a blanket ban or pause in AI.   It’s why we are not legislating. It’s also why we are pro-open source.   Open source drives innovation.   It creates start-ups. It creates communities. There must be a very high bar for any restrictions on open source.  But that doesn’t mean we are blind to risks.   We are building the capability to empirically assess the most powerful AI models.   Our groundbreaking AI Safety Institute is attracting top talent from the best AI companies and universities in the world.  While talent is the key ingredient for an AI ecosystem, access to powerful computers necessary to train and experiment with AI is a close second.  That’s why we’re investing £1.5bn into compute.   Our first cluster of 5,000 of Nvidia’s latest AI superchips will go live this summer in Bristol, alongside the new Dawn computer in Cambridge.   We will soon set out how start-ups and academia will access these powerful new supercomputers. All of this progress is part of our plan to grow the economy. The sector is already worth more than £3.7 billion every year and employs over 50,000 people.    We know open source is a recipe for innovation.     That’s why the AI Safety Institute is today open sourcing what it has built.    The code for its Inspect project – a framework for building AI safety evaluations – is now available to anyone to use.   The AI Safety institute will also soon announce plans for an Open Source Open Day bringing together experts to explore how open source tools can improve safety.    This government’s approach is pro innovation, pro AI, pro open source and pro empiricism.   And it’s working.

We open-sourced Inspect, our framework for large language model evaluations: https://lnkd.in/eZgtjHe8. Inspect enables researchers to easily create simple benchmark-style evaluations, scale up to more sophisticated evaluations, and build interactive workflows.   Sharing Inspect through open source means our approach to AI safety evaluations is now available to anyone to use and improve, leading to high-quality evaluations across the board and boosting collaboration on AI safety testing. We're excited to see the research community use and build upon this work!