The International Center for Future Generations - ICFG

🌐 AI Safety Institutes: how could their structure impact effectiveness? AI governance is evolving rapidly, with the EU, OECD, and UN launching new initiatives to address AI safety more comprehensively. An important part of the AI governance puzzle: the AI Safety Institute Network. Various countries are now establishing these institutions, which will play a key role in shaping governance for advanced AI systems. However, they differ significantly in structure, which could impact their effectiveness. Alexander Petropoulos from our Advanced AI team identified three key structural differences that may impact how well these institutes achieve their goals.

The governance of Solar Radiation Modification is in its infancy - what needs to happen to improve it? Cynthia Scharf from our Climate Interventions program argues two aspects are central: monitoring and transparency. Right now, no one really knows the full extent of research being done today, and currently there’s no global monitoring of who might be testing what, going forward, in the stratosphere. The audio is taken from a recent episode of Carnegie Endowment for International Peace’s podcast "The World Unpacked”. Listen to the full episode - link in comments.

We’re thrilled to have Aaron Maniam join as Senior Fellow in our Advanced AI team! 🙌 Aaron is a Fellow of Practice at the University of Oxford and a global technology policy and public administration expert. He co-chairs the World Economic Forum’s Global Future Council on Technology Policy and is a member of the OECD - OCDE’s Expert Group on AI Futures. Before that, he helped shape Singapore’s policy on the digital economy, digital society, and digital diplomacy and was the founding Head of Singapore's Centre for Strategic Futures. We look forward to his contributions to furthering our #CERNforAI project and many other upcoming projects. Oh, and follow him on LinkedIn if you're into tech policy - he frequently shares unique insights from his work!

How can we determine whether an AI system poses significant risks? It’s a tricky question. AI risks are complex and broad, making it hard to set clear criteria. Recently, the AI governance community has focussed on so-called “risk thresholds” as one potential solution. Risk thresholds are values used to determine when AI systems pose unacceptable risks or require closer monitoring and mitigation efforts. They can include technical factors, such as the scale of a system, and human values, such as social or legal norms. As a submission to OECD.AI‘s public consultation on this topic, Eva Behrens and Bengüsu Özcan from our Advanced AI program suggested refining risk thresholds. Here’s the TL;DR: * AI governance can take inspiration from nuclear safety goals, combining broad and specific risk measures. * Regulating AI based on computing power is a useful tool, but not enough to fully manage risks. Dangerous AI capabilities like self-replication and self-improvement should also be used to assess risk. * Governments, not companies, should define safety standards and enforcement mechanisms, which might include licensing. * If an AI exceeds risk thresholds, it should be shut down until safety improvements are made. * Global standards are needed for AI safety, and organizations like the OECD can lead this effort. Full paper: https://lnkd.in/eYaJ4_25

Over my years working at the nexus of technology and democracy, I’ve noticed we often get caught up in individual rabbit holes—whether it’s fighting disinformation, building digital literacy, or defending human rights in the digital world. Many are rightly addressing these issues, but we often miss the forest for the trees. That’s where I think Marietje Schaake's book succeeds. #TheTechCoup describes how these issues all stem from a single phenomenon: democracy capture by big tech. Whether you agree with that or not, the book elevates the conversation beyond policy silos and makes it accessible for everyone—not just policy experts. It lays out - at length - what we (governments and citizens both) can do to turn it around. So here are my two recommendations: #1: Go read The Tech Coup—especially if you work at the intersection of tech and democracy! #2: Come join us at our event in October, featuring a discussion with Marietje and James Kanter and a book signing afterwards (link in the comments).   See you there!

What is Solar Radiation Modification? Well - we won’t find a better way to explain it than Cynthia Scharf from our climate interventions team. She recently joined “The World Unpacked”, a podcast hosted by Sophia Besch, PhD from the Carnegie Endowment for International Peace. Highly recommend it if you're interested in the geopolitical and conflict-related implications of this emerging technology. Link in the comments!

How can Europe build the industries of the future, like quantum? ⚛️ To succeed, Europe must change its recipe for innovation: - stop spreading our bets too thin - invest in foundational technologies - simplify startups’ access to procurement - put innovators in charge of funding programs. Today, Europe lags behind in the global tech race because of excessive precaution and insufficient institutional capacity. We can’t let the quantum opportunity fall through the same bureaucratic cracks. My take on Euractiv, following a thoughtful and timely discussion with Andrea Rocchetto, Marieke HOOD, Matija Matoković, and Pascal Maillot https://lnkd.in/eVrEy46X #technology #innovation #investment #future #competitiveness #europe #quantum #quantumcomputing

Two exciting updates about our Senior Fellow Marietje Schaake: 🗞️ Marietje Schaake to lead the development of the EU’s AI Code of Practice Yesterday, Marietje was announced as one of four Chairs tasked with leading the development of the AI Code of Practice. Organised by the EU AI Office, this ambitious process brings together hundreds of stakeholders from civil society, industry, and academia. It aims to establish a framework that ensures AI is developed and deployed responsibly across the EU. Marietje will chair Working Group 4 on Internal Risk Management and Governance of General-purpose AI providers, alongside Co-Chairs Markus Anderljung and Anka Reuel. The group will work on how organisations should self-regulate and manage the risks inherent in AI systems. https://lnkd.in/e5E8Hpaz 🗞️ Meet her at our event “Conversation with Marietje Schaake on The Tech Coup” later this month Our upcoming event with Marietje and James Kanter (you’ll know him from the EU Scream Podcast) on 16th October in Brussels is shaping up to be very popular. We haven’t even properly started advertising yet, and seats are already filling up fast. Register here: https://lnkd.in/ea8Ghrg2

🌐 AI Safety Institutes - what they are, and how they’re coming along A major outcome of this year’s Seoul AI Safety Summit was the announcement of a global network of state-run AI Safety Institutes (AISIs). These institutions are currently being set up and are meant to promote the safe, secure, and trustworthy development of AI. The AISIs will play a crucial role in shaping international coordination on developing safe AI, which is why we’re keeping a close eye on them. Here are three AISI’s that are worth paying attention to: 🇬🇧 United Kingdom The UK AI Safety Institute is known for its fast-moving, startup-like approach, prioritizing rapid action, flexibility and top talent acquisition. It leads in empirical evaluations of AI risks and plays a key role in global cooperation, organizing major AI safety summits. It is also conducting object-level safety research and exploring different safety cases. 🇪🇺 European Union The EU AI Office focuses on enforcing the EU AI Act, setting the bar for regulatory governance. Its codes of practice will begin the world-first work to crystalise what AI regulation actually means, but it faces challenges with a slower setup and a more fragmented structure. Its emphasis on compliance and outlier position as a regulator may limit its ability to match the UK's speed in risk evaluation and adaptability and foster multi-stakeholder relations. 🇺🇸 USA The American AI Safety Institute is science-driven, emphasizing measurement and industry collaboration to mitigate AI risks. It benefits from strong partnerships with major AI companies but is still building up its capacity. It is currently struggling to secure funding to operate effectively. Despite this, it remains one of the most important AISIs since nearly all frontier AI companies are based in the US. There are many more AI Safety institutes. Take a deep dive in Alexander Petropoulos’ paper on our website: https://lnkd.in/eaaMe7ni

Our Climate Interventions Program’s “Listening & Learning” approach in action Climate intervention technologies emerge with great uncertainties, controversies, and immature governance. Our program seeks to help strengthen governance through a "listening & learning" approach involving decision-makers and stakeholders. In the last ten days, the team has been doing just that: 🏫 Washington Workshop on Solar Radiation Modification Matthias Honegger and Cynthia Scharf presented our “listening & learning” approach at a workshop featuring top experts and policymakers. The event was hosted by Resources for the Future and The Salata Institute for Climate and Sustainability at Harvard University. 🇺🇳 UN Science Summit & New York Climate Week This week our climate team is also in New York, to engage with the global climate community on the many different views on climate interventions. If you’re in New York, there's still time to connect with Cynthia and Roxanne Cordier! 🤝 A Personal Conversation – Moving Beyond Polarization At an event co-hosted by us and The Alliance for Just Deliberation on Solar Geoengineering (DSG), Matthias moderated a diverse panel including Holly Jean Buck, Hassaan Sipra, Renzo Taddei, Kate Marvel, and Ellen Haaslahti who reflected on their personal aspirations toward improving discussion and consideration of climate interventions. The conversation involved everyone in the room and allowed for a personal exploration of hopes and concerns. Read more on our “listening & learning” approach here: https://lnkd.in/e5d8kwtk