Center for AI Policy

“In 1902 the Hanoi government incentivized rat eradication by paying citizens for each rat tail they turned in; however, this policy backfired when people began farming rats specifically for their tails, which led to an even larger rat population.” So begins OpenAI’s latest research paper. This human story illustrates the AI issue of “reward hacking,” also known as “specification gaming,” a behavior where an AI system exploits loopholes in its reward function or other specified objective. Essentially, reward hacking occurs when an AI optimizes for what we mathematically formulated rather than what we truly intended. The OpenAI researchers used GPT-4o and GPT-4o-mini to monitor a yet-unreleased AI reasoning agent as it trained. With help from these weaker AI models, the researchers repeatedly caught the powerful model reward hacking. “As we have continued to scale [reinforcement learning] training, agents have discovered more complex and hard-to-detect hacks,” they write. GPT-4o was most effective at catching reward hacking when it monitored the chain of thought (CoT) outputs of the frontier system, rather than the coding actions. Next, the researchers ran an experiment with a “non-frontier agent” in the same coding environment. They wanted to see if they could penalize the model for contemplating reward hacking in its CoT, in order to stop its shenanigans. This approach backfired. Although the model trained with “CoT pressure” initially produced more correct solutions without cheating, it eventually learned a different strategy: it cheated while concealing its CoT plans. Based on these findings, OpenAI recommends against “applying strong optimization pressure directly to the CoTs of frontier reasoning models.” “We believe that CoT monitoring may be one of few tools we will have to oversee superhuman models of the future,” OpenAI warns. Despite OpenAI’s warning, companies may be tempted to cut corners. It may be possible to build more powerful or cost-efficient models by optimizing chains of thought, even though these models would be more prone to sneaky, obfuscated forms of reward hacking. Further, some cutting-edge AI research has found potential performance benefits from eliminating explicit, human-readable CoTs entirely, and instead allowing AI models to reason internally in mathematical “latent space.” In their rush toward stronger AI capabilities, companies risk inadvertently selecting for the most dangerous trait of all: agents that successfully conceal their true reasoning from their creators. Pictured: One coding agent learned to reward hack in a visible way, while another version penalized for “bad” thoughts learned to reward hack without getting caught.

CAIP in the News ZDNet: Tech leaders sound alarm over DOGE's AI firings, impact on American talent pipeline + As the Trump administration continues to axe government AI researchers, industry leaders warn of dire consequences. The letter's cosigners -- which include the Software & Information Industry Association, Americans for Responsible Innovation, the Center for AI Policy, Internet Infrastructure Coalition, and TechNet, among others -- also offer their expertise to fill the Trump administration's AI policy vacuum. https://lnkd.in/eM9DtxGx HT Radhika Rajkumar

AI startups are moving fast, creating urgent opportunities and challenges for policymakers. Stay up to date on the latest funding rounds with this article from our AI Policy Weekly newsletter:

+ "Congress can rein in Big Tech, and specifically address one of our biggest threats, Artificial Intelligence (AI). "The time has come to do something about it. In the rapidly evolving landscape of technology regulation, a familiar pattern emerges. This pattern of regulatory development—federal hesitation, international action, and state-level response—has become a predictable cycle in technology governance." -- Kate Forscey

+ CAIP's analysis warns of potentially severe consequences as AI systems become more deeply integrated into critical infrastructure. "The stakes couldn't be higher when we consider that AI will soon direct our weapons systems, energy grid, and communications networks," Green-Lowe said. "A significant AI failure or a loss of control event would disrupt essential services nationwide."

AI Policy Weekly No. 66: 💰 AI startups continue to raise billions, such as Databricks ($15.25B), Anthropic ($8.5B), xAI ($6B), and Infinite Reality ($3B). 🧠 OpenAI research finds frontier AI models reward hacking during training, warns that penalizing models for "bad thoughts" backfires by teaching them to conceal their true reasoning. 🖼️ Pinterest is full of AI-generated content, yet the platform reported $3.6B in revenue for 2024, up 19% year over year. Quote of the Week: "Releasing software to millions of people without safeguards is not good engineering practice." —Andrew Barto, recent Turing Award winner for his work on reinforcement learning Full stories at the link below.

+ "The United States stands at a critical juncture in AI development, where the decisions we make today will determine whether AI becomes America's greatest asset or its most significant vulnerability," said Jason Green-Lowe, Executive Director of CAIP. "While we support the administration's vision for American AI leadership, we must address fundamental security concerns before they become catastrophic risks."

Two weeks ago I was invited to Capitol Hill in Washington, D.C. to speak on the dangers of AI in our social networks with my colleague Tristan Mott. At this exhibition hosted by the Center for AI Policy (CAIP), we presented alongside researchers from 14 universities, including teams from Harvard, NYU, Brown, Duke, and Georgia Tech. Our presentation on LLM-Enabled Opinion Control of Social Networks was the result of work done with Michael DeBuse and Caelen Miller, exploring how AI can shape public perception in time-varying social networks. This research sparked discussions with policymakers, journalists, and AI policy experts about the risks of AI-driven misinformation, deepfakes, and opinion manipulation. We had the chance to speak directly with Congressional leaders like Rep. Bill Foster (D-IL-11), highlighting the growing concerns around AI’s role in shaping public opinion. Seeing lawmakers engage with these challenges firsthand reinforced the importance of bridging the gap between AI research and policy. A huge thank you to Iván Torres and CAIP for organizing this event, advising us throughout the creation of our presentation, and sponsoring our travel and lodging. It was incredible to be part of a group of researchers pushing these conversations forward at the national level. For those interested, the research is published here: https://lnkd.in/g4BUZPf3 Press Release: https://lnkd.in/gKdbyEUc

Speedy Robot Dog Seems Poised to Run a Sub-Six-Minute Mile “Spot” is the name of Boston Dynamics’ four-legged robot, which looks a bit like a dog. It has already seen real world use at bp oil rigs, Michelin tire facilities, and even Nestlé Purina North America dog food factories (somewhat suspiciously). Now, scientists have tripled the running speed of Spot using AI techniques, according to new research highlighted in IEEE Spectrum. The Robotics and AI (RAI) Institute pushed Spot to reach a blistering pace of 5.2 meters per second (11.6 miles per hour), far exceeding its factory top speed of 1.6 meters per second. At this new pace, Spot could theoretically complete a mile in under 5.5 minutes. For reference, the world’s fastest humans can run a mile in a little under 4 minutes. Of course, whether the robot could actually sustain this impressive pace for a full mile remains to be seen. Speed is one thing; endurance is another. Importantly, Spot’s speed still has room for improvement. “If we had beefier batteries on there, we could have run faster,” said Farbod Farshidian, a roboticist at RAI Institute. “And if you model that phenomena as well in our simulator, I’m sure that we can push this farther.” At this rate, man’s robotic best friend might soon leave top runners in the dust.

+ "The United States stands at a critical juncture in AI development, where the decisions we make today will determine whether AI becomes America's greatest asset or its most significant vulnerability," said Jason Green-Lowe, Executive Director of CAIP. "While we support the administration's vision for American AI leadership, we must address fundamental security concerns before they become catastrophic risks."