Epoch AI

We are looking for strong mathematicians to write original and difficult mathematics problems for a new benchmark for AI systems. We'll prioritise candidates who can start soon, so apply as soon as possible! This is a full-time, temporary, and remote contract. A strong candidate would have at least a Ph.D-equivalent expertise in Math, experience with math competitions, and experience solving difficult math problems that require programming. We are also offering referral rewards to whoever lets us know of somebody who might be a good candidate and who we end up hiring for the role, or who ends up submitting at least 4 questions that qualify for the benchmark. Reach out to careers@epochai.org for details. https://lnkd.in/daFfMyGV

Can AI scaling continue through 2030? Our new report examines whether constraints on power, chip manufacturing, training data, or data center latencies might hinder AI growth. Our analysis suggests that AI scaling can likely continue its current trend through 2030. Training state-of-the-art AI models requires a massive amount of computation, growing by 4x every year. If this trend continues, we will see training runs 10,000x larger than GPT-4 by 2030. Achieving such a scale-up would require immense resources. For each key bottleneck (power, chips, data, and latencies), we project potential scale using semiconductor foundries' expansion plans, electricity providers' forecasts, industry data, and our own research. 🔌 Power: Meta's Llama 3.1 405B training used 16,000 H100 GPUs, consuming about 30MW. By 2030, the largest training runs could demand 5GW of power, accounting for efficiency gains and increased training durations. A data center with a >1GW capacity would be unprecedented, but is in line with stated industry plans. Distributed training runs across US states could further surpass this, doubling or even 10x-ing the power a single campus could muster. 💾 Chip production: 16,000 H100s is far from the tens of millions of chips needed to scale 10,000x beyond GPT-4. While GPU production is constrained by advanced packaging and high-bandwidth memory, foundries like TSMC are on track to expand their capacity and meet this demand. Planned scale-ups and efficiency gains could enable 100M H100-equivalent GPUs to be dedicated to a 9e29 FLOP training run by 2030, accounting for the fact that GPUs will be distributed between several labs and between training and inference. This could be much higher if most of TSMC’s top wafers went to AI. 📚 Training data: All indexed web text could be enough for several thousand-fold larger training runs today. By 2030, this stock of data may have grown enough for a 10,000x scale-up. Multimodal data (image, video, audio) could expand AI training scale by ~10x. Synthetic data shows promise in domains like coding/math, but risks model collapse. Synthetic data could enable multiple orders of magnitude more scaling, but with increased compute costs. ⏳ Latency: As models grow, they need more sequential ops per example they are trained on, limiting the size of training runs. Increasing batch size helps, but this has diminishing returns. On modern hardware, these latency constraints would keep runs to ~1e32 FLOP. Exceeding this would require new network designs or lower-latency hardware. Despite these significant bottlenecks, our estimates suggest they won't significantly slow the growth rate of AI training runs. This suggests we could see another major scale-up—comparable to the jump from GPT-2 to GPT-4—by 2030. You can learn more about each of the bottlenecks, and our assumptions, by reading the full report here: https://lnkd.in/dKAnUYGJ

We are looking for somebody to lead the creation of a math reasoning benchmark for AI systems. The ideal candidate will have deep expertise in math, programming and people management experience, and connections in academic math circles. 🗓 6-month temporary role 💻 Fully remote 🌏 We can hire in many countries ⏳ Apply or share by August 18th! https://lnkd.in/enn-_EhU

We’ve just launched our new "Data on AI" page! This is a central hub for our data that includes key insights, interactive visualizations, and documentation. We hope this will be a valuable resource for our audience to examine the trajectory of artificial intelligence. We have what is possibly the largest publicly available database of ML models. This data helps us empirically ground our claims about AI, and has proven valuable for researchers, policymakers, and other stakeholders. Our goal is to make this data as accessible as possible. Currently, the page hosts two datasets. The first is our most comprehensive, and as of today includes 800+ notable AI models. You can learn how quickly the training compute of notable models is growing, compare progress across domains, and more. The second dataset, devoted to large-scale AI models, features 187 models that have likely been trained with over 10^23 floating-point operations. You can learn about which organizations and countries made these models and compare them to regulatory thresholds. We plan to continue building upon this work, creating interactive data pages for more aspects of AI. Check out the new data page at the link below, and follow us to stay informed about AI.

Our new research on data bottlenecks was also featured in the Associated Press! Read their piece here: https://lnkd.in/eMpG4X-s

Are we running out of data to train language models? State-of-the-art LLMs use datasets with tens of trillions of words, and use 2-3x more per year. Our new ICML paper estimates when we might exhaust all text data on the internet. We estimate the size of the available stock of public text data, and forecast when all of of this data will be used for training. We conclude that public human-generated text data will likely be fully used at some point between 2026 and 2032. This won't necessarily halt AI progress, but it may slow it down. AI developers may train increasingly large models on the same amount of data, but those gains will be limited. Eventually they will have to develop and refine alternative data sources. For more details, check out our article summary of the paper! https://lnkd.in/eauDCiYy 

How much does it cost to train frontier AI models? Join us next Wednesday, June 5th, for an insightful webinar on the rising costs of training cutting-edge AI models and the implications for AI research, development, and policymaking. Robi Rahman and Ben Cottier from Epoch AI, along with Nestor Maslej from the Stanford Institute for Human-Centered Artificial Intelligence (HAI), will present key findings from Epoch's in-depth analysis on AI training cost trends. Their research, featured in the 2024 AI Index Report, finds that the cost of training frontier AI models has been increasing at an astonishing rate of 2x to 3x per year since 2016. In this webinar, moderated by Valerie Belu from the Centre for the Governance of AI, our experts will break down the cost of hardware, energy and staff involved in training cutting-edge AI models. They will also discuss the implications of these trends for AI research and development, as well as the potential policy challenges and opportunities they present. https://lnkd.in/ecAsThFZ

We are hiring an operations associate to help us grow sustainably and provide efficient staff support, facilitating our mission of investigating the trajectory of AI for the benefit of society. You might be a great fit for this role if you have an operations mindset, strong communications skills, are service-minded and organized, and care deeply about Epoch’s mission. 💻 Remote role 🌎 We can hire in most countries 🗓 Apply by May 15th! https://lnkd.in/etWp-8xy

We’ve been researching trends in machine learning models in collaboration with the AI Index and the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Here’s a summary of what we found! 1. Before 2014, most notable ML models were released by academia, but industry has taken the lead since then. 3. The US leads in number of notable models, followed by China, France and Germany. 4. The number of parameters in ML models has grown exponentially over time. Some models today have over a trillion parameters. 5. Training compute, or the amount of computation required to train a system, has also been growing exponentially, crossing many orders of magnitude over decades. 6. Because of this growth in training compute, the monetary cost of training has also grown, due to the cost of acquiring and operating AI chips. 7. Most developers do not publish their training expenses, but we often have enough details about the training process, such as the chips used and the duration of training, to produce reasonable estimates of training costs. 8. The most expensive models now cost tens of millions of dollars to train. The original Transformer from 2017 cost only $900 to train, while GPT-4 cost $78 million, and Gemini Ultra cost nearly $200 million! These training costs have also grown exponentially over time. 10. Naturally, since these costs are driven by growing compute, there is a close correlation between training cost and training compute. 11. These escalating training costs have contributed to industry dominance of the field. 12. If trends continue, the cost of training ML models will continue to grow rapidly, and become increasingly economically significant. If you’d like to learn more, check out the full AI Index report below! https://lnkd.in/ea7KF28

We are hiring a researcher to investigate the economic basis of AI deployment and automation. This person work with the team to build out and analyze our integrated assessment model for AI automation, research the economics of AI training and inference, and build models to help forecast AI’s development and its impact on the economy. 💻 Remote role 🌎 We can hire in most countries 🔄 Rolling applications. We will close applications when we make a hire.