Jim Fan’s Post

OpenAI o1 introduces perhaps the most important figure in LLM research since the OG Chinchilla scaling law in 2022. The key insight is 2 curves working in tandem. Not one. People have been predicting a stagnation in LLM capability by extrapolating the training scaling law, yet they didn't foresee that inference scaling is what truly beats the diminishing return. I posted in February that no self-improving LLM algorithm was able to gain much beyond 3 rounds. No one was able to reproduce AlphaGo's success in the realm of LLM, where more compute would carry the capability envelope beyond human level. Well, we have turned the page.

OpenAI Strawberry (o1) is out! We are finally seeing the paradigm of inference-time scaling popularized and deployed in production. As Sutton said in the Bitter Lesson, there're only 2 techniques that scale indefinitely with compute: learning & search. It's time to shift focus to the latter. 1. You don't need a huge model to perform reasoning. Lots of parameters are dedicated to memorizing facts, in order to perform well in benchmarks like trivia QA. It is possible to factor out reasoning from knowledge, i.e. a small "reasoning core" that knows how to call tools like browser and code verifier. Pre-training compute may be decreased. 2. A huge amount of compute is shifted to serving inference instead of pre/post-training. LLMs are text-based simulators. By rolling out many possible strategies and scenarios in the simulator, the model will eventually converge to good solutions. The process is a well-studied problem like AlphaGo's monte carlo tree search (MCTS). 3. OpenAI must have figured out the inference scaling law a long time ago, which academia is just recently discovering. Two papers came out on Arxiv a week apart last month: - Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. Brown et al. finds that DeepSeek-Coder increases from 15.9% with one sample to 56% with 250 samples on SWE-Bench, beating Sonnet-3.5. - Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Snell et al. finds that PaLM 2-S beats a 14x larger model on MATH with test-time search. 4. Productionizing o1 is much harder than nailing the academic benchmarks. For reasoning problems in the wild, how to decide when to stop searching? What's the reward function? Success criterion? When to call tools like code interpreter in the loop? How to factor in the compute cost of those CPU processes? Their research post didn't share much. 5. Strawberry easily becomes a data flywheel. If the answer is correct, the entire search trace becomes a mini dataset of training examples, which contain both positive and negative rewards. This in turn improves the reasoning core for future versions of GPT, similar to how AlphaGo’s value network — used to evaluate quality of each board position — improves as MCTS generates more and more refined training data.

It is *incredibly* easy to game the LLM benchmarks. Training on test set is for the rookies. Here're some tricks to practice magic at home: 1. Train on paraphrased examples of the test set. "LLM-decontaminator" paper from LMSys found that you can beat GPT-4 with a 13B model (!!) on MMLU, GSK-8K, and HumanEval (coding) just by rewriting the exact same test question in different formats, phrasing, or even foreign languages. Easily +10 point gains. 2. It's easy to game the LLM-decontaminator as well. It only checks for paraphrasing, but you can use any frontier model to generate *new questions* that are different on the surface but very similar in solution template/logic. In other words, you attempt to overfit to a close distribution of the test set, but not to individual samples. HumanEval, for example, is a bunch of simple Python questions (i.e. a specific, narrow distribution) that do not reflect the real world coding complexity at all. 3. You can also prompt-engineer the heck out of your generator to fool the LLM-decontaminator or whatever detector. The detector is public but your data gen is private. Take advantage of that. 4. Increasing inference-time compute budget almost always helps. Self-reflection is a technique known for a long time (see Reflexion, Shinn et al. 2023). Also try simple majority voting or Tree of Thought. These thought traces are essentially test-time ensemble methods, and the more the merrier. It's obvious that ensemble of N things > 1 thing if you don't control for inference-time tokens. It's incredible that people still get excited by MMLU or HumanEval numbers in Sept, 2024. These benchmarks are seriously broken, and gaming them can be an undergrad homework project. I would not trust any claims of a superior model until I see the following: 1. ELO points on LMSys Chatbot Arena. It's difficult to game democracy in the wild. 2. Private LLM evaluation from a trusted 3rd party, such as Scale AI's SEAL benchmark. The test set must be well-curated and held secret, otherwise it quickly loses potency. Reading materials: - LLM-decontaminator: https://lnkd.in/gSjbC-uu - Code is open-source: https://lnkd.in/gMBHQPCC - Reflexion: https://lnkd.in/gr8c8EZV - Tree of Thought: https://lnkd.in/gPBfj36d

It's a tradition for hackers to run DOOM in crazy places: thermostats, "smart" toasters, even ATMs. Now they run DOOM purely in a diffusion model. Every pixel here is generated. A while ago, I said "Sora was a data-driven physics engine". Well, not quite, because Sora could not be interacted with. You set the initial condition (a text or initial frame) and may only watch the simulation passively. GameNGen is a proper neural world model. It takes as input past frames (states) and an action from a user (keyboard/mouse), and outputs the next frame. The quality is by far the most impressive I've seen on DOOM. However, this comes with significant caveats. Let's deep dive: 1. GameNGen overfits to the extreme on a single game by training on 0.9B frames (!!). This is a HUGE number, almost 40% of the dataset used to train Stable Diffusion v1. At this point, it's likely memorizing how DOOM renders from every corner of the game in all scenarios. DOOM doesn't have that much content anyway. 2. GameNGen is more like a glorified NeRF than a video gen model. A NeRF takes images of a scene from different view angles, and reconstructs the 3D representation of the scene. The vanilla formulation has no generalization capability, i.e. it could not "imagine" new scenes. GameNGen is not like Sora: by design, it could not synthesize new games or interaction mechanics. 3. The hard part of this paper is not the diffusion model, but the dataset. Authors trained RL agents to play the game first, at various different skill levels, and collected 0.9B (frame, action) pairs for training. Most of the video datasets online do NOT come with actions, which means this method wouldn't extrapolate. Data is always the bottleneck for action-driven world models. 4. There're two practical use cases for game world models in my mind: (1) write a prompt to create playable worlds that would otherwise take game studios years to make; (2) use the world model to train better embodied AI. Neither use cases can be realized. Use case (2) doesn't work because there's no advantage to use GameNGen for training agents than directly using DOOM simulator itself. It'd be more interesting if a neural world model simulates scenes that traditional hand-crafted graphics engines cannot. What's an example of a truly useful neural world model? Elon said in a reply that "Tesla can do something similar with real world video". Not surprising: Autopilot team likely has trillions of (camera feed, steering wheel action) pairs. Again, data is the hard part! With such rich real-world data, it's entire possible to learn a general driving sim that covers all kinds of edge cases, and use that to deploy & verify a new FSD build without physical cars. GameNGen is still a really great proof of concept. At least we know by now that 0.9B frames is the upper bound to compress high-res DOOM into a neural network. Paper: Diffusion Models Are Real-Time Game Engines https://lnkd.in/g9aW_uUK

HuggingFace ships a deluxe robotics tutorial experience to the community, from hardware DIY manual to Jupyter notebooks w/ neural nets! Everything that moves will eventually be autonomous. We need a new generation of talents to work on bridging the world of bits with the world of atoms, and open-source as much as possible along the way. Embodied AI is a mission that cannot be delayed. Kudos to Remi Cadene and team at 🤗! https://lnkd.in/gT6SFUUY

I don’t want Gemini Live. I want an LLM to automatically filter, label, and reprioritize my Gmail according to a prompt. Inbox Zero is the one true LLM killer feature that somehow no one delivers. It doesn’t sound sexy for an eye-popping demo, but simple things go a looong way for productivity. Today’s LLMs (even small tier Llama’s) are more than enough to do a near perfect job. Not everything needs AGI.

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

Going to SIGGRAPH next week! An emerging application of graphics is simulation: creating many worlds to train embodied AI, virtual or physical. The driving principle is very simple: if an AI agent can master 10,000 simulated realities with all kinds of variations, then it may very well generalize zero-shot to our real physical world, which is simply the 10,001st reality. Our team’s research agenda operates on this first principle. In Mar. 2024, NVIDIA CEO Jensen unveiled Project GR00T, our moonshot initiative to create a general-purpose AI brain for humanoid robots. GR00T scales up training massively in simulation. It will enable a humanoid robot to understand multimodal instructions, such as language, video, and demonstration, and perform a variety of useful tasks. GR00T is being baked on NVIDIA’s deep technology stack - what we call the “Three computer problem”: 1. DGX Systems for multimodal foundation model training. These systems are the powerhouse for handling vast amounts of visual, textual, and action data. We tokenize them all into sequence of integers, and DGX helps GR00T internalize these sequences on an enormous scale, powered by the Transformer architecture. 2. OVX Systems for simulation. NVIDIA’s simulation platforms like Omniverse and Isaac Sim will be able to generate an infinite supply of high-quality tokens for GR00T to learn from. In simulation, we train our humanoid robots to see, act, and react, in what we call a “sensorimotor loop”. We are also able to ensure a safe virtual training environment without any physical risks. 3. AGX Systems for hardware-in-the-loop validation. AGX provides the edge computing power, or “mobile brain”, to process a continuous stream of sensor input in real-time. Next week at SIGGRAPH, I’ll be presenting Project GR00T and embodied AI on a panel: https://lnkd.in/gnyyMacf Additionally, Jensen will have live streamed keynote discussions where you might see some exciting demos of GR00T’s tech stack at 2:30 p.m. MT. He’ll also be chatting with Meta CEO Mark Zuckerburg afterwards at 4pm MT. Tune in: https://lnkd.in/gx7jW5sn

LLMs are alien beasts. It is deeply troubling that our frontier models can both achieve silver medal in Math Olympiad but also fail to answer "which number is bigger, 9.11 or 9.9"? The latter question broke the internet recently because none of GPT-4o, Claude-3.5 or Llama-3 could get it right 100% of the time. This makes no sense. But we are able to fit a simple explanation here - training data distribution. AlphaProof and AlphaGeometry-2 are trained on formal proofs and domain-specified symbolic engines. In a way, they are highly specialized towards solving Olympiads, even though they build on a general-purpose LLM base. There are tons of GitHub code data mixed in the training set for GPT-4o, likely way more than math data. In software versioning, "v9.11 > v9.9", which may have skewed the distribution heavily. So the mistake makes sense - kinda. By training AI models, we humans are navigating a vast space of possible intelligences beyond our own. We have discovered a very strange region, like an exoplanet that looks like Earth but is littered with uncanny valleys. Don't get me wrong, AlphaProof is still a great achievement from the DeepMind team worth celebrating: https://lnkd.in/gn5RQNJQ

The power of GPT-4o in the palm of our hands. It's a monumental day - for the first time in history, open weights catch up with the latest frontier models. This figure charts the historic run of open models against the closed ones. The former starts humble, but rise with a much higher gradient and a much more diverse ecosystem. Research like multimodal LM and robot foundation models wouldn't have been possible without white box access to a strong base LM. Llama-3.1 release includes more than just weights: - "Open Source AI is the Path Forward" - a manifesto that clearly lays out Zuckerberg's vision. It explains the commercial strategies, ecosystem positioning, and even geopolitical concerns, basically answering FAQs for OSS LLM in general: https://lnkd.in/gJt_nYZp - A 71-page long paper that treats all LLM researchers to a buffet of training details and analysis. They even discuss large-scale cluster failures and remedies - issues that only emerge with 16K H100s! It's literally early Christmas for my team ;) GPT-4o and Claude-3.5 are great, but I would easily rank Llama-3.1 the No. 1 highlight in the 2024 LLM landscape.