AI at Meta

🎥 Today we’re excited to premiere Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ https://go.fb.me/00mlgt Movie Gen Research Paper ➡️ https://go.fb.me/zfa8wf 🛠️ Movie Gen models and capabilities • Movie Gen Video: A 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. • Movie Gen Audio: A 13B parameter transformer model can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. • Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. • Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.

New open source release — Meta Open Materials 2024: a new open source model and dataset to accelerate inorganic materials discovery. • Open Materials 2024 models: Deliver results putting them at the top of the MatBench-Discovery leaderboard. They use the EquiformerV2 architecture and come in three different sizes: 31M, 86M and 153M. Get the models on Hugging Face ➡️ https://lnkd.in/eMzmfE6W • Open Materials 2024 dataset: Contains over 100 million Density Functional Theory calculations focused on structural and compositional diversity — making it one of the largest open datasets of its kind to train these types of models. Get the dataset on Hugging Face ➡️ https://lnkd.in/eMzmfE6W We’re happy to share this work openly with the community excited for how this work could enable further research breakthroughs in AI-accelerated materials discovery.

We want to make it easier for more people to build with Llama — so today we’re releasing new quantized versions of Llama 3.2 1B & 3B that deliver up to 2-4x increases in inference speed and, on average, 56% reduction in model size, and 41% reduction in memory footprint. Details on our new quantized Llama 3.2 on-device models ➡️  https://lnkd.in/g7-Evr8H While quantized models have existed in the community before, these approaches often came at a tradeoff to performance and accuracy. To solve this, we performed Quantization-Aware Training with LoRA adaptors as opposed to only post-processing. As a result, our new models offer a reduced memory footprint, faster on-device inference, accuracy and portability — while maintaining quality and safety for developers to deploy on resource-constrained devices. The new models can be downloaded now from Meta and on Hugging Face — and are ready to deploy on even more mobile CPUs thanks to close work with Arm, MediaTek and Qualcomm. https://lnkd.in/g7-Evr8H

We’re continuing to see great results as we work with our Meta Movie Gen models. We’ve shared more more examples of what they can do over on Threads https://lnkd.in/g3VEicPK

We are thrilled to announce the release of our latest research paper Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following. https://lnkd.in/gjZWXrby This benchmark addresses a critical gap in evaluating large language models (LLMs) across both multi-turn and multilingual instruction following tasks, providing deeper insights into how these models perform in real-world, complex conversational settings. Key Highlights: 🔹 Over 4,500 three-turn multilingual conversations, spanning 8 languages: English, French, Russian, Hindi, Italian, Portuguese, Spanish and Chinese. 🔹 Comprehensive evaluation of 14 state-of-the-art LLMs including OpenAI o1, GPT4o, Llama 3.1 405B and Claude-3.5 etc. 🔹 Multi-turn challenges: Models exhibit significant performance degradation as the number of conversation turns increases. For example, o1-preview's accuracy dropped from 87.7% on the first turn to 70.7% on the third turn, highlighting the difficulty of maintaining consistent instruction-following across multiple interactions. 🔹 Multilingual complexity: Models generally perform well in Latin-based languages like English and French, but struggle with non-Latin scripts, revealing gaps in the models' multilingual proficiency. 🔹 Release of our open-source dataset and evaluation framework on HuggingFace. Join us in pushing the boundaries of LLM instruction-following capabilities. Let's innovate together! #LLM #AI #MultilingualAI #Research #NaturalLanguageProcessing #AIbenchmark #InstructionFollowing Feel free to connect with us for further discussions or collaborations! Di Jin Chaoqi W. Chloe Bi Karishma Mandyam Hejia Zhang Chen Zhu Ning LI Terry Xu Hongjiang Lv Shruti Bhosale Chenguang Zhu Karthik Abinav Sankararaman Eryk Helenowski Melanie Kambadur Aditya Tayade Hao Ma Han Fang Sinong Wang

Last week we released Meta Lingua, a new lightweight and self-contained codebase design to train language models at scale. Lingua is designed for research and uses easy-to-modify PyTorch components in order to try new architectures, losses, data and more. Meta Lingua repo on GitHub ➡️ https://go.fb.me/mkmtcz More open source work released by Meta FAIR last week ➡️ https://go.fb.me/edk5qw By sharing the code, we hope Lingua will enable researchers to focus on the important work they’re doing while letting the platform take care of efficient model training and reproducible research.

Meta Spirit LM is our first open source multimodal language model that freely mixes text and speech. Details, code and model weights ➡️ https://go.fb.me/2jooyy Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech in inputs and outputs. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations to better understand and generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

Open science is how we continue to push technology forward and today at Meta FAIR we’re sharing eight new AI research artifacts including new models, datasets and more to inspire innovation in the community. More in the video from VP of AI research, Joelle Pineau. This work is another important step towards our goal of achieving Advanced Machine Intelligence (AMI). What we’re releasing today: • Meta Spirit LM: An open source language model for seamless speech and text integration. • Meta Segment Anything Model 2.1: An updated checkpoint with improved results on visually similar objects, small objects and occlusion handling. Plus a new developer suite to make it easier for developers to build with SAM 2. • LayerSkip: Inference code and fine-tuned checkpoints demonstrating a new method for enhancing LLM performance. • SALSA: New code to enable researchers to benchmark AI-based attacks in support of validating security for post-quantum cryptography. • Meta Lingua: A lightweight and self-contained codebase designed to train language models at scale. • Meta Open Materials: New open source models and the largest dataset of its kind to accelerate AI-driven discovery of new inorganic materials. • MEXMA: A new research paper and code for our novel pre-trained cross-lingual sentence encoder with coverage across 80 languages. • Self-Taught Evaluator: a new method for generating synthetic preference data to train reward models without relying on human annotations. Access to state-of-the-art AI creates opportunities for everyone. We’re excited to share this work and look forward to seeing the community innovation that results from it. Details and access to the models, datasets and code released by FAIR today  ➡️ https://go.fb.me/7qchv5

As detailed in the Meta Movie Gen technical report, today we’re open sourcing Movie Gen Bench: two new media generation benchmarks that we hope will help to enable the AI research community to progress work on more capable audio and video generation models. Movie Gen Bench repo ➡️ https://go.fb.me/cppbmy • Movie Gen Video Bench is the largest and most comprehensive benchmark ever released for evaluating text-to-video generation. It includes a collection of 1,000+ prompts that cover concepts ranging from detailed human activity to animals, physics, unusual subjects and more — with broad coverage across different motion levels. • Movie Gen Audio Bench is a first-of-its-kind benchmark aimed at evaluating video-to-audio and (text+video)-to-audio generation. It includes 527 generated videos and associated sound effects and music prompts covering a diverse set of ambient environments and sound effects. To enable fair and easy comparison to our models for future works, these new benchmarks include non cherry-picked generated videos and audio from Movie Gen. In releasing these new benchmarks we hope to promote fair & extensive evaluations in media generation research to enable greater progress in this field.

Following the announcement of Meta Movie Gen, today we’re announcing a new creative industry feedback program — alongside initial early pilot results from our work with Blumhouse and select creators. More details ➡️ https://go.fb.me/dc99hl We believe it’s important to have an open and early dialogue with the creative community. As part of our initial pilot, Blumhouse selected a group of innovative filmmakers to test out the technology in collaboration with our AI researchers — and use generated video clips as part of larger pieces. You can watch the first of these short films, by filmmaker Aneesh Chaganty, today. We’re excited about the potential of this technology and the creative opportunities enabled by these models. We look forward to expanding this program in 2025.