MLCommons

We're looking for an experienced ML platform engineer proficient in Python, Bash scripting, and automation frameworks, to drive and improve the MLPerf benchmark infrastructure. Learn more about the role and how to apply https://lnkd.in/gNHQNSiT

Recent research, funded through an unrestricted grant by MLCommons and published by the Open Data Institute (ODI) and the Pratt School of Engineering at Duke University, explored the impact of standardized data licenses on data sharing in AI development. It provided insights into the motivations behind data sharing within the AI ecosystem. This research emphasizes the importance of understanding why organizations choose to share data and how we can develop better systems to facilitate this collaboration.   Read our latest blog post to discover how standardized licenses can: - Simplify data sharing across industries and academia - Promote fair access to AI technology for everyone - Drive innovation by reducing legal and logistical barriers   Read the full blog post here: https://lnkd.in/g2diaqzp Authors: Lee Tiedrich, Thomas Carey-Wilson, Gefion Thuermer Elena Simperl

We're pleased to announce that our paper on AILuminate v1.0 is now available on arXiv. The benchmark is designed to assess the risk and reliability of LLMs across 12 critical hazard categories, providing a valuable tool for ensuring safer AI deployment. Its development employed an open process that included participants from multiple fields. This technical paper identifies the limitations of our method and of building safety benchmarks generally, including evaluator uncertainty and the constraints of single-turn interactions. The work represents a crucial step toward establishing global standards for AI risk and reliability evaluation while acknowledging the need for continued development in multiturn interactions, multimodal understanding, coverage of additional languages, and emerging hazard categories. Read the Paper: https://lnkd.in/gtMvreVZ Learn more about AILuminate: https://lnkd.in/gNVgWy6c #MLCommons #AISafety #AIRR #AI #ML #AISecurity #AIRisk #AILuminate #Benchmarking Authors: Shaona Ghosh, Heather Frase, Ph.D., CAMS, Adina Williams, Sarah K. Luger, PhD, Paul Röttger, Fazl Barez, Sean McGregor, Ken Fricklas, Mala Kumar, Quentin Feuillade--Montixi, Kurt Bollacker, Felix Friedrich, Ryan Tsang, Bertie Vidgen, Alicia Parrish, Chris Knotz, Alex Presani, Jonathan Bennion, Marisa Ferrara Boston, Mike Kuniavsky, Wiebke Hutiri, James Ezick

MedPerf integrates smart contracts and private data objects to improve accountability and integrity. MedPerf is an open framework developed by the MLCommons Medical AI working group to benchmark medical AI using real-world private datasets. Its core mission is to ensure transparency and privacy, keeping medical data secure without transferring it. Learn more: https://lnkd.in/eCA6V4RQ #MedPerf #MedicalAI #DataPrivacy #smartcontracts #benchmarking Authors: Wenyi Tang (University of Notre Dame), Taeho Jung (University of Notre Dame), Alexandros Karargyris (MLCommons), Micah Sheller (Intel Corporation), Mic Bowman (Intel), Prakash Narayana Moorthy (Intel/MLCommons)

We're #hiring a new Machine Learning Platform Engineer. Apply today or share this post with your network.

Submissions are open for the MLPerf Training v5.0 Benchmark. We are excited to add a new pretraining benchmark, llama3.1 405B, to showcase the latest innovations in AI, for consistency we also renamed GNN to RGAT and SSD to RetinaNet. Come join us and show the world how we are advancing the capabilities of AI! To participate, join the MLPerf Training Working Group here: https://lnkd.in/g8CKEbS6

The MLPerf Power benchmark paper is being presented at IEEE #HPCA2025 today! 🌿💻 Paper Highlights: -MLPerf Power measures AI energy efficiency across diverse applications -1,841 power submissions to MLPerf benchmarks reveal key insights on system scaling and accuracy trade-offs - Measuring power is the first step towards driving "environmentally sustainable AI" Join us in shaping the future of efficient AI - join the working group and add Power measurements to your benchmark submissions! Learn more and read the paper: https://lnkd.in/g9FpyGsk #MLCommons #SustainableAI #EnergyEfficiency #MLPerf #MLPerfPower Arya Tschand, Arun Tejusve Raghunath Rajan, Sachin Idgunji, Anirban Ghosh, Jeremy Holleman, Csaba Kiraly, PhD, Pawan Ambalkar, Ritika Borkar, Ramesh Chukka, Trevor Cockrell, Ollie Curtis, Grigori Fursin, Miroslav Hodak, Hiwot Tadese Kassa, Anton Lokhmotov, Dejan Miskovic, Yuechao Pan, Manu Prasad Manmathan, Liz Raymond, Tom St. John, Arjun Suresh, Rowan Taubitz, Sean Zhan, Scott Wasson, David Kanter, Vijay Janapa Reddi

Join us at #HPCA2025! The MLCommons Power Working Group will present the #MLPerf Power benchmark, measuring AI systems "MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from μWatts to MWatts for Sustainable AI." 📅 Tuesday, March 4, 2025 ⏰ 3:20 PM PST 📍 Session 9B Discover how we're shaping the future of sustainable AI. Don't miss this discussion! #MLCommons #SustainableAI #EnergyEfficiency #MLPerf #MLPerfPower

MLCommons President Peter Mattson shares insights on how AI safety benchmarks can drive responsible AI adoption. Thank you, LatticeFlow AI for the engaging discussion during Davos on risk measurement and governance alignment in AI. 🎬 https://lnkd.in/gxwdKbC5 📰 https://lnkd.in/dmc65u2y

As AI systems become more complex, how do we ensure they are safe? "In the context of AI applications and agents, work is underway to answer that question. I recently found one answer to that in the MLCommons AI Safety Working Group and its tool, AILuminate." Vinton G. Cerf This ACM, Association for Computing Machinery, article by Turing Award winner Vinton G. Cerf highlights critical questions regarding the future of AI systems and the innovative strategies being developed to ensure their safety.   https://lnkd.in/g67uSX4t #AILuminate, #AI, #AIInnovation, #AISafety, #ACM #AIFuture,