Flex Logix Technologies, Inc.’s Post

AI solutions with fixed hardware are incomplete. They require an adaptable data path front-end to support a broad market. Additionally, they need an adaptable memory interface to support novel models that can improve performance and reduce latency.

The landscape of artificial intelligence is undergoing a seismic shift. Large language models (LLMs), once the exclusive domain of tech giants and research institutions, are increasingly accessible to the average consumer. This democratization is made possible by a convergence of factors: the advent of powerful yet affordable graphics processing units (GPUs) like the Nvidia 3090 with its ample 24GB of VRAM, the development of LLMs optimized for consumer hardware, a revolutionary technique known as 4-bit quantization, and innovative tools like Ollama. Quantization is a compression technique that significantly reduces the size of LLMs without much performance impact. This results in models that are roughly 45% smaller, making them far more manageable for consumer-grade hardware. For example, Yi-large and Gemma-2-27B, two powerful LLMs, are reduced to approximately 19GB and 16GB respectively after quantization. Ollama, a cutting-edge tool, takes this a step further by enabling the parallel inference of multiple quantized models on a single GPU. This means that users can run both Yi-large and Gemma-2-27B concurrently on a 3090, leaving ample VRAM for context tokens – the pieces of text that provide context to the models and influence their responses. The Nvidia 3090's 24GB of VRAM proves to be a perfect match for this setup. It can comfortably accommodate the quantized models and their context tokens, while still leaving approximately 4GB of VRAM available for other demanding tasks. This is a testament to the efficiency of 4-bit quantization and the ingenuity of Ollama. The availability of powerful LLMs on consumer hardware has profound implications. It opens the door to a wide range of applications, from personalized chatbots and writing assistants to advanced code generation and data analysis tools. Moreover, it empowers individuals and small teams to experiment with and develop AI-powered solutions, fostering a vibrant community of innovators. The journey towards democratizing AI has just begun. With continued advancements in hardware, software, compression techniques like 4-bit quantization, and tools like Ollama, we can anticipate even more powerful and versatile LLMs becoming available to consumers. This will undoubtedly fuel a new wave of innovation, with the potential to reshape our society in profound ways. The democratization of LLMs is not merely a technological trend; it is a cultural and social phenomenon that promises to empower individuals and communities, democratize knowledge, and unleash the full potential of human creativity.

Qualcomm has launched a new AI Hub, a comprehensive library of pre-optimized AI models ready for use on Snapdragon and Qualcomm platforms. These models are designed to deliver high performance with minimal power consumption, making them ideal for mobile and edge devices. The AI Hub library includes over 75 popular AI and generative AI models, including Whisper, ControlNet, Stable Diffusion, and Baichuan 7B, supporting a wide range of applications such as natural language processing, computer vision, and anomaly detection. All models are bundled in various runtimes and optimized to leverage the Qualcomm AI Engine's hardware acceleration across all cores (NPU, CPU, and GPU), delivering four times faster inferencing times. #qualcomm #ai #aihub #aimodels #snapdragon #generativeai #npu #gpu #cpu #technology #technologynews

A few days ago, engineers at BitEnergy AI, an AI inference technology company, published a report to describe a method of reducing energy needs/costs of AI applications by a staggering 95%. As highlighted in the S&R Associates note on 'Climate Change and ESG Considerations in India’s AI-driven Future' co-authored with Rajat Sethi, the global and dramatic rise in AI use has led to a significant increase in energy needs and costs. LLMs such as ChatGPT require vast amounts of computing power, which means that a lot of electricity is necessary to run these models. For example, the daily requirements of ChatGPT are enough to run approximately 20,000 American households. In this context, the findings of this new report (despite its drawbacks, e.g., new hardware requirements) appear to be important. The report suggests that instead of complex floating-point tensor multiplication (FPM), a new technique using integer addition can be used which works by approximating FPMs without compromising precision. Current AI models, which rely on FPM performed on GPU chips, are extremely compute-intensive. If the findings from this report prove viable, there may be major consequences in the entrenched GPU market, including for existing players like Nvidia and AMD (in this regard, see: https://lnkd.in/gqrfyuE4 and https://lnkd.in/gMA_xrPp). #gpu #ai #esg https://lnkd.in/gtH-Tyab

Nemotron-4 15B - NVIDIA's new AI powerhouse LLM NVIDIA has made another significant leap in the AI domain with its latest language model, Nemotron-4 15B. Trained on an impressive 8 trillion text tokens, this model boasts 15 billion parameters, showcasing remarkable versatility across English, coding, and multiple languages. Nemotron's performance surpasses that of its counterparts in various evaluation metrics, establishing new benchmarks in AI capabilities. Nemotron's edge comes from its comprehensive training regimen, incorporating a mix of English, multilingual texts, and source-code data, aimed at refining the model's performance across a diverse range of tasks. With 32 layers, 6144 units of data processing capability, and 48 attention heads, Nemotron-4 15B represents a sophisticated blend of technology designed to enhance understanding and context in text generation and analysis. This model stands out in its ability to outperform similarly sized transformer models in four out of seven key evaluation areas, directly competing with the leading models in the remaining domains. Notably, Nemotron matches the performance of Qwen-14B in the MMLU benchmarks and code but shows superior results against models like Gemma 7B, Mistral 7B, and even LLaMA-2 34B, particularly in reasoning tasks. It, however, yields the maths crown to Qwen, illustrating the competitive landscape of AI research. NVIDIA's Nemotron-4 15B sets a new standard in the AI landscape, offering unprecedented capabilities in language understanding and generation. As AI continues to evolve, models like Nemotron-4 15B are pivotal in driving forward our understanding and application of artificial intelligence in real-world scenarios. Unfortunately, Nemotron-4 15B is not open source. #NVIDIA #genAI #LLM #Nemotron4 #ai https://lnkd.in/gqbhsauD

How do you deal with uncertainty dealing with AI/ML? In this excellent piece by Karen Heyman from Semiconductor Engineering, she captures interesting insights from several experts who don't always agree! Ashish Darbari CEO of Axiomise, disagrees with the premise that no one knows how the models are implemented. He said “It is more the case that the model’s details are not visible. There is a difference between the two artifacts. My sense is that even if we don’t know what exact model has been implemented, we could still test that model against a design model. In the case of semiconductors, the problem is less severe than the software-as-domain specialists, such as architects, designers and verification engineers, all of whom tend to know what to expect from the design model. This means extensive testing of the design against a black box AI model would reveal acceptable and unacceptable patterns across the I/O, which will make it easier to establish trust in the AI-generated black box model. Coverage models can be developed independently to validate the quality of the black box model. Again, the guide would be a coverage specification obtained from domain experts — designers and architects. Another way to address the black box models is to bring in symbolic AI to build alternative models, which are better in that one can explain these, gain deeper insights, and use these explainable AI models to compare and equivalence check against the black box models to authenticate their validity and completeness. This will allow the developers of black box models to have perhaps more optimized implementations and not reveal the secret sauce, but still get validated against what is open to investigation, such as an explainable AI model.” Other experts who provided insights include Neil Hand, Frank Schirrmeister Steve Roddy Patrick Donnelly. Siemens EDA (Siemens Digital Industries Software) Quadric Arteris Expedera Inc. https://lnkd.in/gn9FPH7g #ai #machinelearning #eda #semiconductors #verification #validation #formalverification #uvm #icdesign

Generative AI and Large Language Models (LLMs) are changing the game, but the industry is shifting towards more specialized compute architectures to run these models efficiently. GPUs are a great example of such specialized architectures. However, the lack of competition has allowed a single company to dominate the market. But, with the increasing demand for specialized compute, new companies have the opportunity to enter the market. Traditional Silicon companies will need to relearn and adapt to these new designs. Learn more about the future of the semiconductor industry with Generative AI: The next S-curve for the semiconductor industry. #AI #SemiconductorIndustry #GenerativeAI Source: McKinsey & Company.

Why is the semiconductor industry buzzing about AI and large language models? AI has the potential to unify disparate data across the IC ecosystem, enhancing collaboration from design to final testing. Key challenges remain—like data sharing and IP protection—but the potential benefits are game-changing. Learn how LLMs could reshape chip design and manufacturing. 📖 Read the article here in #SemiEngineering: https://bit.ly/4e1bOes #AI #ICDesign #LLM #AdvancedPackaging #ML #Tignis #SemiconductorManufacturing

About the future of AI...Many AI researchers believe that the path to ever-more powerful AI requires ever-greater scale: much more data, much more processing power. It's why foundation model developers are investing billions of dollars in building data centers, why new chip architectures are continually being explored, and why Sam Altman appears to be trying to raise trillions of dollars to reshape the semiconductor industry. But I am wondering whether novel architectures and techniques will be able to boost performance without massive new infrastructure. A paper by researchers at Nvidia, which has just released a new foundation-class model, makes this claim: "Our findings indicate that dataset quality and task diversity are more important than scale, even during the pretraining phase, across all architectures." This is the type of thing I am keen to see more of. Story on the new family of models, with link to the paper here: https://lnkd.in/evMSvXU6 cc Wei Ping

Are you curious about the future of AI-powered computing on your laptop? Explore how the LLaVA-Gemma-2B multimodal model leverages the NPU on an AI PC in this blog by Benjamin Consolvo. Read more: https://intel.ly/3zc7SJp #AI #AIPC Hugging Face