Bhavya Shah’s Post

🚨 [AI] Image generation with diffusion models using Keras and TensorFlow 👉 Using Diffusion to generate imagesYou must have heard of Dall-E 2. Published by OpenAI, which is a model that generates realistic looking images from a given text prompt. You can check out a smaller... https://lnkd.in/dgW6tiKv

Since Yann LeCun first introduced Joint Embedding Predictive Architecture (JEPA), the architecture has come a long way. V-JEPA, the latest iteration, learns highly detailed world models from raw videos through self-supervised learning. While many companies are focused on generative AI, V-JEPA shows the potential of non-generative models. In a talk in 2020, LeCun said that if intelligence is a cake, the bulk is self-supervised learning, the icing is supervised learning, and the cherry on top is reinforcement learning (RL). We have reached the bulk of the AI cake. But in many ways, we might be still scratching the surface of what is possible. https://lnkd.in/eEhiT5qp

Musings… According to rumors considered by those who know more than me about it to be probably not far off the mark, the OpenAI GPT-4 model is (probably) a "mixture of experts" (MoE) model with an estimated total of about 1.7 trillion parameters. Each generation of these models (GPT-2, GPT-3, GPT-3.5) has proven to be more capable than the last. Researchers in recent years appear to have been surprised that only two tuning knobs, parameter count and training tokens count, seem to drive the increase in capability. (The other at-least-two tuning knobs they thought should matter… don't matter to us, today.) The not-uncommon expectation that "the only thing that matters in the next several years of AI technology is the data" has a certain grounding. It sure looks like the technology of the model doesn't need to improve, in order for rapid progress in the field to continue. In fact, in theory it's not even necessary for training data to expand, or even get cleaned up. It's likely the models could get quite a bit better for a generation or two merely by doubling the number of parameters a couple times. This will happen even if Moore's Law were to hit the wall, tomorrow, because more GPUs can be thrown at the problem for probably a few cycles. The transformer-based LLMs though, by themselves, probably will struggle to overcome the (poorly named) "hallucination" problem — all of their output is produced in the same way, it's all "hallucinations". Progress along the axis of reduced noise in the output will likely require improvements to the models, or combinations with new types of models in an MoE style network. The successes of the transformer architecture have ignited new thinking and provide a baseline against which the performance of new ideas can be tested and measured. New ideas are being generated at a rapid clip, probably more than we realize are soon discarded. Every few weeks something shiny turns up. What do you think the next year in AI be like? https://lnkd.in/gR83wTpc

Transformer Reel and Real... In the realm of deep learning, the Transformer model, inspired by the movie franchise, doesn't quite metamorphose like its cinematic counterpart. However, it does utilize attention mechanisms to understand data and relationships better. Yet, these mechanisms can be slow due to their quadratic complexity and are limited by input data length. To address this challenge, researchers are devising ways to accelerate attention mechanisms without sacrificing effectiveness. Techniques like making attention matrices sparser or using simpler approximations are being explored in models such as Reformer, Routing Transformer, and Linformer. However, striking the right balance between speed and accuracy remains elusive. Enter the Data Dependent Convolution Network, a novel innovation akin to the shape-shifting abilities of movie Transformers. This network adjusts itself based on input data, allowing the model to efficiently handle long sequences without complexity overload. It can process up to 131,000 elements while maintaining speed. TAt the heart of Orchid lies its innovative data-dependent convolution layer, which dynamically adjusts its kernel based on input data through a specialized conditioning neural network. We've designed two straightforward conditioning networks that ensure shift equivariance in the adaptive convolution operation. This dynamic convolution kernel, combined with gating operations, provides Orchid with remarkable expressivity while maintaining efficiency and quasi-linear scalability, especially for long sequences. We've conducted thorough evaluations of Orchid across various domains, including language modeling and image classification, to demonstrate its performance and versatility. Our experiments show that Orchid not only surpasses traditional attention-based architectures like BERT and Vision Transformers with smaller model sizes but also extends the feasible sequence length beyond the limitations of dense attention layers. This accomplishment marks a significant advancement toward developing more efficient and scalable deep learning models for sequence modeling. In practical terms, Orchid's breakthroughs empower researchers and developers to construct more efficient and potent models for analyzing vast datasets. Whether deciphering lengthy text passages or decoding intricate genetic sequences, Orchid presents a promising solution for accelerating deep learning tasks while upholding accuracy. https://lnkd.in/dQHBARrS #ai #aiinnovation #llm #generativeai #deeplearning

Must-watch, informative overview and detailed walkthrough :) from Andrej Karpathy on LLMs. https://lnkd.in/gJF3pj4w These are new thought-provoking ideas I gained by watching this juicy presentation, some of these are new emphasis or clear articulation of what was simmering in my own mind, articulated so well with evidence in a brisque way. 1. Simplicity of LLM Training and Deployment code: The notion that a powerful tool like an LLM can essentially be boiled down to just two files (parameters and run file) is quite surprising. This simplicity belies the complex operations and vast knowledge these models possess. While the software architecture of the neural network (essentially their simulated-hardware substratum) is well understood and the process of looking for optimal parameters is more are less tamed, the software architecture of the generative model (how concepts are represented in computation) is not. 2. Resource Intensity for Training: The staggering amount of resources required for training an LLM, such as the Llama 2 (70b) model, is remarkable. The need for thousands of GPUs and millions of dollars in funding underscores the significant infrastructure and investment needed for advanced AI research. My respect for Meta AI releasing the Llama2 suddenly went up manifold, even though it was high respect already. 3. Compression of Internet Data: The idea that LLMs are essentially effectively compressing a large chunk of the internet into a usable format for generating responses is intriguing. This perspective provides a unique understanding of how these models leverage vast data sets. I have thought about this for decades and have further distinctions. While prediction is necessary for compression both in the model and coding stages, (well known from Shannon's days), I think that these models go further into kolmogorov style compression which I think is comprehension. What is the difference? Stage 1: replacing repeating sequences with shorter symbols, stage 2 utilizing the distribution probabilities for each context, Stage 3 classification of tokens and substitution (context free and sensitive grammars) and Stage 4 is semantics. I think LLMs do all these. 4. Multimodality and Tool Integration: The evolving capabilities of LLMs to not only process text but also images, audio, and even integrate with tools like browsers and calculators, expand the traditional boundaries of what we expect from AI language models. 5. LLMs as Operating Systems: The analogy of LLMs functioning like an operating system, coordinating various computational resources and tools, is an unexpected but apt comparison. It highlights the versatility and central role these models could play in future technology ecosystems. (continued below)

After working with Machine Learning long enough, you come to realize that the tools you use are less important than the underlying principles they implement. PyTorch or TensorFlow : Whether you choose one or the other, both frameworks revolve around the same core ideas—matrix multiplication and activation functions. These are the building blocks of neural networks, allowing machines to learn from data by adjusting weights and biases. LangChain or LlamaIndex : These tools, despite their different interfaces, both hinge on prompt engineering and retrieval-augmented generation. The essence lies in how well you can frame a problem (through prompts) and retrieve relevant information to enhance the model’s output. scikit-learn or XGBoost : While they offer different algorithms and optimizations, both are grounded in statistical modeling and mathematical optimization. The core challenge is the same: finding patterns in data and optimizing models to make accurate predictions. Pinecone or ChromaDB : At their heart, these tools are designed for the same purpose—storing and retrieving unstructured data as high-dimensional vectors. This vectorization allows for efficient similarity searches, which are crucial for tasks like recommendation systems and semantic search. Claude, GPT, or Llama : Regardless of which model you use, they all share the fundamental mechanism of next-token probability prediction. These models predict the most likely word or phrase to follow, enabling everything from simple text generation to complex conversational AI. Tools will come and go, evolving as technology advances. But the fundamental concepts—matrix operations, prompt design, statistical modeling, vector storage, and probabilistic prediction—remain the bedrock of machine learning. Master these, and you’ll have the versatility to navigate any new tool or framework that emerges, confident in your ability to apply the underlying principles effectively. #DataScience #MachineLearning #GENAI #LLM

𝐖𝐡𝐲 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐛𝐨𝐨𝐦𝐞𝐝 𝐚𝐧𝐝 𝐰𝐡𝐚𝐭'𝐬 𝐧𝐞𝐱𝐭? Well to put it simply, the recent boom has been accredited to Google's 2017 research paper "Attention Is All You Need" which introduced the transformer architecture. Every major generative model, including GPT, Stable Diffusion, Copilot etc. have been built using transformers. Transformers were initially created to solve the language translation problem (because language have varied grammatical structures and contextual vocabulary), they were found out to be extremely general-purpose and can be applied to a variety of ML problems. They proved to be better than RNNs (Recurrent Neural Networks) which only processed one word at a time, and not in parallel. You see transformers were able to capture long-distance relationships between words better, it basically: 1. Embeds each word of the lexicon to a vector counterpart 2. Based on the words in the vicinity, each word calculates its self-attention score against all other words in the sequence 3. Allows transformers to be run in parallel computing self-attention simultaneously So simply, better scalability, more parameters, and enhanced computational efficiency—all optimized for parallelization on GPU chips, the perfect match for transformer architectures. 𝐒𝐨 𝐰𝐡𝐚𝐭'𝐬 𝐭𝐡𝐞 𝐢𝐬𝐬𝐮𝐞? Well the issue is just that, we have parallelization and a process hungry architecture which is super expensive to train. It makes it tempting to throw all the resources at your training endeavors to beat your competitors and build bigger models - using celestial amounts of resources. The key bottleneck of the transformer architecture: 1. It scales quadratically (n²) with the length of the input sequence as each word is compared against all other words 2. As the context window doubles in size, the computational requirement quadruples (Not to mention the additional complexity from calculating the Query, Key and Value matrices) 𝐖𝐡𝐚𝐭'𝐬 𝐧𝐞𝐱𝐭? A 𝐬𝐮𝐛-𝐪𝐮𝐚𝐝𝐫𝐚𝐭𝐢𝐜 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧 that boasts a prowess equivalent to transformers. But is it possible? Well some methods successfully reduce complexity a bit (Flash Attention, Sparse Attention, Locality-Sensitive Hashing, Retentive Networks etc) by sacrificing token to token interactions compromising performance. Each method comes with its own trade-offs and constraints over application space. Architectural adversaries such as State-Space Models (Hyena Architecture) that get rid of attention entirely in sequence modelling tasks boast a 100x faster computation at a 64k context window (𝘗𝘢𝘱𝘦𝘳: 𝘩𝘵𝘵𝘱𝘴://𝘢𝘳𝘹𝘪𝘷.𝘰𝘳𝘨/𝘱𝘥𝘧/2302.10866), the paper however fails to demonstrate performance comparisons with optimized self-attention mechanisms at their benchmarked windows. As of now, no general purpose solution exists that can dethrone the hegemony of attention models and that's why attention is receiving all the attention :) #DeepLearning #AITopics #Attention #GenerativeAI

🚀 Are you building the next generation of AI applications? According to the Berkeley Artificial Intelligence Research (BAIR), state-of-the-art AI results are increasingly obtained by compound systems with multiple components, not just monolithic models. 👀 As we're heading towards a future of compound AI, what should AI teams do and why should they care? Read our latest blog post to explore this trend, the challenges it presents, and how BentoML can be your ally in building and scaling robust compound AI systems. 🔗 Don't miss out on transforming your AI strategies 👉 https://lnkd.in/gVrU7YmQ #AI #CompoundAI #MachineLearning #DataScience #BentoML

The “it” in AI models is the dataset. Posted on June 10, 2023 by jbetker I’ve been at OpenAI for almost a year now. In that time, I’ve trained a lot of generative models. More than anyone really has any right to train. As I’ve spent these hours observing the effects of tweaking various model configurations and hyperparameters, one thing that has struck me is the similarities in between all the training runs. It’s becoming awfully clear to me that these models are truly approximating their datasets to an incredible degree. What that means is not only that they learn what it means to be a dog or a cat, but the interstitial frequencies between distributions that don’t matter, like what photos humans are likely to take or words humans commonly write down. What this manifests as is – trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point. Sufficiently large diffusion conv-unets produce the same images as ViT generators. AR sampling produces the same images as diffusion. This is a surprising observation! It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset. Then, when you refer to “Lambda”, “ChatGPT”, “Bard”, or “Claude” then, it’s not the model weights that you are referring to. It’s the dataset.

Explore the nuances between AI and ML and understand why it's crucial. Replay this riveting webinar featuring experts from CDAO, APL, and CNAS as they delve into the distinction between artificial intelligence (AI) and machine learning (ML). Rodriguez, principal AI technical leader at APL, sheds light on machine learning as data-driven approaches that empower computers to learn from data without explicit programming.