Global Vice President, CTO - Data Sc., ML, LLMs, RAG, DSPy, NLP, Deep Learning (Retail, Supply Chain)
Multi Modal Knowledge Graph #Embeddings - Key Trends As the industry moves towards building complex #LLM products, a few trends are becoming quite important. These trends are very often driven by the need to build higher ROI #GenAI industry products. Specifically, the trends include (but not limited to): a. Context strengthening through better information retrieval & information modeling – specifically, for this post, the use of Knowledge Graphs, semantic web & approaches like the Graph RAG have shown promise. b. Use of real-world inputs, such as multi modal data – a recent design example, I’ve been involved with used drone sent images, sensor data & various forms of metered data, to feed into multiple knowledge graph RAGs. c. Of course, structuring approaches such as advanced RAG, DSPy, agentic flows & others, along with building compound AI systems, through chained inference steps or reasoned function calling. As Graph RAG like approaches arise, it is important to understand the unique needs of Multi Modal Knowledge Graphs. Some base “thought” issues that arise, include: 1. What are the models of using multi modal data in knowledge graphs? 2. How do you populate knowledge graphs with multi-modal data & relationships? 3. What is the nature of the data (especially embeddings) that are maintained in these multi-modal knowledge graphs? It is the last question, on which a fair amount of new research is being done and is critical for defining successful hybrid search approaches when Multimodal LLM-based Graph-RAGs are built. The intent of the blog is to point to key shifting trends in generating Knowledge Graph embeddings for multi modal data, using the backdrop of 2 recent seminal papers. For more details on this line of thinking, see https://lnkd.in/gfPmzBgv Note:: for some early efforts at building multi modal KG RAG pipelines, using LlamaIndex and Neo4j – you can check out - https://lnkd.in/g9xnCSTu
Multifaceted polymath, serial entrepreneur, tech geek, problem solver, builder
2moArindam, I like that you are proposing this very interesting MyGO MMKGC direction and the questions you ask are very sound for multimodality in GraphRAG and KGs. I've been experimenting a little bit with code bases on KGs and have been thinking of how to reapply a similar approach to other modalities than text. I believe some completely new designs will have to come in place for image/audio/sensor search and then for connecting the entities