Qarbine’s Post

Day 10-12 of my #75DaysOfGenerativeAI series. The past few days have been all about building your LLM Twin's Data Pipeline. This one's a code-heavy walkthrough of creating a robust data pipeline for your AI twin project. Key topics covered: - Reading data from MongoDB - NLP-based data cleaning - Generating embeddings - Loading data into QDrant (vector DB) Check it out here : https://lnkd.in/g5Me2b6N

Excited to share my experience at Yesterday''s "Unstructured Data Meetup Berlin" hosted at the Google Berlin office! 📊🇩🇪 Some highlights: • Diving deeper into Milvus, a powerful vector database. It's my second encounter with this technology, and I'm finding a renewed appreciation for its capabilities. https://lnkd.in/eigrK4vn • Also it was good to see how Neo4j is evolving and bringing Graph data closer and useful for generative AI. Effective GraphRAG is definitely one of the best techniques to feed relevant and correct data to Gen AI workflow. • An eye-opening talk by Jakob Pörschmann about alternative solutions to graph databases. It's fascinating how a different perspective can completely reshape our approach to storing graph data. graph2nosql, a python interface to store your knowledge graph in your favourite NoSQL DB. No more GraphDBs and cyper exploding your stack and requiring new skills. A big thank you to Stephen Batifol from Zilliz for organizing this insightful event. These meetups are invaluable for staying current with emerging technologies and connecting with fellow data and AI enthusiasts. Looking forward to the next one! Who else is interested in unstructured data solutions? Let's connect and share insights. Check out Unstructured Data Meetup Berlin https://lnkd.in/eANpfmFa on Meetup for future events. #UnstructuredData #VectorDatabases #Milvus #GraphDatabases #TechMeetup #BerlinTech #DataScience #NetworkingEvent

As AI becomes a top priority for organizations, they must prioritize the right strategy, skills, and data quality. Learn how to overcome integration complexity, data engineering challenges, and security concerns in building AI applications from this AI Cookbook. Discover six reference architectures with MongoDB Atlas and Vector Search to kickstart your journey towards AI-powered solutions. 👇 https://lnkd.in/gbWAD58W #mongodb #nosql #sql #genai #ai #database #developer #llm #RAG #architect #vector #search #mongodbatlas

How many use case cases can you cover with PostgreSQL extensions? 🧐 𝐌𝐨𝐫𝐞 𝐭𝐡𝐚𝐧 𝐲𝐨𝐮'𝐝 𝐞𝐱𝐩𝐞𝐜𝐭. PostgreSQL is built from the ground up to be extensible, so you can choose from over 1000 extensions to meet more use cases with the same solution – saving you time and resources. Join our webinar on July 17th, presented by Mohamed Nsiri, Data Solutions Product Manager at Canonical, to examine the use cases you can cover with just PostgreSQL. We will dive into: - Analytics - Online transaction processing - Geospatial data - Graph data - Search and similarity - AI and ML - Time series and more Register now: https://lnkd.in/deq8AMSA #PostgreSQL #DataSolutions

How will AI benefit developers, teams, and the industry at large? Watch this in-depth discussion produced by Ars Technica and MongoDB with a panel of experts as they discuss potential evolutions in developer experience, building applications, and ways that data is changing the game. https://lnkd.in/geVcc2ka #LoveYourDevelopers #mongodb #nosql #sql #genai #ai #database #developer #llm #RAG #architect #vector #search #mongodbatlas

🌟 Data Showdown: Day 38 - Spark MLlib vs. Spark ML Welcome to Day 38 of our Data Showdown series! Today, let's unravel the differences between two powerful components of Apache Spark's machine learning ecosystem: Spark MLlib and Spark ML. Spark MLlib: - Traditional Machine Learning: Spark MLlib is Apache Spark's original machine learning library, offering a wide range of algorithms and utilities for traditional machine learning tasks. - RDD-based: MLlib operates on Resilient Distributed Datasets (RDDs), making it suitable for large-scale distributed processing of data but requiring developers to manage data explicitly. - Scalability: MLlib provides scalable implementations of popular machine learning algorithms, enabling processing of massive datasets across distributed environments. Spark ML: - DataFrame-based: Spark ML is the newer machine learning library built on top of Spark DataFrames, providing a higher-level API for building machine learning pipelines. -Unified API: ML simplifies the development of machine learning workflows with a unified API, allowing seamless integration with Spark's DataFrame operations and optimizations. - Feature Engineering: ML offers built-in support for feature engineering, transformation, and pipeline management, streamlining the process of building and deploying machine learning models. Key Differences: - API Paradigm: MLlib follows the RDD-based API paradigm, while ML adopts the DataFrame-based API approach. - Ease of Use: ML's higher-level API and integration with DataFrames offer greater ease of use and productivity compared to MLlib's lower-level RDD-based operations. - Feature Support: ML includes features like pipelines, transformers, and estimators out-of-the-box, enhancing the efficiency of machine learning development workflows. Conclusion: Both Spark MLlib and Spark ML are valuable tools in Apache Spark's machine learning arsenal, each catering to different needs and preferences. While MLlib remains a robust choice for traditional machine learning tasks and scalability, ML shines with its simplicity, DataFrame integration, and feature-rich API. Stay tuned for more enlightening comparisons in our Data Showdown series, empowering you with insights to navigate the data landscape with confidence! #DataShowdown #ApacheSpark #SparkMLlib #SparkML #MachineLearning #DataScience101

Announcing Snowflake Copilot - now in public preview! Leveraging Snowflake Cortex, Copilot incorporates genAI into everyday coding activities through natural language. Users can engage with their data using simple text, compose SQL queries, and fine-tune their queries to extract specific insights. This is a huge advantage in data exploration. Check out the article below for more info:

In my college Linear Algebra class we read a paper called the '$25 Billion Eigenvector' that detailed the Google PageRank implementation (obviously Google's valuation has skyrocketed since the publishing of that paper 😂). In a similar fashion, I'm going to refer to Confluent founder Jay Kreps blog post 'The Log' as the $10 billion dollar blog post (I'm assuming that valuation will skyrocket as well)! So what was this blog about and how did it act as a predictor of one of Silicon Valley's most valuable data companies of the 2010s? Jay wrote the blog post while he was still an engineer at Linkedin working on Apache Kafka. He summarized his perspective on write-ahead logs while providing a simple introduction with quite a grandiose forward looking view. Jay also mentioned GoldenGate and other successful commercialization of log streams at the time which were mainly used for database high availability and redundancy of several critical systems that our economy relies on today. However, we are about to see 'The Log' explode in its adoption to heights we've never seen before. What are the driving forces? ✅ AI and Machine Learning – The Log can represent both business and technical state that can be used in downstream Generative AI applications with RAG architectures. Event-driven streams are the most reliable, cost efficient methods to adopt data-driven AI applications. (we will talk about this more later) ✅ Standardization of change stream adoption by data warehouses –BigQuery, Fabric, Snowflake, Databricks and Redshift have all added recent innovations to interop seamlessly with change streams. That means you no longer have to drop and reconstruct views with CDC in the warehouse, you can actually have a pseudo change log in your warehouse and materialize the data in near real-time. This provides a scalable, cost-optimized, low latency method to develop both internal analytics and customer-facing data applications in the data warehouses that were traditionally batch-only and prohibitively expensive/inefficient for these use cases/ ✅ Open Storage – Apache Iceberg, Delta Lake, and Parquet formats give companies an open storage format that multiple compute engines can work with as tables. Streaming append-only into parquet files is a low level adoption path for these formats, but also works as a foundation for table state maintenance (compaction, type mapping, CDC merges). #dataengineering #changedatacapture #apachekafka #apacheflink

Speaking of #LLM, #VectorDatabases are currently a popular trend. Many companies are securing funds to create new vector databases or enhance their existing SQL or NoSQL databases with vector search features. Some top vector databases in the market include #Pipecone, #Chroma, #Weaviate, and more. 🔅 #LLM Vector Databases Overview: ▶ Vector databases aid in extracting crucial parts of a document for Language Model (LLM) applications. ▶ They serve as key elements in LLM systems and are tailored databases crafted to store vectorized data, facilitating swift retrieval of pertinent documents by identifying embeddings similar to a given query. 🔅 Functionality of Vector Databases: ▶ Vector databases enable the storage of documents (sourced from data) and the submission of queries (input by users). ▶ Their primary role is to efficiently return the most relevant document segments to the Language Model (LLM) based on the provided query. 🔅 Why Vector Databases? In essence, vector databases streamline the process of storing documents, receiving user queries, and promptly delivering the most relevant document portions to the #LargeLanguageModel (LLM). Image Credit: DataCamp ---------- 📌 I've been researching about LLM, you might be interested in this collection ▶ https://lnkd.in/gNTq2jgg I also share additional insights on the Machine Learning journey through articles on Medium. Feel free to check them out! https://lnkd.in/grPydFH5 Get UNLIMITED access to every story on Medium with just $1/week ▶ https://lnkd.in/gV_7WhHT Sharing is caring! Do not forget to reshare this to your network. It will help growing the community! #VectorDatabases #GenerativeAI #SQL #NoSQL #DataStorage