Madison Schott’s Post

How to improve your query 99.22%? - add index which someone forgot to add it in the first place :) In my daily work, I usually deal with OLAP workloads and focus on optimizations for analytics. But yesterday on my stream, I ran into an OLTP scenario, and my query was painfully slow. Turns out, I was missing one simple thing: an index. After adding it, the query ran 99.22% faster! It was a great reminder of how different is OLAP vs OLTP and how the right optimizations can make a huge difference, even with small changes. I’m sharing the recording from the stream—check it out and let me know what you think! https://lnkd.in/ejFG9Suj #DataEngineering #SQL #PerformanceTips

Common JOIN Query Indexing Mistakes to Avoid Poor indexing can make JOIN queries painfully slow, leading to table scans and sluggish performance. Here’s how to avoid...... https://lnkd.in/gRNueFmQ

I mugged up "ACID Properties" during my Examinations when I had to write writing 2 - 3 pages answer to score 6 marks. It was most tedious task. Little did I know that SQL Transactions are much more interesting and cool ( like Lightning McQueen ), when actually implemented to facilitate robust operations in real-world scenarios. This article is much beyond the academic notes that answers open-ended questions ( the ones that you'd never ask your professors ) [Sorry!]. More articles could be branched out of this one, explaining technical behind-the-scenes in much more in detail. But more on that later, for sure. Check out the article on Hashnode: https://lnkd.in/dEW5HKVe

Completed "Prompt Compression and Query Optimization" 1. Combine vector search capabilities with traditional database operations to build efficient and cost-effective RAG applications. 2. Learn how to use pre-filtering, post-filtering, and projection techniques for faster query processing and optimized query output. 3. Use prompt compression techniques to reduce the length of prompts that are expensive to process in large-scale applications.

https://lnkd.in/eF5C8Xqh Very interesting lecture on understanding the "Decision Tree", I recall the memory of statistics information when I learned SQL Server 10 years before, SQL Server query optimizer decides whether or not a query should use an index in this way. https://lnkd.in/euiNuxvi

Quick Question: Do the sequence of columns in a query matter in the case of a composite index? Example: Consider a table `test_table` with a composite index on (a, b, c): Given two queries: 1. SELECT * FROM `test_table` WHERE a = 1 AND b = 2 AND c = 3; 2. SELECT * FROM `test_table` WHERE b = 2 AND a = 5 AND c = 3; => Which query will use the composite index we created and why? Answer: Both queries will perform an index scan. The key takeaway is that the query planner does not rely on the order of conditions in the WHERE clause. Instead, it analyzes all provided conditions and determines the most efficient way to execute the query. Therefore, as long as the conditions reference the columns in the composite index, the index can be utilized.

I'm in a situation where I would like to update the date of all the values in the table to have the Check it out: https://lnkd.in/dxjCR3Uu Join the conversation! #sql #sqlserver

Wondering how to read benchmark charts for vector databases? Let's break it down! Benchmark charts typically show vector database performance across two key dimensions: 1. Search Quality (x-axis): measures precision or recall 2. Search Speed (y-axis): often shown as requests per second (RPS) or queries per second (QPS) 💡 The sweet spot? Up and to the right! This indicates better performance in both quality and speed. Vector database algorithms make a conscious trade-off between speed and quality. Finding the right balance is key for your specific use case, and this can be done by configuring various parameters during indexing and search. Benchmarks are an excellent tool to understand how to optimize your vector database performance. They can guide you in making informed decisions about configuration and usage! Want to dive deeper and learn how to benchmark for your own use-case? Join our upcoming webinar with Byron Voorbach: https://lnkd.in/geixgSU2

How To Improve RAG Quality by Storing Knowledge Graph in Vector Database https://lnkd.in/dmhHgmgZ Retrieval-Augmented Generation (RAG) techniques, by integrating external knowledge bases, provide additional contextual information for LLMs, effectively alleviating issues such as hallucination and insufficient domain knowledge of LLM. However, relying solely on general knowledge bases has its limitations, especially when dealing with complex entity relationships and multi-hop questions, where the model often struggles to provide accurate answers.  Introducing Knowledge Graphs (KGs) into the RAG system provides a new solution to this problem. KGs present entities and their relationships in a structured manner, offering more refined contextual information during retrieval. By leveraging the abundant relational data of KGs, RAG can not only pinpoint relevant knowledge more accurately but also better handle complex question-answering scenarios, such as comparing entity relationships or answering multi-hop questions.