Data Solutions Consulting Inc’s Post

Do you find the Snowflake Query Profile confusing? In this article, I discuss analyzing a Query Profile and identifying and resolving potential performance issues. https://lnkd.in/exAnCvNr #snowflake #datasuperhero #snowflake_influencer

We tackled a production issue. We managed to identify the root cause using Snowflake's SEARCH function, which helps find data in rows without knowing the column names. We discovered bad data in the license number. I'm sharing this so you can use the SEARCH function in your investigations. #Snowflake continues to make our lives easier. For example, database name is DB1, Schema name is SCH1, Table name is TB1, search data is "BADDATA01", and you want to search in all columns to identify the row, query would be "select * from DB01.SCH01.TB01 where SEARCH( DB01.SCH01.TB01.*, 'BADDATA01' ) ;" Thank you Somen Swain on sharing this knowledge before which helped me in this case. More details can be found here: https://lnkd.in/esXjx8Xp

SnowFlakes recent improvements in query performance. The snowflake performance Index (SPI) tracks real world improvements in query performance using customer workloads. During the past year snowflake introduced several enhancements 1) Query Optimization Execution - Reduce execution time for complex queries - Improve join queries and handling of skewed data - More support for top K pruning and optimizing queries with special aggregation and patterns 2) Data ingestion and replication - Faster metadata replication and cloning - Large datasets ingestion optimized 3) Platform efficiencies - Reduce cloning operation time for overall speed and reliability. - Improved compression to reduce resource consumption The detailed analysis is found here - https://lnkd.in/gBVpYUVb #snowflake #tech #techblog

Join us as we dig into the pros and cons of query routing and proven strategies for optimizing Snowflake query performance: https://bit.ly/3XBvrEW #dataengineering #snowflakeoptimization

Optimizing Snowflake Query Performance with Clustering In a recent project, we tackled sluggish query performance on a 100M+ row transactional table where users frequently filtered by specific dates. The root cause? Lack of clustering. By adding a DATE column and clustering the table using ALTER TABLE CLUSTER BY (DATE), we reduced query execution times by 75%. Before clustering, queries like WHERE DATE = '2023-01-01' scanned the entire table, driving up latency and compute costs. Post-clustering, Snowflake scanned only relevant clusters, significantly improving performance. This simple yet powerful optimization directly enhanced our reporting SLAs and saved costs. Pro tip: Always align clustering keys with your query patterns for maximum impact!

❄ Here's how to run a 10000$ query on Snowflake:   Choose the biggest warehouse possible.   If it's not big enough, try to create one!   Write an infinite loop query…   …And watch world burn. (I mean $$ in your wallet)   If you think query timeout will save you - it won't.   Default is 2 days.   So before it happens, I would suggest lowering it to a more convenient value like:   ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = 7200   For more tips like this one you can check my video where I've summarized some of them:   https://lnkd.in/djc4trPu   #Snowflake #dataengineering #SnowflakeNinja

Query optimization is crucial for harnessing the full potential of Snowflake's data processing capabilities. After diving into various tuning options recently, I've realized that even small adjustments can significantly enhance performance. In one of my latest experiences, I worked on optimizing a complex query that initially took several minutes to execute. By methodically analyzing the execution plan and testing different strategies, I managed to reduce the runtime to a fraction of the original time. This not only improved efficiency but also encouraged our team to adopt a more analytical approach towards query performance. Through this process, I learned the importance of persistence and a systematic mindset in troubleshooting. Understanding the nuances of query tuning can make a big difference in your data workflows. I encourage you all to explore tuning options in your projects and share your experiences. What strategies have you found effective for optimizing your queries? #Snowflake #DataOptimization #QueryTuning #DataScience #CloudComputing https://lnkd.in/gbGG-ExH