N2S.Global’s Post

We are very excited to hear about Snowflake Artic. Of many possible applications, one key data pipeline step it can revolutionize is data cleanup. Here are a few data cleanup steps I’ve built that could have been replaced by LLMs: ▪️ Standardizing a person’s name across different data sources. Ex: Mapping Robert, Rob, Bob, R. => Robert ▪️ Standardizing a job title across sources. Ex: Mapping Software Engineer, Sr. Software Developer, Software Hacker => “Software Engineer” ▪️ Extracting the City, Zip, Country from the address. Ex: “123 W 83rd Street, NYC, 10024, US” => City:New York, Zip: 10024, Country: USA ▪️ Standardizing dirty event data: Ex: product_clicked, product_clicked_ios, product_clicked_top_of_page, product_clk => product_clicked Often data teams waste time building and maintaining these data cleanup pipelines. RudderStack Profiles makes it easy to integrate LLMs into your c360 data pipeline to handle cleanup. What data cleanup tasks do you want to eliminate with LLMs?

Recently, there has been a buzz in the data industry about Iceberg technology. Leading companies like Cloudera, Databricks, Snowflake, Palantir, and Dremio are all emphasizing the adoption of Iceberg as a common layer for interaction. One key benefit is the decentralization of metadata information from the traditional hive metastore to the file system at a lower level (partition). This approach minimizes interactions and enables O(1) access to essential information. For engineers looking to dive into Iceberg, the best starting point is to read the specification and gain a comprehensive understanding of how it functions. Explore more at: https://lnkd.in/dJ2dugjc

Hi friends, these are the latest enhancements to #snowflake external tables, provided by #minio SME Breena. For your reading.

Need to get the most out of your investment in Collibra's data intelligence platform? Make sure it's integrated across your data governance, management and consumption workflows. When Collibra becomes vital for those core activities, it becomes an indispensable data catalog for your business. We help clients across various industries in driving Collibra adoption through a people-centered approach. For example, we encourage clients to prioritize user roles that depend on trusted data. Our strategy focuses on three groups of people whose work revolves around data  — the heavy, moderate and periodic Collibra users. FSFP's experienced consultants and Certified Collibra Rangers help ensure Collibra's full integration into your business, so you leverage the platform's full capabilities. Interested in learning more? > Download our Collibra Implementation and Adoption informational sheet: https://lnkd.in/gfvHY3u5 > Or email Kate Pingel, MBA, CDMP, one of our Collibra consultants: kate@firstsanfranciscopartners.com #datamanagement #collibra #informationmanagment #dataquality #datagovernance #dataconsultant #dataconsulting #datamanagement Collibra

🌐𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞’𝐬 𝐔𝐧𝐢𝐪𝐮𝐞 𝐓𝐡𝐫𝐞𝐞-𝐋𝐚𝐲𝐞𝐫 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞! 𝐃𝐚𝐭𝐚𝐛𝐚𝐬𝐞 𝐒𝐭𝐨𝐫𝐚𝐠𝐞: Utilizes a columnar storage format, fully managed and optimized by Snowflake, ensuring data is stored efficiently and securely, accessible only through SQL queries. 𝐐𝐮𝐞𝐫𝐲 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠: Employs MPP compute clusters known as virtual warehouses, ensuring that each cluster operates independently without affecting others’ performance, leading to highly efficient query execution. 𝐂𝐥𝐨𝐮𝐝 𝐒𝐞𝐫𝐯𝐢𝐜𝐞𝐬: This layer binds everything together, managing user interactions, authentication, metadata, and query optimization to ensure seamless operations. #Snowflake #CloudComputing #DataEngineering #BigData #TechInnovation #data #dataengineer #dataanalyst #DWS #IT #HCLtech

💡 Unlock the secrets behind the most crucial data-related job roles that power industries worldwide. 🌐 Whether you're a budding data enthusiast or a seasoned pro, this episode is your gateway to understanding the backbone of modern data ecosystems. Ready to embark on an enlightening journey? 📊 Watch now and join our growing community of data aficionados! Don't forget to hit that subscribe button to stay updated on all things data-driven! #AzureData #DataFundamentals #DP900 #datajobs #dataengineer #dataanalyst #databaseadministrator

While I'm thinking about deltalake - highly recommend this article from Matthew Powers, CFA on the pros and cons of hive-style partitioning in a datalake: https://lnkd.in/exd8WKRC

📰 Have you heard the news? Snowflake just announced their Polaris Catalog, an open catalog for Apache Iceberg, with support from Dremio and our Project Nessie! This is a huge step forward in providing customers with the open-source data solutions they deserve. Read all about it in this article from Paul Gillin, via SiliconANGLE & theCUBE: 🔗 https://lnkd.in/dBcvrRjq

Picking a data warehouse is one of the most important choices you’ll make when setting up a modern data stack! Make sure you consider the most important factors. Hint: engineering experience might actually be the most important 😉 and one people don’t think a lot about!