Machine Learning

Unity Catalog vs Apache Polaris (Incubating)… It’s time for some Friday armchair analyses. Unity Catalog was open sourced from Databricks on 6/13 and donated to the LF D&AI Foundation on 6/20. Polaris Catalog was open sourced from Snowflake 7/30 and donated to the ASF Incubator on 8/21 Both of these projects are very exciting advances in the open source community and I look forward to both creating stronger solutions for data governance and ability to interoperate data across diverse tools. It is still very early days for both of these projects, but here are some interesting nuggets so far: 🗿 Table Metadata Format support: Unity Catalog (UC) = Apache Iceberg, Delta Lake, and Apache Hudi Polaris Catalog (PC) = Iceberg only 😢 ⭐ GitHub stars (go smash a star on both or your fav): UC = 2,156 PC = 960 🤼 Github contributions (last 30days): UC = 31 authors | 49 Merged PRs | 19 closed issues PC = 43 authors | 129 Merged PRs | 26 closed issues In 2024 we finally resolved table format wars just in time for the new battle of the catalogs? Which gameboy cartridge are you planning to plug into your data stack? I’ve personally tried one and planning to make a demo on the other soon. Has anyone else tried both and have thoughts to share? At Onehouse we will be supporting both. Good luck to cloud vendors though making sure their entire complex portfolio of products works with both plus their own catalogs they are building? #Snowflake #Databricks #UnityCatalog #PolarisCatalog #DataLakehouse

Join 250+ already accepted attendees to the next Apache Hudi community sync with Ankit Shrivastava from Uber. If you haven't noticed by now.. the Apache Hudi community is incredibly passionate and flourishing group 🤝 Tue, Aug 20, 2024, 12:00 PM EST Event link: https://lnkd.in/ePP4jQ_q

Bloom Filter in Parquet & Lakehouse Table Formats. Parquet filter pushdown is a performance optimization method that prunes irrelevant data from a #parquet file to reduce the amount of data scanned by a query engine. You can skip row groups typically using 2 ways: - column min/max statistics - dictionary filter ✅ Column statistics include minimum and maximum values that allow for range-based filtering. ✅ Dictionaries provide more specific filtering, enabling readers to exclude values that fall within the min and max range but are not listed in the dictionary. Problem with dictionary is that it can consume more space with higher cardinality columns. This results in columns with large cardinalities and widely separated min and max values lacking effective support for predicate pushdown. This is where the 3rd approach comes. ✅ A bloom filter is a probabilistic data structure that allows you to identify whether an item belong to a data set or not. ✅ It either outputs: "definitely not present" or "maybe present" for every data search. By using Bloom filters, you can efficiently skip over large portions of the Parquet file that are irrelevant to your query, reducing the amount of data that needs to be read and processed. In lakehouse platforms like Apache Hudi, users can utilize the native Parquet bloom filters, provided their compute engine supports Apache Parquet 1.12.0 or higher. I also read an amazing blog by the Influx Data Team on Parquet's bloom filter implementation. Link in comments. #dataengineering #softwareengineering

The Data Guy, George Yates talks about where to use #ApacheHudi ⚡ and how to get up and running fast! ⚡ Follow along the Spark quick start ✅

Interested in leveraging an open table format? Follow this short tutorial Thomas Hass and easily leverage your lakehouse across the existing vendor ecosystem 🙏🏻

🌟 Discover the power of storing AI vector embeddings in a data lake for cost-effective storage and seamless integration! Learn how Apache Hudi enables you to manage large datasets efficiently and incrementally bring vectors to your target database. Read more: #AI #DataLakes #ApacheHudi #DataStorage

The dirty little tech secret in 2024. Most AI companies are service companies getting product valuations. 100x valuations on a Saas product. Cool. But this isn’t a Saas product. These companies are white-gloved consulting companies. Call a spade a spade. They masquerade as a product company and trojan horse a platform into the client's infra. I can’t be the only one who sees this, can I? The value doesn’t come from the AI companies ‘product’. The product is a commodity. The consulting is the value add. But hey, who am I to rock the boat? Let the good times roll.

How PENN Interactive uses Apache Hudi 🔥🔥

Curious about how NerdWallet uses Amazon Web Services (AWS) and The Apache Software Foundation Apache Hudi to build a serverless, real-time analytics platform? In this case study, Kevin Chun, a Staff Software Engineer at NerdWallet, outlines how they are scaling a large data ecosystem by building a serverless pipeline that enables streaming data from across the whole company. Drawing inspiration from the data mesh paradigm, they designed a data lake based on Amazon S3 that decouples data producers from consumers while providing a self-serve, security-compliant, and scalable set of tooling that is easy to provision. Services used: Amazon Athena, Lambda, EMR, Kinesis, MSK, QuickSight, Redshift, Glue and many others.

Curious about how NerdWallet uses Amazon Web Services (AWS) and The Apache Software Foundation Apache Hudi to build a serverless, real-time analytics platform? In this case study, Kevin Chun, a Staff Software Engineer at NerdWallet, outlines how they are scaling a large data ecosystem by building a serverless pipeline that enables streaming data from across the whole company. Drawing inspiration from the data mesh paradigm, they designed a data lake based on Amazon S3 that decouples data producers from consumers while providing a self-serve, security-compliant, and scalable set of tooling that is easy to provision. Services used: Amazon Athena, Lambda, EMR, Kinesis, MSK, QuickSight, Redshift, Glue and many others.