Wherobots’ Post

Apache Iceberg has been making waves in the data world recently, with Databrick's acquisition of Tablular, the company behind Apache Iceberg, being the big news over the past 24 hours. During Snowflake's Q4 earnings call, Iceberg also got quite a bit of airtime. It's worth noting that Snowflake and Microsoft (OneLake) have already stated their commitment to supporting this open-source format. Big data formats and tools were big news back when I started my career in data in the early 2000s, and haven't gone away of course. They are if anything increasingly prevalent with the compute and parallel processing capabilities enabled by cloud data platforms today, as well as the huge volumes of data needed to train and tune our AI models. Established in 2017, Apache Iceberg, sits alongside Apache Parquet, as the leading open source standards for analytical storage formats. When diving into any technology or format that I haven't worked with before, in this case, Iceberg, I find that understanding it's context and where it fits in is key. From that perspective, I found the following introductory video, by Kiersten Stokes of IBM, really insightful. She traces the evolution of big data tools from the early 2000s to present, shedding light on where Apache Iceberg fits into the broader technological landscape. Well worth a watch: https://lnkd.in/eq5W2X4w #ApacheIceberg #DataWorld #OpenSource #BigData #TechnologyEvolution

This is what a paradigm shift looks like in real time. I think the amount of momentum in the market surrounding Apache Iceberg in the last few weeks is just a preview of what is to come as more customers adopt Apache Iceberg and embrace OPEN standards for their lakehouse. What an incredibly exciting time to be working in data.

Headless data architecture is the future and an open standard accepted by most is key to it. Apache Iceberg adaptations like the recently announced S3 tables by AWS along with the vanilla version will claim a huge share. Best of breed and fit for purpose compute engines with a unified open storage layer are already becoming a norm. Large scale data platforms with coupled architectures are breathing their last breaths.

Wow, wow! The Lakehouse is just in flames 🔥 🔥 🔥 during the last two weeks. If you missed the latest and hottest news about Apache Iceberg 🧊🧊🧊, help yourself with this quick summary. Despite the news and the vendors behind, one thing is clear, Iceberg is becoming the de-facto solution for Open Lakehouse. May 21, Snowflake Expands Partnership with Microsoft to Improve Interoperability Through Apache Iceberg: https://lnkd.in/d93AXBvA May 30, Dremio Integrates Apache Iceberg REST to Promote Vendor-Agnostic Ecosystem: https://lnkd.in/dZMB7-ve Jun 3, Snowflake introduces Polaris Catalog - An Open Source Catalog for Apache Iceberg: https://lnkd.in/dGHDidfD Jun 4, Databricks Agrees to Acquire Tabular, the Company Founded by the Original Creators of Apache Iceberg: https://lnkd.in/eFniN9t3

Schema evolution, partitioning, performance are just a few of the reasons major tech companies are invested in Iceberg. Snowflake's native support of Iceberg tables combine the features of Apache Iceberg with the robust capabilities of Snowflake. Organizations can efficiently manage and analyze large datasets with improved performance, scalability, governance and flexibility. It's a powerful choice, enabling data teams to derive insights quickly and effectively with #opensource.

To end difference of table formats, Snowflake: Iceberg is the way to go

This quote from Russell Spitzer summarizes table format debate really well: “If you bid on Iceberg, you aren't going to get double-crossed sometime in the future. It's something that you can be a part of, that you can control. That gives a lot of people security in the future” So if you want to future proof your lakehouse architecture, start with Iceberg as the primary format as it’s open and you as a customer can control it.

Hot off the press: Snowflake’s Russell Spitzer has some thoughts about the open format debate. As an Apache Iceberg PMC member himself, Russell can attest to the power of the community and true openness. He emphasizes that Iceberg is THE de facto standard for enterprises, and is confident that it’ll remain so for years to come. Read more in The Register to learn why more enterprises are betting on Iceberg for their open format needs. 👀

In the last post(https://lnkd.in/gp6Cfaqe), we saw about Apache Hudi's Streamer. Hudi has always been built taking inspiration from databases and hence you might see lot of flexibilities and configs. Here is the compilation of configs to use with HoodieStreamer. Feel free to make use of it if you are using HoodieStreamer. https://lnkd.in/gPf2FXTv #ApacheHudi #HoodieStreamer #Datalakes #Lakehouse #configs

If all the goodness that #apacheIceberg has to offer you and that brand new #dataLakehouse you're looking to build seems a little too good to be true, well... you're kind of right. It's not all possible with JUST the open table format that Iceberg gives you. Iceberg might give you the means group and manage a bunch of files in your #dataLake as a single table, but what if there are multiple tables? What if you want to add new tables, update them, move them around? That's the job of a #catalog. 📚 More specifically, an Iceberg catalog implementation (and there are many of them!) keeps track of your Iceberg tables' metadata, makes them discoverable by different compute engines, and allows those compute engines to interact with and change those tables. So, an Iceberg catalog is a single, centralized bridge between the data layer (the Iceberg tables) and the compute layer (all of your favorite compute engines). If that seems exciting to you, that's because it is! A catalog for your Iceberg tables is important! So what are your options? 🤔 Well, there are many. Several are built-in and offered out-of-the-box with Iceberg, such as Nessie, AWS Glue, REST, Hive, and more. But there are even more custom catalogs out there. Some, like #apachePolaris (incubating) are tailored to Iceberg and implement its REST API for better interoperability. In a future blog, I'll be covering more of the differences between catalogs and what you should be considering when choosing which to use. So keep an eye out! 👀 But for now, your next question should be, "Danica, that sounds great, how do I get more familiar with an Iceberg catalog?" I have five (six) words for you: 💥 Apache Polaris (incubating) Quick start #tutorial!💥 https://lnkd.in/dWS2fVpj See you tomorrow for another foray into the world of Apache Iceberg! #dataEngineering #iceberg #adventCalendar