CelerData’s Post

"BigQuery tables for Apache Iceberg use the Apache Iceberg format to store data in customer-owned cloud storage buckets while providing a similar customer experience and feature set as BigQuery native tables." https://lnkd.in/gnpBd56p

🚀 New Blog Alert 🚀 I’m excited to share my first blog post: “Migrate Delta Tables from Azure Data Lake Storage to Amazon S3 using AWS Glue” In today's multi-cloud landscape, connecting data across different cloud providers is more crucial than ever. Our latest post delves into how AWS Glue can streamline the migration of Delta tables from Azure Data Lake Storage (ADLS) to Amazon S3. By leveraging the ADLS2 connector, AWS Glue offers a powerful ETL solution to bridge these cloud environments seamlessly. This guide walks you through the process of leveraging AWS Glue's ETL capabilities to efficiently move and manage your data across cloud environments. Whether you're exploring multi-cloud strategies or optimizing your data workflows, this guide provides practical insights and step-by-step instructions to simplify your data migration process. 🔗 Read the full blog here: https://lnkd.in/gz25q-vN A big thank you to my co-authors Nitin Kumar, Shubham Purwar, Pramod Kumar, and Madhavi Watve for their invaluable insights and collaboration on this project. I hope you find this resource valuable for your data engineering and cloud migration needs. Looking forward to your feedback and any questions you may have! #AWS #DataEngineering #AWSGlue #CloudMigration #MultiCloud #Azure #AmazonS3 #BigData

Today, we announce the newest addition to our data sharing platform at #GoogleNext: Bobsled on BigQuery. Teams that build data products on BigQuery can now securely share data directly to customers in Snowflake, Databricks, AWS S3, Azure Storage and more through a single integration. With Bobsled on BigQuery, you can: - Securely share data to other organizations in any cloud storage, data warehouse or SFTP server through a single integration. - Rollout updates, manage schema changes and maintain pipelines seamlessly. - List on marketplaces and supercharge alliances with data and cloud platforms with no additional development work. Learn more about how Bobsled supports universal data sharing from BigQuery here https://hubs.li/Q02sgCX00 #BigQuery #datasharing #daas #next

The AWS Glue Data Catalog now automatically removes unneeded data files from Apache Iceberg tables to reduce metadata overhead, control costs, and improve query performance. This builds on the catalogs existing automated compaction feature. #aws #awscloud #cloud #amazonsimplestorageservices3 #analytics #announcements #awsglue #awslakeformation #intermediate200 #apacheiceberg

AWS Amplify Launch Week introduces new extensibility features in Amplify Gen 2, including integration with databases, custom authentication, UI components, and GraphQL transforms. #aws #awscloud #cloud #posttypes #amazonbedrock #amazoncognito #announcements #awsamplify #awsappsync #awsclouddevelopmentkit #awslambda #developertools #frontendwebmobile #technicalhowto

A Snowflake Iceberg table is a type of table in Snowflake that employs the Apache Iceberg format for organizing and storing data within the Snowflake data warehouse environment. This format stores the actual data outside of Snowflake in a public cloud object storage location such as Amazon S3, Google Cloud Storage, or Azure Storage. 🧊❄️ To create a Snowflake Iceberg table, you must specify an External Volume and a base location (a directory within the external volume) where Snowflake can store both the table data and metadata. Learn more about creating Iceberg tables in Snowflake❄️  https://lnkd.in/gXj_wAtv #ThinkETL #Snowflake #Iceberg #DataEngineering

This post discusses challenges with traditional data analytics and presents Amazons zero-ETL integration of Aurora PostgreSQL and Redshift as a solution, generally available since October 2024. #aws #awscloud #cloud #amazonaurora #amazonredshift #announcements #intermediate200 #postgresqlcompatible

The construction of big data applications using open source software has become easier with projects like Data on EKS from AWS. This provides blueprints for building data and ML apps on Amazon EKS. #aws #awscloud #cloud #advanced300 #amazonemroneks #awsbigdata #bestpractices

RDS for MySQL zero-ETL with Redshift streamlines data ingestion for customized insights without complex pipelines. It offers filtering, multiple integrations, and CloudFormation support. #aws #awscloud #cloud #amazonredshift #announcements #featured #launch #news #rdsformysql