Estuary’s Post

Do you have data spread across data lakes, data warehouses and databases? Are you struggling with data discovery, monitoring, and governance across these disparate systems? Lakehouse Federation, now generally available, makes it easy to create a unified view of your data estate and have consistent discovery, access controls, lineage and monitoring for all your data, no matter where it lives, with No ETL

📢 Exciting news for all data enthusiasts! Semarchy just released Semarchy 2024.2, a Mainstream Support release that includes a brand new Data Intelligence Module, as well as some amazing new features to their Data Integration and Data Management modules. 👀 Check out some of the highlights of Semarchy Data Integration 2024.2: ✔️ Runtime monitoring improvements ✔️ Updated Snowflake connector ✔️ Updated Azure Cosmos DB connector ✔️ Updated Azure Service Bus connector ✔️ XML as RDBMS String serializer 🎉 And here are some of the highlights of Semarchy Data Management 2024.2: ✔️ Generative AI enrichers ✔️ Run enrichers on button click ✔️ Start workflow from completed batch ✔️ Deletion approval workflows ✔️ Golden errors recycling ✔️ Record errors visible in collections 🔍 Want to learn more and explore the platform? Click on the link to find out: https://hubs.ly/Q02pDvCg0 #mdm #data #datamanagement #dataintelligence #dataintegration #productupdate

Join the upcoming #Couchbase Masterclass India to dive into the future of data management with this session "Beyond SQL - Unleashing NoSQL Brilliance for the BFSI Sector" on August 23!

Join the upcoming #Couchbase Masterclass India to dive into the future of data management with this session "Beyond SQL - Unleashing NoSQL Brilliance for the BFSI Sector" on August 23!

Ever wondered how to choose the perfect partition key to let your DynamoDB database scale effectively? DynamoDB stores data across multiple physical storage units, known as partitions. Each partition can hold up to 10GB of data. The partition key determines the distribution of data across these partitions. If a partition exceeds 10GB, DynamoDB uses a secondary sort key to further distribute data. Choosing an inappropriate partition key can lead to data distribution issues, significantly affecting scalability. Common pitfalls include hitting the 3000 Read Capacity Units (RCU) or 1000 Write Capacity Units (WCU) limits for a single partition key, or exceeding the maximum 10GB storage space per partition key. To avoid these scenarios, choosing the right partition key is crucial. Here are some effective strategies: 1. Use High-Cardinality Attributes - Opt for attributes with unique values, like unique order IDs. This ensures a more even distribution of data. 2. Composite Attributes - Combine multiple attributes to create a unique key, such as 'customerid#productid', especially useful when no single attribute is unique. 3. "Sub-Partitioning" for Write-Heavy Applications - For cases with more than 1000 1KB writes per second, consider adding random digits to the partition key. This creates 'sub-partitions' and increases write throughput. If this complicates reading data too much, a workaround is to hash the partition key to generate the suffix. Avoid using low cardinality attributes like dates or product SKUs as partition keys, as well as IDs that aren’t typically used for data access. In summary, there’s no one-size-fits-all solution. It's essential to analyse your write and access patterns and choose a key that offers the best data distribution while minimising the risk of throttling issues. Remember, a well-chosen partition key is the cornerstone of a scalable, efficient DynamoDB setup.

#day28 𝐀𝐧𝐚𝐥𝐲𝐳𝐞 𝐥𝐚𝐫𝐠𝐞 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬 𝐞𝐟𝐟𝐨𝐫𝐭𝐥𝐞𝐬𝐬𝐥𝐲 𝐰𝐢𝐭𝐡𝐨𝐮𝐭 𝐢𝐦𝐩𝐨𝐫𝐭𝐢𝐧𝐠 𝐭𝐡𝐞𝐦—𝐭𝐫𝐲 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐓𝐚𝐛𝐥𝐞𝐬! 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐚 𝐂𝐫𝐞𝐚𝐭𝐞 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐓𝐚𝐛𝐥𝐞? ➡ An external table is a reference to data stored outside the database, often in cloud storage like an S3 bucket. ➡ This table doesn't store data; it just points to where the data is located, saving space. ➡ Data in these tables typically appears in a single column called 'VALUE,' with a variant-type value that can have multiple properties. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: ➡ Suppose you have sales data stored region-wise in an S3 bucket. As a data analyst, you want to analyze this data in Matillion without importing it into a database. In this case, you would use a Create External Table to reference the data directly from S3. 𝐁𝐞𝐧𝐞𝐟𝐢𝐭𝐬: 1)𝐂𝐨𝐬𝐭-𝐄𝐟𝐟𝐞𝐜𝐭𝐢𝐯𝐞: You avoid charges for storing large datasets in the database, as the data remains in cloud storage. 2)𝐅𝐥𝐞𝐱𝐢𝐛𝐥𝐞 𝐃𝐚𝐭𝐚 𝐇𝐚𝐧𝐝𝐥𝐢𝐧𝐠: The external table can handle semi-structured data, such as JSON, making it easier to work with diverse data formats. 3)𝐐𝐮𝐢𝐜𝐤 𝐀𝐜𝐜𝐞𝐬𝐬: It provides fast access to the data, enabling efficient analysis and reporting without the need for data import. 🙏 Thanks, Anand Jha sir, for sharing the importance of the ETL tool Matillion. I hope you learned how to analyze data in Matillion using external tables without storing it in the database! 😊 #DataAnalytics #ETL #Matillion #Reports #DataVisualization #DataDriven  #DataInsights #dataanalyst #CloudComputing #DataIntegration #snowflake  #DataScience #BigData #DataPipeline #DataTransformation #table #json #DDL #PowerBI #truncate #delete #BusinessIntelligence #day28

2 more days - Join the upcoming #Couchbase Masterclass India to dive into the future of data management with this session "Beyond SQL - Unleashing NoSQL Brilliance for the BFSI Sector" on August 23!

In the realm of data management, Canonic stands out as a versatile platform. Dive into our blog post to uncover the intricacies of Canonic Tables - a hosted MongoDB with a user-friendly graph interface, available for free to all users. #Lowcode #Nocode

AWS zero-ETL could be your new best friend in data management. Builder Rob Koch explains how the service works and why it’s a big deal for anyone dealing with data pipelines. #ZeroETL #DataManagement #DataReplication

#AzureDaily Discover how to synchronize relational data between environments using #Azure Data Factory and SQL Data Sync! Seamlessly maintain data consistency across homogeneous or heterogeneous databases. An efficient, secure, and cost-effective solution! 🔄🌐💻 #MicrosoftAzure #DataSync