DoubleCloud

📣 New feature alert: Kinesis data source connector is now available! 📣 With #RocksetDB going out of the picture, many users are looking for alternatives. 𝗖𝗹𝗶𝗰𝗸𝗛𝗼𝘂𝘀𝗲 𝗰𝗮𝗻 𝗯𝗲 𝗼𝗻𝗲. DoubleCloud provides a great Managed Service for ClickHouse, so we’ve talked to quite a few of the former Rockset users, and most of them ask if we support DynamoDB and its CDC replication. 🤔 🔎 #DynamoDB was often paired with Rockset: the former provided extensive OLTP capabilities, and the latter was used to support analytics workloads, similar to what ClickHouse can do. 𝗧𝗵𝗲 𝗴𝗼𝗮𝗹 𝘄𝗮𝘀 𝗰𝗹𝗲𝗮𝗿: replace Rockset with ClickHouse in this setup. ▶ The problem was setting up CDC. First, we tried an Airbyte connector for DynamoDB, but unfortunately, it didn’t perform well enough to use it in production. That’s why we decided to implement a different solution, combining DynamoDB and Kinesis Data Streams. To support this, 𝘄𝗲 𝗯𝘂𝗶𝗹𝘁 𝗼𝘂𝗿 𝗼𝘄𝗻 𝗻𝗮𝘁𝗶𝘃𝗲 #𝗞𝗶𝗻𝗲𝘀𝗶𝘀𝗗𝗮𝘁𝗮𝗦𝘁𝗿𝗲𝗮𝗺𝘀 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗼𝗿 to cover CDC from DynamoDB to ClickHouse and quickly delivered it to production. 🙌 🚀 📚 To help automate this setup, we also added a new Terraform module: https://lnkd.in/gjJgpgef and provided a setup example on GitHub: https://lnkd.in/gqHT-4YE. #RocksetAlternative

What might be better than 7 Wonders of the World?😌 Only 7 tips for a production #ClickHouseCluster from our latest blog authored by Vladimir Ivoninskii! While #ClickHouse is celebrated for its speed and efficiency, configuring it for production use presents unique challenges. Here are the top tips to ensure optimal performance: 1️⃣ Use multiple replicas: If there’s a failure and you only have one replica and a single host, you’re at risk of losing all your data. 2️⃣ Invest generously in #RAM: Check the full blog to see what ClickHouse developers recommend as a minimum number of gigabytes. 3️⃣ Choose the right table engine: Unless you know for sure why you need a different table engine, use an engine from a #MergeTree family, and it will cover most of your use cases. 4️⃣ Don’t use more than 3 columns for the primary key: Reduce lookup times and increase efficiency by optimizing your primary keys. 5️⃣ Avoid small inserts: Batch inserts to minimize overhead and improve throughput. 6️⃣ Handle duplicate data: Implement strategies to clean and prevent duplicates effectively. 7️⃣ Smart indexing: Use indexes judiciously to speed up queries without burdening storage. 📖 Read the full blog for a deeper dive into each tip: https://lnkd.in/dd2bYRm7 Additionally, for practical demonstrations and more tips and tricks on optimizing ClickHouse, check out our webinar: https://lnkd.in/dU9JEA9b?

In our previous posts, we explored why Apache Airflow remains a preferred choice and introduced some of its modern competitors. Now, let's directly compare these tools to help you decide when and why to choose Airflow in 2024. 📌Airflow vs Dagster 🔵When to Choose Airflow🔵: Opt for Airflow if you need a proven, stable solution with a strong community and extensive integrations. It’s ideal for organizations with established data platforms and those requiring complex workflow orchestration. 📌Airflow vs Prefect 🔵When to Choose Airflow🔵: Airflow is suitable for teams needing a versatile tool that integrates well with existing systems and supports complex workflows. 📌Airflow vs Mage 🔵When to Choose Airflow🔵: Choose Airflow if you need a tool with proven best practices and extensive external system integrations, especially for large-scale operations. 🔎 In conclusion, while modern tools like Dagster, Prefect, and Mage offer unique advantages, Apache Airflow continues to stand strong in 2024 due to its flexibility, strong community support, and proven track record in orchestrating complex workflows. 🔴When Not to Choose Airflow🔴: 🔶Specialized Pipelines: If your pipelines require specialized features that are better supported by Dagster, Prefect, or Mage, such as real-time event handling or asset management. 🔶Operational Costs: Consider the total cost of ownership, including the complexity of setup and maintenance. Tools like Prefect and Mage might offer lower operational overhead for certain use cases. Be sure to check out our webinar on workflow automation, featuring the power of DoubleCloud Managed Apache Airflow, for more use cases and to see our product in action: https://lnkd.in/dJdPz7Pw #DataEngineering #Airflow #Dagster #Mage #Prefect

In our previous post, we discussed why Apache Airflow remains a strong choice in 2024. Now, let’s explore some modern tools that have emerged as notable competitors, along with their main advantages. We'll focus on open-source tools, setting aside proprietary tools like Databricks Workflows for a clearer comparison. 🔎 Dagster: Known for its focus on data quality and observability, Dagster provides a robust framework for building and monitoring data pipelines. Its type-safe design ensures that data flows are reliable and maintainable. While Dagster might not be as feature-rich as Airflow at this point, it’s catching up fast. 🔎 Prefect: Prefect specializes in dynamic workflows and features a user-friendly interface for complex data tasks. Its hybrid execution model enhances scalability and adaptability. Tailored for Data Science applications, Prefect includes tools specifically useful for such contexts, though this specialization might limit its flexibility. Consider pairing it with other tools for comprehensive pipeline management. 🔎 Mage: Mage simplifies data pipeline construction with its low-code platform, making it ideal for users at all technical levels. Its straightforward, accessible approach allows quick setup and deployment. Additionally, it’s important to consider the scalability and integration capabilities of these tools, which are crucial for handling large-scale data operations and ensuring seamless integration with various data sources and systems. With the rise of Data Governance practices, you should also evaluate whether you need support for data catalogues and lineage tooling when selecting tools. In our next post, we’ll compare these modern competitors with Airflow and draw some conclusions. Stay tuned! By the way, have you had a chance to explore any of these tools in a production environment yet? Which is your favorite? #DataEngineering #Airflow #Dagster #Mage #Prefect

Choosing the ideal ETL tool from the many available options can be challenging. Let’s explore why Apache Airflow remains a top choice in 2024. Apache Airflow excels in orchestrating complex workflows and managing task dependencies. Its flexibility supports dynamic pipeline generation, ideal for batch processing and task scheduling. While it can handle real-time data processing, Airflow is traditionally used for orchestrating batch jobs. However, its extensive capabilities can sometimes be overwhelming, offering great flexibility that can be both a benefit and a challenge. Despite newer tools like Dagster, Prefect, and Mage, Airflow has a solid reputation as a reliable and stable tool commonly used in data platforms. It boasts a strong community, numerous integrations with external systems, and a wealth of proven best practices. Recently, there has been progress in integrating Airflow with common LLM providers and enhancing support for Okta/SSO integrations and multi-team functionality. Check out this insightful talk by one of the core Apache Airflow contributors, Jarek Potiuk, to learn more about why Airflow is still a great choice: https://lnkd.in/dgtd8GKU Stay tuned for our next post, where we’ll dive into the modern competitors of Airflow and their pros and cons! #DataEngineering #Airflow

“𝗟𝗲𝘁 𝗺𝗲 𝘁𝗲𝗹𝗹 𝘆𝗼𝘂 𝘁𝗵𝗲 𝗿𝗲𝗰𝗶𝗽𝗲 𝗳𝗼𝗿 𝘁𝗵𝗲 𝗽𝗲𝗿𝗳𝗲𝗰𝘁 𝗼𝗻-𝘀𝗶𝘁𝗲: 𝘄𝗼𝗻𝗱𝗲𝗿𝗳𝘂𝗹 𝗽𝗲𝗼𝗽𝗹𝗲, 𝗮 𝗯𝗲𝗮𝘂𝘁𝗶𝗳𝘂𝗹 𝗰𝗶𝘁𝘆, 𝗮𝗻𝗱 𝗹𝗮𝘀𝘁 𝗯𝘂𝘁 𝗻𝗼𝘁 𝗹𝗲𝗮𝘀𝘁, 𝗮 𝘀𝗽𝗹𝗮𝘀𝗵 𝗼𝗳 𝗯𝗲𝗲𝗿.” - Salvatore, Head of Platform Team! 👏   Last week, our Platform team at DoubleCloud finally met in person for three intense and exciting days! ⚡ Despite being a #FullyRemoteCompany with team members spread all over the world, we seized the opportunity to come together for productive discussions and, of course, some fun. 📸 Check out some shots of the fun times of the team! They shared: “If you ever get the chance, I highly recommend the bike beer tour—it was such an amazing team-building experience.” 🚲 🍻   A big kudos to the entire Platform team for bringing the right amount of energy! 💪 💚   #TeamBuilding #RemoteWork #DoubleCloudTeam #PlatformTeam #TeamSuccess

You may have heard that OpenAI has recently acquired #Rockset, prompting all users to transition to alternative platforms by September 30, 2024. Facing uncertainty about which database to switch to? Join our webinar on July 25th at 18:00 CET, where our tech experts will guide you through a smooth migration to #ClickHouse. Meet Your Guides: Andrei Tserakhau, DoubleCloud Tech Lead, known for his deep expertise in database solutions. Deepan Ignaatious, Product Manager at DoubleCloud, a strategist in product optimizations. What You’ll Learn: • Assessing compatibility between databases • Efficient data export and import strategies • Schema migration and its challenges • Testing and validation for flawless integration • Optimization tips to maximize your database performance Don’t miss out!! Secure your spot to navigate this transition smoothly! Register now and ensure your data strategy is future-proofed

We are expanding our team and hiring for a Territory Sales Manager in India! Join our dynamic awesome team and be part of the growth 🚀 ! Feel free to apply or send me a message. Apply today 🌟 #DoubleCloud #Hiring Find out more: https://lnkd.in/eqAK5BBH

𝗪𝗵𝗮𝘁 𝗮𝗿𝗲 𝘆𝗼𝘂 𝘂𝗽 𝘁𝗼 𝘁𝗵𝗶𝘀 𝗦𝗮𝘁𝘂𝗿𝗱𝗮𝘆, 𝟮𝟳.𝟬𝟳.? 𝗛𝗲𝗿𝗲’𝘀 𝘄𝗵𝗮𝘁 𝘄𝗲 𝘄𝗶𝗹𝗹 𝗯𝗲 𝗰𝗲𝗹𝗲𝗯𝗿𝗮𝘁𝗶𝗻𝗴 𝗶𝗻 𝗕𝗲𝗿𝗹𝗶𝗻! Christopher Street Day (CSD) in Berlin is just around the corner, offering an incredible opportunity to unite in celebration. 🎉   This annual event honors the LGBTQ+ community and champions the principles of equality, diversity, and acceptance. #CSD commemorates the 1969 Stonewall Riots in New York, a pivotal moment when LGBTQ+ individuals stood up against discrimination and oppression. 🏳️🌈   Berlin’s CSD celebration is marked by parades, inspiring demonstrations, lively parties, cultural events, and delicious food. 😋 It’s a perfect occasion to show solidarity, have fun, and advocate for an inclusive and tolerant society. At DoubleCloud, we proudly support these values! To get into the spirit, our Berlin team is already geared up with some 🌈 socks and ‘queer’ beers! 🍻 💚 🧦 We look forward to seeing you there! 🙌

It’s been almost two weeks since our insightful webinar on real-time analytics with #ClickHouse! If you haven’t watched it yet, now is the perfect time. Our Tech Lead, Andrei Tserakhau, covered a range of topics to help you transition from batch data processing to real-time analytics, including: - Introduction to real-time streaming and analytics - Benefits of using ClickHouse - Integrating Kafka with ClickHouse - Demo of DoubleCloud’s solution Don’t miss out on the chance to get valuable knowledge absolutely free! Watch now: https://lnkd.in/dfurBK-U #DataEngineering