D ONE – Data Driven Value Creation’s Post

The lineage Apache Flink really needs - now the blog version! Our latest blog summarizes the insights from Shai Somekh and Colten Pilgreen's killer #FlinkForward presentation. They tackled one of the toughest challenges in real-time data, using data lineage to pinpoint policy violations and stop errors before they impact production. For everyone working in Flink, this goes beyond tracing - it’s about building the confidence to debug faster, uphold policy standards, and maintain control over streaming data pipelines. Check it out and see how data lineage, done right, can transform your approach to observability in Flink. Read it here - https://lnkd.in/dTPw4ekH #apacheflink #datalineage #flinkforward #datorios

Robust Lineage capabilities that truly trace your data is foundational for operational quality. This is a responsible business decision, potentially impacting revenue, reducing unnecessary costs, safeguarding reputation, and enhancing competitive advantage by over $10M.

Apache Flink shines as a top contender in real-time stream processing, boasting robust features like unified batch and stream processing, fault tolerance, and stateful stream capabilities. Its seamless integration with the Big Data ecosystem and strong community support further solidify its position in the industry. Despite facing competition from various stream processing engines and frameworks, Apache Flink stands out for its prowess in handling both batch and stream processing effectively, offering advanced stateful and low-latency processing capabilities. In the realm of large-scale real-time analytics and stateful stream processing applications, Apache Flink emerges as a leading choice, setting itself apart as a formidable competitor in the landscape. #ApacheFlink #StreamProcessing #BigData #RealTimeAnalytics

🚀 Unlocking the Power of Real-Time Data Processing with Apache Flink! After working hands-on with Apache Flink for the past few months, I've learned a ton about how it helps handle real-time data streams and complex processing with low latency. 📊✨ In my latest blog post, I break down: - Why Flink is a game-changer for real-time analytics. - Practical examples and use cases. - How to set up a basic Flink job (with code snippets!). - Lessons learned, tips, and common challenges. If you’re working with real-time data or exploring stream processing solutions, this post is for you. Check it out! 🔗👇 https://lnkd.in/dNB26Aw7 Let’s discuss in the comments! How are you managing real-time data in your projects? 💬 #ApacheFlink #DataEngineering #RealTimeData #StreamProcessing

State of the State in Apache Flink Stateful processing with real-time data is one of Flink's key advantages. As you likely know, with stateful processing comes the challenge of managing state size (and more) - (Yaroslav Tkachenko - https://lnkd.in/dpvq2V5w). However, rather than being a problem, state size is actually a feature. Being able to maintain the context of an entity as part of your pipeline unlocks limitless possibilities. State size has both direct and indirect impacts that must be considered. These include memory usage limits, the shift to using RocksDB, cluster size, and the effects of checkpoints size on performance and cluster costs (Take a look at performance correlation to state and checkpoints size below). The Flink community invests huge effort dealing with state size with upcoming Flink 2.0 (Yuan Mei - https://lnkd.in/d9-pVuNz) But how can you monitor and maintain your state effectively? A common issue is a "stuck" state — information that will never be used or has been kept in state for too long to hold business relevance. How can you query your state, understand it, and make informed decisions? While Flink doesn't provide this functionality out of the box, Datorios offers a solution. With our "State of the State" query feature, you gain the ability to analyze your state, build detailed reports, and make calculated technical and business decisions about its relevance and impact. #apacheflink #Statefulprocessing #datorios #flinkforward2024

Learn how to use Apache Flink for High-Volume Stream Processing in this article. We provide in-depth insights into quantifying workload requirements, optimizing cluster resources, managing distributed state, and efficiently scaling source and sink connectors. This article serves as a guide for implementing Apache Flink in production environments where terabytes of data are processed daily, ensuring effective scaling and performance optimization. https://lnkd.in/d_QU62Mx #apacheflink #streamprocessing #highvolumedata #datascaling

Have you heard of Apache Flink? The biggest tech companies use Flink to process petabytes of data a day with millisecond e2e latency. 🔥🐿️ But how does it work? 👉 Check out this cheatsheet -- For more Big Data content, follow me here: ✅ Stanislav Kozlovski And check out my newsletter - BigDataStream: ✅ https://lnkd.in/gZ28SNt3 #Flink #ApacheFlink #StreamProcessing

At the #KafkaSummit at the #ExCeL in #London, today and tomorrow. The agenda is packed (see: https://lnkd.in/eudQK5YU) It is always a pleasure to attend this #event, meet old (and new) friends and ex-colleagues. Here are the talks I'm looking forward to attend today:   Day 1  • Keynote (from Jay Kreps) "Streams Forever: Unifying the Operational and Analytical Worlds"  • Matthias J. Sax, Sophie Blee-Goldman, and 🗿 Bruno Cadonna: "#KafkaStreams -- Q&A with Committers"  • Stephan Ewen - "Restate: Event-driven Asynchronous Services, Easy as Synchronous #RPC" <-- 💡 check it out! https://restate.dev/  • Tom Scott - "The Streaming #DataLake - What Do KIP-405 and KIP-833 Mean for Your Larger Data Infrastructure"  • Oskar Dudycz - "#EventModeling Anti-patterns"  • Matthias J. Sax - "The Why Behind the How: Kafka #Reliability from First Principles" Day 2  • Tim Berglund "How Do You Query a Stream?"  • David Anderson, Robert Metzger, Ufuk Celebi, "#ApacheFlink Ask Me Anything"  • Igor Soarez "Troubleshooting Long JVM Pauses in Kafka"  • Dave Klein "Streaming Event Data to a Data Lake With Kafka Connect"  • Richard Artoul "Beyond Tiered Storage: Serverless Kafka with No Local Disks"  • Danica Fine and Sandon Jacobs "Closing Session" <-- "But wait, there's more! And you'll just have to attend to find out." 😅 🤣 Looking forward to meet the founders and learn more about some new startups in the thriving Kafka market and ecosystem, including (but not limited to 😉): Streambased, WarpStream, Restate, Lightstreamer, Nstream, Responsive, Axual, Redpanda Data, Conduktor, ... #ApacheKafka #ApacheFlink #ApacheIceberg #StreamProcessing #OpenSource #DurableExecution #Innovation #EventDrivenArchitecture #EventModeling #RealTimeData #KafkaSummit #KafkaSummit2024

🎥 [on-demand] 🔗https://lnkd.in/gwwb3ZNQ - Our "Stream Processing Must Haves" webinar recording is now available to watch! 🎉 We had an amazing session with 🐿️ Ben Gamble🧑🏾🦯 from Ververica | Original creators of Apache Flink®, who joined us to clarify the essentials of real-time stream processing. Ben and Sida Shen shared valuable insights into the benefits, challenges, use cases, and best ways to get started, making it a fantastic resource for anyone interested in real-time data. If you're interested in real-time data processing and its practical applications, this is the perfect place to start! #DataEngineering #StreamProcessing #RealTimeAnalytics #ApacheFlink #DataAnalytics #ApachePaimon #BigData #StreamingData #DataPipelines

There will be companies in 2050 who are still using Apache Iceberg and getting tons of value. Platform agnostic storage is unbelievably powerful. This is the open source future we’ve been waiting for. Hive almost delivered it which is why it had over a decade of longevity. Iceberg is designed the right way to solve this problem universally for decades to come! #dataengineering