Monte Carlo

Will data analysts become data engineers? This is a question that comes up from time to time in the data space. Two things that I think could coalesce to drive consolidation of engineering and analytical responsibilities: - Increased demand—as business leaders' appetite for data and AI products grows, data teams will be on the hook to do more with less. In an effort to minimize bottlenecks, leaders will naturally empower previously specialized teams to absorb more responsibility for their pipelines—and their stakeholders. - Improvements in automation—new demand always drives new innovation. (In this case, that means AI-enabled pipelines.) As technologies naturally become more automated, engineers will be empowered to do more with less, while analysts will be empowered to do more on their own. The argument is simple—as demand increases, pipeline automation will naturally evolve to meet demand. As pipeline automation evolves to meet demand, the barrier to creating and managing those pipelines will decrease. The skill gap will decrease and the ability to add new value will increase. Now, if that doesn’t sound like entirely bad news, that’s because…it isn’t. As Zach Morris Wilson rightly identified in one of his recent newsletters, data engineers are burning out. Endless data quality issues and increasingly complex pipelines are having a demonstrable impact on quality of work-life. On the flip-side, analysts are growing discontent waiting on the hook for that work to get done. The move toward self-serve AI-enabled pipeline management means that the most painful part of everyone’s job gets automated away—and their ability to create and demonstrate new value expands in the process. Data engineers get closer to the business. Analysts get closer to their pipelines. And business stakeholders reap the benefits.

IMPACT 2024 is coming up fast. Don’t miss Tomasz Tunguz, General Partner at Theory Ventures, speaking at 10:30am PT. 🔥 During his session, he’ll share the surprising trends shaping the future of data and AI that will give your team a competitive edge in 2025, including: 🚀 The rise of the post-modern data stack 🚀 What the increasing role of AI means for the data workforce 🚀 The proliferation of open table formats and data product marketplaces 🚀 What it really means to ensure your data is GenAI-ready 🚀 And more! Mark your calendars! Register here: https://lnkd.in/gBGAQ5ti #IMPACT2024 #dataobservability #datatrends #GenAI #datastack

November 10th to 16th is "Dear Santa Letter Week"! 🎅 To all of Data engineers supporting AI, ML, and Analytics teams, is this on your wishlist? 🌟 A "data debt" eraser to clean up all technical debt in our systems 🔮 AI-powered data quality tool that predicts and prevents data anomalies 👓 A pair of glasses that visualize data flows in augmented reality ✨ A magic wand that instantly converts legacy systems to modern, scalable architectures 🌟 A time machine to undo those accidental production deployments #DataEngineer #AI #ML #Analytics #DataQuality #Technology #DearSantaLetterWeek If you happen to be in Chicago on November 12th and attending Databricks #DataAITour2024, stop by at the Monte Carlo booth, Santa delivered few of your wishes early!!!

Ever wondered how data can predict the future? Or how a data-driven approach can help you win Olympic gold? You have a chance to a Data Engineering Course! Announcement towards the end. At Monte Carlo’s IMPACT 2024 on November 14th, you’ll find out the answers to both of these questions – and more. This year, Monte Carlo will feature two incredible keynote speakers: Allan Lichtman, the mastermind behind the Keys to the White House, a model that has successfully predicted U.S. presidential winners for decades, and Allyson Felix, Olympic gold medalist in track and field. They’ll both share how data, statistics, and pattern recognition are not just tools, but gamechangers in shaping our world, achieving our goals, and building trust. Plus, you’ll hear from expert data leaders and practitioners from across industries who will share how their teams are building processes to promote trusted data and AI, including speakers from SurveyMonkey, PepsiCo, Roche, DraftKings Inc., Grammarly, Earnest, Drata, Payoneer, ZoomInfo, Theory Ventures, and more. This yearly event brings the data community together to showcase the latest and greatest trends, technologies, and processes in data quality, LLMs, data and AI governance, and of course, data observability. Don’t miss this! Register here: https://lnkd.in/d_WGjqVc Attendees will also have the opportunity to network with industry peers and participate in exciting giveaways, including tickets to see the Eagles at the Sphere, yearly subscriptions to DataExpert.io’s Ultimate Data Engineering Academy by Zach Morris Wilson, and more! #IMPACT2024 #dataobservability #dataquality #montecarlo #ravenaondata

Join us on November 5th for the next stop on our IMPACT World Tour: London! Enjoy dinner, drinks, and an expert panel featuring Andy T., Director of Data + AI at PrimaryBid and Fernando Velez, VP Data Platform Engineering at Vertex Inc. who will share how their teams are driving trust in data + AI. See you there! Register here: https://lnkd.in/eTVcNUBc

Do you know what's lurking behind your broken dashboard? Check out the link in the comments to read data quality horror stories, and find out how those data teams unmasked their data quality monsters.

What does it take to achieve trusted, AI-ready data at scale? 🤔 Recently, Monte Carlo CEO and co-founder Barr Moses sat down with Surekha Durvasula, AI leader and former CDO at Walgreens Boots Alliance to discuss what she’s seen as the biggest challenges when it comes to delivering reliable data for AI. According to Surekha, the answer is three-fold: 1️⃣ Bring your business along 2️⃣ Practice risk modeling on your use-cases 3️⃣ Take your time getting data quality right Check out more of her tips: https://lnkd.in/gwTZQx2V #datatrust #GenAI #dataquality #datareliability

How three companies lost >$10M due to bad data: 1. Samsung’s $105B “Fat Finger” Data Entry Error In April 2018, Samsung Securities accidentally issued $105 billion worth of shares to employees due to a typo, creating 30 times more shares than in existence. This error led to employees selling $187 million worth of these "ghost shares" within 37 minutes. The fallout saw Samsung Securities' value drop by $300 million. 2. Unity Technologies’ $110M Ad Targeting Error Unity faced a $110M loss in Q1 2022 due to a data quality incident impacting their targeted advertising models. This happened because the predictive models were trained using bad data from a large customer. This led to a 37% drop in Unity's shares and skepticism about their strategic direction. 3. Uber’s $45M Driver Payment Miscalculation In 2017, reports showed that Uber overcharged New York drivers by 2.6% for over two years because it miscalculated its commission. Uber repaid impacted drivers an average of $900, totaling an estimated $45 million. Data quality is not just about data teams performance, it's directly tied to the quality of your ML/AI models. These costly mishaps aren't just cautionary tales—they're wake-up calls for data scientists racing to implement AI solutions. As AI systems become more prevalent, the stakes for data quality grow exponentially. Bad data doesn't just mean incorrect reports anymore; it means AI models making million-dollar mistakes in the blink of an eye. If you want to learn how to safeguard your data and AI systems, the upcoming Data Reliability Summit by Monte Carlo might be what you need. You'll learn from top data leaders about: · Driving trust in data and AI at scale · Building robust data systems for increasing AI adoption · Avoiding million-dollar mistakes with data quality issues Register here (free): https://lnkd.in/eBi4YEgc The cost of bad data is rising exponentially in the AI era. Is your team ready to prevent it?

Don’t miss our expert panel at IMPACT Summit on November 14th, featuring Sri Subramanian, Director of Data Engineering at SurveyMonkey, Stephanie Tignor, Head of Product Data Science at Grammarly, and Casey Maskalenko, Lead Engineer at DraftKings Inc. 🔥 They’ll share how they’re tackling data quality head on, as well as best practices and lessons learned for how to approach implementing data quality management processes, technologies, and organizational changes to ensure their company can fly high with reliable data. Mark your calendars! Register here: https://lnkd.in/gFAhfJ_P #IMPACT2024 #dataobservability #GenAI #AI #datatrust

To mesh—or not to mesh? That’s been the question since Zhamak Dehghani introduced the concept of the data mesh back in 2019. And for good reason! The data mesh is a thoughtful architectural approach that facilitates a holy grail of self-service data products. But for all the hype surrounding the data mesh, there aren’t nearly as many examples of data teams operationalizing it successfully. From the domain expertise required to maintain it to the socio-technical change management required to implement it, building a true data mesh is not for the faint of heart. Fortunately, the data team at Roche stepped up to the plate. As one of the earliest adopters of a data mesh architecture, Roche—a global diagnostics company headquartered in Switzerland—was quick to recognize the potential of the data mesh for large organizations. At a recent event, Harvey Robson (DEng), Global Product Owner of Data Quality and Observability, Global Data Engineer Roberto Münger, Data Architect Santosh Sivan, and Data Engineer Hendrik Serruys shared their experience building a data mesh and what they’ve learned from the experience. Their biggest takeaways? Step 1. Start small. Step 2. Focus on building trust. “If there’s no trust, the platform will fall,” said Santosh. Want to learn more? Check out the full story via link in the comments and find out how the Roche team is driving value from their data mesh today.