Kensu’s Post

📘 News for Data Enthusiasts! 🚀 I recently had the privilege of reading an outstanding book "Data Observability for Data Engineering" written by my talented ex-colleague Michele Pinto and Sammy El Khammal. 📚 If you're a data engineer or scientist, this is a must-read that can truly revolutionize your approach to managing data. 🌐 In the book the authors brilliantly unfolds the intricate world of data quality, the chapter that i like most is "Exploring the seven dimensions of data quality" where they demystify the complexity surrounding this critical aspect of data management. 🛠️ The book is a goldmine of techniques and best practices that are not just insightful but also practical. The standards outlined are what every company should aspire to achieve in their data journey. 🚀 From ensuring data accuracy to improving accessibility, this book covers it all. 🤓 Whether you're a seasoned data professional or just starting in the field, this book provides invaluable insights that will undoubtedly elevate your understanding of data observability. 📈 #dataobservability #datascience #datarngineering #bookrecommendation http://packt.link/1SQFj

Data science project failure. On Sunday, I read a post outlining one reason data science projects fail. Ultimately, it was about data scientists and requestors not being aligned. Which is about communication from both sides. In reality, this issue isn't exclusive to data science. It's incredibly common in the dashboard world. Also in the world of IT development. Actually, in any field where someone is asking someone else to make something for them. Do you know exactly how you want your house to look inside and out? I guess not, hence you use architects and designers to feed you with ideas and expertise. I digress, but the point is that understanding requirements is incredibly hard. Requestors very often fail to articulate what they want. Often, they don't know exactly what they want. In the data world, it's the job of the skilled scientists, developers and analysts to work with requestors and figure out what they want. And why they want it. This ability to understand what requestors really want is what separates the great from the good in the data world. Given how few are able to do this well, it's a hard skill to master. My advice is this; focus on outcomes. Someone wants something for a reason. Working back from that outcome helps identify the data necessary to support that outcome. Understanding the business is key, and understanding the role of the requestor is an advantage. Perhaps you're asked to pull some data. Understand why someone wants that data, what they intend to do with it. What's the business context? Then you can deliver something useful. Perhaps you won't deliver data, maybe you'll be able to deliver the information they want. I would like to cite the post sparking this post; I wanted to comment on it, but was pulled into family duty, and by the time I returned to LI, the feed refreshed, and the post gone. Unfortunately, I don't remember who wrote it. #dashboards #data #datascience #datavisualisation

Sharing with you all some insights inside of Fundamentals of Data Engineering by Joe Reis. ➡ The topic is "Data Maturity and the Data Engineer" ⬅ In the book the author explores 3 stages of data maturity and he starts by "Starting with data", "Scaling with data" and "Leading with data". Stage 1 - Starting with data In this section, Joe gives us an view point about the companies that are starting with data in the early stages of its maturity. As he say "The company may have fuzzy, loosely defined goals or no goals". The data team is small, often with headcount in the single digits and at this stage, a data engineer is usually a generalist and will typically play several roles, such as data scientis or software engineer. In this stage the practicalities of getting value from data are typically poorly understood, but the desire exists. And to move forward and make data evolve we have some key topics to analyze, i'll be brief so the post doesn't get too long. 1️⃣ Get buy-in from key stakeholders, including executive management. Ideally, the data engineer should have a sponsos for critical initiatives to design and build a data architecture to support the company's goals. 2️⃣ Identify and audit data that will support key initiatives and operate within the data architecture you designed. 3️⃣ Build a solid data foundation for future data analysts and data scientists to generate reports and models that provide competitive value. In the meantime, you may also have to generate these reports and models until this team is hired. 💡 And to finish these insights, i would like to share something important too. As we can see this is a delicate stage with lots of pitfalls but here's a tip: Getting quick wins will establish the importance of data within the organization. Just keep in mind that quick wins will likely create technical debt. Have a plan to reduce this debt, as it will otherwise add friction for future delivery. ⏳ In a short while, I'll talk about the other two topics and their insights. #data #dataengineering #fundamentalsofdataengineering #dataengineer

Just updated: "How to Know Your Data Science/ML Project Will Fail Before You Begin." I've refined my article on predicting data science project failures, incorporating valuable feedback and enhancing key sections: Expanded Requirements Section: - Clearer distinction between functional and non-functional requirements - Emphasis on aligning business and technical functionalities - Practical examples to illustrate each point Improved Examples Throughout: - More detailed, real-world scenarios - Clearer illustrations of common pitfalls and success factors - Enhanced actionable insights for data professionals and leaders Whether you're a data scientist, ML engineer, or tech industry leader, these updates offer fresh perspectives on avoiding project failures and maximizing success. Read the full article here: https://lnkd.in/gmFp2rDU What's your experience with data project failures? Have you encountered similar challenges? I'd love to hear from you. #DataScience #MachineLearning #ProjectManagement

How the CRISP-DM Framework Transformed My Approach to Data Science Projects As a data scientist, I've learned that the key to success in any project is having a clear, structured approach that ensures alignment between technical efforts and business goals. Early in my career, I discovered the CRISP-DM (Cross Industry Standard Process for Data Mining) framework, and it has since become an invaluable tool in guiding my work. Its flexibility and focus on results have helped me deliver more impactful, business-driven solutions across various industries. The CRISP-DM framework consists of six key stages that guide the end-to-end data science process: -Business Understanding: This initial phase is about understanding the project from a business perspective. Defining objectives, key questions, and project success criteria ensures that the project stays aligned with business goals from the start. Personally, I believe this is the most important stage. -Data Understanding: Once business goals are clear, the next step is gathering and exploring the data. This phase involves assessing data quality, identifying initial patterns, and recognizing potential challenges. It's essential to know your data well to build an effective solution. -Data Preparation: In this phase, data scientists clean and transform raw data into a suitable format for analysis. This process includes handling missing values, removing inconsistencies, and preparing the data for the modeling stage. It's a critical step that can take up a significant portion of the project timeline. -Modeling: With the data prepared, it's time to apply statistical models or machine learning algorithms to uncover insights. This phase involves selecting and testing various models to identify the best one for the specific problem. -Evaluation: Once a model is built, it is thoroughly evaluated to ensure it meets the original business objectives. Data scientists analyze the model's performance and its effectiveness in solving the problem at hand. -Deployment: Finally, the model is deployed in a real-world environment, where it generates valuable insights or predictions that guide business decisions. Continuous monitoring ensures the model’s long-term performance and value. CRISP-DM has been a game-changer for me, providing a proven, repeatable process that ensures project success. By bridging the gap between business objectives and technical execution, it helps data professionals like myself deliver meaningful insights that drive business growth. #DataScience #CRISPDM #BusinessIntelligence #Analytics #DataDriven#DataAnalyst#DataScientist Jedidiah Ayuba Usara Efemena Ikpro

Data science relies on working with different types of data systems and architecture. For data scientists or analysts just getting started, some of the terminology can be confusing. In this post, Let's dive into definitions and examples for five fundamental data science concepts: - 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲: A centralized repository that stores huge amounts of raw, unstructured data in native formats. Provides flexible, scalable storage for diverse analytics and ML needs. - 𝗗𝗮𝘁𝗮 𝗠𝗮𝗿𝘁: A subset of company data focused on the needs of a specific team or use case. Enables access to tailored data without irrelevant info. - 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲: Automates the flow and transformation of data from sources to destinations. Ingests, processes, and routes data for downstream uses. - 𝗗𝗮𝘁𝗮 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲: Integrates data from multiple sources into one unified repository. Structured for querying, analysis, and reporting. Provides a single source of truth. - 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆: The accuracy, completeness, consistency, and relevance of data. Essential for reliable analytics and decisions. - 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲: Combines data lakes and warehouses. Stores raw data like a lake along with preparing and structuring data like a warehouse. Provides flexibility for diverse data and uses.

HOW TO BUILD A STRONG DATA TEAM!? To build a strong data team and climb the pyramid of Data Science, think of it like constructing a building with different levels, where each level has a crucial role: Data Engineers (Foundation Layer): These professionals are like the architects and builders. They gather all the raw data and organize it, ensuring it’s stored neatly and securely. They prepare the data so that it’s clean and ready for analysis. Goal: Ensure that data is easily accessible and reliable. #DataEngineering #DataManagement Data Analysts (Insight Layer): These experts are like detectives. They examine the data to uncover important trends, patterns, and insights. Their job is to interpret what the data means and help the company make informed decisions. Goal: Extract valuable insights from the data. #DataAnalysis #BusinessInsights Data Scientists (Innovation Layer): These are the masterminds or wizards of the data world. They use advanced tools and techniques, like machine learning, to predict future trends and solve complex problems. They focus on groundbreaking projects that drive the company forward. Goal: Use data to innovate and make powerful predictions. #DataScience #MachineLearning To reach the top of the data science pyramid, each of these layers needs to work together, building on one another to achieve success. #DataTeam #DataSciencePyramid

Data science relies on working with different types of data systems and architecture. For data scientists or analysts just getting started, some of the terminology can be confusing. In this post, we’ll dive into definitions and examples for five fundamental data science concepts: - 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲: A centralized repository that stores huge amounts of raw, unstructured data in native formats. Provides flexible, scalable storage for diverse analytics and ML needs. - 𝗗𝗮𝘁𝗮 𝗠𝗮𝗿𝘁: A subset of company data focused on the needs of a specific team or use case. Enables access to tailored data without irrelevant info. - 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲: Automates the flow and transformation of data from sources to destinations. Ingests, processes, and routes data for downstream uses. - 𝗗𝗮𝘁𝗮 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲: Integrates data from multiple sources into one unified repository. Structured for querying, analysis, and reporting. Provides a single source of truth. - 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆: The accuracy, completeness, consistency, and relevance of data. Essential for reliable analytics and decisions. - 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲: Combines data lakes and warehouses. Stores raw data like a lake along with preparing and structuring data like a warehouse. Provides flexibility for diverse data and uses.

To all my analytics engineers connections or those who want to become analytics engineers: this book is great! Juan Manuel Perafan sent me a copy of this book so I could share my thoughts, and I have been planning to write a review about it for a while now. But I wanted to read it to give an honest review. And like I said, it is great! It covers all aspects of analytics engineering in the right way. It starts with what is analytics engineering, what is an analytics engineer, how it differs from other data roles, and how it all connects to the modern data stack. It discusses a lot about data pipelines, passing through ingestion, transforming, and serving. It also talks about data warehouses and data modeling (super important topic). Then it goes through hands-on guide, data quality, workflow automation, and governance. It all is written in an easy-to-read writing, so you don’t get tired and don’t want to stop reading. It has several tables and illustrations to illustrate complex topics. I think it could cover more deployment types, like WAP and blue-green, I missed it a bit. This book will not make you an expert analytics engineer, such as Joe’s Fundamentals of Engineering will not make you a master data engineer, but it has all the fundamentals you need to put you on the path to becoming one. You can get the book here: https://packt.link/aQ9uz #dataengineering #dataengineer #analyticsengineering #analyticsengineer

Starting off with series of skills essential for excelling in the world of data engineering!  𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐒𝐤𝐢𝐥𝐥𝐬 𝐒𝐞𝐫𝐢𝐞𝐬: 𝐓𝐡𝐞 𝐏𝐨𝐰𝐞𝐫 𝐨𝐟 𝐃𝐚𝐭𝐚 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 🚀 Data modeling: it's the unsung hero of the data engineering world. Often overlooked, yet absolutely essential for building robust data infrastructure. Think of data modeling as the blueprint for your data architecture. It's the process of structuring and organizing data to optimize its accessibility, efficiency, and accuracy. And let me tell you, it's a game-changer! Here's why data modeling deserves more recognition: 1. 𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐃𝐚𝐭𝐚 𝐈𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞: Just like a sturdy foundation supports a skyscraper, solid data modeling supports your entire data ecosystem. It ensures your databases are structured logically, making it easier to store, retrieve, and analyze data. 2. 𝐄𝐧𝐡𝐚𝐧𝐜𝐞𝐝 𝐃𝐚𝐭𝐚 𝐐𝐮𝐚𝐥𝐢𝐭𝐲: Garbage in, garbage out. With effective data modeling, you can enforce data integrity rules, validate inputs, and eliminate inconsistencies. This means cleaner, more reliable data for your analyses and decision-making processes. 3. 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞𝐝 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: Efficiency is key in the world of data engineering. By designing efficient data models, you can minimize storage requirements, reduce query execution times, and ultimately improve the performance of your applications. 4. 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐚𝐧𝐝 𝐅𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲: As your data grows and evolves, so should your data architecture. A well-designed data model allows for scalability, enabling you to adapt to changing business needs and accommodate future growth seamlessly. 5. 𝐅𝐚𝐜𝐢𝐥𝐢𝐭𝐚𝐭𝐞𝐬 𝐂𝐨𝐥𝐥𝐚𝐛𝐨𝐫𝐚𝐭𝐢𝐨𝐧: Clear, standardized data models serve as a common language for data engineers, analysts, and stakeholders alike. This fosters collaboration and ensures everyone is on the same page when it comes to understanding and utilizing data. So, next time you're building a data pipeline or designing a database schema, don't underestimate the power of data modeling. It may not always steal the spotlight, but it's the backbone of successful data engineering initiatives.  #DataEngineeringSkillsSeries #DataModeling #Innovation #QualityAssurance #PerformanceOptimization #FutureProofing #Collaboration #Tech #DataInfrastructure #Analytics #DataEngineering