Seattle Data Guy

If you work in data, then AI is everywhere at this point. But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans. Whether that data is structured or unstructured. With the increasing focus on data and what it can do outside of basic analytics data engineers will have to learn a broader array of tools and skills. Joe Reis 🤓 has touched on this a few times in his recent article where he calls out the fact that “People in different disciplines are starting to learn each other’s craft. I’m starting to see software engineers and analysts learning machine learning/AI. Data scientists are learning to write production-grade code so they can work better with software engineers and integrate ML models into software applications.” In the same way data engineers will need to learn more about what is going on in the world of machine learning and AI to be better prepared for this shift. In this article we’ll review some of those skills and mental shifts data engineers will need to make and the new skills data engineers will need to learn. https://lnkd.in/gTa8Vy3w

I am constantly reminded how solutions like Snowflake, BigQuery and Databricks make my life easier as a data engineer. If you didn't start on SQL Server, Postgres, or Oracle like Jeff Skoldberg and Ryan H. who pointed out some of the ways CDWs make data professional lives easier, then you might be unaware of some of the limitations. So here are a few of the limitations or admin tasks you used to have to perform. 1. Limited Storage - With most cloud solutions you have unlimited storage(which sure, comes with an unlimited bill) but you never have to sit there and wonder if you have a temp table somewhere that is causing storage issues or just generally need to wonder if you need to migrate hardware... 2. Limited compute - This also goes for compute. If you've never had to open up a database activity monitor to see what query is holding up all your other queries, do you even DBA(just kidding nowadays you gotta worry about an accidental 10k query)? 3. General Admin - Ryan Howe covered some of this but he recently came up against having to deal with trying to release space on his database, yet after he released it it wasn't fixed. You can read more about it in the comments below. 4. Query History - Jeff Skoldberg referenced this one. Technically you can find your query history often buried in the sys tables on traditional DBS but cloud data warehouses make it so easy. You can easily find query history as well as metadata about the query, how long it took, its query profile, etc. Now I am sure there are other benefits, which I'd love to hear in the comments below, but I am also sure there are people out there who still prefer using solutions like Postgres for their DW(which I'd also like to hear about)!

Nearly 10 years ago I got my first job in data analytics. In some ways a lot has changed since then. In other ways, not much has changed at all. I wanted to talk to Daniel Palma who has also been working as a data engineer for about 10 years what he has seen as he has worked as a data engineer, consulting and leading teams.

Data engineering terms you should know. Even if you're not a data engineer! Star Schema - A star schema is a multi-dimensional data model used to organize data in a database in a way that is geared toward analytics. It's called a star schema because there is usually a central fact table with several dimension tables that surround it. learn more here - https://lnkd.in/gmyPqQf9 DAG - A DAG or Directed Acyclic Graph in the data engineering world refers to a conceptual representation of tasks. Many are first introduced to this idea via Airflow. There can be simple DAGs such as A->B->C or more complex such as A->B, A->C, C->D, and B->D. learn more here - https://lnkd.in/gsZE3JSU ETL/ELT - Another common concept is ETL/ELTs which letters stand for Extract, Transform, and Load. These patterns can often be built into DAGs but represent a process of pulling data from source systems and then at some point running business logic over it prior to loading it into a data warehouse learn more here - https://lnkd.in/gHswsiq2 Data Connector - In the past, many data engineers had to write/rewrite the same code over and over again to pull data from databases, APIs, and other sources. Now there are several solutions that exist that just act as a layer to connect to hundreds of possible data sources. learn more here - https://lnkd.in/gpp3i5qv Data Lake - The term ‘data lake’ was coined around 2010. Data lakes became popular because they offer a solution to the rapidly increasing size and complexity of data. As defined by TechTarget: "A data lake is a storage repository that holds a vast amount of raw data in its native format until it is needed for analytics applications." learn more here - https://lnkd.in/gST2-F7e Data Warehouse - A data warehouse is a central repository of information that can be analyzed to make more informed decisions. Or as put by Bill Inmon it's a “subject-oriented, nonvolatile, integrated, time-variant collection of data in support of management’s decisions.” learn more here - https://lnkd.in/eWD5Xteh MPP(massively parallel processing) - MPP is a processing paradigm that as the name suggests, takes the idea of parallel processing to the extreme. It uses hundreds or thousands of processing nodes to work on parts of a computational task in parallel. These nodes each have their own I/O and OS and don’t share memory. learn more here -https://lnkd.in/g3BZyyG9 #datanegineer

When I first started consulting, I charged an hourly rate. That was a mistake. I was thinking like an employee charging an hourly rate. And I know I am not alone because I have now had multiple conversations with data engineers and data scientists who are often charging as little as $35 an hour for their time. Which is drastically undercharging. So if you're just starting out here are just a few key values you offer as a consultant that you might not realize you should consider. Experience - You're not just paid for getting the job done, you're getting paid for knowing how to do it better than others because you've done it before. One common question I get during most projects is "How have you seen other companies do it". Temporary Engagement - Hiring and firing employees is expensive. A company has to manage benefits, possibly offer severance, onboarding, and possibly deal with legal problems if they fire their employee. But a consultant’s engagement can end at any time. Meaning you could be saving a company tens of thousands of dollars due to this flexibility alone. Speed - One of the biggest benefits most experienced consultants offer is speed. Due to their experience as well as their focus, consultants can often come in and deliver a specific piece of work faster than an employee who may only occasionally take on a migration or similar project. Risk Reduction - Paying a consultant can also come with some level of risk reduction. Either because a company is trying to figure out the best solution to pick and your perspective could help save 100s of thousands of dollars and the fact that you've likely done a type of project before ensures it'll more likely succeed. And honestly those are just a few of the benefits you bring as a consultant, plus don't forget you've got to handle taxes, benefits, etc. So please charge more than $35/hr(this is some what location based). Now if you do feel stuck and don't know what to charge or how to charge. Or maybe you're stuck on marketing or sales. Then you should consider joining the community of +700 other consultants I have recently started!

Join us on October 3rd and 4th for the Data Engineering And Machine Learning Summit 2024! DEML Summit is a free-to-attend virtual conference where you will hear from data practitioners and learn how they are solving exciting data problems in the real world. 👉 Sign up for the conference here: https://lnkd.in/gcUWqufw (Thank you to Databricks for sponsoring the conference.) #dataengineering

When I was first learning SQL there were several "AH-HAH" moments that helped take me to the next level. Many of these didn't come from books but instead were necessities because of the problems I was solving at the time. Here are a few of those key lessons. 1. Understanding the different joins - ok this one came from a book 2. Finding out about ROW_NUMBER, RANK(), LAG() and LEAD() 3. Learning that you can use a CASE statement inside of an aggregate 4. Figuring out that self-joins can often perform the same action as some window functions 5. Learning to break down my queries into easy to understand CTEs or queries 6. Finally, getting that at the end of the day, it's not about how many SQL clauses you know but how well you understand the data underneath Is there anything you would add?

Data engineering is a difficult role to break into. There are WAY TOO MANY tools, solutions and skills that DEs are expected to know. You don't have to learn everything all at once. But if you are looking to read on some data engineering topics, here are 7 great articles, posts and videos you should check out! 1. Timeless Skills For Data Engineers And Analysts https://lnkd.in/gnxK6vcj 2. Data Warehouses Vs Operational Data Stores Vs Data Lakes – How To Store Your Data For Analytics https://lnkd.in/geZGZ2Q3 3. Data Contracts In ML Pipelines by Chad Sanderson https://lnkd.in/gMzqEQ7X 4. Why Is Data Modeling So Challenging – How To Data Model For Analytics https://lnkd.in/gW7gYSFx 5. The Data Engineer’s Guide to CDC for Analytics, Ops, and AI Pipelines by Rob Meyer https://lnkd.in/esbcDi_q 6. 9 Performance tuning techniques that you can use to tune your Spark Jobs by Sumit Mittal https://lnkd.in/gazPUugJ 7. Normalization Vs Denormalization - Taking A Step Back https://lnkd.in/gP7mVn2m Which articles are you reading?

Cloud service providers such as AWS and GCP offer hundreds of services, and sometimes it can be a little confusing to figure out what solution does what and how data engineers and data scientists might use them. In this article, I wanted to discuss some of the services that are useful to know as a data engineer as well as provide a combination of real-world examples where I have used said services. I’ll also only be focusing on AWS for now. So let’s dive into the cloud services you should know as a data engineer. https://lnkd.in/gqKfD_Rc

I am getting close to four years of putting out data engineering youtube videos and.... We are getting so close to 100k...1.8k away! Over the past few years I have put out videos that range from the traditional "how to become a data engineer" topics to other topics that are a little more nuanced. Here are 7 videos that I have put out over the past few years that I really liked working on! 1. Going From Data Engineer To Head Of Data - How To Run A Data Team Successfully https://lnkd.in/gYU9VrfY 2. What Is Apache Druid And Why Do Companies Like Netflix And Reddit Use It? https://lnkd.in/egVUYJYb 3. Build A Data Stack That Lasts - How To Ensure Your Data Infrastructure Is Maintainable https://lnkd.in/gz2fAXys 4. Data Modeling Where Theory Meets Reality - How Different Companies I Worked At Modeled Their Data https://lnkd.in/gX_V2G2k 5. The Realities Of Airflow - The Mistakes New Data Engineers Make Using Apache Airflow https://lnkd.in/gjp9eyFY 6. What I Learned From 100+ Data Engineering Interviews - Interview Tips https://lnkd.in/gan7WFF7 7. Databases Vs Data Warehouses Vs Data Lakes - What Is The Difference And Why Should You Care? https://lnkd.in/gddxWnpB Also...THANK YOU for all your support. #dataengineering