Seattle Data Guy

With HLTH Inc. right around the corner, our team put together a whitepaper that dives into how you can unlock hidden opportunities to improve patient care by harnessing the power of your untapped data. This is a common challenge we’ve seen in many of the healthcare organizations and companies we work with today and we would love to share with you our insights. Explore the whitepaper to learn about: 🔗 https://lnkd.in/geJk9Ayf ✅ Barriers to Effective Data Utilization ✅ Recent AI Technologies Transforming Healthcare ✅ Opportunities From Untapped Data ✅ How to Unlock Your AI Advantage #Gradient #GradientAI #Healthcare #UnstructuredData #DataReasoning #DataReasoningPlatform #LLM #AIHealthcare

Going from a data engineer or analyst to leading a data team is hard. You have to pick up a whole new skill set fast, learn a whole new set of terminology and not to mention you likely spend a lot less time coding and writing SQL. Just ask Alex Freberg. Not to mention you're often pressured to deliver results quickly, and have stakeholders who like using the the words "This should be quick" and "just" a lot... This is why I have been talking to dozens of data leaders over the past few months to understand how they made the transition and what skills they felt were crucial to being a successful data leader. But I'd also love to hear from you. What skills, articles or books would you recommend to new heads of data out there? Also, if you'd like to learn more, then you can check out this video. https://lnkd.in/gfbFUewJ

Over the past 5 years I have written well over 200,000 words in articles and newsletters on data engineering, infrastructure and more. Basically, I have written more than a books worth of words. Which is why I have started to put together a book to help share not only my experiences working for and leading data teams but also dozens of other data leaders. It's been a great experience thus far, and I have only written two chapters thus far! Truthfully, it takes a lot more focus, rereading and there isn't any instant gratification like when you write an article or a Linkedin post. And thats partially why I have put out one of the chapters in my newsletter. The two parts of it are listed below! Don’t Lead a Data Team Before Reading This - Aligning The Data Team with Business Objectives https://lnkd.in/gen_uaH5 Building Credibility As A Data Leader https://lnkd.in/e9tMvPwc And if you've got any thoughts you'd like to add, or questions you think need to be answered on the subject, feel free to reach out.

There are a lot of "easy fixes" in data that aren't the right solution. They either add unnecessary tech debt or don't really improve the situation. Here are 3. 1. Let's just fix it in SQL - It can seem really easy to fix business logic in the SQL layer rather than from the source. However, this means, in the long-run, anytime that business logic changes, your team needs to update the SQL as well. It's far more impactful to get the source team to ensure the data is right where they create it rather than in the data warehouse. 2. Let's just add more data quality checks - Data quality checks are necessary. But too many become noisy and coupled with no real change occurring leads to people ignoring them. Less critical data quality checks can be aggregated into a dashboard and reviewed on a normal cadence and connected to larger projects. For example, perhaps your data pipelines keep landing late. Instead of being pinged every morning, this is an opportunity for a larger initiative. 3. Let's just build this real quick - This is great for a POC but POCs and temporary solutions often become permanent. Do you have any other examples? Also, if you'd like to learn more about data engineering and leading data teams, check out my newsletter https://lnkd.in/gNwgxpkk

As a consultant, I have been called in to review and, in many cases, replace dozens of half-finished, abandoned, and sometimes forgotten data infrastructure projects. The data infrastructure in a few cases may just need a little tweaking to operate effectively, but other times the project is either so incomplete or so lacking in a central design that the best thing to do is replace the old system. Trust me, I’d love it if I could come into a project and simply change a few lines of code, and then everything would just work. However, so many projects are filled with unclear design decisions or resume-driven development that were never rooted in good planning. Of course, business stakeholders may have also push to get things done quickly. Forcing data teams to take on tech debt that will never be fixed. Don’t get me wrong, you want to get things done and move projects forward. But taking on technical debt is a decision that needs to be made intentionally. Otherwise, like in resume driven development, your data infrastructure might disappear. This begs the question. How do you ensure the data infrastructure you’re building doesn’t get replaced as soon as you leave in the future? In this article I wanted to dive into the problems I often come into that require me to replace the current data infrastructure and how you can avoid it. So let’s dive in. https://lnkd.in/geP3sQ2W

How companies data model varies widely. They might say they use Kimball dimensional modeling. However, when you look in their data warehouse the only part you recognize is the word fact and dim. Over the past near decade, I have worked for and with different companies that have used various methods to capture this data. I wanted to review some of the techniques that are commonly used to model data for analytics. This is part of my unofficial series on data modeling, so if you’d like to learn more, then you can check out some of the prior articles. But for now, let’s dive in. https://lnkd.in/gwV_Fbgm

There are endless posts out there talking about how you can go from 0-100 in a short period of time as a data engineer or analyst. I know, I have written a few articles with that notion. May it be how to break into a certain role or how to go from a junior to a senior quickly. Now don’t get me wrong, I believe all these articles and content have value. There are a lot of little lessons you can pick up along the way on becoming a senior engineer or growing into certain roles that might just need to be explained and don’t need to be experienced. On the flip side, however, I find that sometimes we are often so focused on getting to some end result that we often miss developing a lot of the basic skills along the way. We miss out on the journey. That's why I recently started a series both in video and article form that is a back to the basics. Here are 5 of those articles(let me know if there is something you think I should cover) 1. From Basics to Challenges: A Data Engineer's Journey with APIs https://lnkd.in/gjvesj7z 2. Behind the Scenes of SQL: Understanding SQL Query Execution https://lnkd.in/gmrpYzts 3. Back To The Basics With SQL: Understanding Hash, Merge, and Nested Joins https://lnkd.in/g3NbfBai 4. Embrace Being A Beginner: The Importance of Building a Strong Technical Foundation https://lnkd.in/gxEXPVjH 5. Normalization Vs Denormalization - Taking A Step Back https://lnkd.in/gP7mVn2m And if you'd like to see the video versions, let me know or check out the SDG channel!

❄ ❄ We are excited to announce that Roe AI has officially been accepted to Snowflake's Native App Accelerator. If you are a Snowflake customer, now it’s time to join the ALPHA waitlist: https://lnkd.in/e9CP5VWm The accelerator provides funding, technical support, and Market support for strategic, early-stage startups building a Snowflake Native App. With our Native App "ROE AI Data Agent", hyper-optimized for complex unstructured data, Snowflake customers can extract accurate insights from long documents, audio recordings, and websites. ROE AI data agents orchestrate Cortex LLM, Azure OAI and Google Gemini — all governed within your Snowflake security boundary. As an ALPHA customer, you will get early access to the product, unlimited support sessions, prioritized product feature requests and exclusive discount. Thank you to Snowflake, AWS, and the Snowflake Startup Program for helping bring this vision to life. We can't wait to bring our innovative solutions to the Snowflake Marketplace. Stay tuned for updates as we continue to build.

If you've had to build any data pipelines for analytics, then you're likely very familiar with the extract phase of an ELT or ETL. As the name suggests the extract phase is when you connect to a data source and "extract" data from it. The most common data sources you'll be interacting with being databases, APIs, and file servers(via FTP or SFTP). With my recent focus on going back to the basics, it occurred to me that I have never written about APIs and how we interact with them as data engineers. Now, there are plenty of APIs that have caused me a lot of heartburn in my career and there are others that have been a piece of cake to handle. But it all comes down to how the API is set up and the design choices made when it was built. So, in this video, I wanted to talk about, first, the basics of an API, followed by reviewing some of the issues you'll deal with as a data engineer.

There are WAY too many terms and vocabulary floating around in the data engineering world. Here are some key terms you should know. Change Data Capture - Often referenced as CDC, this process helps track changes that occur in a database, often by reading the log files and generally in a near real-time fashion. Video(Free) - https://lnkd.in/gM7ba9Jr Article(Free)- https://lnkd.in/giDbV_qx Normalization/Denormalization - You'll often hear the term normalization and denormalization when data modeling. If you've ever heard of the one-big-table model, thats a denormalized approach where all your data exists in a single table. Where as normalization reduces duplication of data and often requires more joins. Article(Free) - https://lnkd.in/g3enwMtQ Data Lineage - When you start creating more and more complex DAGs that then are built into other processes, its good to know the entire path. That way as you make changes you can track the impact. Article(Free) - https://lnkd.in/gj48bpdx Unstructured Data - This refers to data that doesn't have a pre-defined schema. Generally we work with tables that have rows and columns where unstructured data can be PDFs, Images, Videos, etc. Article(Free) - https://lnkd.in/gAtDPBHV Article(Free) - https://lnkd.in/gN3-3EPR Column Oriented Database - If you're accustomed to standard transactional databases, you're relying on row-oriented databases. These are databases that store data in rows. This makes a lot of sense when you only need to access a single row at a time. But for analytics its helpful to store data in columns since you often run aggregates on columns. Video(Free) - https://lnkd.in/gJNdC42T Pub/Sub - You'll often hear engineers use the term, Pub/Sub, this stands for publish/subscribe. This is a messaging pattern used in distributed systems where there are two entities, publishers and subscribers. Pubs produce messages on a topic and subs receive said messages. Video(Free) - https://lnkd.in/gV7Rinyj Execution Plan - The execution plan is the processes that take place in order to access the data. This is important to understand because if a query is slow, sometimes this can be improved by looking through said plan to figure out which actions are taking a long time. Article(Free) - https://lnkd.in/gmvirV7P Article(Free) - https://lnkd.in/gmrpYzts Feel free to share your own articles below.