Acheron Analytics

Are you considering becoming a data engineer in 2024 and soon to be 2025? Then here is a free mini crash course with +25 free resources you can use to get up to speed fast. It's broken down by the basics, then goes into higher level concepts like OLTP vs OLAP, data modeling, and finally goes into a few solutions like Airflow, Snowflake, etc. The Basics (Programming, SQL, The Cloud, etc) Data Engineering Vocab 101 1. https://lnkd.in/gRQagRax What Is Data Engineering - Why Is Data Engineering Important? 2. https://lnkd.in/gFmtavpu SQL For Beginners By Alex Freberg 3. https://lnkd.in/gAXcqn8W Python Tutorial - Python Full Course for Beginners Mosh Hamedani 4. https://lnkd.in/gpdCy4mf Python Libraries You Should Know As A Data Engineer - Python For Beginners 5. https://lnkd.in/gzupGDh4 Data Warehouse Tool Kit 6. https://lnkd.in/gVSCTWYi Becoming A Better Data Engineer - Tips On Translating Business Requirements 7. https://lnkd.in/gnQNyZbs Normalization Vs Denormalization 8. https://lnkd.in/gf_ivben Data Modeling - Walking Through How To Data Model As A Data Engineer - Dimensional Modeling 101 9. https://lnkd.in/g4QcjgXy What Is Orchestration by Mage and Matt Palmer 10. https://lnkd.in/g4-rEnWu From Basics to Challenges: A Data Engineer's Journey with APIs 11. https://lnkd.in/gjvesj7z Higher Level Concepts What Is A Data Pipeline 12. https://lnkd.in/ginyTQYM Transactional Databases Vs Data Warehouses Vs Data Lakes 13. https://lnkd.in/gxpbCRPi Why Is Data Modeling So Challenging 14. https://lnkd.in/gd5CQrZs How To Set Up Your Data Analytics Team For Success 15. https://lnkd.in/ey5tbjPQ OLTP vs OLAP 16. https://lnkd.in/gfMqKQHB How to design resilient and large scale data systems by Zach Wilson 17. https://lnkd.in/gWXzeqre Setting Standards For Your Data Team 18. https://lnkd.in/gSpjWpvt How To Come Up With A Data Engineering Project? 19. https://lnkd.in/gZTGK2Cz 7 Data Engineering Projects To Put On Your Resume 20. https://lnkd.in/gPtU5TSm 🚖 Uber Data Analytics | End-To-End Data Engineering Project by Darshil Parmar 21. https://lnkd.in/gAy7mGjJ Some technologies data engineers might need to know. Using AWS Lambda As A Data Engineering 22. https://lnkd.in/gR3HT6Jh The Realities Of Airflow - The Mistakes New Data Engineers Make Using Apache Airflow(ok this one is not free) 23. https://lnkd.in/gRRnpCjh What Is Docker - Docker Intro And Tutorial On Setting Up Airflow 24. https://lnkd.in/g3fPuz8r Spark vs Polars. Real-life Test Case by Daniel Beach 25. https://lnkd.in/gDEeHn5h Snowflake Vs Databricks - 🏃♂️ A Race To Build THE Cloud Data Platform 🏃♂️ 26. https://lnkd.in/gfy-8gDs What are your favorite free resources to learn data engineering? #dataengineering

One of the best ways to grow as a data engineer or analyst is by talking to other experienced data engineers and analysts. But where can you find other analysts and engineers especially in a world where there is remote work and often single person data teams? That's why I put together the Data Engineering and Machine Learning Summit! We already have well over 1600 people signed up, and we've got so many exciting talks planned! Here are just a few of the talks: - Strategies for Success with Next-Gen Data Stack - End to End RAG and Evaluation  - Navigating the Data Ecosystem - 10 most neglected data engineering tasks - And so much more! Also, rumor has it that Alex Freberg might be showing up! If you're interested in coming, or know someone that should attend, then let me know. Also! Special thanks to our amazing sponsor Databricks. It's so awesome to have them come along side and support our conference!

It feels like the list of data technologies that data engineers need to know continues to expand...not contract. There are way too many tools and solutions you can pick from. But really, many of them aren't that different from each other in terms of the problems they are trying to solve. Here are 7 great videos and articles you can go through to get up to speed quickly on what is going on in the data tech world. 1. Snowflake vs Databricks - And the Battle For Iceberg https://lnkd.in/gq-KCFsK 2. Common Pitfalls in Deploying Airflow for Data Teams https://lnkd.in/gGepn-K5 3. Memory Efficient Data Streaming To Parquet Files https://lnkd.in/gpVwKhEK 4. Why is Polars All The Rage? by Daniel Beach https://lnkd.in/gUVATCj5 5. Apache Iceberg - What Is It by Julien Hurault https://lnkd.in/gfntMYZX 6. The Data Engineer’s Guide to CDC for Analytics, Ops, and AI Pipelines https://lnkd.in/esbcDi_q 7. Data Modeling Where Theory Meets Reality https://lnkd.in/g_8FCpuA What articles will you be reading?

What are your best options for using change data capture (CDC) in a data pipeline for analytics, operations, data science/ML, generative AI projects, or all of these projects at once? This guide helps answer that question, both for a single project with a single destination, and as a broader platform across multiple projects and destinations. It first summarizes the major challenges with CDC that you need to consider for CDC-based pipelines - including reliable capture, end-to-end schema evolution and change, snapshots and backfilling, the different types and speeds of loading data, and managing end-to-end data consistency. This guide then continues with a detailed comparison of leading vendors or technologies before moving into a more detailed description of their strengths, weaknesses, and when to use them. If you'd like to read more, you should check out this article - https://lnkd.in/esbcDi_q

If you work in data, then AI is everywhere at this point. But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans. Whether that data is structured or unstructured. With the increasing focus on data and what it can do outside of basic analytics data engineers will have to learn a broader array of tools and skills. Joe Reis 🤓 has touched on this a few times in his recent article where he calls out the fact that “People in different disciplines are starting to learn each other’s craft. I’m starting to see software engineers and analysts learning machine learning/AI. Data scientists are learning to write production-grade code so they can work better with software engineers and integrate ML models into software applications.” In the same way data engineers will need to learn more about what is going on in the world of machine learning and AI to be better prepared for this shift. In this article we’ll review some of those skills and mental shifts data engineers will need to make and the new skills data engineers will need to learn. https://lnkd.in/gTa8Vy3w

Are you trying to learn more about data engineering? It can feel like there is a lot going on and new terms get made up every week. But there are a lot of people out there trying to make it easier to digest. Here are several great articles and posts that can get you up to speed fast on these two topics. 1. Data Modeling - Walking Through How To Data Model As A Data Engineer https://lnkd.in/gqStEWZx 2. Becoming A Better Data Engineer - Tips On Translating Business Requirements https://lnkd.in/gnQNyZbs 3. Databases Vs Data Warehouses Vs Data Lakes - What Is The Difference And Why Should You Care? https://lnkd.in/gnsgeW2s 4. Why is Polars All The Rage? by Daniel Beach https://lnkd.in/gxYPAHqu 5. OLTP vs OLAP - Transactions Vs Analytics https://lnkd.in/gfMqKQHB 6. Data Lakes And Streams - Are Data Lakes Where Streams Go To Die by Rob Meyer https://lnkd.in/gBkTv2wF 7. Using The Cloud As A Data Engineer https://lnkd.in/g2F-MQEn Feel free to share some of your favorite posts below as well!

Running a successful data team is hard. You're often pressured to deliver quickly sometimes feeling as if you aren't given the proper amount of time to consider data modeling or quality. Other times you're having to fend off the 120th "Hey I Have A Quick Ask" request. All of which can keep your data team away from delivering consistently. So the question becomes how do you balance all of these asks, needs and challenges? Well, to help answer these questions I have had several conversations with data leaders that range from Directors of Architecture to heads of data so you can learn from all of them. Below are 7 videos/articles talking about managing data teams, delivering value, and taking your data practice to the next level(who should I talk to next?). 1. 7 Habits Of Effective Data Managers https://lnkd.in/gRixYJCG 2. Onboarding For Data Teams https://lnkd.in/gPqHJuQJ 3. How To Drive Value With Data Science - Managing Data Science Teams In Healthcare with Sule Baptiste https://lnkd.in/gT2PsfQC 4. How Data Analytics Teams Can Deliver What The Business Needs Tom Rampley https://lnkd.in/gijZAwVK 5. Structuring Data Teams And Self Serve Analytics With Ethan Aaron https://lnkd.in/gNG6Qd2N 6. How To Set Up Your Data Analytics Team For Success – Centralized vs Decentralized vs Federated Data Teams https://lnkd.in/gS8fkxW4 7. Navigating the Pitfalls of Data Projects https://lnkd.in/g9wPDvpr I'd also love to start sharing some books on the subject of data teams, anyone have any favorite books. #dataengineering

I hear the term real-time thrown around a lot. So many companies want their dashboards to be in near real-time. Honestly, this has been ongoing since I got into the data industry a decade ago…and well before that. One possible choice is a method called change data capture, also known as CDC. I have seen companies employ multiple ways to use CDC or CDC-like approaches to pull data from databases like Postgres or MongoDB.  This can range from using triggers to reading logs. Of course, this focuses on the analytical component as many companies use CDC to replace or supplement traditional ETL/ELT. But CDC can also be a great way to understand your database and its structure. Databases abstract much of what they do to manage and process large volumes of data quickly. Before going down that rabbit hole, let’s dive into what CDC is and why companies use it. And if you want to learn more about CDC Rob Meyer from Estuary has been putting out some banger linkedin posts! https://lnkd.in/dnJRedgE

It's so exciting to see companies like Keboola ending the year with such great news. They just closed a $32 million series A and I am sure they have some big plans for 2024! A very special congrats to Pavel Dolezal Milan Veverka and Jan Andrs! https://lnkd.in/gPdbKGCm

Apache Druid, like many open-source data solutions, came out of the ad-tech boom. Companies wanted to be able to process and serve what was then "Big Data" we'd now probably consider it less so. But most of the solutions that existed at the time either struggled to do so or were expensive to use. Thus, Apache Druid was developed and has since grown in popularity. In fact, companies like Netflix, Reddit, and Walmart all use Druid because it can help create a real-time application database often in conjunction with a data warehouse. But how does Apache Druid work, and why do companies like using it? https://lnkd.in/egVUYJYb