Data Architect 101 for Data Engineers (Complete Guide)
Understanding how to build end-to-end data systems can be a catalyst in your career growth as a Data Engineer, let's understand the basics of building a data system 👇🏻
The goal of every data project is to solve business problems!
It can be anything from reducing current system costs to building a full-fledged data system helping businesses to make data-driven decisions.
Today I want to take you on a journey of architecting a data system, if you are an aspiring data engineer or have been working in this field for a few years, understanding how to architect a data system can help you see the bigger picture.
What is Data Architecture?
Here is the technical definition you will find in The Fundamentals of Data Engineering Book by Joe Reis and Matt Hoursely
Data architecture is the design of systems to support the evolving data needs of an enterprise, achieved by flexible and reversible decisions reached through a careful evaluation of trade-offs.
In simple terms, before the construction of any building, architects design a blueprint, the same thing we replicate for the data system
Building Architecture contains various components - Foundations, Floor plans, Elevations, Elevators, Stairs, Offices, Restrooms, and many more...
Data Architecture contains Storage, Software, Data flow, Interfaces, Transformation, Staging areas, Data warehouse, Reporting, and many more…
As per the technical definition, decisions should be flexible and reversible, which means if every component in the architecture does not meet the requirements then should be easily reversible
=====There are two parts to it=====
1. Business Needs (Operational Architecture): We focus on the business goals and requirements.
For example in an e-commerce platform
- What is the impact on the XYZ category of product?
- Why is there a delay in product shipping?
- How do we manage data quality from third-party vendors
All focus is on business and how it will impact everyone
2. Technology Integration (Technical Architecture):
We focus on the technical side of things, and how to ingest, store, and transform data. What happens when we have sudden orders spike?
You can read the complete blog here - https://lnkd.in/dqqkJvtE
This is my first post on my newsletter, consider subscribing for quality content :)
Tag someone who might find this helpful 👇🏻
#dataengineer #dataengineering #bigdata
Enterprise Solution Specialist: Advanced Analytics & Data
1wDatabricks is a great option