What components should you consider when designing a batch processing system?

In my experience, for designing a batch data pipeline, we need to gather the schedule time to trigger our pipeline . This is important because. Client may have different schedule to have fresh data in place every day, but that may not align with source team specially when it comes to on-prem databases such as mainframes, oracle etc. we need to sit with them and come to a conclusion so that its not causing contention and any other issues in their business hours. Majority of the time, its midnight to make it safe for both the parties. Second point is to get the connection details. Thirdly we can build our script which in ,my case was sqoop command to extract data and land into a staging area

While designing a batch processing data system the utmost important component to consider is the Data Source. Understanding the data source and identifying the connectivity for those sources is important, it could be RDBMS sources such as Oracle, SQL, AS400, SQL Server, DB2 etc or Flat files from any server. It is also important to understand the specific needs of the business and the users of the batch processing system. Few of the key factors to include while considering data source is the data format, schema, volume of data and access to these sources. Additionally, data source should be secure and protected from unauthorized access to maintain data governance and privacy.

When designing a batch processing system, focus on four key components: data integration (collecting data from various sources), data transformation (cleaning and structuring the data), job scheduling (sequencing and timing of batch jobs), and error handling/monitoring (identifying and addressing issues). These components ensure efficient and reliable data processing.

When it comes to building a data pipeline, the foremost component to consider is Data Source! 1. How do we access and extract the data? 2. What is the volume (size) of the data? 3. What is the Velocity (how frequent data are accumulated) of the data? 4. What is the Variety (structured, semi-structured, or unstructured) of the data? 5. What is the Veracity (quality of data) of the data? 6. Is there any PII (Personal Identifiable Information) or any sensitive data, and what security measures need to be in place to protect it during extraction? By identifying answers for these questions you can significantly lay a foundation for your data pipeline project without any hassles!

Data source plays an important role when designing a batch-processing system. Below are the main considerations: 1) What is the frequency of data loaded/refreshed in the source system and when it is available (time window) for the downstream system to extract the data? 2) How to extract delta records after the last/previous extraction, i.e., based on the timestamp column in the source or any other mechanism? 3) Should a push or pull approach be used to obtain the data from the source? 4) What is the data volume during each extraction, and are there any spikes expected during month-end or year-end periods?

Here is a non exhaustive list of what you need to consider for the storage layer, in no particular order. 1. Schemaless or schema conformant 2. Format used, e.g. parquet 3. Relationships used: relational, graph, a wide table, etc 4. Sensitivity of the data, i.e. PCI, anonymized, etc. 5. NFRs, i.e. resiliency, performance, availability, backup requirements 6. Locality restrictions: e.g. can this hold data of a different jurisdiction 7. Access control mechanisms and integration into your own 8. Does this lock it down to a specific technology 9. Latency e.g. parquet needs a lot of records to be accumulated first before being accessible 10. Access technology and tools 11. (Perhaps the most important) eventually or immediately consistent

Once we understand the the data sources for a batch processing system, understanding the target system to store the data is important. Majorly data storage in the current data engineering batch processing includes - - Hadoop Distributed File System (HDFS) - Amazon Simple Storage Service (S3) - Azure Blob Storage - Google Cloud Storage Few tips for optimizing data storage for batch processing systems: - Storing data in compressed file format - Data Partitioning - Using a data warehouse - Having a distributed file system Carefully selecting a data storage system, businesses can ensure that their batch processing systems are reliable and efficient to process large data.

Selecting the proper data storage is crucial for batch processing system efficiency. 1) Whether data storage is going to be cloud-based (AWS S3, Azure Blob, Google Cloud Storage) or on-premises file-based storage ? 2) One need to select the appropriate storage tier depending on usage requirements and expected query performance. For example, if you choose AWS S3, you need to select S3 Standard Tier for normal access, S3 Infrequent Access for data that is not accessed in real-time, and Glacier for archival data, which helps in reducing latency and storage/access costs. 3) Does the storage support partitioning? This can improve the performance of batch processing by allowing partitioning and parallel processing.

Choosing the right format for data storage is crucial and it can be chosen based on the usecase. There have been significant developments made in this area and we have data formats like Delta lake, Hudi that support ACID transactions, versioning and so. These file formats can be preferred to get advantages of lakehouse capabilities(data lake + data warehouse)

Designing a batch processing system requires careful planning and consideration of these components to meet business objectives, ensure data accuracy, and maintain system reliability and performance. It's also important to adapt the design to the specific needs and constraints of your organization.

Below are points to consider while designing data processing in batch systems: 1) What is the frequency of batch execution? (Daily, hourly, once every 4 hours, etc.) This consideration ensures that data can be processed promptly according to SLAs without overloading the source system or processing environment. 2)Implement efficient re-startability mechanism so that in case of any batch processing failure, it can be restarted without any significant re-processing, saving time and resources. 3) Consider how the design can scale in the long run in case there is an increase in data volume.

Always check with the business about the needed frequency. Sometimes, they mask "real-time" or "near real-time" as hourly or sometimes even four times a day. However, when designing the processing framework or script, be sure that you can adjust it to make data processing frequency adjustable based on the situation easily (no major adjustments to code should be done).

Batch processing is suitable for processing large volumes of data in fixed intervals, such as daily or weekly. It offers scalability for big data applications and provides reliability and fault tolerance. Batch processing is not designed for real-time processing and introduces latency. It is well-suited for complex data processing tasks that require multiple steps and transformations. Examples of applications well-suited for batch processing: - Sales Analysis - Customer Segmentation - Log Analysis Examples of applications where batch processing may not be well-suited: - Fraud Detection - Real-time Personalized Recommendations - Sensor Data Analysis

A short step by step approach to build a data pipeline for batch processing systems: - Define Data Sources: Identify data sources (e.g., databases, APIs). - Extract Data: Use tools like Apache NiFi or cloud services (e.g., AWS Glue) to fetch data on a schedule. - Transform Data: Apply necessary cleaning or aggregation. - Load Data: Store transformed data in target systems (e.g., data warehouses). - Schedule and Monitor: Set regular execution times, monitor for issues, and alert when needed. - Modular Design: Create a reusable, modular design for scalability and consistency. Thoroughly test and validation of the pipeline for data quality and Error Handling is important for recovery mechanisms for reliability.

One major point to focus on is, if your pipelines are majority dependent on external system behavior, you need to build flexible replayable pipelines in case of failure.

I like to think of each data source as the beginning of a new road. With each downstream table or ETL process, the network of roads becomes increasingly complex where its easy to get lost. Having insight to the lineage of each final dataset is valuable information needed to debug quality issues. Setting breakpoints in code is straightforward, but "breakpoints" in a multi-step etl pipeline requires knowledge of the DQ status at each stage in this lineage. Tracking that as its own dataset should be part of early Batch pipeline designs.

Data quality(DQ) is one of the vital blocks of any data processing technique. Often it is considered as an expensive and rather an unnecessary process when the data migration is of core importance or DQ is only considered to be in place when a master data system is built, but it is way beyond this myth as any data pipeline is not complete w/o DQ in place. In many legacy systems where data migration or visualization is the main goal, end data does not fit the purpose if it does not qualify as good data. In DQ the standards and cleansing takes a major role. Maintaining quality dimensions as per international standards, doing data remediation of bad quality data, continuous monitoring & reporting all contribute to overall governance process.

Data Consumption is arguebly the most crucial component in the data workflow because it's where data provides value. Data consumption should often be tailored to the individual needs. A Marketing Manager will need a different kind of dashboard compared to a finance manager. These are due to the operations they perform with the data. Hence, it is important for data professionals to master multiple data consumption formats like dashboards, reports, charts, APIs or even raw tables, and integrate them smoothly with the existing data architecture. A wise effort is creating frameworks around implementing the above formats. These data components should also be regularly maintained, secured, and evaluated if best practices are in place.

Additional to the other points, the below factors also need to be planned for the batch processing data pipelines: -Error Handling: Detect and manage data pipeline issues by logging errors and implementing actions like retries or alerts. -Metadata Management: Catalog and document processed data, including lineage, dictionaries, and transformation information. -Data Documentation: Maintain clear documentation for the pipeline, including job descriptions, transformation logic, and data schemas. -Backup and Recovery: Safeguard data integrity with backup and recovery procedures for system failures or data corruption. -Performance Optimization: Continuously enhance system efficiency by monitoring and optimizing resource usage.

In addition to the components addressed in the article, consider: - Data Lineage: Implement data lineage tracking to trace the origin and transformations of data, aiding in troubleshooting and compliance. - Workflow and Scheduling: Design workflow processes for complex data pipelines, scheduling jobs for optimal resource utilisation and time efficiency. - Malware Scanning: For external data sources, incorporate malware scanning to ensure data integrity and security. - Retention and Archival: Define data retention policies and archival procedures to manage data lifecycle and storage costs.

Please talk and collaborate more with the backend team to help make the data processing easier. Sometimes it is just adding the timestamping or enriching the data to avoid nasty parsing from different data streams.