Data Engineer Things’ Post

If you're interested in learning how to ingest data from Hudi incrementally into Postgres using Spark, you're in the right place! We've prepared a detailed guide and exercises to help you understand and implement this process effectively. Exercises files: https://lnkd.in/eeTkdbzs If you're curious about how to fetch Hudi commit time, check out our blog post detailing four different ways to fetch Apache Hudi commit time using Python and PySpark. 4 Different Ways to Fetch Apache Hudi Commit Time in Python and PySpark:  https://lnkd.in/dXq3-bbC For those who want to implement a landing and staging area, we have additional resources to support your efforts. Incremental Processing Pipeline to Power Aurora Postgres SQL from Hudi Transactional Datalake: Exercises Notebook: https://lnkd.in/eHw9FYGJ Video: https://lnkd.in/d4kuNJHV Explore these resources to deepen your understanding and enhance your skills in managing Hudi data ingestion and processing with Postgres and Spark! Apache Hudi

The new Spark Connect feature introduced since Apache Spark 3.4 provides the means to run Spark code on a remote Spark Connect server using the Data Frame API and unresolved execution plans over gRPC. This very neat feature comes with its own set of limitations when using Delta Tables with the Python API, which is currently not supported. In the following article I explore a workaround to these limitations by embedding Spark SQL directives within the PySpark code, enabling users to perform Delta Table operations like Optimize, Vacuum and Merge using PySpark. #Spark #SparkConnect #DeltaLake #PySpark #SparkSQL

Apache Spark offers two primary methods for performing aggregations: Hash-based aggregation and Sort-based aggregation. These methods are optimized for different scenarios and have unique performance characteristics. Hash-based Aggregation: Hash-based aggregation, implemented by HashAggregateExec in Spark SQL, is the preferred method when conditions allow. This technique creates a hash table where each entry corresponds to a unique group key. As Spark processes the rows, it uses the group key to quickly locate the corresponding entry in the hash table and updates the aggregate values. This method is generally faster because it avoids sorting the data prior to aggregation. However, it requires that all intermediate aggregate values fit into memory. If the dataset is too large or there are too many unique keys, Spark may not use hash-based aggregation due to memory constraints. Key Points: - Preferred when aggregate functions and group-by keys are supported by the hash aggregation strategy. - Generally faster than sort-based aggregation, as it avoids data sorting. - Utilizes off-heap memory to store the aggregation map. - May fall back to sort-based aggregation if the dataset is too large or has too many unique keys, leading to memory pressure. Sort-based Aggregation: Sort-based aggregation, implemented by SortAggregateExec, is used when hash-based aggregation is not feasible, either due to memory constraints or because the aggregation functions or group-by keys are not supported by the hash aggregation strategy. This method involves sorting the data based on the group-by keys and then processing the sorted data to compute aggregate values. While this approach can handle larger datasets, as it only requires intermediate results to fit into memory, it is generally slower than hash-based aggregation due to the additional sorting step. Key Points: - Used when hash-based aggregation is not feasible due to memory constraints or unsupported aggregation functions or group-by keys. - Involves sorting the data based on the group-by keys before performing aggregation. - Can handle larger datasets, as it streams data through disk and memory. In summary, hash-based aggregation is preferred for its speed, provided the dataset can fit into memory and the aggregation functions are supported. In contrast, sort-based aggregation is a fallback option when dealing with larger datasets or unsupported functions, though it is generally slower due to the sorting step. #data #dataengineer #dataengineering #spark #apachespark #sql #python

Hello LinkedIn Community, Let's Learn about 𝗽𝘆𝘀𝗽𝗮𝗿𝗸.𝘀𝗾𝗹.𝘂𝗱𝗳.𝗨𝘀𝗲𝗿𝗗𝗲𝗳𝗶𝗻𝗲𝗱𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 ❇️pyspark.sql.udf.UserDefinedFunction is a class in PySpark, the Python API for Apache Spark, a powerful distributed computing system. This class is used to define user-defined functions (UDFs) in PySpark. ❇️UDFs allow you to apply custom Python functions to columns in Spark DataFrames. This can be useful when the built-in Spark SQL functions are insufficient for your data processing needs. 🔶𝗞𝗲𝘆 𝗣𝗼𝗶𝗻𝘁𝘀 𝗮𝗯𝗼𝘂𝘁 𝗽𝘆𝘀𝗽𝗮𝗿𝗸.𝘀𝗾𝗹.𝘂𝗱𝗳.𝗨𝘀𝗲𝗿𝗗𝗲𝗳𝗶𝗻𝗲𝗱𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻: 🔻Class: It's an internal class within the pyspark.sql.udf module. You don't typically create instances of this class directly. 🔻Creation: Use the pyspark.sql.functions.udf() function to create a UDF object from your Python function. 🔻Arguments: 🔹python_func: The Python function you want to convert into a UDF. 🔹returnType (optional): The expected data type of the UDF's return value. If not specified, defaults to StringType. Follow Shakti Singh Baghel Dharmendra Singh Chandresh Desai Brij kishore Pandey Aurimas Griciūnas Sarika Sinha Archy Gupta #dataintegration #datapipelines #dataarchitecture #datawarehousing #datascience #analytics #dataquality #cloudcomputing #machinelearning #ai #datainfrastructure #dataintegration #datapipelines

Single Node Lakehouse. Lakehouse table formats are predominantly distributed in nature, designed to handle large-scale, complex data workloads. This typically requires setting up clusters with Spark, JVM, and other configurations to manage data in underlying storage systems. But not all workloads requires such setup & dependencies. Sometimes, as an data analyst/scientist you just want to access a dataset (let's say an Apache Hudi table) & carry on with your analysis. Direct access to lakehouse tables for these use cases reduces the ‘wait time’ for data stakeholders, ultimately speeding up the time-to-insight process. With this in mind, the Hudi community has released "hudi-rs". It is a native Rust library for Apache Hudi with Python bindings. "hudi-rs" opens up new possibilities for working with Hudi across a range of use cases that typically don’t require distributed processing. Want to learn more or get involved with the project? Shared the blog in comments! #dataengineering #softwareengineering

Pandas vs Spark (DATAFRAME) Dataframe represents a table of data with rows and columns, Dataframe concepts never change in any Programming language, however, Spark Dataframe and Pandas Dataframe are quite different. In this article, we are going to see the difference between Spark dataframe and Pandas Dataframe. Pandas DataFrame Pandas is an open-source Python library based on the NumPy library. It’s a Python package that lets you manipulate numerical data and time series using a variety of data structures and operations. It is primarily used to make data import and analysis considerably easier. Pandas DataFrame is a potentially heterogeneous two-dimensional size-mutable tabular data structure with labeled axes (rows and columns). The data, rows, and columns are the three main components of a Pandas DataFrame. Advantages: ·      Pandas Dataframe able to Data Manipulation such as indexing, renaming, sorting, merging data frame. ·      Updating, adding, and deleting columns are quite easier using Pandas. ·      Pandas Dataframe supports multiple file formats ·      Processing Time is too high due to the inbuilt function. Disadvantages: ·      Manipulation becomes complex while we use a Huge dataset. ·      Processing time can be slow during manipulation. Spark DataFrame Spark is a system for cluster computing. When compared to other cluster computing systems (such as Hadoop), it is faster. It has Python, Scala, and Java high-level APIs. In Spark, writing parallel jobs is simple. Spark is the most active Apache project at the moment, processing a large number of datasets. Spark is written in Scala and provides API in Python, Scala, Java, and R. In Spark, DataFrames are distributed data collections that are organized into rows and columns. Each column in a DataFrame is given a name and a type. Advantages: ·      Spark carry easy to use API for operation large dataset. ·      It not only supports ‘MAP’ and ‘reduce’, Machine learning (ML), Graph algorithms, Streaming data, SQL queries, etc. ·      Spark uses in-memory(RAM) for computation. ·      It offers 80 high-level operators to develop parallel applications. Disadvantages: ·      No automatic optimization process ·      Very few Algorithms. ·      Small Files Issue #azuredataengineer #azuredatafactory #azuredatabricks #apachespark #pandas #dataengineering #spark

🌟Day 63 : Unlocking the Power of Apache Spark with Python API 🌟#FabricDairies101 If you’re dealing with massive datasets and want to process them efficiently, Apache Spark is the game-changer. But what makes it even better? The Python API, which brings the power of distributed computing into the hands of every data engineer with a few lines of code. Let’s break it down: ✨ Spark 101: Think of Spark as your data engine that can rev up computations at scale. When datasets get too large for single machines, Spark distributes the load across multiple nodes like an army of assistants working in parallel, each handling a piece of the puzzle. 🚀 🛠️ Resilient Distributed Datasets (RDDs): This is Spark’s core. RDDs are your reliable, immutable data containers that can survive any mishap. Even if a node crashes, your data remains intact. It’s like having a photocopy of every critical document—no matter what happens, you’ve got backups. 📂🔄 📊 DataFrames: If RDDs are raw ingredients, DataFrames are those ingredients perfectly prepped and ready to be cooked. They’re SQL-like and optimized for large-scale operations, making them faster and easier to use. 🍳✨ 🎯 Transformations vs. Actions: Here’s the trick with Spark: Transformations (like map() or filter()) are lazy—nothing happens until you trigger an action (like count() or collect()). It’s like preparing your shopping list but only heading to the store when you decide it’s time to cook. 🛒🍽️ 💡 Lazy Evaluation: Spark doesn’t execute transformations until necessary. It’s optimizing under the hood to save on computational costs. Imagine your kitchen preparing all ingredients based on your recipe, but only turning on the stove when it’s time to serve. 🔥🍲 If you're working in data engineering, knowing Spark is a must, and using Python makes it even more intuitive. No matter the scale of your data, Spark has the power to handle it. For official documentation please refer : https://lnkd.in/gArjXkqf Got questions? Let’s chat. 😉 #ApacheSpark #DataEngineering #BigData #PythonAPI #DistributedComputing #DataScience #MachineLearning #DataFrames #ETL #SparkSQL #Day63

🚀 Excited to share some insights on Directed Acyclic Graphs (DAG) and Lazy Evaluation in Apache Spark! 📊 In Apache Spark, the concept of Directed Acyclic Graphs (DAG) is fundamental to understanding the execution of computations. DAG represents the logical execution plan of transformations on RDDs (Resilient Distributed Datasets) or DataFrames. Each node in the DAG represents a task, and edges represent dependencies between tasks. Lazy evaluation in Spark means that transformations on RDDs or DataFrames are not computed immediately. Instead, Spark builds up a DAG of the transformations and only performs the actual computation when an action is called. This lazy evaluation allows Spark to optimize the execution plan by combining multiple transformations and minimizing unnecessary computation. Let's explore an example in PySpark to illustrate these concepts: from pyspark.sql import SparkSession spark = SparkSession.builder \ .appName("DAG and Lazy Evaluation Example") \ .getOrCreate() df = spark.read.csv("data.csv", header=True) word_counts = df.groupBy("word").count() word_counts.show() In this example, the transformations (groupBy and count) are not immediately executed. Instead, Spark builds a DAG to represent the computation plan. The actual computation is triggered when the show() action is called. #BigData #PySpark #DataEngineering #DataScience #DistributedComputing #Python #Hadoop #DataAnalytics #MachineLearning #AI #DataProcessing #Programming #Developer #Tech #Analytics #CloudComputing #DataManagement #DataMining #ParallelComputing #DAG #ApacheSpark #LazyEvaluation

#PySpark PySpark is a Python library that provides a framework for processing big data using Apache Spark. DataFrames are distributed data collections arranged into rows and columns in PySpark. Schema represents the structure of the DataFrame or Dataset.  createDataFrame() is a method used to create a DataFrame. df.show() is a method used to display the contents of a DataFrame in a tabular format. df.printSchema() method is used to display the schema of a DataFrame. Code from pyspark.sql.types import StructType, StructField, StringType, IntegerType data=[(1,'AB'), (2,'CD')] schema= StructType([StructField(name='Id',dataType=IntegerType()),StructField(name='name',dataType=StringType())]) df=spark.createDataFrame(data=data,schema=schema) df.show() df.printSchema() Output: +---+----+ | Id|name| +---+----+ |  1|  AB| |  2|  CD| +---+----+ root  |-- Id: integer (nullable = true)  |-- name: string (nullable = true) #PySpark #Databricks #DataEngineering

This set of questions is mostly for the PySpark Interview, it is a must to know the concept related to it.