𝗥𝗲𝗺𝗼𝘁𝗲 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗦𝗲𝗿𝘃𝗶𝗰𝗲
Yesterday, we have seen how Uber handles peta-byte scale shuffle data on daily basis with the help of RSS. Today, let's dive into the architecture of RSS, its components, and how each component works.
🚀 Service Registry:
• Uber uses zookeeper as a service registry which maintains a long-running connection with each server and updates the status regularly. During Shuffle registering, spark driver retrieves all the available RSS servers from zookeeper.
• Post this, driver selects a server based on several factors such as Latency, shuffle partitions and active connections. After selecting, it passes the details of server to mapper and reducer within shuffle handle.
• During this process, if any instance of RSS, fail to send heartbeat signal, the registry will be automatically deregistered and loses its connection with service registry.
🚀 RSS Client:
• RSS Client implements the shuffle manager interface in which driver chooses the service based on factors mentioned above by querying the service registering, and gets the shuffle handle and retrieve the metadata, and share it with mapper and reducer.
• All mappers and reducers will send shuffle data to a single unique RSS server and retrieve data from same server instead of retrieving from multiple servers, as done generally. Instead of writing the data to a single server, it will replicate the data to other servers just to maintain fault tolerance.
🚀 Fault Tolerance
• Server Busy - If the clients cannot reach the RSS server, it exponentially backs off retry to avoid burdening the servers.
• RSS Server restart- The servers will recover their prior state after restarting by reading the shuffle data files onto the local disk. If map tasks fail under such circumstances, the spark application will try to reattempt these failed tasks with a new task attempt ID.
• Server Goes Down - Replication is useful when RSS servers are unavailable. The shuffle data can be served from more than one RSS server. Under such circumstances, if needed, the Spark driver re-schedules the affected mapper/reducer stages.
#datainfrastructure #dataengineering #kafka #learning
Distributed Systems | Scylla DB | Digital Transformation | E-commerce | SCM | SAAS | Products | Architecture | Engineering @Jio Embibe | WalmartLabs | GE Digital | Target | Times Internet | EdTech | Media | IIOT
2moUber Engineering Did you guys explore the ScyllaDB for efficient h/w utilisations be it ssd or nvms, high throughput , low latency and low bandwidth environments , this actually matters where networks are bad while booking or accepting the cabs ?