Akash Kamble’s Post

This is dam true. Stress doesn’t come from hard work. Stress primarily comes from not taking action over something that you can have some control over – Jeff Bezos

Happy to share that I completed Fundamentals of the Databricks Lakehouse Platform Accreditation.

The biggest skill in today's world is to learn new coming things and new ways to do things, and to unlearn old/known ways to do same.

This early morning, I came across this amazing podcast by FuelEd no YouTube. Arpit Bhayani was the reason I landed up there, because I am one of his students from Asli Engineering. Some important things I felt: 1. Problem solving (structure thinking) 2. Constant learner (learning attitude)! 3. Don't do things that lift the smile off your face. You should have at least two of these 3 P's. 1. Paisa. 2. Power (core competent) 3. Position (growth) Below are the things mentioned in the pod and in his JD for hiring, amazing! 1. Curiosity , 2. Bias for action, 3. Proof of work (serendipity can take you to places; the world needs more heterogeneous people than homogeneous). 4. Being interested in knowing, building, or founding. I am leaving link to pod in comment do listen to him.

Client Server Architecture. Client: Is one who is accessing the service. (Can be any device laptop, mobile or tab). Server: Is one who is providing service. (Present at Internet or in Network) Examples of client-server architecture include: ·       Web servers that deliver web pages to web browsers. ·       Email servers that store and manage email messages accessed by email clients.   Types of Architecture.   1-Tier- Architecture We say it when we have an application and a client on the same machine. For example, you have made a clone of Amazon in the database; the website is on the same machine, and you are using the same machine to access it.   2-Tier- Architecture In this kind of system, we have the client layer and the application layer. In this kind of architecture, we can have more than one client connecting to this application. And we have only one server, which contains the database and applications. This system is used for testing.   3-Tier- Architecture In a three-tier architecture, we have webservers, an application layer, and a database layer. Web servers take the request from the client and send it to the application layer. The application layer is one that contains the code of the application. The database layer is one that contains the database and stores the application data.        #Networking    #WebServers    #EmailServers    #DataManagement    #NetworkInfrastructure       #OneTier    #LocalMachine    #SingleMachine    #IntegratedSystem    #AmazonClone    #LocalServer       #TwoTier    #ClientApplication    #ServerApplication    #DatabaseServer    #TestingEnvironment    #ClientServerModel       #ThreeTier    #WebServers    #ApplicationLayer    #DatabaseLayer    #DataStorage    #CodeLogic    #RequestResponse    #Scalability    #ModularArchitecture

What are the major components of HDFS (Hadoop Distributed File System)? As HDFS uses a distributed computing model, it is made of two or more servers. In our case, let's make it a four-node cluster. So it will contain: 1) One Master Node2 ) Four slave nodes Lates talk about these nodes. 1)Master Node: It is a node using which the client will interact with the HDFS; it is made of high-standard hardware. The master node is also known as the name node. 2)Salve Nodes: This node contains actual data blocks, and this node is built of commodity hardware or low-standard hardware compared to the master node. The save node is also known as the data node. What is the block size in HDFS? In Hadoop 1.0, the default size of the data block was 64MB. In Hadoop 2.0, the default size of the data block is 128 MB. These blocks are really important when it comes to data distribution on slave nodes. Let's talk about them later. How data gets distributed in HDFS. Let's assume we have 1000 MB of files. 1000/128 = 7.8125, which will create 8 bocks as each block will be 128 MB. So each block will contain two data blocks. This will be done by the round-robbin policy.This is how data is stored in HDFS. Image Source:-waytoeasylearn.com    #Hadoop   #BigData   #DataScience   #DataAnalytics   #DataEngineering   #DistributedComputing   #HDFS    #MapReduce   #Spark   #DataProcessing   #DataStorage   #DataManagement   #HadoopEcosystem   #HadoopAdmin   #HadoopDeveloper    #MachineLearning   #AI   #DataMining   #DataArchitecture   #DataInfrastructure

What is Virtualization. ▪ Creating a virtual machine over existing operating system and hardware is referred as Virtualization. ▪ Virtual Machines provide an environment that is logically separated from the underlying hardware. ▪ The machine on which the virtual machine is created is known as host machine and virtual machine is referred as a guest machine. ▪ This virtual machine is managed by a software or firmware which is known as hypervisor. How was it Before Virtualization. ▪ Single OS image per machine ▪ Software and hardware tightly coupled ▪ Running multiple applications on same machine often creates conflict ▪ Underutilized resources ▪ Inflexible and costly infrastructure After Virtualization. ▪ Hardware-independence of operating system and applications ▪ Virtual machines can be provisioned to any system ▪ Can manage OS and application as a single unit by encapsulating them into virtual Two Virtualization Approaches ▪ Hosted Architecture Installs and runs as an application Relies on host OS for device support and physical resource management ▪ Bare Metal (Hypervisor) Architecture A bare-metal virtualization hypervisor does not require admins to install a server operating system first. Bare-metal virtualization means the hypervisor has direct access to hardware resources, which results in better performance, scalability and stability #virtualization #hypervisor #cloud #cloudinfrastructure #operatingsystem

What is Spark? Apache Spark is a general-purpose in-memory compute engine. In Hadoop clusters, we have HDFS is a storage unit. MapReduce is a processing unit/computer engine that can be replaced by Spark as it is a processing unit/computer engine. YARN is a resource manager. Spark is a plug-and-play compute engine. Use it with any storage system, including LOCAL STORAGE, HDFS, Amazon S3, and BLOB. Use it with any Resource Manager YARN or KUBERNETES. Spark is written in Scala. Scala runs on the JVM. Officially, Spark supports Scala, Java, Python, and R programming languages. #Hadoop #BigData #DataProcessing #DataStorage #HadoopEcosystem #YARN #MapReduce #HDFS#spark

What is Hadoop ? Hadoop is an open-source framework designed for distributed storage and processing of large volumes of data across clusters of commodity hardware. Hadoop Evolution In 2003 Google released a paper to describe how to store large datasets. This paper was called as GFS (Google File System). In 2004 Google released another paper to describe how to process large datasets.This paper was called as MapReduce. In 2006 Yahoo took these papers and Implemented it, The implimentation of GFS was named as HDFS (Hadoop Distributed File System), name was changed due to copyright issue, The implimentation of MapReduce was named as MapReduce (unchanged) . Hadoop 1.0 HDFS for distributed storage. MapReduce for distributed processing. In 2009 Hadoop came under Apache Software Foundation and became open source. In 2013 Apache released Hadoop 2.0 to provide major performance enhancements. Yarn was Introduced.   Hadoop 2.0 HDFS for distributed storage. MapReduce for distributed processing. YARN ( Yet Another Resource Negotiator) for Resource management.   Hadoop Ecosystem HIVE : Data warehouse tool built on top of Apache Hadoop for providing data query and analysis. SQOOP: A command-line interface application for transferring data between relational databases and Hadoop. HBASE: A column-oriented NoSQL database that runs on top of HDFS. PIG : A scripting language for data manipulation. Transforms unstructured data into structured format. SPARK: A distributed general purpose in-memory compute engine. OOZIE: A workflow scheduler system to manage Apache Hadoop jobs. #Hadoop #BigData #DataProcessing #DataStorage #HadoopEcosystem #YARN #MapReduce #HDFS          

Big Data System Requirements? There are some important things to consider: 1. Data Storage. 2. Data Process. 3. Scale. Data Storage: Store large amounts of data. Traditional systems are not fit to store such a huge amount of data. Data Process: Process a huge amount of data in an efficient and timely manner. Traditional systems are not capable of handling. Scale : Scale easily to accommodate growing requirements. Traditional systems have serious limitations. Ways to build a system. 1.Monolithic 2.Distributed Monolithic: Monolithic is a single, powerful server. It is difficult to add resources after a certain limit. Fixed resource or limitation to upgrade. Upgrading or updating a system does not mean better performance. With verticle scaling, we increase resources like RAM and storage. Distributed: Many servers come together to build distributed systems and act as a single machine. These servers are referred to as nodes. As it is built out of multiple machines, we can add and remove machines, which makes it scalable. This refers to linearly scalable. Over here, if you upgrade or update the system, you can see a performance improvement. Horizontal sacling refers to adding maxhices to a cluster. real scaling. With the above distinctions, you may have understood that all good big data systems are based on distributed architecture. #BigData #DataStorage #DataProcessing #Scale #Monolithic #Distributed #DataSystems #Architecture #Scalability