Spark

I’m pretty fast in my tasks.

Source — Google Images

1. Spark was founded by Bill Chambers and Matei Zaharia.
2. Spark is an in-memory processing framework.
3. It uses the RAM (majority) of the cluster and processes the programs much faster.
4. It is 100 times faster than Map Reduce programs.
5. It is open source.
6. It is handled by Apache Software Foundation.
7. It’s native language is Scala but it also supports Java, R, Python & SQL.
8. It came with Hadoop v2 in 2012.

#) HISTORY

1. In 2009, the R&D team at university Berkley were working on ‘AMPLab’ . The name given to the project was MESOS.
2. In 2009, there was no YARN because it came with Hadoop v2 in 2012. So, there was no cluster manager. So, they came up with MESOS which is a cluster manager.
3. YARN is very similar to MESOS.
4. So, the team at Berkley started testing MESOS with Map Reduce Programs which were quite slow at that time.
5. So, they wanted to create a new framework to test MESOS.
6. AND, they created SPARK which would process the programs in-memory.
7. In 2012, the project was taken over by Apache.
8. It got famous in 2015 because of it’s fast nature.
9. Companies use 1.6 version of it as it’s very stable but the latest version is 3.3.0 .
10. After v1.6, there was direct v2.0 .
11. In 2012, the founders of SPARK created DATABRICKS which commercially distributes spark.
12. If you want just SPARK then, DATABRICKS is the best.
13. SPARK is an independent project which doesn’t require HADOOP. But, mostly you’ll see it running on top of HADOOP.
14. There are currently more than 10 SQL engines that can run on top of HADOOP.

#) PRESENT LANDSCAPE

Landscape illustration

1. The biggest USP of SPARK is that it is Unified Engine Processing. What 50+ tools are doing on top of HADOOP, SPARK can do alone.

2. Speed is a by-product of SPARK.

3. SPARK consists of Scheduling, Distributing and Monitoring. Spark can use both RAM and Hard Disk.

4. Spark supports Scala, Java, Python and R.

5. Spark 2.0 has an abstraction layer which makes it possible to run these many languages.

#) MODES OF SPARK

1. LOCAL — Running it in local market. Using only for development and testing function.
2. MESOS — Spark was designed on top of MESOS.
3. STANDALONE — If you want to run Spark when you don’t have Yarn installed. Then, spark gives you it’s own cluster manager called STANDALONE.
4. YARN — Most of the companies will be running YARN on it’s cluster.

#) DETAILS

1. Spark has no storage because it’s an execution engine. There is no storage component i.e. you have to provide data somewhere.
2. That means if you are running spark on Hadoop then HDFS has the data.
3. If it is running in local machine then your file system has the data.
4. Spark can also run on KUBERNETES. Kubernetes is container orchestration that means data center manager. In data centers, you can run multiple DOCKER containers. You can manage all of them using Kubernetes.
5. In cloud, you can have multiple options. Let’s say you have to go to AWS you run multiple programs using EMR. Elastic Map Reduce is very fast. It can create a 100–200 machine node cluster in 5–10 minutes. These are disposable clusters i.e. once you have done your job, you have to delete them because you are paying for it.

#) ZOOKEEPER

Zookeeper illustration

1. Spark supports Zookeeper.
2. Zookeeper manages clusters & keeps a list of them.
3. It is a service coordinator.
4. By-default, we’ll be having active and passive namenode.
5. Passive namenode is also known as STANDBY namenode.
6. If, let’s say the active namenode crashes then standby namenode will inform zookeeper and will say now I’m the ACTIVE namenode.

#) READING DATA

1. Spark can read data from almost any file system. Eg:- HDFS, AWS S3, LOCAL.
2. It can read from any NoSQL database & RDMS.
3. If you have MySQL Database or MongoDb it can directly access the data and process it & store it back into the table.
4. It can work with FLUME and KAFKA for high availability of data.

Illustration

#) SPARK ECOSYSTEM

Spark Ecosystem

1. SPARK SQL — In Spark SQL, we can create tables and dataframe. Spark SQL is integrated with HIVE by-default. Let’s say, If I have a Hadoop cluster and I’m running SPARK on top of it, then if I open Spark, I can retrieve data from HIVE. Hive is very slow. So, you can read the Hive tables in Spark as query. These days, Hive is just used for storage &most of the processing is done by Spark.
2. GraphX — It is used for processing graph. it is important because the data stored in social media sites are stored in Graph format. Representation using graph is easy.
3. SPARK STREAMING — It is used for processing real-time streaming data. Even, if your machines are not working your data still will be arriving because of FLUME & KAFKA.

Streaming illustration

#) SPARK PROGRAM

1. DRIVER — When we are running a spark program there will be something known as Driver. It is the master of program i.e. the program runs on master machine in the cluster. It declares transformations and actions in RDD. Once, action is performed on RDD, Spark Context gives the program to the driver.
2. EXECUTOR — It’s slave of the program. Whenever a spark program is run, there will be Driver & Executor. And, when we’ll say LOCAL, a JVM will be created and it’ll have both Driver and Executor. It runs computations and store data on worker nodes.

Illustration

A Yarn container will be allocated. It’s not very efficient as you’ll be given only 1 container in LOCAL MODE. If you’ll run a python code, it will understand it’s logic and will run it inside JVM.

LOCAL MODE is only for testing and development.

Let’s say, I’m running a cluster and have 4 datanodes and I write a spark program. Then,

Illustration

It’s not easy to write a spark program as you have to mention RAM, processor cores & executors.

#) RDD

1. It stands for Resilient Distributed Datasets.
2. They represent your data in RAM.
3. They run parallelly and are fault-tolerant in nature.
4. RDDs are distributed and immutable.

RDD Illustration

#) DAG

1. It stands for Directed Acyclic Graph.
2. It consists of vertices and edges.
3. Vertices represent RDDs and edges represent operations applied to RDDs.
4. This graph is unidirectional i.e. it has only one flow.
5. It converts logical execution to physical execution.

#) SPARK DATAFRAMES

1. Spark Dataframes are distributed collection of datasets organized into columns similar to SQL.
2. It can be created from an array of data from DBs, existing RDDs, Hive Tables etc.
3. They support many data formats such as CSV, Avro, Elastic search etc.
4. It can process large data ranging from KB to PB on a single mode to large clusters.

#) SPARK DATASET

1. Spark Dataset is a data structure in Spark SQL which is strongly typed and is map to relational schema.
2. Datasets have combined powers of both RDD and Dataframes.

#) CHECKPOINT

1. Apache Spark provides an API for adding and managing checkpoints.
2. Checkpointing is the process of making streaming applications resilient to failures.
3. It allows to save data and metadata into a checkpoint directory.
4. In case of failure, it can recover this data & start from whenever it has stopped.

#) TRANSFORMATIONS

1. Transformations are the functions that are applied on RDD that help in creating another RDD.
2. It does not occur until an action takes place.
3. Eg:- map() and filter()
4. map() — Map function iterates over every line in RDD and splits into a new RDD.
5. filter() — Filter function creates a new RDD by selecting elements from current RDD that passes the function argument.

#) ACTIONS

1. Actions help in bringing back the data from a RDD to local machine.
2. Eg:- reduce() and take()
3. reduce() — It’s an action that is applied repeatedly until one value is left.
4. take() — It’s an action that takes into consideration all value from RDD to the local node.