Apache Spark is an open-source data processing engine to store and process data in real-time across various clusters of computers using simple programming constructs. It processes real-time data from Kafka, Flume, Amazon Kinesis, etc. Apache Spark discretized also known as Spark DStream which represents a stream of data divided into small batches.
It supports various programming languages. Also, developers and data scientists incorporate spark into their applications or build spark-based applications to process, rapidly query, analyze, and transform data at scale.
Need for Spark Streaming
- Hadoop is way slower.
- Unification of streaming, batch and interactive workloads.
- Advanced analytics with ML and spark SQL APIs.
- Load balancing.
What is Streaming?
Data streaming is a technique for transferring data so that it can be processed as a steady and continuous stream.
Streaming technologies are becoming increasing important with the growth of the internet.
Why Spark Streaming?
Spark streaming is used to stream real-time data from various sources like Twitter, Stock Market and Geographical System and perform powerful analytics to help businesses.
Spark Streaming Sources
- File system
- Socket Connection
- Kafka
- Flume
- Kinesis
Streaming Context
- Consumes a stream of data in Spark.
- Registers an InputDStream to produce a Receiver object.
- It is the main entry point for Spark functionality.
- Spark provides a number of default implementations of sources like Twitter, Akka Actor and ZeroMQ that are accessible from the context.
Streaming Context – Initialization
- A StreamingContext object can be created from a SparkContext object.
- A SparkContext represents the connection to a spark cluster and can be used to create RDDs, accumulators and broadcast variables on that cluster.
DStream
- Discretized stream (DStream) is the basic abstraction provided by Spark Streaming.
- It is a continuous stream of data.
- It is received from source or from a processed data stream generated by transforming the input stream.
- Internally, a DStream is represented by a continuous series of RDDs and each RDD contains data from a certain interval.
DStream Operation
Any operation applied on a DStream translation to operations on the underlying RDDs.
For example, in the example of converting a stream of lines to words, the flatMap operation is applied on each RDD in the lines DStream to generate the RDDs of the words DStream.
Hadoop Vs Spark
| Hadoop | Spark |
| Processing data MapReduce in Hadoop is slow. | Spark processes data 100 times faster than MapReduce as it is done in memory. |
| Perform batch processing of data, | Perform both batch processing and real-time processing of data. |
| Hadoop has more lines of code, since it is written in Java it takes more time to execute. | Spark has fewer lines of code as it is implemented in Scala. |
| Hadoop supports Kerberos authentication, which is difficult to manage. | Spark supports authentication via a shared secret. It can also run on YARN leveraging the capability of Kerberos. |
Features of Apache Spark
Fast Processing: Spark contains a resilience distributed database (RDD) which saves time taken in reading and writing operations, and hence it runs almost ten to a hundred times faster than Hadoop.
In-memory Computing: In Spark, data is stored in the RAM, so it can access the data quickly and accelerate the speed of analytics.
Flexible: Spark supports multiple languages and allows developers to write applications in Java, Scala, R, and Python.
Fault Tolerance: Spark contains a resilience distributed database (RDD) that is designed to handle the failure of any worker node in the cluster. Thus, it ensures that the loss of data reduces to zero.
Better Analytics: Spark has a rich set of SQL queries, machine learning algorithms, complex analytics etc. With all these functionalities analytics can perform better.
Components of Apache Spark
The components of Apache Spark are as follows:
- Spark Core
- MLlib
- Spark Streaming
- Graph X
- Spark SQL
Spark Core
The Spark Core is a base engine for large-scale parallel and distributed data processing.
It is responsible for:
- Memory Management
- Fault recovery
- Scheduling, distributing and monitoring jobs on a cluster.
- Interacting with the storage system.
Comments are closed.