History of Apache Spark<\/h2>\n
Spark began life in 2009 as a research project inside UC Berkeley’s AMPLab. It was released as open source in 2010 under the BSD license, then donated to the Apache Software Foundation in 2013, becoming a top level Apache project in 2014. Development has continued steadily since, and Spark is now in its fourth major version line, still actively maintained by the Apache community.<\/p>\n
Why Spark Was Needed<\/h2>\n
Before Spark, different jobs typically needed different specialized tools. Hadoop MapReduce handled batch processing. Apache Storm or S4 handled stream processing. Apache Impala or Tez handled interactive queries. Neo4j or Apache Giraph handled graph processing. No single engine covered all of these well, and none of them offered fast, in memory processing with sub second response times.<\/p>\n
Spark’s pitch was to be that one engine, real time streaming, interactive queries, graph processing, and batch processing, all handled through in memory processing that runs meaningfully faster than reading and writing to disk at every step. That combination, plus a consistent interface across all of it, is what set Spark apart from Hadoop and from single purpose tools like Storm.<\/p>\n
Components of Apache Spark<\/h2>\n
![\"Components](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/04\/1-25.png\")
<\/p>\n
- \n
- <\/b>Spark Core, <\/b><\/strong>the foundation everything else runs on top of, providing the execution platform for every Spark application.<\/li>\n
- <\/b>Spark SQL, <\/b><\/strong>which lets you run SQL and HiveQL queries against structured and semi structured data, often meaningfully faster than the same queries running unmodified on older systems.<\/li>\n
- <\/b>Spark Streaming, <\/b><\/strong>which processes live data by breaking it into small micro batches that run on top of Spark Core.<\/li>\n
- <\/b>MLlib, <\/b><\/strong>Spark’s machine learning library, built to take advantage of in memory processing for iterative algorithms that would otherwise be slow.<\/li>\n
- <\/b>GraphX, <\/b><\/strong>a graph computation engine for processing graph structured data at scale.<\/li>\n
- SparkR, <\/b><\/strong>a lightweight front end that lets R users analyze large datasets and run jobs interactively from the R shell, combining R’s usability with Spark’s scalability.<\/li>\n<\/ul>\n
Apache Spark architecture overview<\/h2>\n
![\"apache](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/04\/2-13.png\")
<\/p>\nSpark has a layered architecture where its components stay loosely coupled, which is what makes it straightforward to extend with additional libraries. The architecture rests on two core abstractions: the Resilient Distributed Dataset, or RDD, and the Directed Acyclic Graph or DAG, which represents the sequence of steps in a computation.<\/p>\n
How Apache Spark Works\u00a0<\/strong><\/h2>\n
![\"How](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/04\/3-11.png\")
<\/p>\nLike Hadoop MapReduce, Spark distributes data across a cluster and processes it in parallel. It uses a master and worker style architecture, one central coordinator and many distributed workers. That central coordinator is called the driver<\/b><\/strong>.<\/p>\n
Apache Spark Terminology<\/h2>\n
Spark Context<\/h3>\n
The Spark Context is the entry point of a Spark application. It connects to the Spark execution environment and is used to create RDDs, accumulators, and broadcast variables, access Spark services, and run jobs. Its responsibilities include:<\/p>\n
- \n
- Checking the current status of a Spark application<\/li>\n
- Canceling a job or a stage<\/li>\n
- Running a job synchronously or asynchronously<\/li>\n
- Accessing or unpersisting a cached RDD<\/li>\n
- Programmable dynamic allocation of resources<\/li>\n<\/ul>\n
Spark Shell<\/h3>\n
The Spark Shell is a Spark application written in Scala that gives you a command line environment with auto completion, useful for getting familiar with Spark’s features before writing a standalone application of your own.<\/p>\n
Spark Application<\/h3>\n
A Spark application is a self contained computation that runs user supplied code to produce a result. It can have processes running on its behalf even when it is not actively running a job.<\/p>\n
Task, Job, and Stage<\/h3>\n
- \n
- <\/b>Task, <\/b><\/strong>the smallest unit of work, sent to a single executor. Each partition of an RDD gets its own task.<\/li>\n
- <\/b>Job, <\/b><\/strong>a parallel computation made up of multiple tasks, created in response to an action.<\/li>\n
- <\/b>Stage, <\/b><\/strong>a set of tasks within a job that depend on each other. A job is split into stages because not every computation can happen in one continuous pass, some steps have to wait on others to finish first.<\/li>\n<\/ul>\n
Spark Runtime Architecture<\/h2>\n
The Driver<\/h3>\n
The driver runs the main method of your program. It is the process that creates RDDs, performs transformations and actions on them, and creates the Spark Context. Launching the Spark Shell is itself an example of starting a driver program, and the application ends once the driver terminates.<\/p>\n
The driver splits a Spark application into tasks and schedules them onto executors, through a task scheduler that lives inside the driver itself. Its two key jobs are converting your program into tasks, and scheduling those tasks on executors.<\/p>\n
At a high level, a Spark program takes some input data as an RDD, derives new RDDs from it through transformations, then runs an action to actually compute a result. This builds up a DAG of operations implicitly as you write your code, and when the driver runs, it converts that DAG into an actual physical execution plan.<\/p>\n
The Cluster Manager<\/h3>\n
Spark relies on a cluster manager to launch executors, and in some setups, to launch the driver as well. The cluster manager is a pluggable component, jobs and actions within a Spark application are scheduled by Spark’s own scheduler, by default in FIFO order, though round robin scheduling is also available. Applications can grow or shrink their resource usage dynamically based on workload, freeing resources when idle and requesting more again when demand returns. Spark currently supports this dynamic allocation across standalone mode, YARN, and Kubernetes. Apache Mesos was a supported option in earlier Spark versions, but that support was deprecated in Spark 3.2 and fully removed in Spark 4.0.<\/p>\n
Executors<\/h3>\n
Executors run the individual tasks that make up a Spark job. They launch once at the start of an application and stay alive for its entire lifetime, and if one executor fails, the application can generally continue without stopping entirely. Executors have two main jobs: running the tasks that make up the application and returning results to the driver, and providing in memory storage for any RDDs the user has chosen to cache.<\/p>\n
Resilient Distributed Dataset, RDD<\/h2>\n
![\"Resilient](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/04\/4-10.png\")
<\/p>\nRDDs are\u00a0the building blocks of any Spark application. RDDs Stands for:<\/p>\n
- \n
- Resilient:<\/em><\/strong>\u00a0Fault tolerant and is capable of rebuilding data on failure<\/li>\n
- Distributed:<\/em><\/strong>\u00a0Distributed data among the multiple nodes in a cluster<\/li>\n
- Dataset:<\/em><\/strong>\u00a0Collection of partitioned data with values<\/li>\n<\/ul>\n
RDD Persistence :<\/h3>\n
![\"introduction](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/04\/5-6.png\")
<\/p>\nOne of Spark’s most useful capabilities is persisting or caching, a dataset in memory across multiple operations. Once an RDD is persisted, each node keeps the partitions it computed in memory and reuses them for later actions on that same dataset or on datasets derived from it, which can make later operations dramatically faster, often by an order of magnitude, particularly for iterative algorithms and interactive analysis.<\/p>\n
You mark an RDD for persistence using persist or cache. The first action that computes it will keep it in memory going forward. Spark’s cache is itself fault tolerant, if a partition is lost, Spark automatically recomputes it using the same transformations that built it originally.<\/p>\n
A persisted RDD can use different storage strategies, called storage levels, depending on your priorities:<\/p>\n
- \n
- <\/b>MEMORY_ONLY, <\/b><\/strong>the default, keeps deserialized objects in memory. Partitions that do not fit are recomputed on demand rather than cached.<\/li>\n
- <\/b>MEMORY_AND_DISK, <\/b><\/strong>keeps deserialized objects in memory, but spills partitions that do not fit onto disk instead of recomputing them.<\/li>\n
- <\/b>MEMORY_ONLY_SER and MEMORY_AND_DISK_SER, <\/b><\/strong>available in Java and Scala, store serialized objects instead, which uses less memory at the cost of more CPU work to read.<\/li>\n
- <\/b>DISK_ONLY, <\/b><\/strong>stores partitions only on disk.<\/li>\n
- <\/b>Replicated variants, <\/b><\/strong>such as MEMORY_ONLY_2, work the same as their base level but keep a copy of each partition on two separate nodes.<\/li>\n
- <\/b>OFF_HEAP, <\/b><\/strong>an experimental option that stores serialized data outside the JVM heap, which requires off heap memory to be enabled.<\/li>\n<\/ul>\n
Deploying Spark: Master URLs<\/h2>\n
When you launch a Spark application, the master URL tells Spark where and how to run:<\/p>\n
- \n
- <\/b>local, <\/b><\/strong>runs Spark on one worker thread, with no real parallelism, generally used only for quick local testing.<\/li>\n
- <\/b>local[K], <\/b><\/strong>runs Spark locally with K worker threads, typically set to match the number of cores on your machine.<\/li>\n
- <\/b>local[*], <\/b><\/strong>runs Spark locally using as many worker threads as there are logical cores available.<\/li>\n
- <\/b>spark colon slash slash HOST colon PORT, <\/b><\/strong>connects to a Spark standalone cluster at the given host and port.<\/li>\n
- <\/b>yarn, <\/b><\/strong>runs on a YARN managed cluster, using whichever configuration is set for client or cluster deploy mode.<\/li>\n
- <\/b>k8s colon slash slash HOST colon PORT, <\/b><\/strong>runs on a Kubernetes cluster, the current standard option for containerized Spark deployments.<\/li>\n<\/ul>\n
Deployment Modes Compared<\/h2>\n
YARN can run Spark in two different modes, cluster and client, and standalone mode offers a third option that does not depend on YARN at all:<\/p>\n
\n\n
\n <\/td>\n YARN Cluster<\/b><\/strong><\/td>\n YARN Client<\/b><\/strong><\/td>\n Standalone<\/b><\/strong><\/td>\n<\/tr>\n \n Driver runs in<\/td>\n Application master<\/td>\n Client machine<\/td>\n Client machine<\/td>\n<\/tr>\n \n Requests resources<\/td>\n Application master<\/td>\n Application master<\/td>\n Client machine<\/td>\n<\/tr>\n \n Starts executor processes<\/td>\n YARN NodeManager<\/td>\n YARN NodeManager<\/td>\n Spark worker<\/td>\n<\/tr>\n \n Long running services<\/td>\n YARN ResourceManager, NodeManager<\/td>\n YARN ResourceManager, NodeManager<\/td>\n Spark master, worker<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n ![\"apache](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/04\/6-5.png\")
<\/p>\nReal World Use Cases of Apache Spark<\/h2>\n
![\"apache](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/04\/8-4.png\")
<\/p>\n- \n
- <\/b>Large scale batch processing, <\/b><\/strong>such as transforming and aggregating logs or transaction data across a data lake.<\/li>\n
- <\/b>Real time analytics, <\/b><\/strong>such as monitoring fraud signals or system metrics as events arrive, using Structured Streaming.<\/li>\n
- <\/b>Machine learning at scale, <\/b><\/strong>training models against datasets too large to fit comfortably on a single machine, using MLlib.<\/li>\n
- <\/b>ETL pipelines, <\/b><\/strong>reading from multiple sources, transforming the data, and loading it into a warehouse or lake using Spark SQL and DataFrames.<\/li>\n
- <\/b>Graph analysis, <\/b><\/strong>such as analyzing social networks or recommendation graphs using GraphX.<\/li>\n<\/ul>\n
Common Misconceptions to Watch For<\/h2>\n
- <\/b>Real time analytics, <\/b><\/strong>such as monitoring fraud signals or system metrics as events arrive, using Structured Streaming.<\/li>\n
- <\/b>local[K], <\/b><\/strong>runs Spark locally with K worker threads, typically set to match the number of cores on your machine.<\/li>\n
- <\/b>MEMORY_AND_DISK, <\/b><\/strong>keeps deserialized objects in memory, but spills partitions that do not fit onto disk instead of recomputing them.<\/li>\n
- Distributed:<\/em><\/strong>\u00a0Distributed data among the multiple nodes in a cluster<\/li>\n
- <\/b>Job, <\/b><\/strong>a parallel computation made up of multiple tasks, created in response to an action.<\/li>\n
- <\/b>Task, <\/b><\/strong>the smallest unit of work, sent to a single executor. Each partition of an RDD gets its own task.<\/li>\n
- <\/b>Spark SQL, <\/b><\/strong>which lets you run SQL and HiveQL queries against structured and semi structured data, often meaningfully faster than the same queries running unmodified on older systems.<\/li>\n