Apache Spark RDD Introduction

An RDD, short for Resilient Distributed Dataset, is Spark’s core data structure. It is a collection of elements, partitioned across the nodes of a cluster, that Spark can operate on in parallel. RDDs are exposed through an API, so you work with them as objects and call methods on them to run transformations and actions.

The name itself describes what an RDD actually is:

• Resilient, meaning it can recover from a lost partition automatically, by recomputing it from the sequence of operations that built it, rather than needing the data backed up separately.
• Distributed, meaning the data is spread across multiple machines in a cluster, not sitting on a single node.
• Dataset, meaning it holds a collection of items you can run operations against.

A Simple RDD Example

At its simplest, creating and using an RDD looks like this, taking a small collection of numbers, distributing it as an RDD, then running a transformation and an action against it.

val numbers = sc.parallelize(List(1, 2, 3, 4, 5))
val doubled = numbers.map(x => x * 2)
doubled.collect()

Here, parallelize turns a plain Scala list into an RDD, map is a transformation that doubles each value, and collect is the action that actually triggers the computation and returns the result.

RDD Operations: Transformations and Actions

RDDs support two kinds of operations. Transformations, such as map and filter, create a new RDD from an existing one. Actions, such as reduce and collect, run a computation on the RDD and return a result to the driver program rather than another RDD.

All transformations in Spark are lazy. Writing one does not compute anything right away, it just records what you have asked for, building up a chain against the base dataset. Spark only actually runs that chain once an action calls for a result. This matters for efficiency. If a dataset built through several transformations is only ever used inside a final reduce, Spark can plan around that and avoid returning the larger intermediate dataset at all, sending back only the smaller final result.

Operations are also described as either coarse grained or fine grained. A coarse grained operation applies to an entire dataset at once, which is how most RDD operations work and why they run efficiently across a whole cluster simultaneously. A fine grained operation applies to a smaller, specific subset of the data instead.

Why Spark RDD is Needed

RDDs exist to solve two problems that made the older Hadoop MapReduce model slow for certain workloads: memory efficiency and data sharing.

In Hadoop MapReduce, there is no built in way to reuse data between operations. Each step typically writes its result to an intermediate data store, then the next step reads it back in, which adds real overhead when you are running several operations on the same data. Spark’s RDDs solve this by letting you store data explicitly in memory, using cache or persist, once a computation is finished, so a later operation on the same data does not have to be recomputed or re read from disk.

This matters most in two situations: iterative applications, where intermediate results get reused across multiple passes, and interactive applications, where a user submits several ad hoc queries against the same dataset one after another. Both depend on fast, repeated access to the same data, which RDDs support directly through in memory persistence.

Where and When to Use an RDD

• When you need low level control over transformations and actions, such as map and filter, rather than working through a higher level, more automatically optimized interface.
• When your data is unstructured, such as media streams or plain text, and does not fit naturally into rows and columns.
• When you are willing to give up some of the automatic performance optimizations that DataFrames and Datasets offer for structured and semi structured data, in exchange for more direct control.
• When you want to express your logic through functional programming, building your program from functions applied to data, rather than through domain specific expressions built around a fixed schema.

RDD Compared to DataFrame and Dataset

RDD is Spark’s original abstraction, and DataFrames and Datasets are both built on top of it. A DataFrame organizes data into named columns, similar to a table in a relational database, and benefits from Spark’s Catalyst optimizer, which can significantly speed up common operations like filtering and joining. An RDD has no such built in structure or automatic optimization, but it gives you complete control over exactly how your data is transformed, and it works just as well with unstructured data as with structured data.

For the hands on commands behind creating and transforming an RDD, the Apache Spark RDD Commands guide walks through each one with examples.

Common Misconceptions to Watch For

• Assuming RDDs are deprecated. They are not. DataFrames and Datasets are built directly on top of RDDs, and RDDs remain fully supported and actively used whenever low level control matters more than automatic optimization.
• Assuming an RDD is always slower than a DataFrame. It usually is for structured data, since it misses out on Catalyst’s optimizations, but for unstructured data or highly custom logic, an RDD can be the more practical choice, not just the older one.
• Forgetting to persist an RDD you plan to reuse. Without calling cache or persist, Spark recomputes the full chain of transformations every time you call a new action on the same RDD, even if nothing about the underlying data has changed.

Watch It in Action

The video below, embedded on the original page, walks through these RDD concepts.

That covers the core concepts behind Apache Spark RDDs. For the hands on commands, from creating and transforming RDDs through joins, aggregation, and saving output, see the Apache Spark RDD Commands guide, or explore Prwatech’s Apache Spark training program, which includes placement assistance.