Apache Spark and Cassandra

Apache Cassandra is an open source NoSQL database that stores data as collections of key value pairs. It was originally built at Facebook, then released as open source and moved to the Apache Software Foundation, where it has been developed ever since. Its design borrows ideas from two well known systems: Google’s BigTable for how data is organized into column families, and Amazon’s Dynamo for how data is distributed and replicated across a cluster without relying on a single master node.

Features of Cassandra

• Highly scalable, since you can add or remove nodes to match growing or shrinking demand without redesigning the database.
• Decentralized, with no single point of failure. Every node in the cluster plays the same role, so there is no master node whose failure would take the whole system down.
• Fault tolerant, since data is replicated across multiple nodes. If a cluster keeps four copies of the same data and one node goes down, the remaining three can still serve requests.
• Flexible with data, supporting structured, semi structured, and unstructured data in the same database.
• Evolving transaction support, covered in more detail below, since this is one of the areas Cassandra has changed the most in recent years.

What are the Components of Cassandra?

• Gossip, the protocol nodes use to talk to each other and stay aware of the state of the rest of the cluster.
• Failure detection, which identifies when a node has gone down so the cluster can route around it and begin recovery.
• Replication, which keeps copies of data spread across nodes in a manageable, configurable way.
• Commit log, a crash recovery mechanism. Every write is recorded here first, before anything else happens to it.
• Memtable, an in memory structure that data moves into right after the commit log. A single column family can have more than one memtable at a time.
• SSTable, a file on disk that data gets flushed to once a memtable reaches a set size threshold.
• Bloom filter, a fast, space efficient structure used to check whether an SSTable might contain the data being looked for, checked on every read to avoid scanning files that could not possibly have the answer.

How Reads and Writes Actually Work?

A write in Cassandra is recorded to the commit log first, then moves to a memtable. Once that memtable reaches its size threshold, its contents are flushed to an SSTable on disk. Data is automatically partitioned and replicated across the cluster as this happens, and Cassandra periodically consolidates SSTables together, clearing out data that is no longer needed in the process.

A read works from the other direction. Cassandra checks the memtable first for the most recent data, then uses bloom filters to quickly narrow down which SSTables on disk are actually worth checking, rather than scanning every file on the node.

Where Cassandra Actually Stands on ACID Transactions?

Cassandra has never supported ACID transactions in the full, multi row sense a relational database offers, and that tradeoff is deliberate, it is part of what lets Cassandra scale the way it does. Since version 2.0, Cassandra has offered lightweight transactions using the Paxos consensus protocol, which support compare and set operations on a single row, useful for things like making sure a username has not already been claimed before creating it, but not a substitute for a general purpose transaction.

That is changing. Apache Cassandra 6.0, currently in alpha, introduces a new consensus protocol called Accord, which brings general purpose, multi row ACID transactions to Cassandra for the first time. This is a genuinely significant shift, since it opens up workloads, such as financial transactions or inventory reservations, that teams have historically routed to a relational database specifically because Cassandra could not offer that guarantee.

How Spark Connects to Cassandra

Spark does not talk to Cassandra natively. The connection goes through the Spark Cassandra Connector, a library originally built by DataStax and now developed as part of the Apache Cassandra project itself, which lets Spark read Cassandra tables as DataFrames or RDDs, write Spark data back into Cassandra tables, and run CQL queries directly from a Spark application.

val df = spark.read
.format(“org.apache.spark.sql.cassandra”)
.options(Map(“keyspace” -> “my_keyspace”, “table” -> “my_table”))
.load()

df.write
.format(“org.apache.spark.sql.cassandra”)
.options(Map(“keyspace” -> “my_keyspace”, “table” -> “my_table”))
.mode(“append”)
.save()

This is the piece that actually makes Apache Spark and Cassandra work as a pair, letting Spark’s processing and analytics run directly against data that lives in Cassandra, rather than needing to export it somewhere else first.

When Cassandra is the Right Choice?

• You need to write large volumes of data quickly and reliably, such as sensor readings, event logs, or time series data.
• You need the database to stay available across multiple data centers or regions, even if one location goes offline entirely.
• Your queries are known in advance and can be modeled around specific access patterns, since Cassandra performs best when tables are designed around how they will be queried, rather than normalized the way a relational database would be.

You are comfortable designing around Cassandra’s data model rather than relying on complex joins across many tables, which it is not built for.

That covers Cassandra’s architecture and how Spark connects to it. To go further, explore Prwatech’s Apache Spark training program, which includes placement assistance.