Spark Streaming Kafka Tutorial: Spark Streaming with Kafka

Spark streaming Kafka tutorial, In this tutorial, one can easily know the information about Kafka setup  which is available and are used by most of the Spark developers. Are you dreaming to become to certified Pro Spark Developer, then stop just dreaming, get your Apache Spark Scala certification course from India’s Leading Apache Spark Scala Training institute.

Step by Step Kafka setup for Spark Streaming

Step 1:Download Kafka from this link

Download the current Kafka release from the official Apache Kafka downloads page, kafka.apache.org/downloads, rather than an old, specific version link, since Kafka releases move forward and a hardcoded old version will eventually point at software nobody should still be installing.

Step 2: Extract Kafka and Enter Its Directory

Move into the extracted folder using cd, matching whatever version folder name you actually downloaded.

Step 4:Open a terminal and run

bin/zookeeper-server-start.shconfig/zookeeper.properties

Step 5:Open another terminal and run

bin/Kafka-server-start.shconfig/server.properties

Step 6:Open another terminal and run

bin/Kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test2

bin/Kafka-topics.sh –list –zookeeper localhost:2181 test2

bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test2

Hello World

Hello India

Step 7:Open another terminal and run

bin/Kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test2 –from-beginning

Whatever is typed in producer prompt, will be shown here

HelloWorld

HelloIndia

Reading Kafka Data Into Spark With Structured Streaming

With Kafka running and a topic receiving messages, Spark can read from it directly using Structured Streaming, the current recommended API for stream processing in Spark. The older Spark Streaming API, based on DStreams, still exists but is treated as a legacy engine in Spark’s own documentation, so new work should use Structured Streaming instead.

val kafkaStream = spark.readStream
.format(“kafka”)
.option(“kafka.bootstrap.servers”, “localhost:9092”)
.option(“subscribe”, “test2”)
.load()

val messages = kafkaStream.selectExpr(“CAST(value AS STRING)”)

val query = messages.writeStream
.format(“console”)
.start()

query.awaitTermination()

This subscribes to the test2 topic created above, casts each message’s value from raw bytes into a readable string, and prints incoming messages to the console as they arrive. Try running this while a producer is sending messages in another terminal, and you should see each one appear in Spark’s output shortly after you type it.

Common Mistakes to Avoid

• Following the ZooKeeper startup steps from an old tutorial on a freshly downloaded, current Kafka install. ZooKeeper mode is gone as of Kafka 4.0, so start with the storage format and server start commands instead.
• Copying commands from a page or slide that used an en dash instead of a real double hyphen. If a command fails immediately with an unrecognized option error, retype the flags rather than assuming the command itself is wrong.
• Assuming Spark Streaming and Structured Streaming are the same thing. They are two different APIs. Spark Streaming refers specifically to the older DStream based engine, while Structured Streaming is the newer DataFrame based one that current documentation recommends.

That covers getting Kafka running and reading its messages into Spark using Structured Streaming. To go further, explore Prwatech’s Apache Spark training program, which includes placement assistance.