{"id":654,"date":"2026-09-13T11:25:07","date_gmt":"2026-09-13T11:25:07","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=654"},"modified":"2026-09-14T16:10:44","modified_gmt":"2026-09-14T16:10:44","slug":"spark-streaming-kafka-tutorial","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-kafka\/spark-streaming-kafka-tutorial\/","title":{"rendered":"Spark Streaming Kafka Tutorial"},"content":{"rendered":"

Spark Streaming Kafka Tutorial: Spark Streaming with Kafka<\/h1>\n

Spark streaming Kafka tutorial<\/strong>, In this tutorial, one can easily know the information about Kafka setup\u00a0 which is available and are used by most of the Spark developers. Are you dreaming to become to certified Pro Spark Developer, then stop just dreaming, get your Apache Spark Scala certification course from India\u2019s Leading Apache Spark Scala Training institute<\/a>.<\/p>\n

Step by Step Kafka setup for Spark Streaming<\/h2>\n

Step 1:Download Kafka from this link<\/h3>\n

Download the current Kafka release from the official Apache Kafka downloads page, kafka.apache.org\/downloads, rather than an old, specific version link, since Kafka releases move forward and a hardcoded old version will eventually point at software nobody should still be installing.<\/p>\n

Step 2: Extract Kafka and Enter Its Directory<\/h3>\n

\"Spark<\/p>\n

Move into the extracted folder using cd, matching whatever version folder name you actually downloaded.<\/p>\n

\"Spark<\/p>\n

Step 4:Open a terminal and run<\/h3>\n

bin\/zookeeper-server-start.shconfig\/zookeeper.properties<\/strong><\/p>\n

\"Spark<\/p>\n

Step 5:Open another terminal and run<\/h3>\n

bin\/Kafka-server-start.shconfig\/server.properties<\/p>\n

\"Spark<\/p>\n

Step 6:Open another terminal and run<\/h3>\n

bin\/Kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test2<\/p>\n

\"apache<\/p>\n

bin\/Kafka-topics.sh –list –zookeeper localhost:2181 test2<\/p>\n

\"Spark<\/strong><\/p>\n

bin\/kafka-console-producer.sh –broker-list localhost:9092 –topic test2<\/p>\n

\"Spark<\/strong><\/p>\n

Hello World<\/strong><\/p>\n

Hello India<\/strong><\/p>\n

Step 7:Open another terminal and run<\/h3>\n

bin\/Kafka-console-consumer.sh \u2013bootstrap-server localhost:9092 –topic test2 \u2013from-beginning<\/p>\n

\"open<\/strong><\/p>\n

Whatever is typed in producer prompt, will be shown here<\/p>\n

HelloWorld<\/strong><\/p>\n

HelloIndia<\/strong><\/p>\n

Reading Kafka Data Into Spark With Structured Streaming<\/h2>\n

With Kafka running and a topic receiving messages, Spark can read from it directly using Structured Streaming, the current recommended API for stream processing in Spark. The older Spark Streaming API, based on DStreams, still exists but is treated as a legacy engine in Spark’s own documentation, so new work should use Structured Streaming instead.<\/p>\n

val kafkaStream = spark.readStream
\n.format(“kafka”)
\n.option(“kafka.bootstrap.servers”, “localhost:9092”)
\n.option(“subscribe”, “test2”)
\n.load()<\/p>\n

val messages = kafkaStream.selectExpr(“CAST(value AS STRING)”)<\/p>\n

val query = messages.writeStream
\n.format(“console”)
\n.start()<\/p>\n

query.awaitTermination()<\/p>\n

This subscribes to the test2 topic created above, casts each message’s value from raw bytes into a readable string, and prints incoming messages to the console as they arrive. Try running this while a producer is sending messages in another terminal, and you should see each one appear in Spark’s output shortly after you type it.<\/p>\n

Common Mistakes to Avoid<\/h2>\n